7-neural-networks-with-minibatch-stochastic-gradient-descent-and-ADAM.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment7\n",
    "## CS-5891-01 Special Topics Deep Learning\n",
    "## Ronald Picard\n",
    "\n",
    "In this notebook we will walk through the design, training, and testing of neural networks with multiple hidden layers using minibatch stochastic gradient descent with adaptive moments (ADAM). These neural networks will be used for logistic regression, which is an archaic name for binary classification.\n",
    "\n",
    "The binary classification will be performed on images of handwritten numerical digits. More specifically, the last numerical digit of my student ID. This digit happens to be 9. Therefore, the goal of our neural networks will be to output a the value of 1 when the handwritten numerical digit image input is a 9, and 0 in all other cases.\n",
    "\n",
    "The data set we will be using is the MNIST data set. This is a very popular data set amoung the machine learning community. The data set contains 60,000 images, and each image contains a handwritten numerical digit. Each of the images have been provided with a truth label that corresponds to the handwritten digit within the image from the set {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}. \n",
    "\n",
    "For our case, we only care about when the image is 9. Therefore we will need to re-label the truth labels so that all truth labels with the value of 9 are given to the value of 1, and all other truth labels are given the value of 0. \n",
    "\n",
    "To start we need to import some needed classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import struct\n",
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "import matplotlib.pyplot as pyplot\n",
    "import csv\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we must change our path string to the path of our data file containing the features. (Please note that you must change this string to point to the directory with the data file on your machine data file on your machine.) \n",
    "\n",
    "Second, we much change the string name of the data files to the names of the MNIST data files. (Please note that you may NOT need to change these. Only change them if your MINST data files are named differently.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "## path\n",
    "path = 'C:/Users/computer/OneDrive - Vanderbilt/Vanderbilt_Spring_2019/CS_5891_01_SpecialTopicsDeepLearning/Assignment7/'\n",
    "\n",
    "#Train data\n",
    "fname_train_images = os.path.join(path, 'train-images.idx3-ubyte')  # the training set image file path\n",
    "fname_train_labels = os.path.join(path, 'train-labels.idx1-ubyte')  # the training set label file path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we retrieve the data from the data files as follows. This imports the data into a feature tensor (3-D matrix) in which each index is a feature matrix corresponding to an image. The label data comes in the form of a vector where each index corresponds to the index of the feature matrix (image) of the feature tensor.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The training set contains 60000 images\n",
      "The shape of the image is (28, 28)\n"
     ]
    }
   ],
   "source": [
    "# open the label file and load it to the \"train_labels\"\n",
    "with open(fname_train_labels, 'rb') as flbl:\n",
    "    magic, num = struct.unpack(\">II\", flbl.read(8))\n",
    "    labels = np.fromfile(flbl, dtype=np.uint8)\n",
    "\n",
    "# open the image file and load it to the \"train_images\"\n",
    "with open(fname_train_images, 'rb') as fimg:\n",
    "    magic, num, rows, cols = struct.unpack(\">IIII\", fimg.read(16))\n",
    "    images = np.fromfile(fimg, dtype=np.uint8).reshape(len(labels), rows, cols)\n",
    "\n",
    "print('The training set contains', len(images), 'images')  # print the how many images contained in the training set\n",
    "print('The shape of the image is', images[0].shape)  # print the shape of the image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we need to perform both two steps; feature scaling and feature normalization. Feature scaling consists of converting the 28 X 28 image matrices into 784 X 1 feature vectors. In essence we will flatten the images out into vectors so that we can use an input a vector to our single neuron. Feature normalization is a process of normalizing the pixel data to between 0 <= x <= 1 (for logistic regression). Each pixel comes on a scale of 0 <= x <= 255. Since 255 is the maximum for every pixel we shall divide each pixel by that number (elementwise) in order to normalize each pixel to between 0 and 1 (inclusive).\n",
    "\n",
    "One additional item we need to take care of is relabeling our label (truth) data so that we have a binary classification in which all 9s are converted to 1s and all other labels are converted to 0s."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(784, 60000)\n",
      "(784, 60000)\n",
      "60000\n"
     ]
    }
   ],
   "source": [
    "# feature scaling\n",
    "matrix_side_length = len(images[0])\n",
    "vector_size = matrix_side_length*matrix_side_length\n",
    "\n",
    "scaled_images_feature_matrix = []\n",
    "for image in images:\n",
    "    reshaped_image = np.array(image).reshape((vector_size))\n",
    "    scaled_images_feature_matrix.append(reshaped_image)\n",
    "\n",
    "# convert to numpy array\n",
    "scaled_images_feature_matrix = np.transpose(np.array(scaled_images_feature_matrix))\n",
    "print(scaled_images_feature_matrix.shape) # scaled_images_feature_matrix is a matrix of 60000 X 784\n",
    "#print(scaled_images_feature_matrix[0].shape)\n",
    "\n",
    "# feature normilization\n",
    "normilization_factor = 1/255\n",
    "normalized_scaled_images_feature_matrix = np.multiply(normilization_factor, scaled_images_feature_matrix)\n",
    "print(normalized_scaled_images_feature_matrix.shape)\n",
    "#print(normalized_scaled_images_feature_matrix[0])\n",
    "\n",
    "# re-label for binary classification\n",
    "value_for_1 = 9\n",
    "binary_labels = []\n",
    "for label in labels:\n",
    "    if(label == 9):\n",
    "        binary_labels.append(1)\n",
    "    else:\n",
    "        binary_labels.append(0)\n",
    "\n",
    "# convert to numpy array\n",
    "binary_labels = np.array(binary_labels)\n",
    "print(len(binary_labels)) # binary_labels is a row vector of 1 X 60000\n",
    "#print(binary_labels[0])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to test the efficacy of our neural networks, we need to split up the our label data into two data sets; a smaller and a larger one. The larger set will be the training data that we will use to train our neural networks on. The smaller set will be the testing data that we will used to test the accuracy of our neural nets. The MNIST data set contains 60,000 images. Therefore, we will use 50,000 images for our training data set, and 10,000 images for our testing data set. \n",
    "\n",
    "It is common practice to use a smaller subset of the total data set to debug (ensure it works) and tune hyper-parameters before using the entire time-comsuming data set. This smaller subset is known as a validation set. Therefore, we will first use a validation data set of 600 images. 500 of these images will be used for as our training data set, and the other 100 of these images will be used for our test data set. \n",
    "\n",
    "Thus, we will begin by sifting out a validation set from our total data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(784, 500)\n",
      "(500,)\n",
      "(784, 100)\n",
      "(100,)\n"
     ]
    }
   ],
   "source": [
    "# create a data set\n",
    "size = vector_size\n",
    "\n",
    "number_of_testing_images = 100\n",
    "number_of_training_images = 500\n",
    "number_of_validation_images = number_of_testing_images + number_of_training_images\n",
    "\n",
    "training_images = []\n",
    "training_labels = []\n",
    "testing_images = []\n",
    "testing_labels = []\n",
    "\n",
    "factor = 0\n",
    "for index in range(0, number_of_validation_images):\n",
    "    if(index <= number_of_training_images - 1):\n",
    "        training_images.append(normalized_scaled_images_feature_matrix[:, index + factor])        \n",
    "        training_labels.append(binary_labels[index + factor])\n",
    "    else:\n",
    "        testing_images.append(normalized_scaled_images_feature_matrix[:, index + factor])        \n",
    "        testing_labels.append(binary_labels[index + factor])\n",
    "        \n",
    "# covert to numpy array\n",
    "training_images = np.transpose(np.array(training_images))\n",
    "training_labels = np.array(training_labels)\n",
    "testing_images = np.transpose(np.array(testing_images))\n",
    "testing_labels = np.array(testing_labels)\n",
    "\n",
    "# logger\n",
    "print(training_images.shape) # validation_training_images is a matrix of 784 X 500\n",
    "print(training_labels.shape) # validation_testing_labels is a row vector of 1 X 500\n",
    "print(testing_images.shape) # validation_training_images is a matrix of 784 X 100\n",
    "print(testing_labels.shape)  # validation_testing_labels is a row vector of 1 X 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we move on to the training of our neural network with multiple hidden layers. \n",
    "\n",
    "Part 1 - Feed Forword:\n",
    "\n",
    "For these neural networks we will use multiple hidden layers with between 5-20 units per layer (neurons). The first layer will have an input of a matrix (784 X number_of_images) of vectorized images of 784 X 1, and will output a matrix (# of units X # of images). This matrix will be input into the next hidden layer, which output another matrix (# of units X # of images.) The output layer will take and input matrix that is the output matrix of the last hidden layer and will output a row vector of probabilities which we will convert into binary classifications of 0 or 1. (If P(x) >= 0.5 then we will convert it to a 1, otherwise we will convert to 0.) \n",
    "\n",
    "The model for the units of the hidden layers will be a vecorized linear model Z^[l] = W^[l]^T * A^[l-1] + B^[l], where W is a matrix of 5-10 units (# of units) X # of units or parameter weights, A is the input matrix of vectorized images (784 X # of images) or the output of one of the layers (1 X # of units), and B is a row vector of bias's. (Note: in this case, b will be scalar that applied in a broadcasing manner to save on memory.) The output of this model Z^[l] will be a matrix (5-10 units X # of images). Z^[1] will be subject to an activation function; which for this assignment will be relu (note: we will test tanh once for comparison). \n",
    "\n",
    "Hidden Layer Activation Function: relu activation function is A^[1]  = relu(Z) =  max(Z, 0).\n",
    "\n",
    "The model of the output layer will be a vectorized linear model Z^[1] = W^[l] * A^[1] + b^[l] with a single unit. This linear model will be subjected to a sigmoid activation function.\n",
    "\n",
    "The resultant row vector will then be used to calculate the cost function values in an elementwise manner. The cost function for this binary classification will be L(Y_Predicted, Y_Label) = -Y_Label^[l] * Log(A^[l]) - (1-Y_Predicted^[l]) * Log(1-A^[l]), where Y_Label is the True Label, Y_Predicted is the probability value predicted by the neural network, and A is the activation function value. The resultant cost row vector will be added up and divided by the number of elements in order to calculate the average cost.\n",
    "\n",
    "Part 2 - Back Propogation:\n",
    "\n",
    "The back propogation technique that we will use for training the neural network, will be gradient descent. This involves utilizing the gradient of the cost function to updated the model parameters in our layers. In order to calculate the gradient we will utilize the chain rule. The goal of back propogation is the adjust the parameter weights and bias's of our model to accurately perform binary classification. In general the chain rule can be used to find the gradient of the cost function (vecorzied rates of change) with respect to the model parameters. The following is the chain that we will utilize. \n",
    "\n",
    "\n",
    "Generalized Chain Rule for N layers: \n",
    "\n",
    "dL(A^[n], Y)/dW^[l] = ∏(i = n to l) (dl(A^[i], Y)/dz^[i]) * dZ^[i]/dA^[i-1] * dA^[i-1]/dZ^[i-2] *....* dz^[l]/dW^[l];\n",
    "\n",
    "dL(A^[n], Y)/dB^[l] = ∏(i = n to l) (dl(A^[i], Y)/dz^[i]) * dZ^[i]/dA^[i-1] * dA^[i-1]/dZ^[i-2] *....* dz^[l]/dB^[l];\n",
    "\n",
    "\n",
    "\n",
    "Output Layer - Back Propogation:\n",
    "\n",
    "The partial derivative of the cost function with respect to the output layer sigmoid activation function is found by the following:\n",
    "\n",
    "dL(A^[n], Y)/dA^[n] = -Y/A^[n] + (1-A^[n])/(1-A^[n]).\n",
    "\n",
    "\n",
    "Due to the chain rule, the derivative of the cost function with respect to the linear model Z^[n] is found by the following:\n",
    "\n",
    "dL(A^[n], y)/dz = dL(A^[n], y)/dA^[n] * dA^[n]/dZ^[n].\n",
    "\n",
    "The derivative of the sigmoid activation function is da/dz is found by the following:\n",
    "\n",
    "dA^[n]/dZ^[n] = sigma(Z^[n]) * (1-sigma(Z^[n]))\n",
    "\n",
    "Therefore, the derivative of the cost function with respect to the output of the linear model is found by the following:\n",
    "\n",
    "dL(A^[n], Y)/dA^[n] * dA^[n]/dZ^[n]. = (-Y/A^[n] + (1-Y)/(1-A^[n])) * (sigma(Z^[n]) * (1-sigma(Z^[n]))) = A^[n]-Y. (For convienence we will say dZ^[n] = A^[n]-Y.)\n",
    "\n",
    "Now we can extrapolate the chain rule to all the paramters of the linear model our output layer.\n",
    "\n",
    "dL(A^[n], Y)/dW^[n] = dZ^[n] * dZ^[n]/dW^[n] = dZ^[n] * A^[n-1] = A^[n-1] * dZ^[n] (we will change our notation to dW^[n] = A^[n-1] * dZ^[n] for convienence)\n",
    "\n",
    "dL(A^[n], Y)/dB^[n] = dZ^[n] * dZ^[n]/dB^[n] = dZ^[n] (we will change our notation to dW^[n] = dZ^[n] for convienence)\n",
    "\n",
    "\n",
    "\n",
    "Hidden Layers - Back Propagation:\n",
    "\n",
    "dL(A^[n], Y)/dZ^[l] = ∏(i = n to l) (dl(A^[i], Y)/dz^[i]) * dZ^[i]/dA^[i-1] * dA^[i-1]/dZ^[i-2] *....* dA^[l]/dZ^[l]\n",
    "\n",
    "dL(A^[n], Y)/dZ^[l] = dZ^[l+1] * dZ^[l+1]/dA^[l] = W^[l+1] * dz^[l+1] * (element-wise) dA^[1]/dZ^[1]. The reason this is element-wise is because we are propgating from a single neuron to a layer with multiple neurons (we shall rename this dz^[l] = W^[l+1] * dz^[l+1] * (element-wise) dA^[1]/dZ^[1] for conveinience) \n",
    "\n",
    "dA^[1]/dZ^[1] depends on the activation function we are using in the hidden layer (in this case relu): The derivative of relu activation function is dA^[l]/dZ^[l] = if Z^[l] > 0 then 1 else 0.\n",
    "\n",
    "\n",
    "dL(A^[n], Y)/dW^[l] = dZ^[l] * dZ^[l]/dW^[l] = dZ^[l] * X^T (we will change our notation to dW^[l] = dZ^[l] * A[l-1]^T for convienence)\n",
    "\n",
    "dL(A^[n], Y)/dB^[l] = dZ^[l] * dZ^[l]/dB^[l] = dZ^[l] (we will change our notation to dB^[l] = dZ^[l] for convienence)\n",
    "\n",
    "\n",
    "Find vector averages:\n",
    "\n",
    "m = # number of images\n",
    "\n",
    "dW^[l] = 1/m * (A^[l-1] * dZ^[l])\n",
    "\n",
    "dB^[l] = 1/m * (dZ^[l])\n",
    "\n",
    "\n",
    "Finally, we will update our the weights and bias's of the layers.\n",
    "\n",
    "\n",
    "W^[l]:= W^[l] - alpha * dW^[l]\n",
    "\n",
    "B^[l]:= B^[l] - alpha * dB^[l]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first thing we have have to do is initialize our weights and bias's. There are multiple ways to initialize weights and bias's. Typically we will set our values based on either a uniform distribution between or a normal distribution with a some reasonable mean and standard deviation. There is some flexibility in the initalization of the weights but in general they need to be small (not to small) and varied. The weights need to be different so that the gradients with respect to each other are different. In other words we don't aways want the relative rates of change to be 0. Additionally, we do not want to reach saturation on our output activation function where the gradients are 0 (vanishing gradiants). For this assignment we wills stick with with a uniform random between -.1 and .1. We will also set a random seed each time so that we for our random values to be the same (or similar if there are more layers)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature Size: 784\n",
      "Weights Shape: (20, 784)\n",
      "Bias Shape: (20, 1)\n",
      "Velocity Weights Shape: (20, 784)\n",
      "Velocity Bias Shape: (20, 1)\n",
      "RMSProp Weights Shape: (20, 784)\n",
      "RMSProp Bias Shape: (20, 1)\n"
     ]
    }
   ],
   "source": [
    "# initialize weights & bias\n",
    "np.random.seed(10)\n",
    "print('Feature Size: ' + str(size))\n",
    "\n",
    "lower_bound = -.1\n",
    "upper_bound = .1\n",
    "\n",
    "#mean = 0.015\n",
    "#std = 0.005\n",
    "\n",
    "# hyper-parameters: hidden layers\n",
    "hidden_layers = 2\n",
    "units_array = [20, 10]\n",
    "Weights = []\n",
    "Bias = []\n",
    "V_dW = []\n",
    "V_dB = []\n",
    "R_dW = []\n",
    "R_dB = []\n",
    "for i in range(0, hidden_layers):\n",
    "    if(i == 0):\n",
    "        _W = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], size]))\n",
    "        _B = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], 1]))\n",
    "        _V_dW = np.float64(np.zeros([units_array[i], size]))\n",
    "        _V_dB = np.float64(np.zeros([units_array[i], 1]))\n",
    "        _R_dW = np.float64(np.zeros([units_array[i], size]))\n",
    "        _R_dB = np.float64(np.zeros([units_array[i], 1]))\n",
    "        Weights.append(_W)\n",
    "        Bias.append(_B)\n",
    "        V_dW.append(_V_dW)\n",
    "        V_dB.append(_V_dB)\n",
    "        R_dW.append(_R_dW)\n",
    "        R_dB.append(_R_dB)\n",
    "    else:\n",
    "        _W = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], units_array[i-1]]))\n",
    "        _B = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], 1]))\n",
    "        _V_dW = np.float64(np.zeros([units_array[i], units_array[i-1]]))\n",
    "        _V_dB = np.float64(np.zeros([units_array[i], 1]))\n",
    "        _R_dW = np.float64(np.zeros([units_array[i], units_array[i-1]]))\n",
    "        _R_dB = np.float64(np.zeros([units_array[i], 1]))\n",
    "        Weights.append(_W)\n",
    "        Bias.append(_B)\n",
    "        V_dW.append(_V_dW)\n",
    "        V_dB.append(_V_dB)\n",
    "        R_dW.append(_R_dW)\n",
    "        R_dB.append(_R_dB)\n",
    "    \n",
    "# output layer\n",
    "_W = np.float64(np.random.uniform(lower_bound, upper_bound, [1, units_array[i]]))\n",
    "_b = np.float64(np.random.uniform(lower_bound, upper_bound)) # b will be added in a broadcasting manner\n",
    "_V_dW = np.float64(np.zeros([1, units_array[i]]))\n",
    "_V_dB = np.float64(np.zeros(1))\n",
    "_R_dW = np.float64(np.zeros([1, units_array[i]]))\n",
    "_R_dB = np.float64(np.zeros(1))\n",
    "Weights.append(_W)\n",
    "Bias.append(_b)\n",
    "V_dW.append(_V_dW)\n",
    "V_dB.append(_V_dB)\n",
    "R_dW.append(_R_dW)\n",
    "R_dB.append(_R_dB)\n",
    "\n",
    "Weights = np.array(Weights)\n",
    "Bias = np.array(Bias)\n",
    "V_dW = np.array(V_dW)\n",
    "V_dB = np.array(V_dB)\n",
    "R_dW = np.array(R_dW)\n",
    "R_dB = np.array(R_dB)\n",
    "\n",
    "\n",
    "for index in range(0, len(Weights) - 1):\n",
    "    Weights[index] = np.where(Weights[index] != 0, Weights[index], np.random.uniform(lower_bound, upper_bound))\n",
    "\n",
    "#print(train_X.shape)\n",
    "#print(np.ravel(train_Y).shape)\n",
    "\n",
    "print('Weights Shape: ' + str(Weights[0].shape)) # matrix with a size of # of units X 784\n",
    "print('Bias Shape: ' + str(Bias[0].shape)) # vector with a size of the # of unit\n",
    "print('Velocity Weights Shape: ' + str(V_dW[0].shape)) # matrix with a size of # of units X 784\n",
    "print('Velocity Bias Shape: ' + str(V_dB[0].shape)) # vector with a size of the # of unit\n",
    "print('RMSProp Weights Shape: ' + str(R_dW[0].shape)) # matrix with a size of # of units X 784\n",
    "print('RMSProp Bias Shape: ' + str(R_dB[0].shape)) # vector with a size of the # of unit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we implement our minibatch stochastic gradient descent algorithm. The only difference betwen minibatch stochasic gradient descent and general full-batch gradient descent is that during every epoch (run) we split up our training data into minibatches based on the specified minibatch size. If the data does not evenly split, then the last batch will be smaller; so that we utilize all of the trainig data during every epoch. Then during evey epoch we train once on each minibatch. Since we are training on minibatches, our path the extrema of the function we are tryign to find will not as direct. Therefore, our cost and test accuracy will not decrease every epoch; however, the general trend will be decreasing. Minibatch gradient decense will help prevent us from getting stuck on local extrema, as well as increase the speed at which the code runs.\n",
    "\n",
    "We will also include ADAM in our back propogation. This is a combination of RMSProp and Momentum. Momentum updates the network paramters based on the exponentially weighted moving average of the gradient which helps speed up the convergence rate and prevent the network from getting stuck on local minima. RMSProp penalizes momentum by the exponentially weighted moving average of the square of the gradient which prevents the network from focusing on specific features of the training data, and therefore, helps to prevent overfitting. We will start by setting both of our momentum and RMSProp hyperparamters to .9 with a learning rate of 0.1 since this provides good results.\n",
    "\n",
    "We will also collect data on the accuracy of our networks as a function of training iterations. To do this we will need to find the number of inaccuracate binary classifications (false positives & false negatives). This will be acommplished using our test data set. We will send our test data set through the network and compare the results with the true labels of the test data set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Main Loop Epoch: 100\n",
      "Number Of Minibatches: 10\n",
      "Cost: 0.1101922786244274\n",
      "Main Loop Epoch: 200\n",
      "Number Of Minibatches: 10\n",
      "Cost: 0.017543898313142785\n",
      "Main Loop Epoch: 300\n",
      "Number Of Minibatches: 10\n",
      "Cost: 0.05816910301022392\n",
      "Main Loop Epoch: 400\n",
      "Number Of Minibatches: 10\n",
      "Cost: 0.05842772204814277\n",
      "\n",
      "Results:\n",
      "\n",
      "\n",
      "Run Time: 6.019030570983887 seconds\n",
      "Cost: 0.05483394259959518\n",
      "Accuracy: 94.0 %\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XmYXFWd//H3J+mEkJCEQMKWAAFBUNkJ+yJgBpFVHBAYdmFQfuOAu6IzI24zroDMOCiLG6CIqMAAghiJGlEg7EtAwpqQQDpAkg4hgSTf3x/nVPXtSnenqSWVrnxez1NP3f2cc7fvPefeuqWIwMzMDGBAszNgZmarDwcFMzMrc1AwM7MyBwUzMytzUDAzszIHBTMzK1sjgoKk30o6tdn5sPqQdKqk39Z7WluRpO0l3VWnZU2U9Gw9ltXD8nvd1pKmSDqth3FbSWrZ5/Ml7SLpz32ZtqFBQdKzkiY2Mo2+iIj3RcRPGrFsSSMkXSTpeUkLJU3P/aMbkV49STpf0ps536XPloXxO0m6V9Ki/L1TYZwkfUPSy/nzTUnqJo0TC8t+XdLyYnrV5DsifhIR76v3tG+VpJm5TAslzZP0F0lndbceepi/5hORpKMlPShpgaS5kn4vabM6pv1V4FuF+YplflHSFZKG1VKGvpB0pqTJ3QyfKekAaOy2rkUxj80SEfcBr0ta6frp9zUFSW1NTHswMAl4F3AIMALYG3gZ2L2K5TWjLL+IiHUKn6dzXgYDNwBXAaOAnwA35OEAZwHvB3YEdgAOBz5cufCIuLq0bOB9wKxiepXTN3N7Vul9uRzjSSfPzwOXroqEJW0D/Ag4FxgJbAF8H1hep+WPA/YF/q9iVKnMuwB7AJ+pR3rWGIVj6mq6OUYrNS0oSDpc0gP5CutOSTsUxn1O0lOSOiQ9JunowrjT8hXZhZJeAc7Pw6ZI+rakVyU9U4yIkiZLOrMwf2/TbiHpTznt30v6nqSreijGKcBmwNER8VhELI+IORHxlYi4JS8vJG1VWP6PJX01dx+QryI+K+lF4EeSpkk6vDB9W74C3CX375nX17x8hXhALduhFwcAbcBFEbEkIi4GBByUx58KfCciZkbEC8B3gNOqSSivg09LehhYlIf9m6Sn83Z4VNKRhenLV415/YSkDyvV0l6VdHGV0w5UquW9nNP+175eyUfEvIi4HjgBOEPStnmZR+b9vEOpNvnvhdn+lKcp1Zx2k7S1pDtyHuZKulLSyB6S3RmYHhGTI+mIiOsiYmZe7gBJn8/H0lxJ10ga1VPa3Sz/YOCeiFjSQ5lnAb8DijXIIZIukDRD0kuS/lfSkMp5C9tifGHYVZLO76GsK6WK2oSkQyQ9IWm+pO+S9t/SuIH5HPKypKdIF3XFZa0r6UeSZuf988uSBhTS+WOef17eVw6uIr/rS7pFUnveF/9P0tg87gRVNNvl88R1ubvH9azcTJe3/YvAZXkRk4F/kDSot3w1JSjkE9wPSVFrfeAHwI2S1sqTPAXsR7r6+RJwlaSNC4vYA3ga2AD4WmHYE8Bo4JvAFVKP1fjepv0ZcHfO1/nAyb0UZSJwa0RU1QySbQSsB2xOuvr+OenEUvJeYG5E3Jd3mJtJVfr1gE8Bv5I0pob0j5D0Sj7xnl0Y/i7goej6HpSH8vDS+AcL4x4sjKvG8aSaROkE+Hdgn9z/NeBnkjbsZf5DgV1JJ8qT1HuzZU/Tnk3apjsAE4APvNVCRMRfgRdJ+y/AQuCkXI4jgHPVGfT3z/OUak73kE5cXwU2Bt4JbAkUA0nRvcD2kr4j6UCt2IzzCeCwnM444DWgFAS7S7vS9qTjpFuSNiWdTKcXBn+bVGPZAdiaVIP6Qk/LaBRJGwDXAZ8jHeczScd9ydmkoLcjqVb/wYpFXAW8DryNtC8cBpxeGL838DDpPHEhcEUV2RxAOmFvRjr+3wS+m8ddD2wjaevC9CcBV+bula3nccA6edn/DyAiniPtX8VlrigiGvYBngUmdjP8EuArFcOeAN7dw3IeAI7K3acBz1eMP410xVTqHwoEsFHunwycubJp8wpcCgwtjL8KuKqHfN0OfH0l6yCArQr9Pwa+mrsPAN4AhhTGbwV0lPJAqvL9R+7+LHBlxfJvA06tcvu8E9gEGEjayWcDJ+Rx/w5cUzH91cD5uXsZsG1h3Na5rOolvQOAmd0MnwmcspK8PgIclrvPBCbn7rac7p6FaX8NfKqKaf8EnFEYd0g6RHrM00zggG6GTwU+28M8/wN8q7Cte1x+nuYY0tV6T+P3Bn4JzAUWky62SvvOkxSOKWBTYAnpZNSXtH9U2lcryrww76NBqimMzOMG5DxsXph+P+DJ3D0ReLZiW4yvONbO7yEvZ5KOzXkVn+WlbVCxrT8ETCnMPyDv36cVtvWZhfGHltYHMJYUENYqjD8ZuL2QzuOFcSNyWUa/lf2km+kmAO2F/suAL+XunfI2HtTH9bwYGNxNGi8Be/eWj2Y1H20OfDJXveZJmkfaYTcBkHSKOpuW5gHbkaJ9yYxulvliqSMiFuXOFdqsVzLtJsArhWE9pVXyMumKrhbtEbG4kJ/pwDTSFfxQ4EhS7QXSeju2Yr3t210e1PUGb7dPZERq8poVEcsi4k7SVcoxefRC0s5eNIJ0Muhu/AhgYemoqkKX9azUzPdgoZzb0nUfqPRioXsRPW/73qbdpCIfvW373owFXgGQtJdS82W7pPmkE0qP5ZC0kaRrJb0gaQHpIqLH6SPizog4NiJGk67+DwLOy6M3A/6vsA4fJp28NuhjOV4Fhncz/PCIGA68h1Q7XC8P3whYCyhut5veQnorMyUi1i1+gFk9TNtlW0bEctLJudvxwHOF7s1J5XipUI7vAcWaauU+BL3vcyuQNEzS5UrNiguAP9B1W/8EODF3n0S6//cmfVvPL0XEG90kO5wUTHvUrKAwA/haxQYeGhE/l7Q5KUJ+FFg/b/hHKLQHknbsRpgNrJdPxiWb9jL974H3dlNtL1pEqo2UbFQxvruylJqQjgIey4EC0nq7smK9DYuIr1cuIAo3eKPvT2QEnev5UWCHiia4HfLw0vgdC+N2LIyrRnk9KD0BdQmpil/aBx6n6z7QCLNJ1e6S3rZ9tyTtSTp5TMmDrgF+BWwaESOBy+ksR3fb/hukq/ntI2IEqWbbp3JHxN2kZoft8qCZwD9U7C9DIuLFHtKu9BDw9l7S+wPp6r70dNJLpJrvNoX0RuZyV867NJezt2OjFrMpbL98P2BcT+NJAbRkBum4Xa9QjhERsQP19RlSE9DueVsfVBwZEVNy3vchnQ9KTUd9Wc8rbN98boVUg+zRqggKg/JNkdKnjXTS/4ikPZQMk3SYpOHAMFKB2gEknU7nTt5QkdrcppJuXg+WtBepHbgnV5J2oF9J2lbpxt76+QbPoXmaB4B/UrqxdQjw7j5k5RpSe+fZdNYSIB2AR0h6b17eEKWb1eO6XcpKSDpK0qi8DXYHziE9cQSpyW0ZcI6ktSR9NA//Q/7+KfAJSWMlbQJ8knRVWw/r0LkPSOkhgW3rtOzeXAt8TNImSjdkP93XGSWNVLoZ/jPgxxExLY8aTqp9Ls4B4/jCbHOAUOEx4Dz9a8D83Gb/qV7SfLfSTc8Ncv87SPvr3/Ik3wf+U/kRVUkbqPOGfXdpV/odsJs6nzjrzoXAoZK2i4hlpKB3kaQxeb8ap55vwj4InJj35cNItd56uQnYKe/jbcDHgeK9t9K2HitpfVLTLAARMQP4I/BtpUfOByg9wrt/DfkZ3M15cDgp+Lya8/Af3cx3JekC6bWI+FvO31tdzyXvBn6faxs9WhVB4RZS+1zpc35ETAX+mdS++irpRtVpkJo0SE+y/JUUEbcH/rIK8llyIrAXqWnoq8AvSFc0K4j0VMZE0lXs7cAC0k3q0UDpyYFzSQfqvLzs61eWgYiYTSr/3jn90vAZpNrD50knzBmkE1e12/F40rrvIJ3kvxH59xy56vl+0hNW80httO8vVEl/QHpU8WFSTe7mPKxmEfEQ6Ybo3aQrum3pXJ+NdAkpGD5Muol7M+mKrDe/Vfq9xfOkm5rfIjURlZwN/JekDtJ2u7Y0IiI6gP8C7srNABOAL5JufM4HbiTVMnryKnA08EjOwy15+d/J4y8AbgUm5fTvBHbrJe0uIj1d9Gd6uTDKtY6r6bwZ/klSU8zduQy/o+cbm+fk/M8Djs3lrYuIeAk4jrQ9XibVBIr70CWkx8kfBu4h3ZQuOol0gfoYaT3/ktpqMrfR9Tz4b6TtMzLn706gu2ben5Iuiq+sGP5W1nPJiaQLhV6p+ibgNYOkX5BuKn2x2XmxVUvSEaRHct/W7Lw0i6TtgcsiYs9m52VNlJum5wDbRcQzNSxnZ+C/I2KltTEHhQpKz2u/AjxDasK5HtgrIu5vasas4fIBuB+p1rcx8BvgjxHRYxOOWSNJ+gzpCc63/DuIavW3X4+uChuRHlNcn3Sj7mwHhDWGSL+JuI7Urn8T6XcyZqucpJmk3y4ctUrTdU3BzMxK+v27j8zMrH5Wq+aj0aNHx/jx45udDTOzfuPee++dGxG1vOqmi9UqKIwfP56pU6c2OxtmZv2GpOdWPlXfufnIzMzKHBTMzKzMQcHMzMocFMzMrMxBwczMyhwUzMyszEHBzMzKWiMoLFoEV14JfmWHmVlNWiMofPKTcMopMHlys3NiZtavtUZQmJn/erWjo/fpzMysV60RFErU6L/wNTNrba0RFHwvwcysLlorKLimYGZWEwcFMzMrc1AwM7MyBwUzMytzUDAzszIHBTMzK3NQMDOzMgcFMzMrc1AwM7Oy1ggKJQ4KZmY1aY2g4NdcmJnVRWsFBdcUzMxq4qBgZmZlDgpmZlbmoGBmZmUOCmZmVtZaQcHMzGrSWkHBwcHMrCYOCmZmVtbQoCDp45IelfSIpJ9LGtKQhBwUzMzqomFBQdJY4BxgQkRsBwwEjm9UeoCDgplZjRrdfNQGrC2pDRgKzGpoasuXN3TxZmatrmFBISJeAL4NPA/MBuZHxO8qp5N0lqSpkqa2t7dXm1jXbzMzq0ojm49GAUcBWwCbAMMknVQ5XURcGhETImLCmDFjqkvMQcHMrC4a2Xw0EXgmItoj4k3g18DeDUnJQcHMrC4aGRSeB/aUNFSSgPcA0xqSkoOCmVldNPKewl3AdcB9wMM5rUsblFjXbzMzq0pbIxceEV8EvtjINHJCXb/NzKwq/kWzmZmVOSiYmVmZg4KZmZW1RlAo8S+azcxq0hpBwTUFM7O6cFAwM7MyBwUzMytzUDAzszIHBTMzK3NQMDOzMgcFMzMrc1AwM7MyBwUzMytrjaBQ4qBgZlaT1ggKpWDg11yYmdWktYKCawpmZjVxUDAzszIHBTMzK3NQMDOzMgcFMzMrc1AwM7MyBwUzMytzUDAzszIHBTMzK2uNoFDioGBmVpPWCAp+zYWZWV20VlBwTcHMrCYOCmZmVuagYGZmZQ4KZmZW5qBgZmZlDgpmZlbmoGBmZmUOCmZmVtbQoCBpXUnXSXpc0jRJezUkIQcFM7O6aGvw8r8L3BoRx0gaDAxtaGoOCmZmNWlYUJA0AtgfOA0gIt4A3mhIYn7NhZlZXTSy+WhLoB34kaT7JV0uaVjlRJLOkjRV0tT29vbqUnLzkZlZXTQyKLQBuwCXRMTOwGvA5yoniohLI2JCREwYM2ZMdSk5KJiZ1UUjg8JMYGZE3JX7ryMFifpzUDAzq4uGBYWIeBGYIWmbPOg9wGMNSqzrt5mZVaXRTx/9K3B1fvLoaeD0hqbmoGBmVpOGBoWIeACY0Mg0ckJdv83MrCr+RbOZmZU5KJiZWVlrBIUSBwUzs5q0RlDwL5rNzOqitYKCawpmZjVpjaAwc2b6dlAwM6tJawSF9ddP3w4KZmY1aY2gUOKgYGZWk9YJCpKDgplZjRwUzMyszEHBzMzKHBTMzKzMQcHMzMpaJygMGOCgYGZWo9YJCpJfc2FmVqPWCgquKZiZ1aRPQUHSlX0Z1lQOCmZmNetrTeFdxR5JA4Fd65+dGjgomJnVrNegIOk8SR3ADpIW5E8HMAe4YZXksK8cFMzMatZrUIiI/4qI4cC3ImJE/gyPiPUj4rxVlMe+cVAwM6tZX5uPbpI0DEDSSZIukLR5A/P11jkomJnVrK9B4RJgkaQdgc8AzwE/bViuquGgYGZWs74GhaUREcBRwHcj4rvA8MZlqwoOCmZmNWvr43Qdks4DTgb2y08fDWpctqrgoGBmVrO+1hSOA5YAH4qIF4GxwLcalqtq+DUXZmY161NQyIHgamCkpMOBxRGx+t1T8GsuzMxq0tdfNH8QuBs4FvggcJekYxqZsbfMzUdmZjXr6z2FLwC7RcQcAEljgN8D1zUqY2+Zg4KZWc36ek9hQCkgZC+/hXlXDQcFM7Oa9bWmcKuk24Cf5/7jgFsak6UqOSiYmdWs16AgaStgw4j4tKQPAPsCAv5KuvG8+nBQMDOr2cqagC4COgAi4tcR8YmI+DiplnBRozP3ljgomJnVbGVBYXxEPFQ5MCKmAuMbkqNqOSiYmdVsZUFhSC/j1q5nRmrmoGBmVrOVBYV7JP1z5UBJZwD39iUBSQMl3S/ppmoy2Gf+RbOZWc1W9vTRx4DfSDqRziAwARgMHN3HNM4FpgEjqsphX/kXzWZmNes1KETES8Dekg4EtsuDb46IP/Rl4ZLGAYcBXwM+UUtG+5CYawpmZjXq0+8UIuIO4I4qln8R6f8XenzNtqSzgLMANttssyqSKC/IQcHMrEYN+1VyfnHenIjo9d5DRFwaERMiYsKYMWNqSdBBwcysRo18VcU+wJGSngWuAQ6SdFXDUnNQMDOrWcOCQkScFxHjImI8cDzwh4g4qVHpOSiYmdVu9XqpXS0cFMzMatbXF+LVJCImA5MbmoiDgplZzVxTMDOzMgcFMzMra52g4NdcmJnVrHWCgl9zYWZWs9YKCq4pmJnVxEHBzMzKHBTMzKzMQcHMzMocFMzMrMxBwczMyhwUzMyszEHBzMzKHBTMzKysdYKCX3NhZlaz1gkKfs2FmVnNWisouKZgZlYTBwUzMytzUDAzszIHBTMzK3NQMDOzMgcFMzMrc1AwM7MyBwUzMytzUDAzs7LWCQoDBvgXzWZmNWqdoDBwICxb1uxcmJn1a60TFAYNgqVLm50LM7N+rXWCQlsbvPlms3NhZtavtU5QGDTIQcHMrEatExTa2tx8ZGZWo9YJCq4pmJnVzEHBzMzKWicouPnIzKxmrRMUXFMwM6tZw4KCpE0l3SFpmqRHJZ3bqLQABwUzszpoa+CylwKfjIj7JA0H7pV0e0Q81pDU3HxkZlazhtUUImJ2RNyXuzuAacDYRqXnmoKZWe1WyT0FSeOBnYG7uhl3lqSpkqa2t7dXn0gpKPhNqWZmVWt4UJC0DvAr4GMRsaByfERcGhETImLCmDFjqk+oLbeE+U2pZmZVa2hQkDSIFBCujohfNzItBg1K325CMjOrWiOfPhJwBTAtIi5oVDplpaDgm81mZlVrZE1hH+Bk4CBJD+TPoQ1LrdR85JqCmVnVGvZIakRMAdSo5a/AzUdmZjVrrV80g5uPzMxq0DpBwc1HZmY1a52g4OYjM7OatV5QcPORmVnVWicouPnIzKxmrRMU3HxkZlaz1gsKbj4yM6ta6wQFNx+ZmdWsdYKCm4/MzGrWekHBzUdmZlVrnaDg5iMzs5q1TlBw85GZWc1aLyi4+cjMrGqtExTcfGRmVrPWCQpuPjIzq1nrBIVRo2DAALjzzmbnxMys32qdoDB6NJx2Glx+OSxf3uzcmJn1S60TFAC22SY1H73+erNzYmbWL7VWUBg+PH13dDQ3H2Zm/ZSDgpmZlbVmUFi4sLn5MDPrp1orKKyzTvp2TcHMrCqtFRTcfGRmVpPWDAqHHw5nnw3PPNPc/JiZ9TOtFRRKzUcA3/8+bLstRDQvP2Zm/UxrBYVSTaHkjTdg+vTm5MXMrB9qraBQrCmUTJq06vNhZtZPtVZQKL0pFWDkSNh8c7j55ublx8ysn2mtoFA0Zw4cdxz89repu1k6OuCss+DVV5uXBzOzPmq9oPDHP6anjgYPhqOPhmXLmvvm1CuugMsug29+s3l5MDPro7aVT9LP7L9/Z/cWW6TvGTOakxfofGPra681Lw9mZn3UejWFojFjYK21mhsU3nij67eZ2WqstYPCgAGw6abw85/DvHnNycNLL6XvZqW/Onj5ZfjNb7oOmzbNvyGxVau9ves+9/rrsGRJZ/+iRfD886s+X6uZ1g4KkG4yz5yZ/pnt+ONhwYK+zbdoUZqvO8uWwZVXdu5QlS/ge+yxtANCZ1B46KHul/XcczB//orDZ8/uXEal5cvhySdTPu67D/78586ayKpsplq8OP1/xauvpjwVD7CSJUvSev/ABzp/M3LfffDOd8IFFzQmX+3tff9b1jfeaE5wmj07XSj0tI91p9Evemxvh1mzqpt3+fIU/Pti8eK0zhcsSCfh+fM7j8sFC+CVV9Lx151q18ELL8AGG8BXvpKCwWuvwbhxsM8+ndMcf3x6YnHx4r4vs6/HW+nY6A+/nYqIhn2AQ4AngOnA51Y2/a677hp1d9xxEWkX7PxsuWXEz34Wcf75EcuXRyxbFrF4cfp+9dX0ffDBado33oh4/vmIX/0qdUdEXH11Gvcv/xLx8MOp+7rr0rg330z9b3tbWvaBB3ame/HFEY8/npYfEbF0aRq+884r5hsihg1L3cuWpeUuXpz6v/KVNP4974loa0vd//RPEddem7ofe6xzOfPmdc4XkfJU8tJLqb+0DopeeSVi+vSIp5+OeOGFlP4bb0TMmhWxcGGaZtSoiH32iVh33YjDD09pX3991+Xsu29n+X/wgzTssstS/y67dJ+v0nqeOzflo9LcuRFz5qTu+fMjZs5M6/Lll1M+N9gg4pRTOtfdwoVdl18s49ixaT9Ytiyio6Nz3IIFaZml+ZYv7yx3RMSiRSvmNyLNE5HysnRpmq64/t98M6K9vev+eOONnduhUmnYlCkRa60VcfnlaRnF5c2cmdZDqb+Un+XL07Yr5amU146OrmUtOeigiPHjI5Ys6Zp2pY6OtO8U18GXv5zKMmVK53zFfaq0febNi5AivvCFzmMM0nZYvjzi7W9P/W1taV8r5vvii9O4v/41lXPBgu7zt3x5SqdU7lmzIr7//a7rfPTozu7S9in1T5q04vJeeSXioosiDjss5eXll9O0Rx+d1kd7e8SMGWl9L1uW0n799TT/pz4VsdFG6dzykY9EDBmS8ldadkdH57RVAKZGPc/b9VxYlwXDQOApYEtgMPAg8M7e5mlIUHjttYiHHor49rcj7rwz7bwjR64YKCBiwoSIAQMiTj21c9iYMV2nKZ38Sp/SSfm9700b9tZbO8ftsUfa+SZOTCeq4nxf+EIKJKX+hQsjZs9OB/i0aZ3Dr7yys3uDDSIeeCCdhIvLWm+9lI899kj9X/5yxIsvpoNn6NA0bOjQiHPPjdhii4gnn4z43/9Nww84IGK33dIyJ09OB8hzz3VNY+TIiB13jNhhh9S/664R99/f/TocMCBNP3RoxI9/vOL6ffbZrut3/PiI++5L4844Ix28pXUKEYMHR9x1VzqgJk+O2HTTNHzQoIif/jRi3LiuaVx+eWf3+9/f2b3//p0n09LJ/iMf6TrvsGERzzwT8d3vphPXLrtEbLVVxHe+E/HZz6ZpPv/5iHPOSdO+8ELEhRem9X/rrRFXXJG20c03dwa99dZLJ7wZM1K6O++84jo74oiI00+POPLItB8sWpROQn/6UzqZ3HprWueV8519dsTw4Z3b6C9/SRc8w4dHPPVUxH//d/fbqPS57bbOYDdrVtp2EPGNb0Qce2zEO96Rgv7JJ0c88UTE3/+eLnZK8++wQzoZPv/8isfU2WdHbLxxxA03pOPiox+N8oVRT/mp3F8gHbN33tn99G1tEffck7bp/Plp+957b8TAgWn8xz+eTr6jRvW+Hn75y65pnHBC2ralYPH1r3edvvJYrvy8/e1pW26+ecTUqZ3DDz20M28/+EG6QCxdNG21VTpXVaE/BYW9gNsK/ecB5/U2T0OCQncmTYrYaafeN2xfPjvu2Nk9YEA6gXU33S23RHziE7WnV/qsvXYKdNttl07ekybVb9mlz8CBEf/5n+mk15fpiwfK0KErPxBXNh7Siakv05VOOMVP8Uqw9BkxImKddVL3Wmul71L/yj5SfdZpsf/yy1Mw6Ov8pcBU/Oy/f8T3vtcZHEqf0gm+8vOP/xjxrW+lYDx4cNcADBGbbNK3vLzrXbWti5626zveEXHTTeniYNCgFcf/8pddLwQGDerc1tKK83S3Hs45p7O7ePEzalTEBz7QddrK9dPb55RTUkCtTHP06M4LlLXXTp/KeU88serTWb2DQiMfSR0LFB/7mQnsUTmRpLOAswA222yzBman4KCD4P774cUX4emn4amnYODA1Na4fHlqt5w9O/3nc0Rq9x03DjbbDJ54It283m671EZ5+eXp+/HHU9vo2LGpXTIiLfO11+CQQ2DffdPjsjvtBPfck9ojIbWtL1oEjz6afpE9ZEhKf/jwlL+2tjTvRhultsjp0+HAA2H77eGmm9I0e+wB116b7lvsvXf6XcbIkelG+7bbpqewnnoKHnkEdtsNHn44PZX1oQ/BbbfB0qUpjRtvTPnt6Ehvmt1335TH7bZLv/94/XXYccc0zZIlsO666X7CTjul9fPkk/CXv8Dpp6dXjlxySecvy/fbD265Jc3X0ZF+Q/L3v6f1/PzzKY+jRsHUqSlfL7yQyvn443D99WkdjR2blrfJJjB6NEyZkrbJEUek/M6aldbjuHEwcWJ6xcmAAanM06al6UeMSMtYuDBtt5NPhquvhg03THlZtCiV66ST4IYb0ra+++607I9+NK3ztjZYf3148MGU53Hj0vIWLoTx41Ob+H77wR13wNZbpzSnTEnt5SNHprw98wwccwwcdRTsvnvaD7fdFubOTfvg4MGpbXvMmJSvrbdObd6HHJLGzZ+f7kedeWbKz957w+9/D1tumdqt77orraMRI2DYsFS+xYvTuho8OO3znaaZAAAHRUlEQVQz11yT8jNyZMrzdtvBzjunBzMmTIC1106fO+9M6/Gpp+CAA9Lyjjwy7XPPPZfWxdKlsMsuaR0880y6PzFqVLq/NmZMSnvp0nRcTJwIe+0FF14Ip54KkyenY2zePDj22JQ2pOPl2mvTfrfNNrDxxqn8EyfCs8+m5f3iF2ldjBuX7rHNmZPKse666XiISNt5woS0fw4Zkrb5hz+ctseGG6b7g8OGwcEHw9vfnh6KmDcv3SN57TUYOhTOOCPtv6Vt+O53p/zccEO6L/G3v8EHP5jyvcsu8MADKV9SGr7VVvCjH8Hb3paOmVmz0vlhww3TMXzKKavm3NcHSoGmAQuWjgXeGxFn5v6Tgd0j4l97mmfChAkxderUhuTHzKwVSbo3IibUa3mNfPpoJrBpoX8cUOWjDWZmtio0MijcA2wtaQtJg4HjgRsbmJ6ZmdWoYfcUImKppI8Ct5GeRPphRDzaqPTMzKx2DX33UUTcAtzSyDTMzKx+Wv8XzWZm1mcOCmZmVuagYGZmZQ4KZmZW1rAfr1VDUjvwXBWzjgbm1jk7qzuXec2xJpbbZe67zSNiTL0ysVoFhWpJmlrPX/T1By7zmmNNLLfL3DxuPjIzszIHBTMzK2uVoHBpszPQBC7zmmNNLLfL3CQtcU/BzMzqo1VqCmZmVgcOCmZmVtbvg4KkQyQ9IWm6pM81Oz/1IumHkuZIeqQwbD1Jt0t6Mn+PysMl6eK8Dh6StEvzcl49SZtKukPSNEmPSjo3D2/ZcksaIuluSQ/mMn8pD99C0l25zL/Ir59H0lq5f3oeP76Z+a+FpIGS7pd0U+5v6TJLelbSw5IekDQ1D1vt9u1+HRQkDQS+B7wPeCdwgqR3NjdXdfNj4JCKYZ8DJkXE1sCk3A+p/Fvnz1nAJasoj/W2FPhkRLwD2BP4l7w9W7ncS4CDImJHYCfgEEl7At8ALsxlfhU4I09/BvBqRGwFXJin66/OBaYV+teEMh8YETsVfo+w+u3b9fzD51X9AfYCbiv0nwec1+x81bF844FHCv1PABvn7o2BJ3L3D4ATupuuP3+AG4B/WFPKDQwF7iP9l/lcoC0PL+/npP8n2St3t+Xp1Oy8V1HWcaST4EHATYDWgDI/C4yuGLba7dv9uqYAjAVmFPpn5mGtasOImA2QvzfIw1tuPeQmgp2Bu2jxcudmlAeAOcDtwFPAvIhYmicplqtc5jx+PrD+qs1xXVwEfAZYnvvXp/XLHMDvJN0r6aw8bLXbtxv6JzurgLoZtiY+Y9tS60HSOsCvgI9FxAKpu+KlSbsZ1u/KHRHLgJ0krQv8BnhHd5Pl735fZkmHA3Mi4l5JB5QGdzNpy5Q52yciZknaALhd0uO9TNu0Mvf3msJMYNNC/zhgVpPysiq8JGljgPw9Jw9vmfUgaRApIFwdEb/Og1u+3AARMQ+YTLqfsq6k0kVbsVzlMufxI4FXVm1Oa7YPcKSkZ4FrSE1IF9HaZSYiZuXvOaTgvzur4b7d34PCPcDW+amFwcDxwI1NzlMj3QicmrtPJbW5l4afkp9Y2BOYX6qS9idKVYIrgGkRcUFhVMuWW9KYXENA0trARNLN1zuAY/JklWUurYtjgD9EbnTuLyLivIgYFxHjScfsHyLiRFq4zJKGSRpe6gYOBh5hddy3m33zpQ43bw4F/k5qh/1Cs/NTx3L9HJgNvEm6ajiD1I46CXgyf6+XpxXpKayngIeBCc3Of5Vl3pdURX4IeCB/Dm3lcgM7APfnMj8C/EceviVwNzAd+CWwVh4+JPdPz+O3bHYZaiz/AcBNrV7mXLYH8+fR0rlqddy3/ZoLMzMr6+/NR2ZmVkcOCmZmVuagYGZmZQ4KZmZW5qBgZmZlDgrW8iQty2+mLH3q9jZdSeNVeJOtWX/X319zYdYXr0fETs3OhFl/4JqCrbHy++2/kf/P4G5JW+Xhm0ualN9jP0nSZnn4hpJ+k//74EFJe+dFDZR0Wf4/hN/lXyYj6RxJj+XlXNOkYpq9JQ4KtiZYu6L56LjCuAURsTvwP6T375C7fxoROwBXAxfn4RcDf4z03we7kH6ZCumd99+LiHcB84B/zMM/B+ycl/ORRhXOrJ78i2ZreZIWRsQ63Qx/lvQHN0/nF/G9GBHrS5pLenf9m3n47IgYLakdGBcRSwrLGA/cHulPUpD0WWBQRHxV0q3AQuB64PqIWNjgoprVzDUFW9NFD909TdOdJYXuZXTeqzuM9P6aXYF7C28ANVttOSjYmu64wvdfc/edpLd3ApwITMndk4CzofzHOCN6WqikAcCmEXEH6c9k1gVWqK2YrW585WJrgrXzP5uV3BoRpcdS15J0F+kC6YQ87Bzgh5I+DbQDp+fh5wKXSjqDVCM4m/Qm2+4MBK6SNJL0xssLI/1fgtlqzfcUbI2V7ylMiIi5zc6L2erCzUdmZlbmmoKZmZW5pmBmZmUOCmZmVuagYGZmZQ4KZmZW5qBgZmZl/x+c3OcGKQfavQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZkAAAEWCAYAAAC0Q+rDAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xm8XHV9//HXOzdAbiAhZJMtISpYdjBGVgXrVkREUSsgFJRNLFWwautWl7qBZbHWKqJA0SKKBYHixqLwK1WgYTUQEVRkNwESICSELJ/fH9/v8Z5MZu499+aeuXMn7+fjMY85c9bPmTlzPuf7/Z5FEYGZmVkdxox0AGZm1r2cZMzMrDZOMmZmVhsnGTMzq42TjJmZ1cZJxszMauMkY2Zto+T/JO04TPN7TNIrhmNeTea9kaQlkrZsMfxESdf0M/2Nko6sI7aRln/H2yRtN9C465RkJN0vaVn+If4k6XxJm6zLPIdbjvG1FcZ7oaTVkr7WjrhGiqQtJJ0r6VFJz0j6jaTPSNp4pGMbiKR3SPqlpKWSrmsyfHdJt+Tht0javTRMkk6T9ER+fUmSmszjiLw9L8nb9urS5yXrEPv2klYOMM6pklbk36X4bf5V0vRBLGeddmySZkm6TNLjkp6SdKekdw7jst8OPBwRd+dpinVeImmxpBskzRlq/FVJGicpJG3d0P9USd8CiIjlEbFJRDxSdzyDUY5xpES6wPIs4NMDjTscJZk3RcQmwGzg5cAnBjsDSWOHIY51dRSwCDhM0kbtXHC71l/SZOBXQC+wd0RMAF4HTAJePIT5tft3exL4MnBqk1g2BC4H/hPYDLgAuDz3BzgBeAuwG7ArcBDwnsb5RMSFeceyCfAG4JHic+5Xtwvy7zIF+GtgFjBX0rQ2LBvgIuAeYAYwFXg38Pgwzv9E4DsN/S7I3+004Ebg+8O4PBtmpf/9pcAbJU3pd4KIGPILuB94benzvwBX5u5NgXOBR4GHgc8BPXnYu4D/JWXCJ4HP5f7HA/OBZ4C7gdm5/5bAJcBC4A/A+0vL/DRwMfDtPN1dwJw87DvAamAZsAT4h37W5XfAe4E/AW9vGLYTcHWO9U/Ax3L/HuBjedpngFtIf85ZQABjS/O4Djiu1fqTdvI/B54g/akvBCaVpp+Rf9SFeZyvAhvl6XcpjTc9r++0Juv4OeDXwJgW38Fg4/4isBjYuTT+tLz86fnzQcDtebxfAruuyzaX53kccF1Dv9fn7Uylfg8AB+TuXwInlIYdC9w4wHJeBTzUpP8MUkJ7HPg9cGJp2L7AbcDTwGPAF3P/Bfm7XZJfL20y31OBbzX024D0nyj+I9OAn+Tt4MkcxxZ52BnAKuC5vIwzcv+vAw/lmG4G9mqxvgJWANv38528Ergp/563Avv2t+yGacfn+U9ttc6kg9UAJpT6HQLcmZf5P8COpWGPAa/I3d8DPlEadgBwX4v1GJeXs3Wr36BxHNJ/68f5e/wVafu/pjTtG4F7c5xnkhLmkaXh7yEl8CeBHwFbNSzneNK+ZBFwVj+/wVrbSWnYJ0n7yGeAecAbS9/908B2pXG3BpaS9zMVvucPkfavS0v9/wc4tL//0bC1yUiaARxI+oNBOpJcCWwLvJS0EziuNMmepD/odODzkv6alDCOAiYCBwNPSBoD/DdwB7AV8BrgFEl/VZrXwaQNbBJwBWkHTET8DWlH86ZIR6JfahH7K0lf+PdICeuo0rAJwDXAT0nJblvg2jz474HD83pPBI4h/WhVrLH+pD/4F/MydiDtyD6dY+gBrgT+SEoEWwHfi4jlOeZyFcXhpA1/YZNlvha4NCJWV4xxoLj/mZT4Di8NfwdwfUQskDQbOI/055oCfAO4oqaS4k7AnZG3/OzO3L8Yfkdp2B2lYZXl3+LHpKS1JWlH9jFJ++dRvgp8ISImAtsBl+X++wGroq9UdBsVRMQK0vb/ytxrDHA2MBN4Ye53Vh73g8D/kQ4KNsmfIe0QdyH9BpcDP5C0QZNlBSmBfCNXTTZWJc3K6/NxYDKp1uIySZv1s+yyHYCnI6JpyShvF39D2qEtyf32Ar5GKlFNIR04XjZCtR/nkBLEC0gHpMcUAyRtTtp3fJB0ILAQmFMafhhwCvCmPP1tpFJ32RtI+8rZwLslvWoIMd4D7EM6yD8N+J6kqRGxFPgv1txXHAH8KCIWV/yeDyXVfJRLLvNJtQOt9ZeBBnqRSjJLSJnvjznIXtKXuBzoLY17OPCL3P0u4IGGef0MOLnJMvZsMu5HgfNz96dZ82hiR2BZQ4yvHWA9vgVclrv3Jh1tTS/FfVuL6e4B3tyk/ywGLhE8MEBMbymWm2NaWJ5fw/fzILl0AswF3tFinvdSOuoejrhJiev3pc//CxyVu78OfLbJd7b/Om53zUoy/0RKvOV+FwKfzt2rKB2hkxJAUCr5NFnOq2goyQD7A/c29PsM8PXcfTNpJzylYZztgZUDrFfTI1TSzunXLabZC3i09HmNo+cm44t0IPQXLYZPJdVIzCfVAswll7qATwHfbBj/evKRbIVlvwa4v8k6LyftQ1aRSnz7loafD3y8YZo/Anvm7nUtyTyVl128nqNJSSZ3rwZmleZxJnnfQ6qOva40rCevy5H58y+AI0rDNyDtZ15QWs6c0vArgFMGs520GPc3wF+Vtt37SsN+DRw8iO/5nU3mfwbwtf5iGI6SzFsiYlJEbBMRfxsRy4Bt8pf4aG7MW0w6ii03YD7YMJ8ZpKJio22ALYv55Hl9jPTjFB4rdS8FxlU90pHUS6r7vhAgIn5FKv0UjZ2t4hpo2EDWWH9J0yV9T9LDkp4mHeVMLS3njxGxVsNxRNwEPAvsL2l7UknrihbLfALYYojxNo2bVMXXK2lPSdsAuwM/zMO2AT7Y8NvNIJUA1iDpY6UG9rOHENcSUmmybCKp2qDZ8InAksj/lEHYBpjVsE5/D2yehx9NavP5raSbGkrcQ7UV6QgaSRMknSfpgbydXEXfdtKUpI9KukfSU6SqmHGtpomIxyPiwxGxQ16n35JKq5DW/ciGdZ9Dk9+zhUXAhCb9vxMRk0jb5u9I21BhG1JJsbzMaaTvZDjslPdfk3IMX24x3uakBF3e/v9Y6t6yPCwiVpGqbwvbAGeX1mEhqaanXFps3I8Nug1Q0rH5ZI1iOdvS91v/P6BH0t5KJ8VsQap6LeIb6Htu/O9D+j0X9xdTXacwP0g6Opla+gEnRkS5eqLxz/0gzRufHwT+UN4QImJCRBxYMZaBdiKHkHY4X1M6HfIx0hdbVJm1iqu/Yc/m9/Glfps3jNMY1xdzv10jVbUcSdqoi+XM7CdxXpDH/xvgvyLiuRbjXQMckqsgmxl03JGq3i4mlfjeSWqTK3bsDwKfb/jtxkfERY0LjogvRF9V0okt4uvPXcCu0hpnjO2a+xfDy8X63UrDBuNB4DdNtsdD8nrMj4hDSQdUXwEuzScfDDaZAX9uZD2IVPcN8BHSjunleTt5PX3bCY3LkfQ64H2k7XwSqZprWcM0TUXEAtLR+iylsw8fJB1Bl9d944g4q9mym5gPTJDUKsEtIFWtfrE0zoPAJ5tsQ5c2mcWz9L/trovHSOs3o9RvZqn70fKw/B9r3EG/q2E9eiPiluEKUNJLgH8jlaom56R5H/m3zgdU36ZvX/G9SNWxRXwDfc/Nft8dWLMaei21JJmIeJR0hHWGpImSxkh6caneuplvAR+S9DIl2+Yj45uBpyX9o6ReST2Sdpb08orh/Al4UT/Djya1G+xCOoLandR4u7ukXUhtIZtLOkXpvPkJkvYsxfxZSdvlmHeVNCVSe8jDpKO+HknHMPDZWxPIVY+StgI+XBp2M2kjPlXSxkqnX+5bGv4d0k7kSNJG1MqZpIR6Qf5ukbSVpDMl7TrEuAG+S6qvPSJ3F74JnJhLOcqxv1GpnWvQckzjgLHAmPw9FG0L15GqW96ff6e/y/1/nt+/Dfx9Xt8tSXXn/zGEMG7IsZySlz82/+6zc/+j8jawilQVE6RqlgWko8iZLee85rpuIGknUgKfQEpY5O6lpO1kKmufzdm4vU8gVcssBDYktaON62e5p0vaMX/Xm5LOBpsXEc+SDmb+WtJr8vDe3F3szPv9r+VajutI7VOtxrmTdMRdtOmcA7xP0py8DW0i6WBJ45tMfjtwkKRJ+T/0vlbLGax84PbfwGfyeu9K2t4LVwAvl3RQ3iY/TErohbOBT0j6CwBJm0l62zqE1JO3v+K1Ianks5r0W4+RdCKpJFP2bVK76eGsua8YzPdMXoeNSfvNa1uNAwzv2WUNwzal76yWp0gNXYflYe8CbmgyzYmkOvslpDMjirrgLUmnVj5GKnLfWCyX1Cbzn6V5zKLUrgC8mVT9tRj4UMPytiIVWXdpEsuPgdNz9875i1yUY/hI9NW7foK+szn+j74zUd6Q+y8m1Vtez5ptGzc0LG8n0tlpS0h/lg9Sag8gHTVdRt/ZZ19pmP6a/Hu0bGMofZfn5fV4hlRn+ylg/FDiLs33PlKVzoYN/Q/I38tiUqL8AaUzhwa5vb0r/7bl13+Uhr80f4fLSGc+vbQ0TMCXcoxP5u6BvqtX0frssotJO9VFpHao/fKwi/Pv8wypzvvA0nSnkXYAi4Hdm8z3VFJCeIZ0VP5b0pHpFg3bwQ15O/kN8LeU2nrI9e45ri+Rqq2/Qzqz6GFS+86f2zGaxHBOnn5JjvVy1jwjad+8/EWkxHkFsGWzZbeY/9uAHzasc+MZdfvneCfnzwfn3/Up4BFS20tvHlZuk9mYVLX3NGl/8yGG9+yyzUknALU6u+xNef1bnV12LKn0/DSpqu3sVrHQ0L7UJMbG/8F9edjp+ftfmLe3tdrJ8u93T5P5VvqeS+P/DfDdgf63yiPbKCfpPNI1HYO+TsmsXXJ15k2kqqO7Rzqe9ZGk7wJ3R8Tn1mEeIiWkwyLit/2O6yQz+imdWno76cj9DyMbjZl1Kknbkkr5O0TEwwONPxx877JRTtJnSVWL/+IEY2atSPoSqRrxn9uVYMAlGTMzq5FLMmZmVptOuDHlgKZOnRqzZs0a6TDMzEaVW2655fGIaNfNVZsaFUlm1qxZzJ07d6TDMDMbVST9ceCx6uXqMjMzq42TjJmZ1cZJxszMauMkY2ZmtXGSMTOz2jjJmJlZbZxkzMysNk4yQ/X978NXvgKDvS3P8uVw/vmDn87MbBRykhmqww6Dk0+G3w3y6ctXXQXHHAO33lpPXGZmHcRJZijKpZDlywc37TP5ycRLlgxfPGZmHcpJZihWrerrXrGi9XjNLFu25ruZWRdzkhmKlSubd1exdOma72ZmXcxJZijKpReXZMzMWnKSGYp1STIuyZjZesRJZihckjEzq8RJZijcJmNmVomTzFC4JGNmVomTzFC4TcbMrBInmaEoJ5bBVpe5JGNm6xEnmaEoJxaXZMzMWnKSGQq3yZiZVeIkMxRukzEzq6TWJCPpZEnzJN0l6ZRS//dJuif3/1KdMdTCbTJmZpWMrWvGknYGjgf2AJ4HfirpR8DWwJuBXSNiuaTpdcUwLFauBAl6etKNMSOat8k8/zxssEEatz/DWZJZvhw22mjd52NmVpM6SzI7ADdGxNKIWAlcDxwCvBc4NSKWA0TEghpjGLx99oG99krdy5bBZpvB2LEpeYwdmz4//XTf+CtWpGfDbLQRHHTQwPMfbEnmoIPgJS9J3UccAVOmpO4zz4Rx4+Dss6vNx8xsBNSZZOYB+0maImk8cCAwA3gJ8EpJN0m6XtLLm00s6QRJcyXNXbhwYY1hNvjVr+Cmm1L3E0+s/dyXJUvg0Uf7Pq9Y0ffgsh//eOD5D7Yk86Mfwb33pu7vfheefDJ1z5+f3n/zm2rzMTMbAbUlmYiYD5wGXA38FLgDWEmqotsM2Av4MHCxtHYdU0ScExFzImLOtGnT6gqzf60SwVNP9XWvXDm4qq/hapPxCQRmNgrU2vAfEedGxOyI2A94ErgXeAi4NJKbgdXA1DrjGLJWO/DG6rKqCWPFir72nHVNDsX0PoHAzDpYbQ3/AJKmR8QCSTOBtwJ7k5LKq4HrJL0E2BB4vM44hqzVDrwxyVRNGMX8pHVPDsX0LsmYWQerNckAl0iaAqwAToqIRZLOA86TNI901tnRERE1xzE0QynJRLQ+w6yY3+TJsGhR/+M2avyKXJIxs1Gg1iQTEa9s0u954Mg6lztsWu3A+2uTWbECNtyw//lNnpxOKuhv3EbPP998Xi7JmFkH8xX/ZatXr9k9lJJMfzv9YlhxGvJQThhonJdLMmbWwZxkyp57rq972bKB22TGjFm7Taa/nX4xrEgyg0kQ5WWsWuWSjJmNCnW3yYwu5Z3+smUDl2R6e9c8YwyGtyRTvn1NObZyYnNJxsw6mEsyZeWd/tKlA7fJ9Pau3SYznCWZVtVw5So6l2TMrIM5yZQNtiQzfny9bTKN8RTKic0lGTPrYE4yZVVLMkXbTVFdVldJpjGe8nyG66JOM7MaOcmUVSnJbLJJeh8zJt0Uc+XKNO6ECal/O0oyRUlq4sR0avOqVf3Px8xshDjJlDWWFpqVNDbeOL2PHZteRUmmSulkuEoyRZvQUM5SMzNrIyeZssa2lWYljfHj0/sGG6RX0SZTpXQy3CUZJxkz63A+hbmsvNM/++y+W+yX9fam9yLJXHVVqjbbf//Uv0pJZvLk9H7BBelZMXvs0Xz8c89tHltjkrn88vRsmblz0+cXvxje977WcZiZtYmTTFmRBMaMgf/939T9ilfADTek7v33h+23h4cfhn33halT07Dly/sSR39J5rnn0m1kpk+HnXaCa69N87jooubjX3bZ2rEBPPNMen/5y+FnP0vJ6L77UvIZMyYt55hj+qr2zMxGiKvLyorSwsMPw+LF6XXeeX3Dr7sulXAWL4Yrr4TTTusbNnFiei9fmNloxYpU+tloI5g3D3bbrf8qszFj4Ljj1owN+hLO3nunJ2cuXw7PPgsnnwynn772+GZmI8RJpqzYeRdVYpCSQitF+wzAppum9/JV+o2KJFPo7e0/GaxalUojjY8GKKbZYIMUw7PPpuG9vX2xO8mYWQdwkikrdszl5NFfkikno6IkM5gkM358/9Vrq1alM9h6e1MiKRTTbLBBGrZoUV88RexOMmbWAZxkypYtg56eNRPB2H6arcrDiutk+qsuW7lyzWkGKsmsXJni6e1d887PxTRjx6ak8sQT6fP48X2Jz2ecmVkHcJIpW7p0zVIM9F+SKdt445QQhrsk09OTxisnmcaSTPGIApdkzKzDOMmUFe0aZVWTTG9v33UzrQylTaa/kkzRJlNwScbMOoyTTNm6lGTGjx98kumvJBORXs1KMuUkU06KLsmYWYdxkilrVpLpr02mrLc3jTtcbTLF/ciKkkz5kc9FYiraZAouyZhZh3GSKWtWkunpqTbtupRkItYet0hWRSLpr02m4JKMmXUYJ5myZiUZqdq0Q22TWb063Um5UWNJxm0yZjYKOcmUNSvJVDXUkgw0TwjlJOM2GTMbpZxkypqVZKoaapsMNE8IjSWZcpWa22TMbJRwkinr5JJMWX9tMj096SacLsmYWQeoNclIOlnSPEl3STqlYdiHJIWkqXXGMCjrWpIZSpsM9F+SKW4rU9aquqxIRr29LsmYWUeo7Vb/knYGjgf2AJ4HfirpRxFxr6QZwOuAB+pafmXLlvW1dzz77LqXZJYuTacbb7opPPkkTJqU7qYMqbqsv5JMBCxcmPoX1W4DlWTKw4qEU9xq5k9/Gtq6mFnnmDQp3bl9lKrzeTI7ADdGxFIASdcDhwBfAs4C/gG4vMblD2zVKpg1CxYs6OtX3Oiyqp12grvuSjv8sWPh6qvTRvG5z8EnPgHvehecfz5st1165suLXtQ3bZEU9twTPvnJtCF9/OPpAWTF82x6etaOqXzvsuLuz9CXwCZOhAsvTC8zG91+8hM44ICRjmLI6kwy84DPS5oCLAMOBOZKOhh4OCLuUD+nB0s6ATgBYObMmfVE+PzzKcEcfHD6EceMgbe8Ze3x7rgDNtus+Tyuuy4lD1izlHLdden97rvTe7NxyqWQM86AQw9N3c89Bw89lLp7euDYY1PiGjcOjjoqPT9GSsO22QZ+8IM0vPg+L7gAbr216rdgZp1sxx1HOoJ1UluSiYj5kk4DrgaWAHcAK4GPA6+vMP05wDkAc+bMaXK14rAEmd733hve+97W4+26a+thU6emF6yZQIo7Izdq1iZTKLejLFmS3nt60lM3jz8+fT7xxFSSKc/n7W9fcz577pleZmYjrNaG/4g4NyJmR8R+wJPA/cALgTsk3Q9sDdwqafM64+gnwPRe9YLLgZRPTy6STOOJAOVxGttayicAFI9YbrzjQDFN1dvdmJmNoLrPLpue32cCbwW+HRHTI2JWRMwCHgJmR8RjdcbRUjuSTONZXlVLMkWSaUwmxTRVb9xpZjaC6j4cviS3yawAToqIRTUvb3CGO8mUL8QsnmTZeHpyf0lm6VKYMiUlqIFKMk4yZjYK1JpkIuKVAwyfVefyBzTcSabZNTL9lWSanZpcJJlym0yZSzJmNoqs31f8tyPJLF3a9+RK6LtmBtY+970oyYDbZMysKwyYZCRdIumNkrovIRVJZswwrVqz+5YtW7ZmaaY8TmNyK0oy4JKMmXWFKnvXrwPvBO6VdKqk7WuOqX3aUZJZvXrNB471d9uZwZRknGTMbBQYMMlExDURcQQwm3QK8tWSfinp3ZJG956uHUkG1rxmpr8ks2xZuuhT8tllZtYVKtUT5TPE3gUcB9wG/Csp6VxdW2Tt0GlJprgLdG+v22TMrCsMuKeSdCmwPfAd4E0R8Wge9H1Jc+sMrnZ1nsJcVk4yrcZZsSLdS6148JjbZMysC1Q5HP5qRPy82YCImDPM8bRXXSWZTTftuxPzU09VK8kUj2CuWpJxkjGzUaBKddkOkiYVHyRtJulva4ypfYpTi4c7yRSN98V71eoyqF6ScXWZmY0CVZLM8RGxuPiQr9o/vr6Q2qiukszkyel9XZLMQA3/5WtvzMw6VJUkM0ale/JL6gE2rC+kNqqrTaZ4/kuRZM44Y+1xWimqywqtqssGSlZmZh2gSpL5GXCxpNdIejVwEfDTesNqk+FOMhdcALNnw1vfmp7zctRRaw7fYAP47GfX7Hf66TBjRnqY2Utekh4rUL4TQGOS2WMPeOEL4ZBDhidmM7MaVanY/0fgPcB7AQFXAd+qM6i2Ge4k86Y3pRfASSel9zvugNNOS90PPACbNzzV4IMfTK+ycmJpTDL77AO///3wxGtmVrMBk0xErCZd9f/1+sNps+FOMs2Uq74ab4jZSn9JxsxsFKlyncx2wBeBHYFxRf+IeFHLiUaLdiSZcmJpvLV/K+XGficZMxvFqrTJnE8qxawE/hL4NunCzNGv3SWZqte2lBOLT1U2s1GsSpLpjYhrAUXEHyPi08Cr6w2rTdpdkqnK1WVm1iWqHCY/l2/zf6+kvwMeBqbXG1abtLskU5WTjJl1iSolmVOA8cD7gZcBRwJH1xlU27gkY2ZWq35LMvnCy3dExIeBJcC72xJVuwz3Q8uacUnGzNZj/e5dI2IV8LLyFf9dpVNLMj67zMy6RJU2mduAyyX9AHi26BkRl9YWVbuMhjYZn11mZqNYlT3YZOAJ1jyjLAAnmSrcJmNm67EqV/x3VztM2WgoyTjJmNkoVuWK//NJJZc1RMQxFaY9mfRYAAHfjIgvS/oX4E3A88DvgHeXHyXQVi7JmJnVqsppVVcCP8qva4GJpDPN+iVpZ1KC2QPYDTgo36LmamDniNgV+C3w0aGFPgxckjEzq1WV6rJLyp8lXQRcU2HeOwA3RsTSPN31wCER8aXSODcCb68e7jDr1CRTNPaPGVNvbGZmNRvKBSLbATMrjDcP2E/SFEnjgQOBGQ3jHAP8ZAgxDI/hfvxyM0XCmD27+jRF6cWlGDMb5aq0yTzDmm0yj5GeMdOviJgv6TRS9dgS4A7STTaL+X48f76wxXJPAE4AmDmzSk4bgnaUZADuvDM9mKyqIrn49GUzG+WqVJdNGOrMI+Jc4FwASV8AHsrdRwMHAa+JiLVOKsjTngOcAzBnzpym46yzdiWZXXYZ3PhFkql612Yzsw41YHWZpEMkbVr6PEnSW6rMXNL0/D4TeCtwkaQDSCWhg4v2mhHTriQzWE4yZtYlqrTJfCoinio+5NONP1Vx/pdIuhv4b+CkiFgEfBWYAFwt6XZJZw826GHT6UnGbTJmNspVqfRvlogqNRZExCub9Nu2yrRt0elJptPiMjMbpColmbmSzpT0YkkvknQWcEvdgbVFpyaZ8inMZmajWJW92PtIV+d/H7gYWAacVGdQbdOpScYlGTPrElXOLnsW+EgbYmm/Tk8yLsmY2ShX5eyyqyVNKn3eTNLP6g2rTZxkzMxqVWUvNrV8A8t8htj0+kJqo3Y8GXMonGTMrEtU2Yutzte5ACBpG5rclXlU6vSSTKfFZWY2SFVORf44cEO+wSXAfsB76gupjTo1yRRnl3VaXGZmg1Sl4f+nkmYDe5GeC/OBiHi89sjaoVOTjKvLzKxLVNqLRcTjEXElcDdwoqR59YbVJk4yZma1qnJ22RaSTpF0M3AX0AMcXntk7dDpSabT4jIzG6SWSUbS8ZJ+DlwPTAWOAx6NiM9ExK/bFWCtOj3JuCRjZqNcf20y/w78CnhnRMwFkNQdZ5UVnGTMzGrVX5LZEvhr4ExJLyDdUqa77j3fqUnGZ5eZWZdoeaicG/u/HhH7Aa8BngIWSJqfH0A2+rXj8ctD4TYZM+sSVc8ueygiTo+IlwFvAZbXG1abdGpJxtVlZtYlBv0Q+Yi4B/hMDbG0n5OMmVmt1u+9WKcnmU6Ly8xskJxkoPN25i7JmFmXqHIx5rVV+o1KnZ5kOi0uM7NBatkmI2kcMB6YKmkz0n3LACaSTm8e/To1yfjxy2bWJfpr+H8PcAopodxCX5J5mnSh5ujXqUmmiKfT4jIzG6SWSSYi/hX4V0nvi4h/a2NM7dOpDy3r1LjMzAapyq3+/03SPsCs8vgR8e0a42qPTi3JOMmYWZcYMMlI+g7wYuB2YFW5/gERAAANeUlEQVTuHYCTTF2KOxE4yZjZKFflYsw5wI4RMeibY0o6GTie1J7zzYj4sqTJwPdJJaP7gXdExKLBzntYdHqS6bS4zMwGqcqh8jxg88HOWNLOpASzB7AbcJCk7YCPANdGxHbAtfnzyOjUJNOpcZmZDVKVksxU4O780LI/37MsIg4eYLodgBsjYimApOuBQ4A3A6/K41wAXAf846CiruqBB2DhwtbD77knvXfaztxtMmbWJaokmU8Pcd7zgM9LmgIsAw4E5gIviIhHASLiUUnThzj/gZ12GnztawOP12lJZost0vs++4xsHGZm66jK2WXXS9oG2C4irpE0nvQI5oGmmy/pNOBqYAlwB7CyamCSTgBOAJg5c2bVydZ04olwwAHNh82bBx/7WLGwoc2/LjvumOLbfvuRjsTMbJ1UObvseNLOfjLpLLOtgLNJz5jpV0ScC5yb5/MF4CHgT5K2yKWYLYAFLaY9BzgHYM6cOUN7Iucuu6RXM5tt1tfdaUkGYKedRjoCM7N1VqXS/yRgX9KV/kTEvUClKq6iKkzSTOCtwEXAFcDReZSjgcsHF/IwKbd3dGKSMTPrAlXaZJZHxPPKO2JJY0nXyVRxSW6TWQGcFBGLJJ0KXCzpWOAB0iOe289JxsysdlWSzPWSPgb0Snod8LfAf1eZeUS8skm/J6hQ1Va7cmJxkjEzq0WV6rKPAAuBX5Numvlj4BN1BtUWLsmYmdWuSkmmFzgvIr4JIKkn91taZ2C1c5IxM6tdlZLMtaSkUugFrqknnDZykjEzq12VJDMuIpYUH3L3+PpCahMnGTOz2lVJMs9Kml18kPQy0hX8o5uTjJlZ7aq0yZwM/EDSI/nzFsCh9YXUJk4yZma16zfJSBoDbAhsD/wF6Zb9v4mIFW2IrV7lJOMbUZqZ1aLfJBMRqyWdERF7k2542T1ckjEzq12VQ/irJL1N6rI9sZOMmVntqrTJ/D2wMbBK0jJSlVlExMRaI6ubr/g3M6tdlVv9T2hHIG3nkoyZWe0GrC5TcqSkf8qfZ0jao/7QauYkY2ZWuyptMl8D9gbemT8vAf69tojaxUnGzKx2Vdpk9oyI2ZJuA8i369+w5rjq5yRjZla7KiWZFfmmmAEgaRqwutao2sFJxsysdlWSzFeAHwLTJX0euAH4Qq1RtYOTjJlZ7aqcXXahpFtIDxoT8JaImF97ZHVzkjEzq13LJCNpHHAisC3pgWXfiIiV7Qqsdk4yZma166+67AJgDinBvAE4vS0RtYuTjJlZ7fqrLtsxInYBkHQucHN7QmoTX/FvZla7/koyf77TcldVkxVckjEzq11/JZndJD2duwX05s/dce8yJxkzs9q1TDIR0dPOQNrOScbMrHbr79O6/NAyM7Pa1bp3lfQBSXdJmifpIknjJL1G0q2Sbpd0g6Rt64yhJZdkzMxqV1uSkbQV8H5gTkTsDPQAhwFfB46IiN2B7wKfqCuGfjnJmJnVru56orGkEwbGAuOBR0j3QCtOGtg092s/Jxkzs9pVuQvzkETEw5JOBx4AlgFXRcRVko4Dfpyfsvk0sFez6SWdAJwAMHPmzOEP0EnGzKx2dVaXbQa8GXghsCWwsaQjgQ8AB0bE1sD5wJnNpo+IcyJiTkTMmTZtWh0BNu82M7NhU2d12WuBP0TEwohYAVwK7AvsFhE35XG+D+xTYwzVOMmYmdWiziTzALCXpPGSRLqL893AppJeksd5HTDyd3R2kjEzq0WdbTI3Sfov4FZgJXAbcA7wEHCJpNXAIuCYumKozEnGzKwWtSUZgIj4FPCpht4/zK/O4SRjZlYLX+oOTjJmZjVxkgEnGTOzmjjJgJOMmVlNnGTAScbMrCZOMuAkY2ZWEycZcJIxM6uJkww4yZiZ1cRJBvzQMjOzmnjvCi7JmJnVxEkGnGTMzGriJANOMmZmNXGSMTOz2jjJmJlZbZxkzMysNk4yZmZWGycZMzOrjZOMmZnVxknGzMxq4yRjZma1cZIxM7PaOMmYmVltnGTMzKw2TjJmZlYbJxkzM6tNrUlG0gck3SVpnqSLJI1T8nlJv5U0X9L764zBzMxGzti6ZixpK+D9wI4RsUzSxcBhgIAZwPYRsVrS9LpiMDOzkVVbkinNv1fSCmA88AjwOeCdEbEaICIW1ByDmZmNkNqqyyLiYeB04AHgUeCpiLgKeDFwqKS5kn4iabtm00s6IY8zd+HChXWFaWZmNaotyUjaDHgz8EJgS2BjSUcCGwHPRcQc4JvAec2mj4hzImJORMyZNm1aXWGamVmN6mz4fy3wh4hYGBErgEuBfYCHgEvyOD8Edq0xBjMzG0F1JpkHgL0kjZck4DXAfOAy4NV5nP2B39YYg5mZjaDaGv4j4iZJ/wXcCqwEbgPOAXqBCyV9AFgCHFdXDGZmNrJqPbssIj4FfKqh93LgjXUu18zMOoOv+Dczs9o4yZiZWW2cZMzMrDZOMmZmVhsnGTMzq42TjJmZ1cZJxszMauMkY2ZmtXGSMTOz2jjJmJlZbZxkzMysNnU/GbOzXX89/OEPIx2FmVnXWr+TzH77pZeZmdXC1WVmZlYbJxkzM6uNk4yZmdXGScbMzGrjJGNmZrVxkjEzs9o4yZiZWW2cZMzMrDaKiJGOYUCSFgJ/HOLkU4HHhzGc0cDrvH7wOq8f1mWdt4mIacMZzGCNiiSzLiTNjYg5Ix1HO3md1w9e5/XDaF9nV5eZmVltnGTMzKw260OSOWekAxgBXuf1g9d5/TCq17nr22TMzGzkrA8lGTMzGyFOMmZmVpuuTTKSDpB0j6T7JH1kpOMZLpLOk7RA0rxSv8mSrpZ0b37fLPeXpK/k7+BOSbNHLvKhkzRD0i8kzZd0l6STc/+uXW9J4yTdLOmOvM6fyf1fKOmmvM7fl7Rh7r9R/nxfHj5rJONfF5J6JN0m6cr8eX1Y5/sl/VrS7ZLm5n5dsX13ZZKR1AP8O/AGYEfgcEk7jmxUw+Y/gAMa+n0EuDYitgOuzZ8hrf92+XUC8PU2xTjcVgIfjIgdgL2Ak/Lv2c3rvRx4dUTsBuwOHCBpL+A04Ky8zouAY/P4xwKLImJb4Kw83mh1MjC/9Hl9WGeAv4yI3UvXxHTH9h0RXfcC9gZ+Vvr8UeCjIx3XMK7fLGBe6fM9wBa5ewvgntz9DeDwZuON5hdwOfC69WW9gfHArcCepCu/x+b+f97OgZ8Be+fusXk8jXTsQ1jXrUk71FcDVwLq9nXO8d8PTG3o1xXbd1eWZICtgAdLnx/K/brVCyLiUYD8Pj3377rvIVeJvBS4iS5f71xtdDuwALga+B2wOCJW5lHK6/Xndc7DnwKmtDfiYfFl4B+A1fnzFLp/nQECuErSLZJOyP26YvseO9IB1ERN+q2P52p31fcgaRPgEuCUiHhaarZ6adQm/UbdekfEKmB3SZOAHwI7NBstv4/6dZZ0ELAgIm6R9Kqid5NRu2adS/aNiEckTQeulvSbfsYdVevdrSWZh4AZpc9bA4+MUCzt8CdJWwDk9wW5f9d8D5I2ICWYCyPi0ty769cbICIWA9eR2qMmSSoODsvr9ed1zsM3BZ5sb6TrbF/gYEn3A98jVZl9me5eZwAi4pH8voB0QLEHXbJ9d2uS+T9gu3xWyobAYcAVIxxTna4Ajs7dR5PaLIr+R+WzUfYCniqK36OJUpHlXGB+RJxZGtS16y1pWi7BIKkXeC2pMfwXwNvzaI3rXHwXbwd+HrnCfrSIiI9GxNYRMYv0n/15RBxBF68zgKSNJU0ouoHXA/Polu17pBuF6noBBwK/JdVjf3yk4xnG9boIeBRYQTqiOZZUD30tcG9+n5zHFeksu98BvwbmjHT8Q1znV5CqA+4Ebs+vA7t5vYFdgdvyOs8DPpn7vwi4GbgP+AGwUe4/Ln++Lw9/0Uivwzqu/6uAK9eHdc7rd0d+3VXsr7pl+/ZtZczMrDbdWl1mZmYdwEnGzMxq4yRjZma1cZIxM7PaOMmYmVltnGRsvSZpVb7zbfEatjt2S5ql0t2yzdZH3XpbGbOqlkXE7iMdhFm3cknGrIn8fI/T8jNdbpa0be6/jaRr83M8rpU0M/d/gaQf5ue/3CFpnzyrHknfzM+EuSpfvY+k90u6O8/neyO0mma1c5Kx9V1vQ3XZoaVhT0fEHsBXSffQInd/OyJ2BS4EvpL7fwW4PtLzX2aTrtyG9MyPf4+InYDFwNty/48AL83zObGulTMbab7i39ZrkpZExCZN+t9PemjY7/PNOR+LiCmSHic9u2NF7v9oREyVtBDYOiKWl+YxC7g60kOnkPSPwAYR8TlJPwWWAJcBl0XEkppX1WxEuCRj1lq06G41TjPLS92r6GsHfSPp/lMvA24p3WXYrKs4yZi1dmjp/Ve5+5ekOwQDHAHckLuvBd4Lf37Y2MRWM5U0BpgREb8gPaBrErBWacqsG/joydZ3vfnpk4WfRkRxGvNGkm4iHYwdnvu9HzhP0oeBhcC7c/+TgXMkHUsqsbyXdLfsZnqA/5S0KemOumdFemaMWddxm4xZE7lNZk5EPD7SsZiNZq4uMzOz2rgkY2ZmtXFJxszMauMkY2ZmtXGSMTOz2jjJmJlZbZxkzMysNv8f0Q8jYgp5EzcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# gradient descent\n",
    "detailed_logger = False\n",
    "main_logger = True\n",
    "main_logger_output_epochs = 100\n",
    "L2 = False\n",
    "Dropout = False\n",
    "momentum = False\n",
    "adam = True\n",
    "hidden_layer_relu = True\n",
    "hidden_layer_tanh = False\n",
    "hidden_layer_sigmoid = False\n",
    "\n",
    "# hyber-parameters\n",
    "alpha = .1;\n",
    "epsilon = .85\n",
    "keep_prob = .9\n",
    "number_of_epochs = 500\n",
    "batch_size = 50\n",
    "momentum_coef = .9\n",
    "RMSProp_coef = .9\n",
    "epsilon = 1e-20\n",
    "t = 0\n",
    "\n",
    "# copy initalization\n",
    "W = Weights.copy()\n",
    "B = Bias.copy()\n",
    "\n",
    "# data arrays\n",
    "cost_array = []\n",
    "accuracy_array = []\n",
    "interation_array = []\n",
    "\n",
    "# rename\n",
    "X_train = np.float64(training_images).copy()\n",
    "Y_train = np.float64(training_labels).copy()\n",
    "\n",
    "X_test = np.float64(testing_images).copy()\n",
    "Y_test = np.float64(testing_labels).copy()\n",
    "\n",
    "#m = size\n",
    "m = number_of_training_images\n",
    "\n",
    "def model(W, B, A):\n",
    "    return np.dot(W, A) + B\n",
    "\n",
    "def activation_relu(Z):\n",
    "    Z = np.where(~np.isnan(Z), Z, 0)\n",
    "    Z = np.where(~np.isinf(Z), Z, 0)\n",
    "    return np.where(Z > 0, Z, 0)\n",
    "\n",
    "def activation_tanh(Z):\n",
    "    return np.tanh(Z)\n",
    "\n",
    "def activation_sigmoid(Z):\n",
    "     return 1/(1 + np.exp(-Z))\n",
    "\n",
    "def loss(A, Y):\n",
    "    epsilon = 1e-20\n",
    "    return np.where((Y == 1), np.multiply(-Y, np.log(A + epsilon)), -np.multiply((1 - Y), np.log(1 - A + epsilon)))\n",
    "    #return np.multiply(-Y, np.log(A)) - np.multiply((1 - Y), np.log(1 - A)) \n",
    "    \n",
    "def cost(L):\n",
    "    return np.multiply(1/L.shape[1], np.sum(L))\n",
    "\n",
    "def cost_L2(L, W, epsilon):\n",
    "    L2 = np.multiply(epsilon/(2*W.shape[1]), np.multiply(W[len(W)-3], W[len(W)-3]).sum() + np.multiply(W[len(W)-2], W[len(W)-2]).sum() + np.multiply(W[len(W)-1], W[len(W)-1]).sum())\n",
    "    J = cost(L)\n",
    "    return L2 + J\n",
    "\n",
    "def prediction(A):\n",
    "    return np.where(A >= 0.5, 1, 0)\n",
    "    \n",
    "def accuracy(prediction, Y):\n",
    "    return 100 - np.multiply(100/Y.shape[0], np.sum(np.absolute(Y - prediction)))  \n",
    "    \n",
    "def forward_propagation_return_layers(W, B, A, A_layers, Z_layers, layer, D, keep_prob):\n",
    "    if(layer < len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        Z_layers.append(Z)\n",
    "        if(hidden_layer_relu == True):\n",
    "            A = activation_relu(Z)\n",
    "        elif(hidden_layer_tanh == True):\n",
    "            A = activation_tanh(Z)\n",
    "        elif(hidden_layer_sigmoid == True): \n",
    "            A = activation_sigmoid(Z)\n",
    "        if(Dropout == True):\n",
    "            _D = np.float64(np.where(np.random.uniform(0, 1, A.shape) < keep_prob, 1, 0))\n",
    "            D.append(_D)\n",
    "            A = np.multiply(A, _D)\n",
    "        A_layers.append(A)\n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Training Data: ' + str(layer))\n",
    "        A_layers, Z_layers, D = forward_propagation_return_layers(W, B, A, A_layers, Z_layers, layer, D, keep_prob)\n",
    "    elif(layer == len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        Z_layers.append(Z)\n",
    "        A = activation_sigmoid(Z)\n",
    "        if(Dropout == True):\n",
    "            _D = np.float64(np.where(np.random.uniform(0, 1, A.shape) < keep_prob, 1, 0))\n",
    "            D.append(_D)\n",
    "            A = np.multiply(A, _D)\n",
    "        A_layers.append(A)\n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Training Data: ' + str(layer))\n",
    "            print('Forward Propagation Training Data Complete')\n",
    "    return A_layers, Z_layers, D\n",
    "\n",
    "def forward_propagation(W, B, A, layer):\n",
    "    if(layer < len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        if(hidden_layer_relu == True):\n",
    "            A = activation_relu(Z)\n",
    "        elif(hidden_layer_tanh == True):\n",
    "            A = activation_tanh(Z)\n",
    "        elif(hidden_layer_sigmoid == True): \n",
    "            A = activation_sigmoid(Z)\n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Testing Data: ' + str(layer))\n",
    "        A = forward_propagation(W, B, A, layer)\n",
    "    elif(layer == len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        A = activation_sigmoid(Z)  \n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Testing Data: ' + str(layer))\n",
    "            print('Forward Propagation Testing Data Complete')\n",
    "    return A\n",
    "\n",
    "def dZ(dZ, W, Z):\n",
    "    Z = np.where(~np.isnan(Z), Z, 0)\n",
    "    W = np.where(~np.isnan(W), W, 0)\n",
    "    dZ = np.where(~np.isnan(dZ), dZ, 0)\n",
    "    Z = np.where(~np.isinf(Z), Z, 0)\n",
    "    W = np.where(~np.isinf(W), W, 0)\n",
    "    dZ = np.where(~np.isinf(dZ), dZ, 0)\n",
    "    if(hidden_layer_relu == True):\n",
    "        return np.multiply(np.dot(np.transpose(W), dZ), np.where(Z > 0, 1, 0))\n",
    "    elif(hidden_layer_tanh == True):\n",
    "        A = activation_tanh(Z)\n",
    "        return np.multiply(np.dot(np.transpose(W), dZ), 1- np.multiply(A, A))\n",
    "    elif(hidden_layer_sigmoid == True): \n",
    "        A = activation_sigmoid(Z)\n",
    "        return np.multiply(np.dot(np.transpose(W), dZ), np.multiply(A, (1-A)))\n",
    "\n",
    "def dW(dZ, A):\n",
    "    return np.multiply(1/dZ.shape[1], np.dot(dZ, np.transpose(A)))\n",
    "\n",
    "def dW_L2(dZ, A, W, epsilon):\n",
    "    return np.multiply(epsilon/Z.shape[1], W) + dW(dZ, A)\n",
    "\n",
    "def dB(dZ):\n",
    "    return np.multiply(1/dZ.shape[1], np.sum(dZ))\n",
    "\n",
    "def backward_propagation(W, B, Y, A_layers, Z_layers, _dZ, alpha, epsilon, layer, D, V_dW, V_dB, R_dW, R_dB, t):\n",
    "    if(layer >= 0):\n",
    "        if(layer == len(W) - 1):\n",
    "            _dZ = A_layers[layer+1] - Y\n",
    "        elif(layer >= 0):\n",
    "            _dZ = dZ(_dZ, W[layer+1], Z_layers[layer])\n",
    "        if(Dropout == True):\n",
    "            _dZ = np.multiply(_dZ, D[layer])\n",
    "        if(L2 == True):\n",
    "            _dW = dW_L2(_dZ, A_layers[layer], W[layer], epsilon)\n",
    "        else:\n",
    "            _dW = dW(_dZ, A_layers[layer])\n",
    "        _dB = dB(_dZ)\n",
    "        if(adam == True):\n",
    "            epsilon = 1e-6\n",
    "\n",
    "            # ADAM - RMSProp + Momentum\n",
    "            V_dW[layer] = np.multiply(momentum_coef, V_dW[layer]) + np.multiply(1-momentum_coef, _dW)\n",
    "            V_dB[layer] = np.multiply(momentum_coef, V_dB[layer]) + np.multiply(1-momentum_coef, _dB)\n",
    "            R_dW[layer] = np.multiply(RMSProp_coef, R_dW[layer]) + np.multiply(1-RMSProp_coef, np.multiply(_dW, _dW))\n",
    "            R_dB[layer] = np.multiply(RMSProp_coef, R_dB[layer]) + np.multiply(1-RMSProp_coef, np.multiply(_dB, _dB))\n",
    "            \n",
    "            # index decay in bias correction\n",
    "            t = t + 1\n",
    "            \n",
    "            # correct bias for initial rounds\n",
    "            V_dW[layer] = np.multiply(V_dW[layer], 1/(1-np.power(momentum_coef, t)))\n",
    "            V_dB[layer] = np.multiply(V_dB[layer], 1/(1-np.power(momentum_coef, t)))\n",
    "            R_dW[layer] = np.multiply(R_dW[layer], 1/(1-np.power(RMSProp_coef, t)))\n",
    "            R_dB[layer] = np.multiply(R_dB[layer], 1/(1-np.power(RMSProp_coef, t)))\n",
    "            \n",
    "            val1 = 1/(np.sqrt(R_dW[layer])+ epsilon)\n",
    "            val2 = 1/(np.sqrt(R_dB[layer])+ epsilon)\n",
    "                      \n",
    "            W[layer] = W[layer] - np.multiply(alpha, np.multiply(V_dW[layer], val1 ))\n",
    "            B[layer] = B[layer] - np.multiply(alpha, np.multiply(V_dB[layer], val2 ))\n",
    "        elif(momentum == True):\n",
    "            V_dW[layer] = np.multiply(momentum_coef, V_dW[layer]) + np.multiply(alpha, _dW)\n",
    "            V_dB[layer] = np.multiply(momentum_coef, V_dB[layer]) + np.multiply(alpha, _dB)\n",
    "            W[layer] = W[layer] - V_dW[layer]\n",
    "            B[layer] = B[layer] - V_dB[layer]           \n",
    "        else:\n",
    "            W[layer] = W[layer] - np.multiply(alpha, _dW)\n",
    "            B[layer] = B[layer] - np.multiply(alpha, _dB)\n",
    "        if(detailed_logger == True):\n",
    "            print('Backward Layer: ' + str(layer))\n",
    "        layer = layer - 1\n",
    "        W, B, t = backward_propagation(W, B, Y, A_layers, Z_layers, _dZ, alpha, epsilon, layer, D, V_dW, V_dB, R_dW, R_dB, t)\n",
    "        if(detailed_logger == True):\n",
    "            print('Backward Propagation Complete')\n",
    "    return W, B, t\n",
    "    \n",
    "\n",
    "def shuffle(X, Y, number_of_training_images):\n",
    "    random_array = np.random.permutation(np.arange(number_of_training_images))\n",
    "    return X[:, random_array], Y[random_array]\n",
    "    \n",
    "start_time = time.time()   \n",
    "# main loop\n",
    "for epoch in range(1, number_of_epochs):\n",
    "    \n",
    "    # logger\n",
    "    if(main_logger == True and epoch % main_logger_output_epochs == 0):\n",
    "        print('Main Loop Epoch: ' + str(epoch))\n",
    "    \n",
    "    # saftey check\n",
    "    if(adam == True and momentum == True):\n",
    "        print(\"ERROR! Please Select Either Adam OR Momentum OR Neither, Not Both.\")\n",
    "        break\n",
    "\n",
    "    # saftey check\n",
    "    if(hidden_layer_relu + hidden_layer_tanh + hidden_layer_sigmoid != 1):\n",
    "        print(\"ERROR! Please Select Only 1 Hidden Layer Activation Function\")\n",
    "        break\n",
    "    \n",
    "    # shuffle data\n",
    "    X, Y = shuffle(X_train.copy(), Y_train.copy(), number_of_training_images)\n",
    "    number_of_batches = int(np.floor(number_of_training_images/batch_size))\n",
    "    split_index = number_of_batches*batch_size\n",
    "\n",
    "    # parse into minibatches\n",
    "    X_minibatches = np.split(X[:, 0:split_index], number_of_batches, axis=1)\n",
    "    if not(split_index == number_of_training_images):\n",
    "        X_left_over_portion  = X[:, split_index:number_of_training_images]\n",
    "        X_minibatches.append(X_left_over_portion)\n",
    "    \n",
    "    Y_minibatches = np.split(Y[0:split_index], number_of_batches, axis=0)\n",
    "    if not(split_index == number_of_training_images):\n",
    "        Y_left_over_portion  = Y[split_index:number_of_training_images]\n",
    "        Y_minibatches.append(Y_left_over_portion)\n",
    "    \n",
    "    number_of_minibatches = len(Y_minibatches)\n",
    "    \n",
    "    # logger\n",
    "    if(main_logger == True and epoch % main_logger_output_epochs == 0):\n",
    "        print('Number Of Minibatches: ' + str(number_of_minibatches))\n",
    "\n",
    "    for index in range(0, number_of_minibatches-1):\n",
    "        X_minibatch = X_minibatches[index]\n",
    "        Y_minibatch = Y_minibatches[index]\n",
    "\n",
    "        # forward propogation training data set\n",
    "        A_layers, Z_layers, D = forward_propagation_return_layers(W, B, X_minibatch, [X_minibatch], [], 0, [], keep_prob)\n",
    "        L = loss(A_layers[len(A_layers) - 1], Y_minibatch)\n",
    "        if(L2 == True):\n",
    "            C = cost_L2(L, W, epsilon) \n",
    "        else:\n",
    "            C = cost(L) \n",
    "\n",
    "        # backpropogation\n",
    "        W, B, t = backward_propagation(W, B, Y_minibatch, A_layers, Z_layers, 0, alpha, epsilon, len(W) - 1, D,  V_dW, V_dB, R_dW, R_dB, t)\n",
    "        \n",
    "    if(epoch % main_logger_output_epochs == 0):\n",
    "        print('Cost: ' + str(C))\n",
    "\n",
    "    # forward propogation test data set\n",
    "    A_test = forward_propagation(W, B, X_test, 0)\n",
    "\n",
    "    # accuracy\n",
    "    _prediction = prediction(A_test)    \n",
    "    _accuracy = accuracy(_prediction, Y_test)  \n",
    "\n",
    "    # storage for plotting\n",
    "    cost_array.append(C)\n",
    "    accuracy_array.append(_accuracy)\n",
    "    interation_array.append(epoch)\n",
    "\n",
    "\n",
    "end_time = time.time()\n",
    "run_time = end_time - start_time\n",
    "    \n",
    "print('')\n",
    "print('Results:')\n",
    "print('')\n",
    "    \n",
    "print('')\n",
    "print('Run Time: ' + str(run_time) + ' seconds')\n",
    "print('Cost: ' + str(C))    \n",
    "print('Accuracy: ' + str(_accuracy) + ' %')    \n",
    "print('')\n",
    "print('')\n",
    "\n",
    "\n",
    "pyplot.figure()\n",
    "pyplot.plot(interation_array, cost_array, 'red')\n",
    "pyplot.title('Learning Curve - ' + str(len(X[0])) + ' Training Data Set (Relu Hidden Layer)')\n",
    "pyplot.xlabel('Epochs')\n",
    "pyplot.ylabel('Cost')\n",
    "pyplot.show()\n",
    "\n",
    "# plot percent accuracy curve\n",
    "pyplot.figure()\n",
    "pyplot.plot(interation_array, accuracy_array, 'red')\n",
    "pyplot.title('Percent Accuracy Curve - ' + str(len(X_test[0])) + ' Test Data Set (Relu Hidden Layer)')\n",
    "pyplot.xlabel('Epochs')\n",
    "pyplot.ylabel('Percent Accuracy')\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As shown, our validation set worked, so now we can move on to the full data set, and begin our evaluation and exploration.\n",
    "\n",
    "First, we need to split up our full data set into testing and training data. We will use 50,000 images as the training data set and 10,000 images as the testing data set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(784, 50000)\n",
      "(50000,)\n",
      "(784, 10000)\n",
      "(10000,)\n"
     ]
    }
   ],
   "source": [
    "# create a data set\n",
    "size = vector_size\n",
    "\n",
    "number_of_testing_images = 10000\n",
    "number_of_training_images = 50000\n",
    "number_of_validation_images = number_of_testing_images + number_of_training_images\n",
    "\n",
    "training_images = []\n",
    "training_labels = []\n",
    "testing_images = []\n",
    "testing_labels = []\n",
    "\n",
    "factor = 0\n",
    "for index in range(0, number_of_validation_images):\n",
    "    if(index <= number_of_training_images - 1):\n",
    "        training_images.append(normalized_scaled_images_feature_matrix[:, index + factor])        \n",
    "        training_labels.append(binary_labels[index + factor])\n",
    "    else:\n",
    "        testing_images.append(normalized_scaled_images_feature_matrix[:, index + factor])        \n",
    "        testing_labels.append(binary_labels[index + factor])\n",
    "        \n",
    "# covert to numpy array\n",
    "training_images = np.transpose(np.array(training_images))\n",
    "training_labels = np.array(training_labels)\n",
    "testing_images = np.transpose(np.array(testing_images))\n",
    "testing_labels = np.array(testing_labels)\n",
    "\n",
    "# logger\n",
    "print(training_images.shape) # validation_training_images is a matrix of 784 X 500\n",
    "print(training_labels.shape) # validation_testing_labels is a row vector of 1 X 500\n",
    "print(testing_images.shape) # validation_training_images is a matrix of 784 X 100\n",
    "print(testing_labels.shape)  # validation_testing_labels is a row vector of 1 X 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we must reset our weights and bias's. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature Size: 784\n",
      "Weights Shape: (20, 784)\n",
      "Bias Shape: (20, 1)\n",
      "Velocity Weights Shape: (20, 784)\n",
      "Velocity Bias Shape: (20, 1)\n",
      "RMSProp Weights Shape: (20, 784)\n",
      "RMSProp Bias Shape: (20, 1)\n"
     ]
    }
   ],
   "source": [
    "# initialize weights & bias\n",
    "np.random.seed(10)\n",
    "print('Feature Size: ' + str(size))\n",
    "\n",
    "lower_bound = -.1\n",
    "upper_bound = .1\n",
    "\n",
    "#mean = 0.015\n",
    "#std = 0.005\n",
    "\n",
    "# hyper-parameters: hidden layers\n",
    "hidden_layers = 2\n",
    "units_array = [20, 10]\n",
    "Weights = []\n",
    "Bias = []\n",
    "V_dW = []\n",
    "V_dB = []\n",
    "R_dW = []\n",
    "R_dB = []\n",
    "for i in range(0, hidden_layers):\n",
    "    if(i == 0):\n",
    "        _W = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], size]))\n",
    "        _B = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], 1]))\n",
    "        _V_dW = np.float64(np.zeros([units_array[i], size]))\n",
    "        _V_dB = np.float64(np.zeros([units_array[i], 1]))\n",
    "        _R_dW = np.float64(np.zeros([units_array[i], size]))\n",
    "        _R_dB = np.float64(np.zeros([units_array[i], 1]))\n",
    "        Weights.append(_W)\n",
    "        Bias.append(_B)\n",
    "        V_dW.append(_V_dW)\n",
    "        V_dB.append(_V_dB)\n",
    "        R_dW.append(_R_dW)\n",
    "        R_dB.append(_R_dB)\n",
    "    else:\n",
    "        _W = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], units_array[i-1]]))\n",
    "        _B = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], 1]))\n",
    "        _V_dW = np.float64(np.zeros([units_array[i], units_array[i-1]]))\n",
    "        _V_dB = np.float64(np.zeros([units_array[i], 1]))\n",
    "        _R_dW = np.float64(np.zeros([units_array[i], units_array[i-1]]))\n",
    "        _R_dB = np.float64(np.zeros([units_array[i], 1]))\n",
    "        Weights.append(_W)\n",
    "        Bias.append(_B)\n",
    "        V_dW.append(_V_dW)\n",
    "        V_dB.append(_V_dB)\n",
    "        R_dW.append(_R_dW)\n",
    "        R_dB.append(_R_dB)\n",
    "    \n",
    "# output layer\n",
    "_W = np.float64(np.random.uniform(lower_bound, upper_bound, [1, units_array[i]]))\n",
    "_b = np.float64(np.random.uniform(lower_bound, upper_bound)) # b will be added in a broadcasting manner\n",
    "_V_dW = np.float64(np.zeros([1, units_array[i]]))\n",
    "_V_dB = np.float64(np.zeros(1))\n",
    "_R_dW = np.float64(np.zeros([1, units_array[i]]))\n",
    "_R_dB = np.float64(np.zeros(1))\n",
    "Weights.append(_W)\n",
    "Bias.append(_b)\n",
    "V_dW.append(_V_dW)\n",
    "V_dB.append(_V_dB)\n",
    "R_dW.append(_R_dW)\n",
    "R_dB.append(_R_dB)\n",
    "\n",
    "Weights = np.array(Weights)\n",
    "Bias = np.array(Bias)\n",
    "V_dW = np.array(V_dW)\n",
    "V_dB = np.array(V_dB)\n",
    "R_dW = np.array(R_dW)\n",
    "R_dB = np.array(R_dB)\n",
    "\n",
    "\n",
    "for index in range(0, len(Weights) - 1):\n",
    "    Weights[index] = np.where(Weights[index] != 0, Weights[index], np.random.uniform(lower_bound, upper_bound))\n",
    "\n",
    "#print(train_X.shape)\n",
    "#print(np.ravel(train_Y).shape)\n",
    "\n",
    "print('Weights Shape: ' + str(Weights[0].shape)) # matrix with a size of # of units X 784\n",
    "print('Bias Shape: ' + str(Bias[0].shape)) # vector with a size of the # of unit\n",
    "print('Velocity Weights Shape: ' + str(V_dW[0].shape)) # matrix with a size of # of units X 784\n",
    "print('Velocity Bias Shape: ' + str(V_dB[0].shape)) # vector with a size of the # of unit\n",
    "print('RMSProp Weights Shape: ' + str(R_dW[0].shape)) # matrix with a size of # of units X 784\n",
    "print('RMSProp Bias Shape: ' + str(R_dB[0].shape)) # vector with a size of the # of unit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we re-run minibatch stochastic gradient descent with ADAM on the full data. We will first utilize minibatches of 50 each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Main Loop Epoch: 100\n",
      "Number Of Minibatches: 1000\n",
      "Cost: 6.328639783644706e-08\n",
      "Main Loop Epoch: 200\n",
      "Number Of Minibatches: 1000\n",
      "Cost: 0.011801970848913768\n",
      "Main Loop Epoch: 300\n",
      "Number Of Minibatches: 1000\n",
      "Cost: 0.021871124168560296\n",
      "Main Loop Epoch: 400\n",
      "Number Of Minibatches: 1000\n",
      "Cost: 0.0\n",
      "\n",
      "Results:\n",
      "\n",
      "\n",
      "Run Time: 1264.1851885318756 seconds\n",
      "Cost: 0.0\n",
      "Accuracy: 98.36 %\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZYAAAEWCAYAAABFSLFOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJztnXm4HFW1t9+Vk4khCWRiSAKJEJUQphDDLFxEjYAMyhA+EFAwAhfhCopEFAEHRJRJuVxQUcQwikLECCIEvMiUoGFIQq4hhExkJPM8rO+PvYuu06e7T5/TVV3VVet9nn66ateuqrWr9t6/WnvVIKqKYRiGYURFh6QNMAzDMLKFCYthGIYRKSYshmEYRqSYsBiGYRiRYsJiGIZhRIoJi2EYhhEpuRIWEfmLiJyTtB1G8ojIOSLyl6jzGi0RkX1E5OWItnWMiMyKYltltl/xXIvI8yJybplle4pIZp/fEJFhIvK/1eSti7CIyCwROaYe+6qEqn5GVe+JY9si0l1EbhGR2SKyWkRm+PnecewvSkTkGhHZ5O0Ofh8KLd9fRF4VkbX+f//QMhGRG0Rkqf/9WEQkinVD+c4M2bVORLaGbW1PmVX1HlX9TNR524qIzPVlWi0iy0XkHyIyutRxKLN+zZ2ZiJwsIq+JyEoRWSIifxOR3SLc9/eBG0Prhcu8QER+JSLb1VKGahCR80Xk2RLpc0XkKIj3XNdC2MakUNV/AutEpNXjkxmPRUQ6JrjvzsDTwN7ASKA7cCiwFBjRju0lUZYHVXX70G+mt6Uz8BjwO2BH4B7gMZ8OMBo4CdgP2Bc4HvhKreuGUdWxgV3AZ4D5YVuL8ydZF9rJZ3w5BuI64G8Bd9VjxyLyEeDXwKVAD2AQ8D/A1oi23x84HPhT0aKgzMOAg4ArotifEQ+hNjWWEm20Baoa+w+YBRxTZtnxwGRgOfACsG9o2ZXA28AqYCpwcmjZucA/gJuB93FXRecCzwM/AZYB7+AqcLDOs8D5ofUr5R0E/N3v+2/A7cDvypThfGAhsH2FY6DAnqH53wDf99NHAXOBbwILgHuBacDxofwdgSXAMD9/sD9ey4HXgKNqOD/XVCjbp4B5gITSZgMj/fQLwOjQsvOAl2pdt4KtRwFzS6TPBb4BvAFs9GnfBmb6czgFOKHonD0bOraKazAzfH24rZ15m4BbcBcVM4GvAlqhPHOLzx1wCK5j/6ifPwHXRlb54/edUN753p7V/vcxYDAwwduwxNenHmX2PwqYVMG+Djihe9tv6wFgx3L7LrH+l4AnKpUZuAl4LDTf1afNwbWr/wa6+mXHALOKzsXA0Lq/A66p0E6frXQOivPgLhSnAyuAW3F9zrmhc32zP85vAxeHzzWwA0603/P7uA7oENrPc3795b6ufKot9cSn9wLGA4t9XfwT0M8vOwN4uSj/N4HfV3uc/blfAPzap+8OrAE6VWqniXosIjIMuBvXSHsBdwLjRKSLz/I2cATuSupa4HcisktoEwfhTkhf4AehtOlAb+DHwK8qDCtUynsf8Iq36xrgCxWKcgyu8bRrWMazM9ATd+JGA/fjKkbAp4ElqvpPEekH/Bknpj2BrwOPiEifGvb/WRF5X0SmiMiFofS9gdfV1yrP6z49WP5aaNlrRcvau257GIXzaHr4+f8DDvPzPwDuE5GdKqx/LHAgcABwVivDt+XyXoirD/sCw4HPtbUQqvoirjEf4ZNWA2f5cnwWuFREjvfLPu7XCTy4iYDg6sYuwBDgQ8B3yuzuVWAfEfmpiPxHiSGpy4Dj/H764zqV2yrsu5h9cG2sJCIyANd5zwgl/wR3YbcvTiQHAleV20ZciEhf4Pe4C9zeuM79oFCWC3EXT/vhRiZOK9rE74B1wB64unAc8MXQ8kNxF0K9cALzq3aY2QH4BbAbru/YhBNAgEeBj4jI4FD+s3AXGtD6ce4PbO+3fRGAqr6Lq1/hbbakkupE9aOMxwLcAXyvKG06cGSZ7UwGTvTT5wKzi5afC8wIzW+Lu6LZ2c8/S3OPpWRefyA3A9sWXQmVu6p/CvhRK8egNY9lI/5qwaftibtC3dbPjwWuDl113Fu0/SeBc9p5foYAu+KuwA7FXWGd4Zd9B3igKP9Y/FUhsAV/Ze3nB/uySi3rVrD1KMp7LGe3Us43geP8dCkv5OBQ3j8AX29H3r8D54WWjaSNHotPnwR8s8w6PwduDNWTstv3eU4BJlZYfijwMM4jWY+72Avq3b8JtUdgALAB16FVs+9fB/W8qMyrff1W4K94j8pvdz2weyj/EcC//XStHstmnIcQ/m2lhMeC87aeD63fAdc2zg2d6/NDy48NjgfQDycqXULLvwA8FdrPW6Fl3X1ZerelnpTINxxYHJr/BXCtn97fn+NOVR7n9UDnEvtYCBxayY6kYyy7A5f7oOVyEVmOq7i7AojI2SIyObRsKO7KIWBOiW0uCCZUda2fbDEO30reXYH3Q2nl9hWwFHd1WAuLVXV9yJ4ZuOGwz4rItrjhkPv84t2BU4uO2+GlbCgKfJe820VVp6rqfFXdoqov4K54TvGLV+MqfZjuuE6h1PLuwGrfumpZtz00O0cicq4PSgfH6KM0rz/FLAhNr6V8vamUd9ciOyrVm0r0ww3xIiKHiMizIrJYRFbgOqWy5RCRnUXkIRGZJyIrcRcxZfOr6guqeqqq9sZ5IUcDY/zi3YA/hY7hG7gOsG+V5VgGdCuRfryqdgM+gfNSe/r0nYEuQPi8Pd6G/bXG86q6Q/iHG9IrRbNzqapbcR18yeXAu6Hp3XHlWBgqx+1A2GMurkNQuc61QES2E5FfirtpaCXwDM3P9T3AmX76LFwsdRPVHeeFqrqxxG674QS5LEkLyxzgB0UneltVvV9Edsep7cVAL18B3sRdCQe0twNqjfeAnr5DDxhQIf/fgE+3cmfLWpxXFLBz0fJSZQmGw04EpnqxAXfc7i06btup6o+KN6ChwLdWf7dL4HGAi03sWzScuK9PD5bvF1q2X9Gy9q7bHj44huLuarsDN1wR1J+3aF5/4uA93BBCQKV6UxIRORjXAT3vkx4AHgEGqGoP4JcUylGq3tyA8yr2UdXuOO+8qnKr6iu4IZShPmku8MmiutZVVReU2XcxrwMfrrC/Z3BeRnDX2EKc9/6R0P56+HIXr7vZl7NSu6qF9widPxHpQPNz22w5ToQD5uDafM9QObqr6r4R2gfupodBwAh/ro8OL1TV573th+H6kmAYrJrj3OL8+n4ZnCdblnoKSycR6Rr6dcQJxwUicpA4thOR40SkG7AdrmCLAUTkixQqe6yoG0ecBFwjIp1F5BDc2HY57sVVpEdE5KMi0kFEeonIt0TkWJ9nMvD/RKRJREYCR1ZhygO4MdwLKXgr4BriZ0Xk0357XUXkKH8HTpsRkRNFZEd/DkYAl+Du5gI3fLgFuEREuojIxT79Gf//W+AyEeknIrsCl+OukGtdt1a2p1B/RETOx3kscfMQ8F8isquI7Ii7oaAqRKSHiASe6W9UdZpf1A3nQa/3ojMqtNoiQCV0e7jPvwZY4WMYX6+wzyPF3Ybb18/vhavrL/ks/wP8UPztxyLS19tYbt/F/BX4mBTuBCzFzcCxIjJUVbfghPMWEenj62R/EflUmXVfA8707eA4nOceFY8D+/v20RH4GhCOYwbnup+I9MINUQOgqnNwwfmfiHsUoYO427M/XoM9nUv0od1wArbM23B1ifXuxV1krVHVl7x9bT3OAUcCf/NeT1nqKSzjcWOOwe8aVZ0EfBk3ZrwMF8A7F9zwDPBT4EWcuu6DuyOjXpyJuztnKS4Q+iDu6qgFqroBNyb5Fi7eshIX+O8NBA+GXYprsMv9th9tzQBVfQ9X/kP9/oP0OTgv5lu4jnMOrgNr7/kchTv2q3Cd/Q3qn/fxrvBJwNne9i8BJ4Vc5Dtxd6K8gfMo/+zTalq3VlT1dVyQ+RXcleVHKZyLOLkDJ6hv4ALjf8ZdGVbiL+Kex5mNCxTfiBvuCrgQuF5EVuHO+UPBAlVdBVwPvOyHNIYD38UFk1cA43DeTjmWAScDb3obxvvt/9Qvvwl4Anja7/8F3J1n5fbdDFWdD/wvFS7MvPczlsINBpfjhpVe8WX4K+WDxZd4+5cDp/ryRoKqLgROx52PpTiPJFyH7sA9ZvAGMBEX6A9zFu4CeSruOD9MbR7VkzTvQ7+NOz89vH0vAKWGu3+Luyi/tyi9Lcc54EzcxUZFpP3D2flCRB7EBdu+m7QtRuMgIp8FblHVPZK2JSlEZB/gF6p6cNK25BE/RL8IGKqq79SwnQOAn6lqq16hCUsZRORjuODpO7jhqEeBQ1T1X4kaZqQa34iPwHmuuwB/BJ5T1bLDUYYRJyJyBe6u3NaGuSKj0Z5Qric7424j7YULYF5oomJUgeCemfk9Ls7xOO4ZLMOoOyIyF/dsy4l13a95LIZhGEaUJH27sWEYhpExGm4orHfv3jpw4MCkzTAMw2goXn311SWqWstrn6qm4YRl4MCBTJo0KWkzDMMwGgoRebf1XNFgQ2GGYRhGpJiwGIZhGJFiwmIYhmFEigmLYRiGESkmLIZhGEakmLAYhmEYkWLCYhiGYUSKCYthGEaULFgAjz3Wer4M03APSBqGYaSao4+GadNgwwboXOn7ZtnFPBbDMIwoeftt95/jF/yasBiGYRiRYsJiGIYRB+axGIZhNCCLFsHixS3Tly2D+fNLr6MKU6fGa1fOMWExDKNx2Wkn6Nu3Zfruu0O/fqXXufVW2HtvePHFeG3LMSYshmFkj1Wryi+bONH9z5wZrw02FGYYhmFEigmLYRiGESkmLIZhGIYRDSYshmEYcWAei2EYhhEpJiyGYRhGpJiwGIZhGJFiwmIYhmEY0WDCYhiGEQfmsRiGYRiRYsJiGIZhRIoJi2EYRs6Iu+M3YTEMwzCMaDBhMQwjn4jEu33zWAzDMIxIMWExDMMwIsWExTAMw4iEYIjNhMUwDMMwosGExTAMI0oCT8U8FsMwDCNSTFgMwzCMSDFhiQcRGSki00VkhohcWWL5biIyQUT+JSKvi8ixcdpjGIYROxa8j09YRKQJuB34DDAEOENEhhRl+zbwkKoeAIwC/jsuewzDMIz6EKfHMgKYoaozVXUj8ABwYlEeBbr76R7A/BjtMeJk4kRYuzZpKwwjeSx4H6uw9APmhObn+rQw1wBnichcYDzw1VIbEpHRIjJJRCYtXrw4DluNWliyBEaMgLPPTtoSw0gPJiyxUOpFPMVH+gzgN6raHzgWuFdEWtikqnep6nBVHd6nT58YTDVqYs0a9z9xYrJ2GEaaMGGJhbnAgNB8f1oOdZ0HPASgqi8CXYHeMdpkGIYRLxa8j1VYJgKDRWSQiHTGBefHFeWZDXwCQET2wgmLjXUZhhE/Oe744yY2YVHVzcDFwJPANNzdX1NE5DoROcFnuxz4soi8BtwPnKtqZ9swjAbGgvd0jHPjqjoeF5QPp10dmp4KHBanDYZh5BjV8t9dse+xxIY9eW8YRnZJsnM3YTHqypIlMGNG0lYYRvZJonO34H28Q2FGGfbYA1auzHXFM4y6sHUrNDWVXmbtLzbMY0mClSuTtsAw8kES4mHBexMWwzAyTKXO3YL3sWHCYhhGdrEYSyKYsBiGkV22bk1u3yYshmEYGSTHnXuSmLAYhpFd7DmWRDBhMQwju9hQWCKYsBiGkV3MY0kEExajdnLcgIyUY8KSCCYsRu3kuAEZKceEJRFMWIzayXEDMlJOpRiL1dvYMGExascaqJFWzGNJBBMWo3Zy3ICMlGOvdEkEExajdnLcgIyUY7cbJ4IJi1E7OW5ARsqxobBEMGExaifHDchIOZXqptXb2DBhMWrHGqiRVsxjSQQTFqN2ctyAjJRTKcZiwfvYMGExDCO7mMeSCCYsRu3kuAEZKceEJRFMWIzayXEDMlJOkrcb5xgTFqN2TFiMtJLkXWE5bhcmLEbt5LgBGSnHhsISwYTFqJ0cNyAj5ZjHkggmLEbt5LgBGSknybcb57hdmLAYtZPjBmSkHHvyPhFMWIzasQZqpBUbCksEExajdnLcgIyUY283TgQTFqN2ctyAjJRjHksixCosIjJSRKaLyAwRubJMntNEZKqITBGR++K0x4iJHDcgI+WYsCRCx7g2LCJNwO3AJ4G5wEQRGaeqU0N5BgNjgMNUdZmI9I3LHiNGctyAjJRjwftEiNNjGQHMUNWZqroReAA4sSjPl4HbVXUZgKouitGe9JGVip2VchjZI4nbjYO3Jue4XcQpLP2AOaH5uT4tzIeBD4vIP0TkJREZGaM96SMrFS8r5TCyRxJ1M9hnjttFbENhQKmPHRQf6Y7AYOAooD/wvyIyVFWXN9uQyGhgNMBuu+0WvaVJkZWKl5VyGNnDYiyJEKfHMhcYEJrvD8wvkecxVd2kqu8A03FC0wxVvUtVh6vq8D59+sRmcN3JSsXLSjmM7GFP3idCnMIyERgsIoNEpDMwChhXlOdR4D8ARKQ3bmhsZow2pYscVzzDqAtJeCxxf5myAYhNWFR1M3Ax8CQwDXhIVaeIyHUicoLP9iSwVESmAhOAb6jq0rhsSh1ZEZagHNagjLRhQ2GJEGeMBVUdD4wvSrs6NK3AZf6XP7JS8SxYaaSVJITF2oM9eZ8oWal4WSmHkT3slS6JYMKSJFmpeFkph9FYVFPvbCgsEUxYkiQrFS8r5TAai7QKi8UaTVgSJSsdclbKYTQW1dQ7u904EUxYkiQrFS8r5TCyhwXvE8GEJUmyUvGyUg6jsah1KCxuctwuTFiSJCsVLyvlMBqLtMZY6rX9FGPCkiRZqXhZKYfRWKQ1xmLBexOWRMlKh5yVchiNhXksqcWEJUmyUvGyUg4je1jwPhFMWJIkKxUvK+UwGgsL3qcWE5YkyUrFy0o5jMYirTGWem0/xZiwJElWKl5WymE0FuF6V64O2pP3iVCVsIjIvdWkGW0kKx1yVsphNBZpFZZ6bT/FVOux7B2eEZEm4MDozckZOa54hlEXbCgsESoKi4iMEZFVwL4istL/VgGLgMfqYmGWyUrFsw99GUlgHktqqSgsqnq9qnYDblTV7v7XTVV7qeqYOtmYXbJS8ez2SiMJahWWuMlxe6h2KOxxEdkOQETOEpGbRGT3GO3KB1mpeFkph9FYmMeSWqoVljuAtSKyH3AF8C7w29isygtZqXhZKYfRWFQjLEnGWHJMtcKy2X+f/kTgVlW9FegWn1k5ISsVOyvlMBoL81hSS8cq860SkTHAF4Aj/F1hneIzKydkpeJlpRxG9jBhSYRqPZbTgQ3Al1R1AdAPuDE2q/JCVipeVsphNBa1DoXFTY7bRVXC4sVkLNBDRI4H1quqxVhqJSsVLyvlMBoLGwpLLdU+eX8a8ApwKnAa8LKInBKnYbkgKxUvK+UwGou0C0uOqTbGchXwMVVdBCAifYC/Ab+Py7BckJWKnZVyGI1F2oUlx+2i2hhLh0BUPEvbsK5RjqxUvKyUw8ge9kqXRKjWY3lCRJ4E7vfzpwPj4zEpR2Sl4mWlHEZjYR5LaqkoLCKyJ7CTqn5DRD4HHA4I8CIumG/UQlYqXlbKYTQW9kqX1NLacNYtwCoAVf2Dql6mql/DeSu3xG1c5slKxctKOYzGwp68Ty2tCctAVX29OFFVJwEDY7HIaDysgdaHLVvg5pthw4akLUkH7fVY6vXS1By3i9aEpWuFZdtEaUguyUrFy0o50s6vfw2XXQbXX5+0JY2DCUsitCYsE0Xky8WJInIe8Go8JuWIHFc8ox2sWuX+V6xI1o60YB5LamntrrD/Av4oImdSEJLhQGfg5DgNywVZqXj2oS8jCeyVLqmltQ99LVTVQ4FrgVn+d62qHuJf81IRERkpItNFZIaIXFkh3ykioiIyvG3mNzhZqXhZKYfRWKTdY8kxVT3HoqoTgAlt2bB/A/LtwCeBubhhtXGqOrUoXzfgEuDltmw/E2SlYltDNZIg7cKS4/YQ59PzI4AZqjpTVTcCD+C+51LM94AfA+tjtCWdZKXiZaUcRvYoNRQWt7AEQ8I5bhdxCks/YE5ofq5P+wAROQAYoKqPV9qQiIwWkUkiMmnx4sXRW5oUWal4WSmH0ViU81ha82TiFhbz4GMVllKR3A+OtIh0AG4GLm9tQ6p6l6oOV9Xhffr0idDEhMlKxctKOYzGor3CUi9y3C7iFJa5wIDQfH9gfmi+GzAUeFZEZgEHA+NyFcDPSsXLSjmMxiKtHosRq7BMBAaLyCAR6QyMAsYFC1V1har2VtWBqjoQeAk4wT/Vnw+yUrGzUg6jsahGTJKIsZSyI2fEJiyquhm4GHgSmAY8pKpTROQ6ETkhrv02FFmpeFkph9FYpNVjseB91a/NbxeqOp6i1+ur6tVl8h4Vpy2pJCsVLyvlMLKHBe8TwT7WlSRZqXhZKYfRWNhQWGoxYUmSrFS8rJTDaCzSfldYjjFhSZKsVPqslMNoLNIaY6m075xgwpIkWal4WSlHLSxZAmvXJm1FvkirsFjw3oQlUbJS8bJSjlro0wc+9rGkrTCKSSLGYsF7E5ZEyUrFy0o5amXq1NbzRIEdb0daPZZK+84JJixJkuOKZxg10967wozYMWFJkqwIi33oq77YcXaYx5JaTFiSJCsVLyvlMBqLtD7HYsF7E5ZEyUrFs2BlfbHjXJmkhcXagwlLomSl4mWlHEZjUc2DkPbkfSKYsCRJVipeVsrRKFiMxVHNUNiWLfWzx/gAE5YkyUqHnJVyGI2FxVhSiwlLkmSl4mWlHI2CHW9HWoWllB05w4QlSbJS8bJSDqOxSKuwWPDehCVRslLxslKORsFiLNVjHksimLAkSVYqXlbKYTQW7Q3em0cROyYs9SaL34rISjkaBTvejrS+0sWC9yYsdceEJXvkvfxJkdYYSyk7coYJS70xYcke9X5WwmIsDhOW1GLCUm9MWLJHvYZb8n6cqyXpGIvFcExY6o4JS/aot7Dk/XgHpP2VLjnGhKXemLBkD/NYksGC96nFhKXeZFFY8k69Oq9gPxZjcViMJbWYsNSbLApLUI68dnj1Fpas1JtaMWFJLSYs9SbLwpJX6i0sRmUseJ84Jiz1xoQle1iMJRnMY0ktJiz1JsvCkpXytJV6PcdiMZbmtPeusLix82PCUncaVVgWLoQbbyxtcyOVIw4sxpIM5rGkFhOWetOownLmmXDFFTB5cstljVSOODBhSQYTltRiwlJvGlVYli93/5s2tVzWSOWIA3tAMl0kHbwvZUfOMGGpN40qLJXIe4dnHksymMeSWkxY6k0jCMu6dXDDDbB5cyEtCEiWCkyWa6jz5qW3jFFSb2Gx244d9kqX1BKrsIjISBGZLiIzROTKEssvE5GpIvK6iDwtIrvHaU8qaARh+eEP4cor4de/ri5/qYY6eTL07w933hm9fWnDhCUZ0vpKl1J25IzYhEVEmoDbgc8AQ4AzRGRIUbZ/AcNVdV/g98CP47InNTSCsKxc6f7XrKkufylhmT7d/T/7bGRmpZZ63W4cHF8TFkdah8LMI4rVYxkBzFDVmaq6EXgAODGcQVUnqOpaP/sS0D9Ge9JBIwhLpfvwK91uHF6WpxfxWYwlGaoRFgveJ0KcwtIPmBOan+vTynEe8JdSC0RktIhMEpFJixcvjtDEBGgEYQmo1r5KwpIHbCgsXZjHkjhxCkupnqXkkRaRs4DhwI2llqvqXao6XFWH9+nTJ0ITS+4Mvv71wlBOHNsvNZ0m2ioKlRpSWssYJSYsyZD2obAcE6ewzAUGhOb7A/OLM4nIMcBVwAmquiFGe6pj5kz46U/hs5+NZ/uNICwBpeyrdmjBhsKix2IszUnrK13MY4lVWCYCg0VkkIh0BkYB48IZROQA4E6cqCyK0ZbqCSpDXAHZRhCWSh6LCUtLLMaSDGn3WHJ8nmITFlXdDFwMPAlMAx5S1Skicp2InOCz3QhsDzwsIpNFZFyZzdWfuGIEjSAsAdV6LKWwGEt8+zGPxWHB+9TSMc6Nq+p4YHxR2tWh6WPi3H8qaSRhKUX4ockAi7HUdz8mLJUxjyVx7Mn7Yup5FZP2imcxluqw51iSIe1DYTnGhOWxx+CddwrzcXcSjSAsUcRYipdlGYuxJENaX+liHku8Q2ENwUknwfbbw6pVbr7UUE+UNIKwVKJaYclTJ2hDYcmQ1le6mLCYxwLA6tWFaROWaDyWIF9ayxgldrtxMljwPrXkW1hKnfhS3xuJa59pr3il7Ks2eB+3QKcJ81jSRdIxllJ25Ix8C0spEQk6RLvduDTmsbTEYizJkMbgvZ0bwISlZVogLO++C0uWxLv/tFbCSnd0VSMsr74KZ59dfhtZwzyWZEi7sOSh7pfBhKWYQFg2bYJ+ld6Z2U4aqeKVapTVCMtFF7VclmXsduNkqKYt1evclCIPdb8MJizFhGMDGzdGv89GEJbAYykVJ0kyGJpWzGNJhtY8lg4dzGNJCBOWYuyusAKljoU9ed+ScOcVZ3ktxlIdwfFpajJhSQgTlmJMWApeSXs9lnqUUQS++tV4tt1Wwp1XnN6EeSzNac1jSVpYcowJSzEmLIVjEMVQWJxl/PnP49t2W6iXsFiMpTlpF5a0tu86YMJSTVqUNELFK+WxBHGXNMRY0nbczGNJhtbaUlOTBe8TwoSlmKg8lquvhueea5kelbBs3RrPzQVQ2mMJbE1DjKWtncWUKbBsWfR2BIQ7+jg7MouxNMc8ltRiwlJMVMLyve/BUUe1TI+q4l18MXTp0v71K1FrjCX8cGkcjautgjp0KBx6aPR2BJjHkgwmLKkln8IydSqcdRasXdtyWRTCUqlCRVXx7rjD/ccRE2prjKWY1h5Qq5UN7fiC9VtvRW9HQPiYWIwledIiLDkmn8Jy0UUwdiz8/e8tl0XRUYevqNesab4s6iua9etr38bChTBxYmG+Vo+ltfy10haPpR5j7OaxJENrHkvHjuaxJEQ+haVnT/c/b14h7ZFHnAhEISzhK+pZs5ovi7ritefqvZjhw2HEiMJ8KY+lvcH7ODyqtghLFMenNeotLDnusJpRzVCYvd04EfIpLH37uv+5cwtpp5wCl1wSvbCEX8kPzSvbV797lqUuAAAUzElEQVQKp50W3b7aS/g4QLTB+zjussu7sKTNY/nZz+CVV+q/39Yu0so9eR8n5rEAef3Q1447uv93322e/s47sPfetW8/3JlVGgoDePjh2vYVxVBYwJYtza/ywiISpJnH0pK8P8dyySXuv94dqQXvU0s+PZagY5oxo3l6hw7RXGGHO/vWhKVWouw4A7tLeSxpEpa2lLnewlKP243TJCxp7DzTIiw5Jp/CEnSgxXeFdehQ36GwKIhDWNrrsZQi6aGwKD26cuQ5xhLXs1TVYMH71JJPYVm3rnS6SPTCEvZY1q6Ff/2r9u2HibLjDI5Lez2W4unibURF2obC6nW7cRo9lnJtKcy4cc1vlIkKC96nlnwJy+9/Dy+8UL4zLuWxtKcRl/NYzj4bzj237durdl+1UsljqfRsS96FJWqP5cUXoWtXdxv41q3w5z+745rGGEtrwrJ1K5x4Inz849Hv24L3qSVfwftTT3X/J59cenlTU8uOcNOmtj/hXs5jKfXcTK1E6bHUGmMpni7eRlRkXVhuuMHZ/Y9/wHvvubcs3HdfOj2WUg8ZhwkurGbOjH7fFrxPLfnyWALa4rG0J0ZQ6a6wqImy4wyuPmuJsRQ3pjhiLG0pcyPGWAKbO3cu3Ln47rvpjLG05rEUxxjrQVIxFlX3XjojR8IS7iTLNQbVlsLSnuBkuDOrpmENGtT2fQTEeVdYWETaIixhMTGPpe0E52HdOnfVDe6419NjUa0uLpKksKTNY7npJjj88NI25Yz8CMvy5YXpclex69eX91jWrXNj3tXQVo9l1qz2V8I4gve1eix5FpYobjcObF6xormw1DPG8stfQv/+rd9sUmoobNOmQl1atSp62wLSFrz/4x/L25cz8iMs779fmC53lbV2bcuhm6AT+8EPYNiw6ipL0DE0NRWu2BYsgMWLy6/T3iu7ej3HUpy2aJG7i+6++1o26nDHX81dQ20lzcISpceyYkXhVTqbN9fXY/nb39x/ay/vLHV+Tz8dtt3W2Zm0x1K8LE6K32BhwpIDwt/jKNfBr11b3mOZMQPmz3e/AFX44Q9bvg8s6Mx69XIeyyWXwC67VLYvLHxtoa0d57hx5ffVFo9l+nT3//OfV/ZY1q1rPcAbsHEj/OlP1eUrtivMI4/AmDFuuhFjLMHxWr68ML1yZTIxltb2VUpYgiv3Z55pu8dy//1OTKsZHajmrjBoeU5a81gWLIDvfrft57J46NCEJQeEO9OwOIQpJSwbN8IRR8CDD7r5qVMLy2bNgquugs99rvk6QWffs6e7YvvZz1q3r70fompLx/nqq+7Wz29+s5AWrvwTJrg714JjEN52sbCExaKSxwKwZEl19l1zDZxwAjz7bOV8YTEt5b2ccgr86EfOlnDe9gzL3XEHfOMblfNE/RxL0KmuWFHomJcvLz8U9vbbMG0avP568w75wQfha1+rzZZgKHfOHHjzzZbLiy8awsf4+efb7rHcfrv7r+YzB9U8IAltF5avfAWuu849mgDu/I4dW7mNbt3asn4Ft4gH53DsWPfaqByQH2EprhQ9erTMU85jef75wvyUKXDbbbDTTu5WUCj8BwSd2aBB1T8Q2RZhCTeUtngsDz3k/sNxn/D6d94JRx5ZOAbhchULS9jrq+SxFOetRNBxLVhQOV9YTCoNiy1a1Lx87RkWu+gi+MlPKueJ0mNZtcoJCrj/lSvd9LJlhW3PmAH9+hXy7bknDBkC++0HF15Y2NaoUXDLLa3bpApPPuk+nRBsM+D9951w7bYb7LNPy3XDHsvmzc1fkzRvXnlhee65yhdFQbnbwtat7uJk9mw3H45PhSklLHPmuPpx000FzyOw/fLL3feb7r67/L5L1XFVuP566N7dlfess+BLXyq/jc2bC3cBFvc7DUZ+hKV4+KdXr8L0nDnuSrmUsBRfbV92GVx6qeu0gje6Bh3pli3uSitoMMOGVd9Azj/fPa9QDW3tLIMyjRvn/sMNulTjDhpU+Iq5krAU015hCbZdzqMMqCQsc+YUpmfNKn2s2vMUeKWbMKIUlrD9xR5LeNvz5zsPtJhgiDLMokXN52+7DbbZxtWLyZPd275HjnSfTjjppOaextKlTrgCNm1yx2LhQtdevvzlwrJ165rfbjtvXumhsHfecV9X/fznSx6CD8r3/vtw7bXNz/GKFe4CadUq56UFqLqLuGuvLXTegbC05rHcfbcTzo98xIlIcFyDC6sJE9x/8Utri+0t5p573IgGFP4rec1XXQUDB7pje9ttbqTk3/8unz/FxCosIjJSRKaLyAwRubLE8i4i8qBf/rKIDIzNmE6d3F0uxx3n5rfZprCsf39XqdasadlR/eMfzefDVzlPP+3+ly51onDppbDXXoWKeeCB1ds3c6YbBijnnh9xRKEhhjvLYmGYNAmOPdaV5d13XXk7dXI3HwTDC+EgY6kx8nnz3DpQ6OhKBe8DW9oyFHb//XDjjaXLGIh/MFywcKE7LkEn98YbLc9R8b52260wPWtW8+OzYQP84Q/ufD/xBPzzn6XtCAhf6QZXwaUId1zTpsHVV7vObfVqN1Z/ww2V9xMmON4dOjQXliVLWnaQs2e3vLBYurTlNotjgJde6o7L9Oku0B4+P88+6+ra+PFuvnhIas4cN9S4887w6KPNl4WF5ROfaOmxBMczOO7jx7cU7KCM770H3/6280Aee6yw/NZbnc2jRjlvIGD06EK9CrZRrbD84hfuv1g45s1z6wade/CQ57JlLo4UrvetXawE/cjzzzsv8oYbnBcD8Nprbv0HHnDzb77phs3Aecu1DmcmgarG8gOagLeBDwGdgdeAIUV5LgL+x0+PAh5sbbsHHnig1sRzz7mRzx13dP+dOrn0u+8ORkSr/22zTfllnTurvv1227f5zjstbV60qLB840bVadMK85de2jzvfvu59IcfVv3851tu/+MfV91lF7fNt95SnTmztB0f/rD7Hz9edetWVRE3f/zxbj/nnefmBwxQPeWUwnqrVrljGt7WTTe5dbZuLaSdeKLq979fsHvjRtWePQvLH39ctW/fwvzLL7v/E05QHTOmkD5zZmEbs2Y13+/VV6t+61uF+U9/WvWLX2ye5623CuuvWaP67ruF+dmzm+f94hdL16nrrit9DK+/vjC9aVNV1VPvusvlP/po1R12UN1rLzcv4s5bePtjxqhOn95yv6tWqa5cWZh/4AG37cmTVU86qZB+773Nj3GpX/fuzef/+tfC9N57t6y7p5+uOmiQ6gUXqPbqpXrFFYXlK1c6O77znULaE080L/+ee7r0k08u2HbNNW7Zrbe2rS0FZV2xovk+grY/YoTqhg2qXbqoDh3acv0LLnD1IZjfay9Xh885x80/+GBhm3fe2Xzdww6rzsbVq91/376qu+3mpr/ylZb5pk6trv5UAJikGk9/X/yLb8NwCPBkaH4MMKYoz5PAIX66I7AEkErbrVlYtm51FfWll1RPPdU1TFXVtWtVm5oqV4IOHdz/hz7UPD3ckY4aVZhWbVtDCH6DBrlK1q+f6s47N18WdPDBb8cdXYX/8IddQw7Se/VS7dq1ed7hw1W/+1033aWL29agQaVtOO20wnSfPoXprl1Vd921ed7tty9MB4IU/Jqa3PpDhqh+5CMt9zNkiPsFdhx1lPvv1q38PsICtMcehW0EDTP4bbNN82MCqttu23x+l13c8dtrL9eRizg7hwxpeZ7D9oZ/vXuXPoZduhSm99yz9LrFv759XT179NHCuqWOG6j26KE6cGDL9D33LHTQ4OrQkCEt60OfPoU6Xe2vWNzCvz32cMf8+ONVv/e9Qv0M140hQwp1vFMn1Z12al7+Uvb06FFYr9qfiOp//mfh+JXaR9euqoMHu+l77mm5jW7dCsd32LBC+YP23rVr8/MmUrgQuOMO1bFjm2+vVNnC5y/of5qa3LbC9bxfP7ef4CKhHdRTWOJ8V1g/IDRgzFzgoHJ5VHWziKwAenmB+QARGQ2MBtgtPNTRHkTc8AQUgtnghsamT3dxiJNPdkMCBx7ohrUOP9y9XqNrV+eKX3MNPPWU+40Y4eIz117rxqmPOAL239/lBTd0MGGCc3e7dHHp++zjxswPOqjwHEiXLs62YFy6qcnd1RL8Dxvm7vpZtcrNjxrltvnii24/HTrA9tu7oYWddnLDSKqw665uWGv+fPcCzH79XDC2qcmNkau68vXs6YaVzj7bDRddeaUbMlqypDCE0b+/K08wjDhggBuOWbfOjQ3PmuWGyoYNg8GD4eCD3Rh+cAODCBx2mPsPXrcRvmnhK1+BK65w5fzRj9wx33Zbt2zpUveBtlWr3D769XNpxUN5I0e6dc45xwVi161zdm/e7OIDS5e6dRcudDdwhGNv++7rjkd46OTII10MYtEit41SQ4dDhrhtiriYWvfuzo5f/hK6dXPbbMvdUfvv74YzL7rI7feCC9yw0ezZsMcebshm550LQ5qf+ATssIMLDt9yS2F46dBDYbvtCjGuoUPdUOG8ee58L1zozkP//q6ObN7slvXp426gGDDAndOddnL1c/16Z8822zjb7roLevd2dWnePHdsDjjAncf+/d0dlJs2uWOzZElh2G74cHcL/t//XrjzKmDo0EI9A1evgqHRQw91dpxzjrtbr3t3Z38wXLhmjbNn4UL3XsChQ935LY757b23u7EmGNo67DB3t+Tdd7uhzNmznc2BDZ/8pIsl/epXbl9btrihzocfbv7g9X77uX0+/DB86lNuHzNmuDqxaJEb0rrnHjffqZNrf+vXw9FHOxs3bXJ1bf58d4z693fnvVu3whBn8JHClCNOyGLYsMipwKdV9Xw//wVghKp+NZRnis8z18+/7fOUGCh2DB8+XCdNmhSLzYZhGFlFRF5V1eH12Fecwfu5wIDQfH+g+NaJD/KISEegB9DOJwUNwzCMNBCnsEwEBovIIBHpjAvOjyvKMw44x0+fAjyjcblQhmEYRl2ILcbiYyYX4wL0TcDdqjpFRK7DBZHGAb8C7hWRGThPZVRc9hiGYRj1IdYPfanqeGB8UdrVoen1wKlx2mAYhmHUl/w8eW8YhmHUBRMWwzAMI1JMWAzDMIxIMWExDMMwIiW2ByTjQkQWAxVeM1qR3hQ91Z8DrMz5wMqcD2op8+6q2idKY8rRcMJSCyIyqV5PnqYFK3M+sDLng0Ypsw2FGYZhGJFiwmIYhmFESt6E5a6kDUgAK3M+sDLng4Yoc65iLIZhGEb85M1jMQzDMGLGhMUwDMOIlFwIi4iMFJHpIjJDRK5M2p6oEJG7RWSRiLwZSuspIk+JyL/9/44+XUTkNn8MXheRYclZ3n5EZICITBCRaSIyRUQu9emZLbeIdBWRV0TkNV/ma336IBF52Zf5Qf95CkSki5+f4ZcPTNL+WhCRJhH5l4g87ufzUOZZIvKGiEwWkUk+raHqd+aFRUSagNuBzwBDgDNEZEiyVkXGb4CRRWlXAk+r6mDgaT8PrvyD/W80cEedbIyazcDlqroXcDDwn/58ZrncG4CjVXU/YH9gpIgcDNwA3OzLvAw4z+c/D1imqnsCN/t8jcqlwLTQfB7KDPAfqrp/6JmVxqrfqprpH3AI8GRofgwwJmm7IizfQODN0Px0YBc/vQsw3U/fCZxRKl8j/4DHgE/mpdzAtsA/gYNwT2B39Okf1HPcN5AO8dMdfT5J2vZ2lLU/rhM9GngckKyX2ds/C+hdlNZQ9TvzHgvQD5gTmp/r07LKTqr6HoD/7+vTM3cc/HDHAcDLZLzcfkhoMrAIeAp4G1iuqpt9lnC5PiizX74C6FVfiyPhFuAKYKuf70X2ywygwF9F5FURGe3TGqp+x/qhr5QgJdLyeI91po6DiGwPPAL8l6quFClVPJe1RFrDlVtVtwD7i8gOwB+BvUpl8/8NX2YROR5YpKqvishRQXKJrJkpc4jDVHW+iPQFnhKRtyrkTWW58+CxzAUGhOb7A/MTsqUeLBSRXQD8/yKfnpnjICKdcKIyVlX/4JMzX24AVV0OPIuLL+0gIsHFYbhcH5TZL++B+/R3I3EYcIKIzAIewA2H3UK2ywyAqs73/4twFxEjaLD6nQdhmQgM9neTdAZGAeMStilOxgHn+OlzcDGIIP1sfxfJwcCKwLVuJMS5Jr8CpqnqTaFFmS23iPTxngoisg1wDC6gPQE4xWcrLnNwLE4BnlE/AN8oqOoYVe2vqgNxbfYZVT2TDJcZQES2E5FuwTTwKeBNGq1+Jx3kqVMw7Fjg/3Dj0lclbU+E5bofeA/YhLtyOQ83rvw08G//39PnFdzdcW8DbwDDk7a/nWU+HOfqvw5M9r9js1xuYF/gX77MbwJX+/QPAa8AM4CHgS4+vaufn+GXfyjpMtRY/qOAx/NQZl++1/xvStBfNVr9tle6GIZhGJGSh6EwwzAMo46YsBiGYRiRYsJiGIZhRIoJi2EYhhEpJiyGYRhGpJiwGIZHRLb4N8oGv8jehC0iAyX0FmrDyDJ5eKWLYVTLOlXdP2kjDKPRMY/FMFrBfx/jBv9NlFdEZE+fvruIPO2/g/G0iOzm03cSkT/676e8JiKH+k01icgv/DdV/uqfokdELhGRqX47DyRUTMOIDBMWwyiwTdFQ2OmhZStVdQTwc9w7q/DTv1XVfYGxwG0+/TbgOXXfTxmGe4Ia3DczblfVvYHlwOd9+pXAAX47F8RVOMOoF/bkvWF4RGS1qm5fIn0W7kNbM/0LMBeoai8RWYL79sUmn/6eqvYWkcVAf1XdENrGQOApdR9qQkS+CXRS1e+LyBPAauBR4FFVXR1zUQ0jVsxjMYzq0DLT5fKUYkNoeguFGOdxuPc9HQi8Gnp7r2E0JCYshlEdp4f+X/TTL+DevAtwJvC8n34auBA++EBX93IbFZEOwABVnYD7qNUOQAuvyTAaCbsyMowC2/ivNAY8oarBLcddRORl3MXYGT7tEuBuEfkGsBj4ok+/FLhLRM7DeSYX4t5CXYom4Hci0gP3ptqb1X1zxTAaFouxGEYr+BjLcFVdkrQthtEI2FCYYRiGESnmsRiGYRiRYh6LYRiGESkmLIZhGEakmLAYhmEYkWLCYhiGYUSKCYthGIYRKf8ffBwQciAOogQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaoAAAEWCAYAAAA3h9P4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzsnXe8HFX5/9/PbemkBwIkBBKktxCRIkWadGkiIAgiIEWliHxR+aGIgiCIIiiggKAIihRBEQglQTqhlwChJ5BAQklyU++9Ob8/zhzm7OzM7Oze3bt7d5/367Wv6TNnZnfPZ57nPOc5YoxBURRFUWqVpmoXQFEURVHSUKFSFEVRahoVKkVRFKWmUaFSFEVRahoVKkVRFKWmUaFSFEVRahoVKkVRehVieVJE1i/T+eaIyBfLca6Yc/cRkXYRWTVh+3Eicm/K8Y+JyGGVKFu1Cb7HZ0Rk7UL7lkWoRORtEVkSfCEfiMg1IjKwHOcuF0EZd86w35oiskJEft8T5aoWIjJaRK4SkdkislBEXhGRs0VkQLXLVggROUhEHhGRxSIyJWb7piLyVLD9KRHZ1NsmInK+iHwUfC4QESnHsd5+Xw/+C+3B/2KFt9zejfteV0Q6C+zzSxHpCL5T973+VkRGFXGdblWOIjJORG4TkXkiMl9EnheRQ8t47QOB94wxLwfHuHtuF5FPReQhEZlUavmzIiJ9RcSIyOqR9b8UkT8BGGOWGWMGGmPer3R5isEvY7UwthPvxcBPC+1bTotqb2PMQGAi8HngzGJPICItZSxPqXwD+AQ4WET69OSFe+r+RWQY8CjQD9jKGDMI2AUYAowv4Xw9/b19DPwG+GVMWdqAfwF/BYYC1wL/CtYDHAvsC2wCbAzsBXy7u8f6GGOuDyqngcDuwPtuOVhXaa4NvtPhwFeBccA0ERnZA9cGuAF4FRgDjAC+Ccwr4/mPA/4SWXdt8GxHAo8Bfy/j9ZQy49UZtwB7isjw1AOMMd3+AG8DO3vLvwL+HcwPBq4CZgPvAT8HmoNtRwIPY1X1Y+DnwfpjgOnAQuBlYGKwflXgZmAu8BbwPe+aPwX+AVwXHPcSMCnY9hdgBbAEaAdOT7mXN4DjgQ+AAyPbNgAmB2X9APhRsL4Z+FFw7ELgKeyfdBxggBbvHFOAo5PuHysU9wMfYf/c1wNDvOPHBF/u3GCfS4E+wfEbefuNCu53ZMw9/hx4AWhKeAbFlvs84FNgQ2//kcH1RwXLewHPBvs9Amxcht/d0cCUyLpdg9+ZeOveBXYL5h8BjvW2fQt4rLvHppRxB2BWzPoxWFGcB7wJHOdt2wZ4BlgAzAHOC9Z/GHwv7cFns5jz/hL4U2RdK/b/5P5fI4H/Br+hj4NyjA62XQR0AUuDa1wUrP8DMCso0xPAlgn3K0AHsG7KM9kWeDz4LTwNbJN27cix/YPzj0i6Z+zLsgEGeev2A54Prvk/YH1v2xzgi8H8jcCZ3rbdgNcT7qNvcJ3Vk76D6D7Y/+WdwXN8FPvfudc7dk9gRlDOX2NF9zBv+7exLwEfA/8BVotc5xhsPfQJcHHKd5D3O/G2nYWtXxcCLwJ7es9+AbC2t+/qwGKCOirDcz4NWzcv9tb/D/ha2v+o7G1UIjIG2AP7RwP7VtoJTAA2w1YGR3uHfAH7Rx0F/EJEvooVnW8AKwH7AB+JSBNwB/AcsBqwE3CyiHzZO9c+2B/aEOB2bCWOMeZwbIWzt7FvtRcklH1b7IO/ESt63/C2DQLuBe7CCuYE4L5g86nAIcF9rwQchf3yspBz/9g/+nnBNdbDVmg/DcrQDPwbeAcrJqsBNxpjlgVl9l0mh2D/AHNjrrkzcIsxZkXGMhYq98+w4nmIt/0gYKox5kMRmQhcjf2TDQeuAG6vkMW6AfC8Cf4BAc8H693257xtz0W2lXpsZoLv8U6s8K2KrQx/JCLbB7tcCpxrjFkJWBu4LVi/HdBlQuvsGTJgjOnA/ne2DVY1AZcDY4E1g3UXB/t+H3gS+1IyMFgGW6luhP3+/gXcJCKtMdcyWBG6InDRRt1i44L7+TEwDOt5uU1EhqZc22c9YIExJtZCC35Th2MrxfZg3ZbA77GW3XDsi+ttVfLgXIkVmZWxL8RHuQ0isgq23vk+9mViLjDJ234wcDKwd3D8M1jr32d3bD07EfimiOxQQhlfBbbGGhnnAzeKyAhjzGLgn+TWM18H/mOM+TTjc/4a1nvjW1DTsV6KZNJULOsHa1G1Y1X0naCw/bAPcxnQz9v3EOCBYP5I4N3Iue4GToq5xhdi9v0hcE0w/1Ny30zWB5ZEyrhzgfv4E3BbML8V9s1tlFfuZxKOexX4Ssz6cRS2TN4tUKZ93XWDMs31zxd5PjMJrCRgGnBQwjln4L3Bl6PcWPF701t+GPhGMP8H4JyYZ7Z9N393cRbV/8OKt7/ueuCnwXwX3ts+VggM9gWh5GNTyrgDEYsK2B6YEVl3NvCHYP4JbEU+PLLPukBngWcS+6aMreBeSDhmS2C2t5zzFh+zv2BfxNZJ2D4C61WZjvVkTCOw/oCfAH+M7D+V4I06w7V3At6Ouedl2PqnC2t5buNtvwb4ceSYd4AvBPPdtajmB9d2n6XEWFTB/ApgnHeOXxPUW1jX8hRvW3NwL4cFyw8AX/e2t2LrqJW960zytt8OnFzM7yRh31eAL3u/3de9bS8A+xTxnA+NOf9FwO/TylBOi2pfY8wQY8waxpgTjDFLgDWChzk7aOT8FPs27TfszoycZwzWdI2yBrCqO09wrh9hvyTHHG9+MdA361uTiPTD+vOvBzDGPIq1wlwjcFK5Cm0rRM79i8goEblRRN4TkQXYN6YR3nXeMcbkNagbYx4HFgHbi8i6WIvv9oRrfgSMLrG8seXGuiv7icgXRGQNYFPg1mDbGsD3I9/dGKw1kYOI/MgLPLi8hHK1Y61an5Wwboy47SsB7cb+Y7pzbDGsAYyLPI9TgVWC7Udg28BeE5HHI16DUlkN+yaPiAwSkatF5N3gN3YP4W8sFhH5oYi8KiLzsW6lvknHGGPmGWN+YIxZL7in17AWN9h7Pyxy75OI+S0k8AkwKGb9X4wxQ7C/6zewvz/HGliL1b/mSOwzKQcbBHXfkKAMv0nYbxWsyPv/nXe8+VX9bcaYLqwr2rEGcLl3D3Ox3irfao3WgUW3iYrIt4IAGHedCYTf9YNAs4hsJTbQaDTWjezKV+g5R+sNsN/np2llqnR4+kzsm84I74tcyRjju0uif/KZxDfozwTe8n8QxphBxpg9MpalUGWyH7bi+b3YcNU52Afs3H9J5UrbtiiY9vfWrRLZJ1qu84J1Gxvr+jkM++N21xmbIr7XBvsfDvzTGLM0Yb97gf0Cd2ocRZfbWDfiP7CW56HYNkpXwc8EfhH57vobY26IXtgYc64JXVvHJZQvjZeAjUVyovE2Dta77b6bYZPItlKPLYaZwCsxv+X9AIwx040xX8O+0F0C3BIEdBQriMBnDdd7YdsCAM7AVm6fD35juxL+xoheR0R2Ab6L/Y8MwbrslkSOicUY8yHWahgnNqJ0JvZN3r/3AcaYi+OuHcN0YJCIJInkh1gX83nePjOBs2J+f7fEnGIR6b/77jAHe39jvHVjvfnZ/rbg/xmt5I+M3Ec/Y8xT5SqgiHwO+B3WuhsWCO/rBN918FJ2HWE9c6OxrmVXvkLPOe77XY9cl3oeFRUqY8xs7NvaRSKykog0ich4zxcfx5+A00Rkc7FMCN7QnwAWiMj/iUg/EWkWkQ1F5PMZi/MBsFbK9iOw7SgbYd/GNsU2am8qIhth24ZWEZGTxfaNGCQiX/DKfI6IrB2UeWMRGW5s+9B72DfIZhE5isJRdYMI3KgishrwA2/bE9gf8y9FZIDY8NhtvO1/wVYmh2F/TEn8GivK1wbPFhFZTUR+LSIbl1hugL9hfdBfD+YdfwSOC6wtCcq+p9h2v6IJytQXaAGagufg2kumYN0/3wu+p+8E6+8PptcBpwb3uyq2PeDPZTi2GB4K7uPkoOwtwW9mYrD+G8HvpwvrVjJYl9GH2LfZsYln9hCRVhHZAPsCMQgregTzi7G/sRHkR+hG/yuDsC6muUAbtk2yb8p1LxSR9YPvaTA2Su9FY8wi7MvUV0Vkp2B7v2DeCULq/zTw1EzBttcl7fM89s3ftXFdCXxXRCYFv7+BIrKPiPSPOfxZYC8RGRL8/76bdJ1iCV4c7wDODu57Y+x/xXE78HkR2Sv4Pf8A+1LguBw4U0TWARCRoSJyQDeK1Bz8/tynDWuBrcB+100ichzWovK5DtsGfQi59Uwxz5ngHgZg69z7kvYBKhP1F9k2mDBiaD62AfDgYNuRwEMxxxyHbcNox0adOP/2qtjQ1zlYF8Bj7rrYNqq/eucYh9fOAnwF68r7FDgtcr3VsCb0RjFluRO4MJjfMHignwRlOMOEvuQzCSNlniSM8tk9WP8p1hc7ldy2noci19sAGzXYjv3TfB+vjQP7BnYbYVTgJZHj7w2+j8R2E+9ZXh3cx0KsH/onQP9Syu2d93Wsi6ktsn634Ll8ihXbm/Cisor8vR0ZfLf+58/e9s2CZ7gEG1W2mbdNgAuCMn4czEs5jk0o6w4kR/39A1sxf4Jt09su2PaP4LtdiG0D2MM77nxsJfIpsGnMeX+JFZWFWOvgNewb8ujIb+ih4Df2CnACXtsXQTtEUK4LsO77v2Ajvt7Dtnd91q4TU4Yrg+Pbg7L+i9xIsW2C63+CFd/bgVXjrp1w/gOAWyP3HI103D4o77BgeZ/ge50PvI9ti+oXbPPbqAZg3ZQLsHXVaZQ36m8VbEBWUtTf3sH9J0X9fQtrxS/Aug0vTyoLkfa2mDJG/0OvB9suDJ7/3OD3ltduGHx/r8acN9Nz9vY/HPhbof+8BDsrdYKIXI3tt1N0PzZF6Q0ErtnHsW6wl6tdnkZERP4GvGyM+Xk3ziFYUTvYGPNa6r4qVPWD2NDfZ7FWwFvVLY2iKPWIiEzAehvWM8a8V2j/cqC5/uoEETkH6yb9lYqUoiiVQEQuwLpEf9ZTIgVqUSmKoig1jlpUiqIoSk1TC0lgy8KIESPMuHHjql0MRVGUXsVTTz01zxjTUwmLS6JuhGrcuHFMmzat2sVQFEXpVYjIO4X3qi7q+lMURVFqGhUqRVEUpaZRoVIURVFqGhUqRVEUpaZRoVIURVFqGhUqRVEUpaZRoVIURVFqGhWqcvDmm3DPPdUuhaIoSl1SNx1+q8rnPgddXaB5ExVFUcqOWlTloKur2iVQFEWpW1SoyokKlqIoStlRoSonS5ZUuwSKoih1hwpVsZx7LohAR0f+tsWLe748iqIodY4KVbGcd56dxllPKlSKoihlR4WqWFxkX2dn/jYVKkVRlLKjQlUsTqiWLw/XtbXZqQqVoihK2VGhKpY4oerTx05VqBRFUcqOClWxqFApiqL0KCpUxeIL1YMPwoIFxbn+5syBp5+uXPkURVHqDE2hVCwrVtjp/Pmw/fYwcWJxFtXGG8PcuZpuSVEUJSNqURWLE5g5c+z06adDoVq0qPDxc+fmnmfhwlw3oqIoipKDClWxOIH54INw3dKldlpMG5Xrh7XSSvDlL5enbIqiKHWIClWxRC0qCHP8FSNUCxaE81OmdLtYiqIo9YoKVbG4NipfqNy6LK4/hy9UiqIoSiIqVKXiu/5cG1MxbU0qVIqiKJnQqL9Sue22cN4J1OLFcNxx0NwMBxwAO+6Ye4yfdmnBgtASUxRFURJRoSoHy5bZ6fTpYXvT73+fH4L+6afh/IIFuRnY29th4MCKFlNRFKU3oq6/cuCEav789P0++SScnz8f/vjHcNl3JfZW3ngD/vvfapdCUZQ6Qy2qcuAsJ99iisNvl7r7brjhhnD5o49g/Pjyl60nWXtt+yy0M7OiKGVELarucPnluctRoVq6FGbMCJf9NqpZs3L3rYc8gSpQiqJUABWq7jBkSO5y1PV35JHwuc+FIuT6W0F+1F89CJWiKEoFUKEqhqjFsNJKucvRKL4777TThQvt1Leoon2u6kmoNJpRUZQyokJVDF/5Su5ya2u245z15AtVe3vuPrUqVEcdBSLFHaO5CxVFKSMqVMVwxx25y83N2Y6LE6qPP87dp1aF6pprij/GRUEqiqKUARWqNJ59FrbYIjk1UkvGoMk4oYpaHbUqVKWgFpWiKGVEhSqNk0+GJ5+Exx+3y6uskru9kEXlKuyoULlhQXzqSajUolIUpYyoUKXhxKN/fzudPx9OOy3cXsiichX2SSfZYT2cUEWDMPxr1Sp+xCLAm2/Cb34Tv69aVIqilBEVqjTcmFF9+tg+UUuWwNCh4fasbVTvvANXX927hSoqPl/+MpxySm62jaR9FUVRukFFhUpEThKRF0XkJRE5OVi3iYg8KiIviMgdIhJTa4OI7CYir4rI6yJyRiXLmYcxNrTcuey6usIKediwcL+sbVQAzz1ns08ADBqUv33x4vC6tdhxNio+rs+Y3+7mUNefoihlpGJCJSIbAscAWwCbAHuJyNrAn4AzjDEbAbcCP4g5thm4DNgdWB84RETWr1RZ87j2WthzT3j3Xbvc2RkKlbOodt01u0UFNq/fCSfY+SSL6qqr7HWvvbb0sleKqFA1BT+dOKFSi0pRlDJSSYtqPeAxY8xiY0wnMBXYD1gHeDDYZzJwQMyxWwCvG2PeNMYsB24EvhKzX2X45z9zlzs6wnDyYcOsxXP33cVZVD6DB+cut7ZaoXrtNbtc7QS1y5aFnZQdfqZ3CIVq9uz44x3z5pW3bIqiNByVFKoXge1EZLiI9Af2AMYE6/cJ9vlqsC7KasBMb3lWsK5neOKJ3OXOzjCPn582qRiLyidqUQ0ZYoXKtYn161faecvFVlvllzHJotp8c7jrrvh9b74ZRo6ERx6pTDkVRWkIKiZUxpjpwPlYq+ku4DmgEzgKOFFEngIGAXF+orhUCHkNNyJyrIhME5Fpc+fOLVvZ86yHzs5QRFwEIKRbVH/9K/znP/DYY/nbCglV377Fl7mcPPNM/rqoUPkiHb1HZ1E5gXr44fKVTVGUhqOiwRTGmKuMMRONMdsBHwMzjDGvGGN2NcZsDtwAvBFz6CxyLa3Vgfdjzn+lMWaSMWbSyJEjy1fwaMqgjg4b9Qe5IpImVGPGwB57wCab5G9LEip3jX79bHZ1EXj00eLLXwl8oTrqqLD9DvIty/32g733DoNGoumiFEVRiqDSUX+jgulYYH/gBm9dE3AmcHnMoU8Ca4vImiLSBhwM3F7JsubQFHksnZ3xQpXm+nN5AOM690aFasCAXKutb1+YOtXOX3JJ9nKXGz/60BeqaFql6HNYtgz+/e9QqKLtXYqiKEVQ6X5UN4vIy8AdwInGmE+wEXyvAa9graRrAERkVRG5EyAIvvgOcDcwHfiHMealCpc1pBwWlROquISuUaHq08dmHHd9qZqbrXhBcvqmnqCjIxTttEi+H/8Ynn46f/3AgXaqQqUoSjeo6Ai/xphtY9b9FvhtzPr3sQEXbvlO4M5Kli8zWSyqvn3DfSA9s/qXvmQ7zN59d3jsihXh8V1doVBVsyPw4sX2HlesKBxyvm3eVx0+HxUqRVG6gWamiCPaN6izMwwQ8F15vkUVDYBIE6p1182NlGtttWLgXH9dXWEZkiyq++6DF19MvkY5WLw4vMdogEmUuDGonMhqG5WiKN1AhSqOaKXsXH/NzbniFLWofNKEyrnTLrzQBlI4q8UXKieMSUK1886w0UaF76U7OIsKCltUcdk0XCSmWlSKonQDFao4opWyc/1FxcgXrWjfpySham4O262+/32b8aKpKVmoCrn+li+Hl19O36dUfIuqFKH63//s1KWiUhRFKQEVqijGJFtUUaEqxaKKC8BoarLi5AuVa68qFEzxgx/ABhvkhot3B19wirGo4lIpuchFdf0pitINVKiixFW4SRZVOYXKt6hWrEi3qHwxcWNlzZoVf71i8UW6GIsqro3K4e5LURSlBFSofJYuhba2/PVJFpXf36o7QuXaqPyovzSh8kWj3H2V/OsVY1FlPaeiKEqRqFD5vP12/HpnUcV13nWUw6JylpIvVNEBCyG34nd9lcrlXquEUC1YUJtDlyiK0itQofJ588349TfdZBOspuXgyxpMkSZUDl+o4ir4NKH68Y/hrLNy9z/iCLj44uSy+/htYosWhUJVKDw9ij/ApN/+piiKUiQqVD5vxKUdBKZNs9M0q2DvveHss8NlX6gefhj2CRLGpwmViwb0hQryRcIXKuf6c+NlnXsunHNO7v7XXQennppc9qRzF9NGFcUJqEMj/xRFKREVKp8koXK4UW3jWHnlXEvGb7/aems4+mg7n8Wi8oMpwI6PdbuX6tAXE2fxuPGyustll4XzP/lJeB//+IcVwd/9Ltt5oqMYq1ApilIiFU2h1Ov48MP07W5MqjicW/Cmm+Dqq/O3u5RIxbr+AA491E6dRecLlRPPjz9Oj7zLyp1e1qqPPgrbyB5/PIwwzIJaVIqilAm1qHzi2mF8yyjNoho2zE4PPDC3sne4caziMq4Xcv05nnwS3n8/V6icy++ll+z4V9F9o7z4Ijz0UO7gkI89Bu+9Z+c//dS6Cf/6V7tcalLcJKFatAjuvTd329135+ZJVBRF8VCh8okTKr/CjYvAczihSqI7FpVjiy1gwoR4oZoyJWwHc/uOH59/jo02sglkv/CFcN2++1pxWr7cCsnQoaGwFhtE4YgKlRP5Y46BXXYJ3azPPAO77QannFLadRRFqXtUqHziKuW0nH0+5RCqQhYV2Oi5OKGKI4uVsmKFzck3eTLMm2fXDRuWO5Lx/vvDwQcXPpdPkkXl0j25ZedO7U4aqEIuW0VRejUqVD6FhCputF7HkCHp5y6HReXIKlSFWL7cCsaKFfY8kyfb9b5FBTb0vtgRlKPBFC7Yw4mxa29zrtVS29fuvdcGsvhuT0VR6goVKh9fqP71L9sB2AnLpEnw4IPJx6aN9gvZhMpV1tGovyi+UC1cCOuvD0ceGb9vmgAsWpQrdDfeaKdRi6q5OXfZkSZevkU1YEDYmbrcQvXoo3b62GOlHa8oSs2jQuXYaCPbzuMYPhzWWCO0qNZbL39k3mJwFX1aUlpXWTuLKsnt+J3vhPPt7Xa/L34xfl9ffKP9wNrbcy0dN0ZWVKiamuKFarXV4q8JuRbV+PFhm5QTqhUrrDW03XbhcrFcc03YJaCpyJ/yl74Ehx1W/DUVRelxVKgc0UEInUg4YYmrqIuhqcm60NJy/UWFqpA70dHWFlpsUXyhimaH8IVq7NhwfdT1l2RRxXWAdhk6fItqwoR8oVq2DC66KNynFKE6/fRwvlihmjIFrr+++GsqitLjFPx3i8jNIrKniDSWqJVbqNw50lx/LqqwWKFqbc0PXnD4QhXNB9jeHrr+/AjBrBZVVqEaP96mp+rqCgVl2bLcTPVpEZVJ+O7Wpib4wx+s+PzoR+luWkVRehVZOvz+AfgmcImI3AT82RjzSmWLVQO4LOqFhOqKK7ILyoAB2aL+li6F6dNhxx1hxozC5+2OUDmLavx4uP9+Oz94cG7EYDEWVf/+9px+XsSxY23gxrx5uffoC1UpFlVUqE44IVw+77zklFeaIFdRehUFrSRjzL3GmK8DE4G3gcki8oiIfFNEMsZu90KcReUqtSShOvZYOOigbOdMEypjwsr60kutiOy7b7bztrUlC9Utt4Tz0c67fjCFs6ja2qwA+El2S7Go/PtcZRU7/eCDZIsqTqhuvDF9nK2oUPnEPWeHL9h33JG8n6IoNUEmd56IDAeOBI4GngF+ixWuyRUrWbVxb/7OIimH62/HHWGrrfLXJ7Wv7Lln8rn8IUdaW5PbqI4/PpyPs6hmz7aBD6NH23XOknRtalC8RQW5QrHyynY6Z05uG1Wa62/RIjjkENhjj/j7cuVKKoufvT2Knxfx8MOT91MUpSYo6PoTkVuAdYG/AHsbY2YHm/4uItMqWbiq4ipOlzW8HEJ16aXx6+OE6qKLCod/uxD2NIvKJxrC3d5ugxzGjw87LPsDR/bvbwMwmprix+JKs6h8EXFC9cEHoVA9/niuWzJqUTlLb86c5PvxrxEN588qVG6sLFcuRVFqjiwW1aXGmPWNMed5IgWAMWZShcpVfYYPt9MJE+zUWRyVIE6o2trCynP//fO39+0bikpaG5VPdKgPX6hcxR4VKrCCEFeRxwnVgQfa6frrh+t81587z29/C089Fe4TtaicmKTdly9U0YjGNKHy+44ZU3o+Q0VReoQsQrWeiHwWLSAiQ0XkhLQDejXHH28rMmfN3HqrDWzYa6/KXTNJqMCmGPr738P1LkihtTWcTwpPT3IHOhYssNF4hYSqmNDvk0+2WdfXWy9cN2iQLavv+oviC5UxoVC5ezAmXxh9oYomDB46NP4YyB8SpVyjIyuKUhGy1EDHGGM+G9/CGPMJcEzlilRl+vTJjeIbMADWXbeyrqE4IXDBHIMH57b3jBkTbvdFyxcYx0YbpV/3tdes+22ttcJ73nLLcLvvxhsxIv/4OBFobrZuRP95iVj334cfJj9H3yJqarIdciFXLE+IvB/5z+2jj/LLccop8c82mnZq4cL4MimKUhNkEaomkbB2EZFmIKZWrBOyJqEtJ2kWVRS/4nbtRm7fxx8PQ8wBVl01//jzz7fDagwbFqY1GjbM7jtlSu5YWk4Im5pg001tJonFi8Nh7YsJ83btXUlC5aeF8hkwIGy/uvzy3G2+RRUVquXLrXsR8tu/osl6VagUpabJIlR3A/8QkZ1EZEfgBuCuyharivQWoYJciwrs0B5+WqM4oTr0UNh1V+vafPddu861A22/fa670AmhE4SddrJWlnODDh6cfE9RWlut9ZYkbkntRAMG5IrY978fzvtRg1F3nguCgfxAi2jy4e4K1XvvwTrrhMKvKEpZySJU/wfcDxwPnAjcB5yeekRvpjcLVXQ+zl03apSdDhsGM2fa+aSABSdU0fKNHw+/+hXcfHP8cY7rrgsDJpxQJY1vtXx5rrg42tpyRezXvw7nfXdhVKh8caq0UP35z9aNesUV3TuPoiixZOnwu8IY8wdjzIHGmAML2/24AAAgAElEQVSMMVcYY0rId9NL6I1C5e/ru8OiQ20MGxbu60fFJQmVexbRzPAicNppsPrq8cc5Dj8cJk4My5gkRo4FC+Ccc3LXucEcfS65xE59oYpz/TmiQhUtQ1So3njDukizEs0EDzYLx5lnlpYaSlGUHLLk+ltbRP4pIi+LyJvu0xOFqwrVEKq4IUJKtaj8yjJ6Xj9IxB/oMSk60J03Lerv8sthhx2St/vn6uhIF6oHHwyzoTuWLcsXqpNOslNfqNx8//42iKQYiyoa9bfTTnDGGfnil4QTKr/97fjj4Re/sO16iqJ0iyyuv2uw+f46gS8B12E7/9YnhcaVqgSlWlROSJIsquh5/W1ZLCoXbZj2TL797dzw+SSyWFTRDPYQL1RgAySWLrUZ1HfdNVz/6KOw4Ybds6hcVGDWSM84oXLnqMbvSVHqjCxC1c8Ycx8gxph3jDE/BXasbLGqSDUSlqaFp0fxrR/XLuMHNfgVY7SS9MPcfYuqkOuvUD+qLON0JVlUvmDGDUcf5/pz65cts8EdfgLcAQNCUXREo/wKtVE5d11Se1qUOKFyFp5fNkVRSiKLUC0NhviYISLfEZH9gFEVLlf1qBWhymJRPfusnbo+R5Ddosri+stiUUF8eqUoSRbV++/bsHoI78dn2bL4DrlOXOKEqk+fdNdfIYvKRRN2dNi2pjjB+uSTUADjhMptS/oeFUXJTBahOhnoD3wP2Bw4DDiikoXqcXxxKnVI9O5QqlC5QAW/Y29WoXIpoqLrfZxQFbKosrjIkiyqvn1Di+zVV/OPS3L9LVhgp1ksqmKj/pxQLVliw/i//e386w8bFr4guN9MnFBV4/ekKHVGag0UdO49yBjTboyZZYz5ZhD591jacb0OX6h6i0UlAvfdZ0PM/Qoyq1CNG1e4XElRf6XQ1haGpx9yCLz1Frzzjt2W5jpMEqpPg2QpAwbkWoT9+xe2qAoFUzjXn0vL9I9/xJfNJfl15/ev44TK7+ulKEpJpApVEIa+uZ+ZohhE5CQReVFEXhKRk4N1m4rIYyLyrIhME5EtEo69IDhuuohcUmoZMuG/9fYWoTLGRvFFQ8T9c6UJlT+ibxJZLaostLaGrr8xY6xQjh1rt6UJVVIblQtWGDgwbGPr29feY7TvVTGuvyM8Z4ETquh3Ef2NuA7JfhSiEyoNT1eUbpNlhN9ngH8Fo/t+9u83xtySfAiIyIbYnIBbAMuBu0TkP8AFwNnGmP+KyB7B8g6RY7cGtgE2DlY9BGwPTMlQ3uLxhao3uf7iiAZTiIQVa9wYUWk4iypLZfvII7mDLcady7n+oveWljy3kEXlC1U0pZR/Dp80199114XzSUIVzdSeJlRqUSlKt8nyqjwM+Agb6bd38MmSSnw94DFjzGJjTCcwFdgPMIB7hR4MvB9zrAH6YnMK9gFagQ8yXLM0eqNFlUTU9edbK/62LAaqE7Ys0W9bbWXzASbR1mYr766u/IjGtLIkCdVXv2qnAwfmC100uOPAA0P33u2354fTT5mSn0cQ4IADwrL7RF2FTqj86EIVKkUpGwUtKmPMN0s894vAL4LRgZcAewDTsMEZd4vIhVih3Drmmo+KyAPAbECwY2JNj+4nIscCxwKMdW6kUqhFiypLeHocUaH63/9g443zt4G1HtKE2ZWhHJVta2sYABEXsn3WWbYf1S0RQz3J9ecYMCC0qNy9RIWlqwtuuAGOOQa+8pX48xx/PBx3XPy26Pmi5XHL6vpTlIqQZYTfa7AWTg7GmKPSjjPGTBeR87HD1bcDz2E7DR8PnGKMuVlEDgKuAnaOXHMC1iJzDTCTRWQ7Y8yDkWtcCVwJMGnSpNJNod5oUSVZIdE2qo02ghNPhMsuyxeqQsOwF2NRFaKtLay0o6mdAM4+205bW3OFMcmicviuP/fdxYXLz5sH11xTfLkhu0XlC5X7TUVFftkya70dd1y2sH5FUTK5/v4N/Cf43Id122Uaac4Yc5UxZqIxZjvgY2AGNrTdvTbfhG3DirIf1m3YboxpB/4LbBmzX3nwherggyt2mUSKEaq0dqCk8zqBKjZ6r9wWlSNt1F5fgA8+2H430UERfeJcf3HP7kc/gqO8dysXTLLVVuG6JOsnTaiMiRcqR/TZXXCBHVzyL/Wb3EVRyk2WpLQ3e5/rgYOADbOcXERGBdOxwP7YIULexwZGgG33mhFz6LvA9iLSIiKtwf55rr+y4YTq4ovtcA09TZxQJYlKMW/h7hzu/C1ZYmc8ym1ROdLcl66sDz8Mm21m56OZ0X3iLKosnWxXW83uP2lSuM51PI7in+/993P7ey1dWpxQPfCAnfovR11d8PTThcusKA1KkTUXAGsDWRuEbg7aqDqAE40xn4jIMcBvRaQFWErQxiQik4DjjDFHA//EitgLWLfjXcaYO0ooazZcpVGOMOxSKMbScZZJFhelu5+oZVXstcohVMVaVG1toShHR+T18duo3PeYJaWTExDfrbjNNvH7+kLlj/cFtt3NnSOaqgnyrTTXd2zu3HDduefaNronnoDPf75w2RWlwcjSRrWQ3DaqOdgxqgpijNk2Zt1D2AwX0fXTgKOD+S4gJh1Ahai2UBVz3WLEprtCVSmLKk2oXFn79AmP+fhjm3j2xBPzgyFaW/PPlyQ4PnFClUSahbZgQRjevmSJFSbfWnLXiQbpzJljp8aE1tS778YLlTH2eE1wqzQoWVx/g4wxK3mfzxljCoyY18voDUK1YeBtddaCc4tlOW93Lapyt1Gluf7iLKqPP7bHJI0o7M7nrMwsfcSc+E6YUHhfJ1Rxrr3588NoxunTrbi/9Va43T27Lbaw5XTLHwS9LQ47DG67zc4ntZFdfrk97weV66GhKLVMlvGo9hORwd7yEBHZt7LF6mGcUFXrjTWLUE2dCk8+aTNR/O9/cNVV2c9bajBFOS2q7rj+Fi60lXySuzPaRgUwY0Z87kCHu6ef/ASuvz5b2d+MGYbtww/zBcy/rhOfp56yrsGoUP3tb/n7RnGdkGfENecqSv2TxYT4iTHms7ArY8ynwE8qV6Qq0BssqmHDwob/L36xcMdf6H4whQsjLybSMInuuP7AClWhvmUjR4brJkyAz30u+TpOMFpbYa8C/df//W8bNfjGG/nb9tzTTv3hSvwoxag16gTywQfhzjtzt0Xdg9/8JvzmN+Hzj7PoqsGzz9qoybQgF0UpI1lq5rh9SgnCqF1qSaguuig5CWqp5y3V9fe1r9mh4X/+8+6XpVjXX2trboTjwIGw9dZw6KHhOucy69/f9hNzEXU+f/1r/HV8KzGLEJ93Hrz3XvJ2P8mv76KLClVnZ5j1furU3G2+RWUM/PnPcMoptSdUP/+5tS7vv7/aJVEahCw18zQR+bWIjBeRtUTkYuCpShesR6klodpyyzA9ULnOG51mpaUFzjwzvoNusfjWUZo16JfVF6oBA6yInXlmuM4PrDjhhPhEu1//evx1fAGJWmonnBB/jC9A0cjCNdYI512gBMAPfxh2ZnbX3X57WGWV/GhGX6g++iicd0Ll8htWm3K6hBUlA1lqru9ik8r+HfgHNh3SiZUsVI9TS0JVznayqCVVwQT0BfHFIO0533IL7LGHdaVFhQqKd18mkVbJJrkmfQHyXX2Qa1H5+7W3w09/Gi53dtp7GDYs33XmC5XvZnRCVSuutnJ2W1CUDGTJ9bcIOKMHylI9akmoylmGqCVVTaHKOtLtDjvYT/QYJ1RJ7VTFkhbJ6Lsmv/Sl0KXoW1TRF4okiyruui0tVuiiwuOXyReqZ56x07T+ZD2JWlRKD5Ml6m+yiAzxloeKyN2VLVYPU69CFQ2mqNb9QWkCE2dRlUuoslpU/vV8AYoKVZJFFaWz055z2LB84fGHI/Hbw156yU7VolIalCw114gg0g8AY8wnwKjKFakK1JJQVcL1VwsWlXvGaUOBRKmW68+3qPzrzZoVzqdZVLNnJ5/bmGSL6q23wucUHdwRaseiKmf/umKZPz93/DClIchSM68IcvUBICJrEJNNvVdTS0JVr64/Z6Ucf3z2Y8rl+ouLMtx99+T9fYGMCtWo4B2tuTm3A7JvUX30UXLnZHfOOIvqt7+FCy+083FCWitCVU3X35AhMHx4z19XqSpZXk9/DDwkIi6Wdjt6Mr1RT1BtoYqOylsuSg1LrwTbbw8vvwzrrZf9mHK5/mbOtG610aPt8ltv2ai7JHxxilpwo0fbTr7NzfY8Cxday2LoUHjttbDv1oABuf2p+vULw8udRRVnGdx/P5x+erwI1Ep4uvsO4qy+nkBdjg1HlmCKu0RkInaYDcGOJTWv4iXrSaotVI1gUUFxIgX5/aigNKFKi9CLw4++iz4zl56pudme1z/32mvb8nV05Ifgb7013HefnXcWVRwuwi+uMvbbsBwHHWRTON11V/L9lBv34lMrwqnUPZlqRWPMPGPMv4GXgeNE5MXKFquHqSWhKqf1U0vBFKVQ7jaqqVPDwIQ00kZ89oUqDidQUaHyhamlBTbPy8uce1xHhxXnL3853BbNzr5iBdx0E9zdw7FNTsjd8CaKUmGyRP2NFpGTReQJ4CWgGTik4iXrSWpJqOrZoiqWuDaq7gj5dtvB+usX3s+3qKJC5bdRxZEkVL7l1dJis6QPGUIevlD16QMnnRRuiwrVCy/El6HSOGtPhUrpIRJrRRE5RkTuB6YCI7BDcMw2xpxtjKnSP6RC1LtQOYHqbUIVZ1H1xD2kWVTOBdkdoWpttWIVFwHpC1Vra25QRlSo3n03vgyVxrVNqVApPURarXgZ1no61BhzpjHmeeot2s9RS0JViWCK3kpLS3gPafkBy02aUBWy7LK6/qLrHE6InVD5qZqiQlXOoIKFC+HSS7MNyFmsRfXwwzBlSuH92tuzl0FpKNJqslWBG4Ffi8irInIOUKbeljVGLQlVJSwq98fvbRYVWPdfc3P2zBalcsUVdtrSYlM47bgjnHZavlBtu621qn6SMICAs7iiwhonVHGuPxcwkUWoyhl194MfwHe/C//9b+F9i7WovvhFm+GjnGVQGorEWjEIoPiDMWY7YCdgPvChiEwXkXN7rIQ9QS0JVSWCKXqzUPXpEyakrSTHHmufU0cHjB1rI/R+9av8MaLGjrXWx047xZ/HCVKhNiqI72vlxGj58nyhciI2a5Z1+5XTonIJb+fPt8/h8ceT93XX9UdHXrIEnn/ezr/wQhh6Hw1e+eCD3IElfVwH6AceqF7ou1KTZI36m2WMudAYszmwLxATJ9uLqSWhqoRFVe376w5OqKIkCUW5iVpU0azpUZwgRYXKz0CfZlE5oXIWlX+c27bmmjYTRjkrc2exLl8OV19ts/jffnv8vnGuv6OOgk02gddfh403hhODvNVuZGrHKqvAWmvFn9c9lwsvtON/KUpA0TWXMeZVY8zZhffsRVS7Iu+pNqreaFG1teUL1eLFPddvKGpRFWori7Oo2tvjs11ELap11skXqubmMAN7V1fuKMFxwRQLF9r1WdIbdXaG1rYvVK+/budfjOmF8vHHofXl9+t67DE7dS7UtHyHjqjQ+l0Pnn228PFKw9ALX7ErQC0JVSXbqHojffrkD7vRr1/5cv4VImpRFRJ7ZyWljU7syh7tAL3SSvltVO54J5D9+oVluPfe8FhjrJCttZa1tr71rfRydnXZ8//f/+WWd/ny8Fq+aw9sotzhw8OOy75QOSF2KaAmTsx/dj7332+PcQIHud9pOUaVVuoGFSqoLaGqhEVVD21U1cL9Nm66KRxuIw1XwUatBV+4nADtuqt1szn69s23qPxtjhEj7NRPftvVZS2veUHSmHfeSS+ns4quvDK3fL5QtbfnHvP++7nLvlD55QP7W/Ofweqr527/9a/t9LXXwnW+UEXPpzQ0WTr83pdlXa+mloRKo/5y6ds3eSDDnsD9NlZeOVvmd1fBRtMd+ULlV8i77pp77AMPwO9/ny9UvuvQlclPUtvRYVMpOaIReV/9KlxzTbjsjnVtbq58fvon36KaOhW22CL3nHEWlV9GX6iigR9ujK+kTPX9+lnX5LbbJreVKQ1Dov9ERPoC/YERIjIUm+cPYCVs6Hr9UG2h8q2oSoxH5e6vNwrVOefUhkWV1dJ1QhUNJU8SqjgxOvFEG8zgWxX+vBMhf2j6NKGaNQv++U/7+eY37bokoVq+PGyX84XqtNPII82i6upKFypXPv8c/m+/Xz97Pw89ZD+92X2tdJs0R/+3gZOxovQUoVAtwHYGrh+qLVQaTJHMHntU9/qu0s7623Bik1Wo/PV+ZZ7m+otLBrtgARx9tJ0fOTJXqCZPttMhQ2DGDLjxRpvCCcKADj8juhOPG2+En/3MuhinTcu/ZppF1dUVbh84MDlC0T+Hf/99++aK04oVvTNqVSkLaf2ofmuMWRM4zRizljFmzeCziTHm0h4sY+WpJaGK/uHLcd7e7PqrNsVaVAccALvsYqP0fLJYVFELKUmo4vjjH8PAhFVWyRWqGTPsdMQI25H5rLPCoe5d+Lv7bSxfnisq3/++HaIljiSLqrk51/U3YEByny//HL64t7XlBmPoYIkNTZZhPn4nIlsD4/z9jTHXVbBcPUu1hcoXEG2jqi2KtahWWgnuuSd/fRaLyh+/Kq2NqhCrrAJvvx0uOwH48MNQMNy1nOvPhbMvXZorVGPGJF9n2TIbAbjTTrlCNWhQrutv4EB77aRzxM13duZ2DdAxqBqagkIlIn8BxgPPAu6XYwAVqnLhKovf/a6851Wh6j7f+55t11lzzeKP3XLLMHLOr8iTBmZMs6ii/bmi+ILnLCpj7HfuhMpvw3LRge6353fi9YNXCoWJ77yzvY4vpK2t9j/lu/5c+HyUJIuqoyPXolKhamiydEaZBKxvTB23ZlZbqPr1q0xjcT2kUKo2Rx5pP6Xw6KPhvJ9hwhcg/zvxhcqlUHIUyqvnb19lFSsKHR1WwKLtZRAmiXW/eWdRLV5sr92vny1zNEQ9jhUrcoW4qSnfooL88HbILlSaUqmhyVIzvwikjNtdB1RbqCqFWlS1g//s4zorb7ddbuDIRx/lCtUaa6Sf33cbujGz4iLrHK5PmLNyfKFatswK3MCB2YSqqyvXompuzg+mAJsnMUqSUC1frhaV8hlZauYRwMsicreI3O4+lS5Yj6JCpfQkUaGaNctmDL/+ehtlBzY03BeqSZPCjBBx+ELlrDcnVEuX2g63cQEhTqCiFlUxQrViRe6544Ip4ujfP1+oJk2y7siOjvw2qrRMF0pdk8X199NKF6LqqFApPUlUqFZbLZzfbLNwvjUyqo7f4Vgk113sC5XrsLvaanafpUutO3DsWHjkkdxzRi2qRYuswLjUVdE0SnF0deWKSpLrz2fjjWHmTCtUW21lw+SXLbOC2t5uRfujj8L9zz8/OSBDqXsK1szGmKnA20BrMP8k8HSFy9WzqFApPUlankI/gi4qVH6F79IoOZxQTZmSn7l96VJ7zmhuQci3qDo7Q9ffgAG5YuG45x644IJwecWKXGvHWVRR159j551tZoo+few+jz0Gd98dltPdt594+JFHctv8lIYiSwqlY4B/AkFaZFYDbqtkoXqcehUq545x055K5Kqkk/Y99OsHp54av58f2RcnVIMH2z5PvlDtv78N0ujb11pVUebPh4MOCvtVdXbmuv7i8hvusgtMmBAux1lU7e3wla/Y5ahQHX64zTLfp09uu5QTqrhBMj/+OPtAjcVgDJxwAjz4YPnPrZSNLDXXicAWwOMAxpgZIjKqoqXqaepVqNz9fO97th0kLg2O0vMUemFYeWU79cPJo4waBdOnh8vz54cC5YeU33qrne65Z3hen+hwKS4IIi5rvY8fPNHVFf6H7rwTTjnFTh3R8ziL0VlUDidUcW1pn3xSmTaqFSvgD3+wnzoObO7tZKmZlxljPosNFZEWbD+qgojISSLyooi8JCInB+s2FZHHRORZEZkmIlskHDtWRO4JRhR+WUTGZblmSdS7UA0YAJddlhsirfQ8TkgKZbkYOdJO3Yi3cYwenbvsC1VcFockiyqKb1GljWkVTZK7YoVtX9p9d3t/vjhFgync/ScJVdx1ixWpjz6yyX0LiU/WaMLOTpvxPS6Csie46qr48P4GIUvNPFVEfgT0E5FdgJuAOwodJCIbAsdgrbFNgL1EZG3gAuBsY8ymwFnBchzXAb8yxqwXnKNyLan1KlTaJlVbXHONzfRQKMnu8OF2GidUTmzShGqrrfKP69s336KKs+xcEESfPnbY+CSiFlVXV/j/aWrKdT9GLSrnwosK1YIF1n2ZZdDHQgJ0+OE2ue/zz6fvl1WorrnGppP65S+z7V9O5syxeRz32afnr10jZKmZzwDmAi9gE9XeCZyZ4bj1gMeMMYuNMZ3AVGA/rDXmxvMeDOS9JojI+kCLMWYygDGm3RhTAQd1QL0KlVJbHHSQHX23kOvPjRIcF8jgIgRdRgnHihWhAI4YAf/6V+72vn1h6NDcdXFtQc7119YWjtIb134TZ1H5baJ+m1VWoerqgvHjs4lH3OjDPu+9F5YtDf9aUYE0JgzgcNGPU6eGFmtXFzz+eOGydofXXrNuewiziTQgWWrmfsDVxpivGmMOBK4O1hXiRWA7ERkuIv2BPYAx2IzsvxKRmcCFwA9jjv0c8KmI3CIiz4jIr0Qkz18iIscG7sNpc+fOzVCkBOpNqI44otolULrDWmvZ6YEH5m877jg73WCD/G1pVkzfvvl5++LyB/quv699za7baKNwuzuHb63HWVS+AETLMnFieP2oK238+GwW1cYbp29354hGTibtB6EwO666CrbeGm65JRThBx4ILZvzzrNpsqIh/+VknXXCTPflHFmhl5GlZr6PXGHqB9ybsO9nGGOmA+cDk4G7gOeATuB44BRjzBjgFOCqmMNbgG2B04DPA2sBR8Zc40pjzCRjzKSRzq9fCvUmVFddFT8UhNI7GDbMvsH/MOYd7uij7Rv95pvnbyskVEOGwA03hOvSLKo+feDMM2303pAhtjxLloTRgb4QuWAK36KKE6oBA2zZXcWbJFTlyELhzlGocvevFY0qfPNNO3355dzzuPRTzz1np87iqTQNHLWbpWbua4z5rHt6MN8/Zf/PMMZcZYyZaIzZDvgYmAEcAdwS7HITtv0pyizgGWPMm4Hb8DZgYpZrlkS9CVVzsw7l3dvp3z+5jXHgwNA9GD3G38fHWU9+QE2cReXaqNra7P/BuRP798/t4+S7EVesyLWookLlRyP65YoTqtGjiwtwSMKdo1AyX/9a0Zc7V+4lS+IFz30/lYoWjJ5XhSqVRSLymUiIyOZAptd1F8YuImOB/YEbsG1SboCbHbHiFeVJYKiIjPT2eznLNUui3oRKqX+ibVRgLR9HNGDDvbj4FW6cReVcf4WGFVl33dANGbWooq4/d53oy1NUqFpabOWfxfUH8cl2HS4rRiGh8q8VFSoX5l9IqCpFVMRVqFI5CbhJRP4nIv8D/g58J+P5bxaRl7FRgicaYz7BRgJeJCLPAecCxwKIyCQR+ROAMaYL6/a7T0RewI4u/Mci7qs4VKiU3kbcb9W3cpL6LhUapNO5/gq17QBsu62dumCKJIvKnSs6ZEjfvrmV8Z572mlWiyrNve3O4YToBz+w/QmT9gMrfE8/bSMr581LF6pRo+Cpp+x8Vovq1FPhqKPit02ZEqaPckTzLGb5TuqUVIkWkSagDVgXWAcrGK8YYzL9kowx28asewjIc7AbY6YBR3vLk4ECLaZlQoVK6Y38/e828MK1+fjuwCSh8ivcJKHq6CgcQg/h/yUumMJx/PHh/ytOqJxV1NxsQ8ChMkJ14YV2eskl8fu5811+uQ3Lnzw5XajmzrUfyC5UF19sp1dfnb/t9NNtpOJLL8EXvmDXRYVKLap4jDErgIuMMR3GmBeNMS9kFalehQqV0hs56CCbbdzhC1XUrZfm+vMFpLPTBk6kZaVwuHPFBVM4Dj88FJSo669v3zCA4f/9v9AiLKdQdXWlC0nU9eeeyfLlYXmvvRZefz35HOVoo3IuSv/ZRRMCu23nnlu4f1idkaVmvkdEDhCp496jKlRKPeC7/kTg29/O3xZnUfnHLVliK+ksFpU7V1wwhaOpyYaj77OPjUb16dcv7JPkWwtZhSqtjcq3qPyw82i7T9SickLV0ZErQOeck61MpRJXB8VZVEuXwo9/bMcvayCy1MynYqPzlovIAhFZKCIpSch6ISpUSm/GCUM0EvDyy8P58ePt1P+Nu0rZPy5taI4ovusvGkzhl62tzXZA3nDD3ON9S84XqkIBEI6srj/fGooOFZIkVMuXZy+HE7TXXoMnnii8/8MPw1tv2fkHH4RLLw0tS2Nse9WsWfFC5bKVNNhAkgWdnsaY+k8Q536QKlRKb6StzVay0cwTPk6o4lx/cccV4/orZFEl4QuVHyhwzjnWFViINKFy4tHZmZsKas6c3I7PUdefK0cpQrXOOrnLSXzxi+F+e+6ZK0gdHfClL9nsIn/6U+5xLS02OS/kt/fVOVmG+RAROUxE/l+wPCYpkWyvRS0qpTcTZxlFceHscSISd1w5LaokkiyqM8+0Y14VwheqhQutIM2cCe+8E67v6sodVDKavzAa9edbVMUkwi01M07UanL3NG9efhuVb1GpUOXxe2Ar4NBguR24rGIlqgZqUSm9mYMOstM4yyga2RcnHKUKlR9MUYpF5QdXREOvs0S4+UK16abhKMbjxoXrOztzh0vJ6vrr6CjOohpVppGP/ETEURFrblaLKoUvGGNOBJYCBH2hYnoK9mKcXz6uA6Si1DqXXWatiLhhXGbPzk1m6guHi4+KE7higymi/aii+8SRZFEVOs7hB1O4dEdRokIVdRdGXX/umSxZkl2osuyXNTLQT0Sc1kbVYJlnsghVR5AQ1gAE2SIqMIJZFXHZous4sFGpY2HcsJoAABWVSURBVFpbrSURx9Ch4bAhkCsA7vdeSddfKW1U0XJCfJ+vLPksu7qsULlyRCMFoxaVW25vz+76i2bSWHvt/MTQWYMffIsqzvWnFlUilwC3AqNE5BfAQ9iMEvWDS8KpKPVOnKVSyWCKUi2q6PKIEfnH+0KVlLXBWVTONVcoPN15V9rbs1tUURF6/XW47rrcdWmh9D6+UEWT5DY1hdsbLEtFQaEyxlwPnA6cB8wG9jXG3FTpgvUoKlRKo+BbOM4dFSdKPWFR+e6rQq4/3yp0+EKVZFE6oXLHRwUj6vqLs6gK1Q1ZchNmHc3Ad/1Fj1mxIhSqao00XCUSWyxFpC9wHDABO2jiFUEm8/pDhUppFHwBcBVxayv86lfwzDPwt7/ZdcW0UaUFU2S1qAq5/uKsPl901lgjHILEx7n+Bg/OTdnkcMLUv7/d5sRz0aLQoorL8h53jjTihCqu3SpNqLq6Qndggw3jk2ZRXQtMworU7thBDusTFSqlUfAFwFXEbW1w2mm5Q9gX6/pLSqGUtY2qkOsvLlDEr6yT2pOcRbXSSulCNWhQsuuvUOBCnFsv2t4dJyxxAucLVZz159apUH3G+saYw4wxVwAHAvWbs0OFSmkU4iwqF+1aaAiQKFmS0pbDotptN9hll/zjo8PYx5FVqAYOtNucUC1alN31F21LgnxhzSpUfhtVnEWlQpXHZ0+xbl1+DhUqpVGIa6NyIlFsdu7uWlRZ26h+97v4ITp8cUpqJ3JCleT6c8cNGGCFwwmVH55eilAtW2Y/jz8eni/KyzFD7PkWld9RGeDVV8PUSw0mVGm/zE28nH4C9AuWBTDGmJiR23opS5eqUCmNQZzrL05gshANpuiORZXm+ksSO1+ckiyqri6btWLAgHSLasAAK1K+UHXHolq2DM46Cy64AJ59Nt49uEVMgh/foopmu/BzFmaNIqwTEl93jDHNxpiVgs8gY0yLN18/IgVqUSmNQ5zrz60r1aIqNTNFVtdfd4Rq4UK7X2tr/kCNkB9M4bK5L12ab1GNGgX//Gf+NeKECmySWoAXXsjPiJGE7w70O2pHWbrUWsTLlmUPo+/FaM4gUKFSGge/0o+mDivWosri+itHZoqkc/gVdFJlfeaZtlJvbrb/8TTX3zPP2MzlkOv6c+11ffrEB3UkCZULiX/3XTj00Ph90iiUP9BFKR52WPHn7mWoUIEKldI4xFlUpQpVlmCKNIvKv15arj93jmgkXRaLyj9fkutPJD/Tg3P9NTfntuEVkyHDtcHNnGmnm22WXsYohdx7TiBvvLG48/ZCVKhAhUppHOLaqJwQOHHImkqsuxaVT5pF5coXFZNoMEVatobm5mShamnJP3bJEnvOpqbCQpVkUX36qZ2+/bad7rtvcvlKIS5bR52iQgUqVErjkNZGVWxbVXctKp8sbVRRoYpaVGkh9UlC5QQu7tglS+xx7nkUK1QuL98rr9hp1kSy/vP/xjfg/PNhp53CddHzNECOUhUqUKFSGgdfOJJcf1mtID+YotwWVZzrL82i6upKt6iirr/p0+GMM6xFlSRUixZ1z6JyQuUsqqyJZP0hQ4YOhdNPzxWnaJaOLFlEejkqVKBCpTQOvnBcfTXst1/YduJbDsWcq9QRfiGsdLMEU5TDonJRfxtsYC2VDz9MFyq/jaq1tTih8ocXceV/4AH4yleSywm5QuUEyn8eUaHq3z/9fHWAChWoUCmNg1/hbbgh3HJLfmaKUlx/pfSjgjAyzvVfijsuaxtVVtefMWFn5yVL7P3GHdvens31lxRMEV3frx/ssAPcdhucemr+/m4U5pEjc4+JMnhw7rJaVA2CCpXSKKRZOK5CLtb1lxZMUaj95Oyz7XT06PiyQFjmM87I3acYi8p3/fn9k+bPt2JQCddfnFD55Ymy8sp2GidUvihHLSgVqgbAdZpToVIagSzh4sVaVK+/Hh9MIVJYqA491P4HhwyJL4t/vkMOCa2hLbe013z77TCYo5BF5fpRzZkTrp850wpEXPuWs6iyCtWRR+au90cMhsJC5QTHd/3FCVW0DFkSCPdyVKhcT3AVKqXRKVao3P4XXmgDB0pNxRRHocjB5maYMQPWXBN+9rPsFtWyZTB7drjeCVWWNip3jiiuDoluW7IkN4Tc3+4/I1f3OHekL1TOekoTKrWoGgDXuKpCpTQ6pQZTOErtOFzsdcCW0UXT3XdfNovKWR4usSvY//8qq4THbrIJTJ5s553rz38uaZGF0fakZctyLcUki+rJJ+GDD2Cjjeyyfx/ueH8Yk2hd1QDD0qtQqVApiqVU11/0+Kx9p7KSZFH5/cCyBFM4yyM6wKJvUY0eHUbVxbn+0lyZcdaWX68kCdXQodaK2n13u+xbT8OG5a+L3qf2o2oA+vWzYarbbFPtkihKdSnV9eeIWlRxI9iWQpxQRYMtsrj+nEXlZyGHXKHy0ynFBVOkEWfZ+GXyt8eljzrkEJsO6bvfDbfFCVX0pboBktIWmS65DhkwwHaoU5RGp9iov2pbVP72LK4/JxRxFpUb+8kXKpfMNqtLNM6i8l2FSRaVL5Jf+1ru8c6688XID57YYouGECq1qBRFyaXW2qjiXFtxFlWhXH+ugndJYh1DhuSKhZ8h3Xf9FcqtV4xFFSdUcTiLyrk5x4+Hn/409zxJg0bWESpUiqJYXIVXbD8qR6WEqtC1RbK5/lwb1Sef5O7bt28oRiK5ARBNTWH4+fjx6WWKs6ii1/HLE7dPFFdmJ1SXXRZ2DAb7HNSiUhSlYXBCVWvBFHH4ZXRtYWkBUb5FBWFGDMgXqpaWMPtDc7MdTwpg3Lj0MhXj+vOFNu15O2syOjaWfx4VKkVRGgZX4ZUqVNWyqF5+2U6zuv4gX6icILh7cG1Dzc3wzjt2fuzY9DIVcv3FiVNbW7aoPRUqRVEUQouh1BFjSx3SvhT8a7z/vp1GK/Goq83vGBvtiOtca040XNtQUxN89at2fr317LR//9zjjz7aTp1F5wtPW5uN5ksqf5Lbb8KEXGFNEqqWFhUqRVEaiFVXtZFuxx9f2vHOGsk67lJ3iLPaopX4woW5+/sVf1SonPvQiYxvUZ1+un0uzh04f35udosrrwwjBKNla22Fv/41OeluklC98ko4TAikW1QaTNE9ROQkEXlRRF4SkZODdZuKyGMi8qyITBORLVKOX0lE3hORSytZTkVRAvr0Kb0DadJwHJUgzmqLszb8eT+ZayGh8i0qkdz2r5aW8Nwbbhhuj7Mo29py+2JFy5bkrvTD4iG0+OIGmFSLqnREZEPgGGALYBNgLxFZG7gAONsYsylwVrCcxDnA1EqVUVGUMpI0HEclyGJRRff329SibVROqOLaqJJ46SV48MH8MvkCkyREhVx/UZwYuWu8/bYNs28QoaqkM3k94DFjzGIAEZkK7AcYwMVXDgbejztYRDYHVgbuAiZVsJyKopSDnnT9ZbGofKKC4wtVnz6hteXW+xZVEuuvH1+mLKHnhVx/UaJCtcYa4bIKVbd4EfiFiAwHlgB7ANOAk4G7ReRCrEW3dfRAEWkCLgIOB3ZKuoCIHAscCzC2UESOoiiW//wnHPuonLhKtxYtKiceQ4fath/f9dfSAnvsARdfHAZGuE6/xbT/xIXnJ5WpWIvKuf6iwqnBFN3DGDMdOB+YjLWKngM6geOBU4wxY4BTgKtiDj8BuNMYMzNmm3+NK40xk4wxk0b6g40pipLMHnvA5puX/7yuHadW2qh8nIhstZWdRtvhmprg5JPDgAtnYbmk1VmICz+vlOvPv6YGU3QPY8xVxpiJxpjtgI+BGcARwC3BLjdh27CibAV8R0TeBi4EviEiv6xkWRVF6SY9KVSFLKof/jB+/yuvhP33h912Sz+/E6qlS4svk0hh116xQnXrrbbbwOqr51+zASyqinZ4EJFRxpgPRWQssD9WgL4LbA9MAXbEilcOxpive+c4EphkjDkjup+iKDVELVlU554bv/9qq8HNNxfO7N4diyqLUBXbRrX55vCXv8SfR4Wq29wctFF1ACcaYz4RkWOA34pIC7CUoI1JRCYBxxljjq5wmRRFKRcjRsC8eXbeCVW1+lEVykzhUygEv7uuPzdfyPWXVuas11Sh6h7GmG1j1j0E5DnIjTHTgDyRMsb8GfhzBYqnKEp3mTs3rPRryaKKUmxap1pz/SWh2dMVRVGKoJbaqKIUm9bJpVsq1fVXSIiKdf2lXbMBLCoVKkVRyoMTqp7O9Re9fhylWlSVdv2VQ6gWLCjO8uuFqFApilIe0oSi3MQJT5oYFSueTqhc/6ViytSTrr/mZujogO226955ahwVKkVRykNPClWc8KS11ZRqURVDJaP+knDPoScCWKqICpWiKOWh2hZVR0dx+6dRLqHqCdcf9Ey7YBXpAWeyoigNQbmF6p574IMP4rfFhZenWVRxFti119qhTeLojlD51+sJ1x+oUCmKomSi3EK1yy7J25YsyV9XrOvvG99I3r+Uit8fKqSQRVVoe1ZUqBRFUYrAF6qjjgozkFeCxYvtdN117SCDUF7XX1rW9EL0dDAF9KzbtQqoUCmKUh78yvKquFzTZWTRIjv91rfgrbfg979PF6qeCJn30zL1ZIdf6L5lVuNoMIWiKOWhJ9/qnVANGAB7723nXWb0OIq1qBx77pl9XzfY4rHHlj97ehLlih6scdSiUhSlPHTHXVYsTqj697eZ0JcvT7cqSrGoOjuLu6eBA61V19wMd9xh1/VEZopynKfGUaFSFKX34UbXXXttOy3k+ipFREuxwpwgFhKQfv3sNmeFlYrrkKxCpSiKUmOcfjrsvDN8/vPZ9i+ULb3cFIrq698fnngCPve57l1n+fL069QJKlSKovQ+mpuzi1Q1yOKS22ST7l+nQYRKgykURVHKTbmCJQrhIh3r3PWnQqUoilJuytWhtxDOoqpzoVLXn6Io3eP662HWrGqXIp677oLJk3v+uj0VjdcgFpUKlaIo3ePQQ6tdgmS+/GX76Wl62qKq8zYqFSpFUeqHddeFiROrXYqet6hUqBRFUXoJ06dXuwSWnkpt1CBtVBpMoSiKUm6am20n41JTN2XFjey7wQaVvU6VUaFSFEUpN83NPWPlHHMMvPdebbg7K4gKlaIoSrnpKaESSR78sY5QoVIURSk3LS11H+DQk6hQKYqilJuWlroPcOhJNOpPURSl3Bx1FGy9dbVLUTeoUCmKopSbbbaxH6UsqOtPURRFqWlUqBRFUZSaRoVKURRFqWlUqBRFUZSaRoVKURRFqWlUqBRFUZSaRoVKURRFqWlUqBRFUZSaRowx1S5DWRCRucA7JR4+AphXxuL0BvSeG4NGvGdozPsu9Z7XMMaMLHdhykndCFV3EJFpxphJ1S5HT6L33Bg04j1DY953Pd+zuv4URVGUmkaFSlEURalpVKgsV1a7AFVA77kxaMR7hsa877q9Z22jUhRFUWoatagURVGUmkaFSlEURalpGl6oRGQ3EXlVRF4XkTOqXZ5yISJXi8iHIvKit26YiEwWkRnBdGiwXkTkkuAZPC8iE6tX8tIRkTEi8oCITBeRl0TkpGB93d63iPQVkSdE5Lngns8O1q8pIo8H9/x3EWkL1vcJll8Pto+rZvm7g4g0i8gzIvLvYLmu71lE3haRF0TkWRGZFqyr29+2T0MLlYg0A5cBuwPrA4eIyPrVLVXZ+DOwW2TdGcB9xpi1gfuCZbD3v3bwORb4Qw+Vsdx0At83xqwHbAmcGHyf9Xzfy4AdjTGbAJsCu4nIlsD5wMXBPX8CfCvY/1vAJ8aYCcDFwX69lZOA6d5yI9zzl4wxm3r9per5tx1ijGnYD7AVcLe3/EPgh9UuVxnvbxzworf8KjA6mB8NvBrMXwEcErdfb/4A/wJ2aZT7BvoDTwNfwGYoaAnWf/Y7B+4GtgrmW4L9pNplL+FeV8dWzDsC/wakAe75bWBEZF1D/LYb2qICVgNmesuzgnX1ysrGmNkAwXRUsL7unkPg3tkMeJw6v+/ABfYs8CEwGXgD+NQY0xns4t/XZ/ccbJ8PDO/ZEpeF3wCnAyuC5eHU/z0b4B4ReUpEjg3W1fVv29FS7QJUGYlZ14jx+nX1HERkIHAzcLIxZoFI3O3ZXWPW9br7NsZ0AZuKyBDgVmC9uN2Caa+/ZxHZC/jQGPOUiOzgVsfsWjf3HLCNMeZ9ERkFTBaRV1L2rZd7Bhq8jQr7ljHGW14deL9KZekJPhCR0QDB9MNgfd08BxFpxYrU9caYW4LVdX/fAMaYT4Ep2Pa5ISLiXkT9+/rsnoPtg4GPe7ak3WYbYB8ReRu4Eev++w31fc8YY94Pph9iX0i2oEF+240uVE8CawfRQm3AwcDtVS5TJbkdOCKYPwLbhuPWfyOIFNoSmO/cCb0JsabTVcB0Y8yvvU11e98iMjKwpBCRfsDO2ACDB4ADg92i9+yexYHA/SZoxOgtGGN+aIxZ3RgzDvufvd8Y83Xq+J5FZICIDHLzwK7Ai9TxbzuHajeSVfsD7AG8hvXr/7ja5Snjfd0AzAY6sG9X38L65e8DZgTTYcG+go1+fAN4AZhU7fKXeM9fxLo3ngeeDT571PN9AxsDzwT3/CJwVrB+LeAJ4HXgJqBPsL5vsPx6sH2tat9DN+9/B+Df9X7Pwb09F3xecnVVPf+2/Y+mUFIURVFqmkZ3/SmKoig1jgqVoiiKUtOoUCmKoig1jQqVoiiKUtOoUCmKoig1jQqVohRARLqCjNXuU7Ys+yIyTrwM94qi5NPoKZQUJQtLjDGbVrsQitKoqEWlKCUSjA90fjAe1BMiMiFYv4aI3BeMA3SfiIwN1q8sIrcGY0c9JyJbB6dqFpE/BuNJ3RNkmEBEviciLwfnubFKt6koVUeFSlEK0y/i+vuat22BMWYL4FJsvjmC+euMMRsD1wOXBOsvAaYaO3bURGyGAbBjBl1mjNkA+BQ4IFh/BrBZcJ7jKnVzilLraGYKRSmAiLQbYwbGrH8bO2jhm0Ey3DnGmOEiMg879k9HsH62MWaEiMwFVjfGLPPOMQ6YbOzAd4jI/wGtxpifi8hdQDtwG3CbMaa9wreqKDWJWlSK0j1MwnzSPnEs8+a7CNuO98Tma9sceMrLDK4oDYUKlaJ0j69500eD+UewWb0Bvg48FMzfBxwPnw12uFLSSUWkCRhjjHkAO0DgECDPqlOURkDf0BSlMP2CEXQddxljXIh6HxF5HPvSd0iw7nvA1SLyA2Au8M1g/UnAlSLyLazldDw2w30czcBfRWQwNhP2xcaON6UoDYe2USlKiQRtVJOMMfOqXRZFqWfU9acoiqLUNGpRKYqiKDWNWlSKoihKTaNCpSiKotQ0KlSKoihKTaNCpSiKotQ0KlSKoijK/9+gBgC7AW75x9M7yAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# gradient descent\n",
    "detailed_logger = False\n",
    "main_logger = True\n",
    "main_logger_output_epochs = 100\n",
    "L2 = False\n",
    "Dropout = False\n",
    "momentum = False\n",
    "adam = True\n",
    "hidden_layer_relu = True\n",
    "hidden_layer_tanh = False\n",
    "hidden_layer_sigmoid = False\n",
    "\n",
    "# hyber-parameters\n",
    "alpha = .01;\n",
    "epsilon = .85\n",
    "keep_prob = .9\n",
    "number_of_epochs = 500\n",
    "batch_size = 50\n",
    "momentum_coef = .9\n",
    "RMSProp_coef = .9\n",
    "epsilon = 1e-20\n",
    "t = 0\n",
    "\n",
    "# copy initalization\n",
    "W = Weights.copy()\n",
    "B = Bias.copy()\n",
    "\n",
    "# data arrays\n",
    "cost_array = []\n",
    "accuracy_array = []\n",
    "interation_array = []\n",
    "\n",
    "# rename\n",
    "X_train = np.float64(training_images).copy()\n",
    "Y_train = np.float64(training_labels).copy()\n",
    "\n",
    "X_test = np.float64(testing_images).copy()\n",
    "Y_test = np.float64(testing_labels).copy()\n",
    "\n",
    "#m = size\n",
    "m = number_of_training_images\n",
    "\n",
    "def model(W, B, A):\n",
    "    return np.dot(W, A) + B\n",
    "\n",
    "def activation_relu(Z):\n",
    "    Z = np.where(~np.isnan(Z), Z, 0)\n",
    "    Z = np.where(~np.isinf(Z), Z, 0)\n",
    "    return np.where(Z > 0, Z, 0)\n",
    "\n",
    "def activation_tanh(Z):\n",
    "    return np.tanh(Z)\n",
    "\n",
    "def activation_sigmoid(Z):\n",
    "     return 1/(1 + np.exp(-Z))\n",
    "\n",
    "def loss(A, Y):\n",
    "    epsilon = 1e-20\n",
    "    return np.where((Y == 1), np.multiply(-Y, np.log(A + epsilon)), -np.multiply((1 - Y), np.log(1 - A + epsilon)))\n",
    "    #return np.multiply(-Y, np.log(A)) - np.multiply((1 - Y), np.log(1 - A)) \n",
    "    \n",
    "def cost(L):\n",
    "    return np.multiply(1/L.shape[1], np.sum(L))\n",
    "\n",
    "def cost_L2(L, W, epsilon):\n",
    "    L2 = np.multiply(epsilon/(2*W.shape[1]), np.multiply(W[len(W)-3], W[len(W)-3]).sum() + np.multiply(W[len(W)-2], W[len(W)-2]).sum() + np.multiply(W[len(W)-1], W[len(W)-1]).sum())\n",
    "    J = cost(L)\n",
    "    return L2 + J\n",
    "\n",
    "def prediction(A):\n",
    "    return np.where(A >= 0.5, 1, 0)\n",
    "    \n",
    "def accuracy(prediction, Y):\n",
    "    return 100 - np.multiply(100/Y.shape[0], np.sum(np.absolute(Y - prediction)))  \n",
    "    \n",
    "def forward_propagation_return_layers(W, B, A, A_layers, Z_layers, layer, D, keep_prob):\n",
    "    if(layer < len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        Z_layers.append(Z)\n",
    "        if(hidden_layer_relu == True):\n",
    "            A = activation_relu(Z)\n",
    "        elif(hidden_layer_tanh == True):\n",
    "            A = activation_tanh(Z)\n",
    "        elif(hidden_layer_sigmoid == True): \n",
    "            A = activation_sigmoid(Z)\n",
    "        if(Dropout == True):\n",
    "            _D = np.float64(np.where(np.random.uniform(0, 1, A.shape) < keep_prob, 1, 0))\n",
    "            D.append(_D)\n",
    "            A = np.multiply(A, _D)\n",
    "        A_layers.append(A)\n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Training Data: ' + str(layer))\n",
    "        A_layers, Z_layers, D = forward_propagation_return_layers(W, B, A, A_layers, Z_layers, layer, D, keep_prob)\n",
    "    elif(layer == len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        Z_layers.append(Z)\n",
    "        A = activation_sigmoid(Z)\n",
    "        if(Dropout == True):\n",
    "            _D = np.float64(np.where(np.random.uniform(0, 1, A.shape) < keep_prob, 1, 0))\n",
    "            D.append(_D)\n",
    "            A = np.multiply(A, _D)\n",
    "        A_layers.append(A)\n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Training Data: ' + str(layer))\n",
    "            print('Forward Propagation Training Data Complete')\n",
    "    return A_layers, Z_layers, D\n",
    "\n",
    "def forward_propagation(W, B, A, layer):\n",
    "    if(layer < len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        if(hidden_layer_relu == True):\n",
    "            A = activation_relu(Z)\n",
    "        elif(hidden_layer_tanh == True):\n",
    "            A = activation_tanh(Z)\n",
    "        elif(hidden_layer_sigmoid == True): \n",
    "            A = activation_sigmoid(Z)\n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Testing Data: ' + str(layer))\n",
    "        A = forward_propagation(W, B, A, layer)\n",
    "    elif(layer == len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        A = activation_sigmoid(Z)  \n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Testing Data: ' + str(layer))\n",
    "            print('Forward Propagation Testing Data Complete')\n",
    "    return A\n",
    "\n",
    "def dZ(dZ, W, Z):\n",
    "    Z = np.where(~np.isnan(Z), Z, 0)\n",
    "    W = np.where(~np.isnan(W), W, 0)\n",
    "    dZ = np.where(~np.isnan(dZ), dZ, 0)\n",
    "    Z = np.where(~np.isinf(Z), Z, 0)\n",
    "    W = np.where(~np.isinf(W), W, 0)\n",
    "    dZ = np.where(~np.isinf(dZ), dZ, 0)\n",
    "    if(hidden_layer_relu == True):\n",
    "        return np.multiply(np.dot(np.transpose(W), dZ), np.where(Z > 0, 1, 0))\n",
    "    elif(hidden_layer_tanh == True):\n",
    "        A = activation_tanh(Z)\n",
    "        return np.multiply(np.dot(np.transpose(W), dZ), 1- np.multiply(A, A))\n",
    "    elif(hidden_layer_sigmoid == True): \n",
    "        A = activation_sigmoid(Z)\n",
    "        return np.multiply(np.dot(np.transpose(W), dZ), np.multiply(A, (1-A)))\n",
    "\n",
    "def dW(dZ, A):\n",
    "    return np.multiply(1/dZ.shape[1], np.dot(dZ, np.transpose(A)))\n",
    "\n",
    "def dW_L2(dZ, A, W, epsilon):\n",
    "    return np.multiply(epsilon/Z.shape[1], W) + dW(dZ, A)\n",
    "\n",
    "def dB(dZ):\n",
    "    return np.multiply(1/dZ.shape[1], np.sum(dZ))\n",
    "\n",
    "def backward_propagation(W, B, Y, A_layers, Z_layers, _dZ, alpha, epsilon, layer, D, V_dW, V_dB, R_dW, R_dB, t):\n",
    "    if(layer >= 0):\n",
    "        if(layer == len(W) - 1):\n",
    "            _dZ = A_layers[layer+1] - Y\n",
    "        elif(layer >= 0):\n",
    "            _dZ = dZ(_dZ, W[layer+1], Z_layers[layer])\n",
    "        if(Dropout == True):\n",
    "            _dZ = np.multiply(_dZ, D[layer])\n",
    "        if(L2 == True):\n",
    "            _dW = dW_L2(_dZ, A_layers[layer], W[layer], epsilon)\n",
    "        else:\n",
    "            _dW = dW(_dZ, A_layers[layer])\n",
    "        _dB = dB(_dZ)\n",
    "        if(adam == True):\n",
    "            epsilon = 1e-6\n",
    "\n",
    "            # ADAM - RMSProp + Momentum\n",
    "            V_dW[layer] = np.multiply(momentum_coef, V_dW[layer]) + np.multiply(1-momentum_coef, _dW)\n",
    "            V_dB[layer] = np.multiply(momentum_coef, V_dB[layer]) + np.multiply(1-momentum_coef, _dB)\n",
    "            R_dW[layer] = np.multiply(RMSProp_coef, R_dW[layer]) + np.multiply(1-RMSProp_coef, np.multiply(_dW, _dW))\n",
    "            R_dB[layer] = np.multiply(RMSProp_coef, R_dB[layer]) + np.multiply(1-RMSProp_coef, np.multiply(_dB, _dB))\n",
    "            \n",
    "            # index decay in bias correction\n",
    "            t = t + 1\n",
    "            \n",
    "            # correct bias for initial rounds\n",
    "            V_dW[layer] = np.multiply(V_dW[layer], 1/(1-np.power(momentum_coef, t)))\n",
    "            V_dB[layer] = np.multiply(V_dB[layer], 1/(1-np.power(momentum_coef, t)))\n",
    "            R_dW[layer] = np.multiply(R_dW[layer], 1/(1-np.power(RMSProp_coef, t)))\n",
    "            R_dB[layer] = np.multiply(R_dB[layer], 1/(1-np.power(RMSProp_coef, t)))\n",
    "            \n",
    "            val1 = 1/(np.sqrt(R_dW[layer])+ epsilon)\n",
    "            val2 = 1/(np.sqrt(R_dB[layer])+ epsilon)\n",
    "                      \n",
    "            W[layer] = W[layer] - np.multiply(alpha, np.multiply(V_dW[layer], val1 ))\n",
    "            B[layer] = B[layer] - np.multiply(alpha, np.multiply(V_dB[layer], val2 ))\n",
    "        elif(momentum == True):\n",
    "            V_dW[layer] = np.multiply(momentum_coef, V_dW[layer]) + np.multiply(alpha, _dW)\n",
    "            V_dB[layer] = np.multiply(momentum_coef, V_dB[layer]) + np.multiply(alpha, _dB)\n",
    "            W[layer] = W[layer] - V_dW[layer]\n",
    "            B[layer] = B[layer] - V_dB[layer]           \n",
    "        else:\n",
    "            W[layer] = W[layer] - np.multiply(alpha, _dW)\n",
    "            B[layer] = B[layer] - np.multiply(alpha, _dB)\n",
    "        if(detailed_logger == True):\n",
    "            print('Backward Layer: ' + str(layer))\n",
    "        layer = layer - 1\n",
    "        W, B, t = backward_propagation(W, B, Y, A_layers, Z_layers, _dZ, alpha, epsilon, layer, D, V_dW, V_dB, R_dW, R_dB, t)\n",
    "        if(detailed_logger == True):\n",
    "            print('Backward Propagation Complete')\n",
    "    return W, B, t\n",
    "    \n",
    "\n",
    "def shuffle(X, Y, number_of_training_images):\n",
    "    random_array = np.random.permutation(np.arange(number_of_training_images))\n",
    "    return X[:, random_array], Y[random_array]\n",
    "    \n",
    "start_time = time.time()   \n",
    "# main loop\n",
    "for epoch in range(1, number_of_epochs):\n",
    "    \n",
    "    # logger\n",
    "    if(main_logger == True and epoch % main_logger_output_epochs == 0):\n",
    "        print('Main Loop Epoch: ' + str(epoch))\n",
    "    \n",
    "    # saftey check\n",
    "    if(adam == True and momentum == True):\n",
    "        print(\"ERROR! Please Select Either Adam OR Momentum OR Neither, Not Both.\")\n",
    "        break\n",
    "\n",
    "    # saftey check\n",
    "    if(hidden_layer_relu + hidden_layer_tanh + hidden_layer_sigmoid != 1):\n",
    "        print(\"ERROR! Please Select Only 1 Hidden Layer Activation Function\")\n",
    "        break\n",
    "    \n",
    "    # shuffle data\n",
    "    X, Y = shuffle(X_train.copy(), Y_train.copy(), number_of_training_images)\n",
    "    number_of_batches = int(np.floor(number_of_training_images/batch_size))\n",
    "    split_index = number_of_batches*batch_size\n",
    "\n",
    "    # parse into minibatches\n",
    "    X_minibatches = np.split(X[:, 0:split_index], number_of_batches, axis=1)\n",
    "    if not(split_index == number_of_training_images):\n",
    "        X_left_over_portion  = X[:, split_index:number_of_training_images]\n",
    "        X_minibatches.append(X_left_over_portion)\n",
    "    \n",
    "    Y_minibatches = np.split(Y[0:split_index], number_of_batches, axis=0)\n",
    "    if not(split_index == number_of_training_images):\n",
    "        Y_left_over_portion  = Y[split_index:number_of_training_images]\n",
    "        Y_minibatches.append(Y_left_over_portion)\n",
    "    \n",
    "    number_of_minibatches = len(Y_minibatches)\n",
    "    \n",
    "    # logger\n",
    "    if(main_logger == True and epoch % main_logger_output_epochs == 0):\n",
    "        print('Number Of Minibatches: ' + str(number_of_minibatches))\n",
    "\n",
    "    for index in range(0, number_of_minibatches-1):\n",
    "        X_minibatch = X_minibatches[index]\n",
    "        Y_minibatch = Y_minibatches[index]\n",
    "\n",
    "        # forward propogation training data set\n",
    "        A_layers, Z_layers, D = forward_propagation_return_layers(W, B, X_minibatch, [X_minibatch], [], 0, [], keep_prob)\n",
    "        L = loss(A_layers[len(A_layers) - 1], Y_minibatch)\n",
    "        if(L2 == True):\n",
    "            C = cost_L2(L, W, epsilon) \n",
    "        else:\n",
    "            C = cost(L) \n",
    "\n",
    "        # backpropogation\n",
    "        W, B, t = backward_propagation(W, B, Y_minibatch, A_layers, Z_layers, 0, alpha, epsilon, len(W) - 1, D,  V_dW, V_dB, R_dW, R_dB, t)\n",
    "        \n",
    "    if(epoch % main_logger_output_epochs == 0):\n",
    "        print('Cost: ' + str(C))\n",
    "\n",
    "    # forward propogation test data set\n",
    "    A_test = forward_propagation(W, B, X_test, 0)\n",
    "\n",
    "    # accuracy\n",
    "    _prediction = prediction(A_test)    \n",
    "    _accuracy = accuracy(_prediction, Y_test)  \n",
    "\n",
    "    # storage for plotting\n",
    "    cost_array.append(C)\n",
    "    accuracy_array.append(_accuracy)\n",
    "    interation_array.append(epoch)\n",
    "\n",
    "\n",
    "end_time = time.time()\n",
    "run_time = end_time - start_time\n",
    "    \n",
    "print('')\n",
    "print('Results:')\n",
    "print('')\n",
    "    \n",
    "print('')\n",
    "print('Run Time: ' + str(run_time) + ' seconds')\n",
    "print('Cost: ' + str(C))    \n",
    "print('Accuracy: ' + str(_accuracy) + ' %')    \n",
    "print('')\n",
    "print('')\n",
    "\n",
    "\n",
    "pyplot.figure()\n",
    "pyplot.plot(interation_array, cost_array, 'red')\n",
    "pyplot.title('Learning Curve - ' + str(len(X[0])) + ' Training Data Set (Relu Hidden Layer)')\n",
    "pyplot.xlabel('Epochs')\n",
    "pyplot.ylabel('Cost')\n",
    "pyplot.show()\n",
    "\n",
    "# plot percent accuracy curve\n",
    "pyplot.figure()\n",
    "pyplot.plot(interation_array, accuracy_array, 'red')\n",
    "pyplot.title('Percent Accuracy Curve - ' + str(len(X_test[0])) + ' Test Data Set (Relu Hidden Layer)')\n",
    "pyplot.xlabel('Epochs')\n",
    "pyplot.ylabel('Percent Accuracy')\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As illustrated the after 500 epochs with minibatches of 50 the cost became approximately 0.0 (or too low for python to estimate) and the test data accuracy reached 98.36%. These results are very good. The test accuracy is high because minibatch stochastic gradient descent inately provides a form of regularization, combined with the ADAM (momentum and RMSProp) which prevents us from getting stuck on local minima, and from focusing too much on specific features based on large gradients. It is important to note that having a cost of zero usually means we have overfit the training data; however, in this senario that doesn't appear to be the case since we still have a very high test accuracy.\n",
    "\n",
    "We now wish to explore the impact of adjusting the momentum hyper-parameter size for Adam. We will re-run the algorithm with smaller momentum hyper-paramter of .5 see what the results we achieve.\n",
    "\n",
    "First we reinitialize our weights and bias's."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature Size: 784\n",
      "Weights Shape: (20, 784)\n",
      "Bias Shape: (20, 1)\n",
      "Velocity Weights Shape: (20, 784)\n",
      "Velocity Bias Shape: (20, 1)\n",
      "RMSProp Weights Shape: (20, 784)\n",
      "RMSProp Bias Shape: (20, 1)\n"
     ]
    }
   ],
   "source": [
    "# initialize weights & bias\n",
    "np.random.seed(10)\n",
    "print('Feature Size: ' + str(size))\n",
    "\n",
    "lower_bound = -.1\n",
    "upper_bound = .1\n",
    "\n",
    "#mean = 0.015\n",
    "#std = 0.005\n",
    "\n",
    "# hyper-parameters: hidden layers\n",
    "hidden_layers = 2\n",
    "units_array = [20, 10]\n",
    "Weights = []\n",
    "Bias = []\n",
    "V_dW = []\n",
    "V_dB = []\n",
    "R_dW = []\n",
    "R_dB = []\n",
    "for i in range(0, hidden_layers):\n",
    "    if(i == 0):\n",
    "        _W = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], size]))\n",
    "        _B = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], 1]))\n",
    "        _V_dW = np.float64(np.zeros([units_array[i], size]))\n",
    "        _V_dB = np.float64(np.zeros([units_array[i], 1]))\n",
    "        _R_dW = np.float64(np.zeros([units_array[i], size]))\n",
    "        _R_dB = np.float64(np.zeros([units_array[i], 1]))\n",
    "        Weights.append(_W)\n",
    "        Bias.append(_B)\n",
    "        V_dW.append(_V_dW)\n",
    "        V_dB.append(_V_dB)\n",
    "        R_dW.append(_R_dW)\n",
    "        R_dB.append(_R_dB)\n",
    "    else:\n",
    "        _W = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], units_array[i-1]]))\n",
    "        _B = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], 1]))\n",
    "        _V_dW = np.float64(np.zeros([units_array[i], units_array[i-1]]))\n",
    "        _V_dB = np.float64(np.zeros([units_array[i], 1]))\n",
    "        _R_dW = np.float64(np.zeros([units_array[i], units_array[i-1]]))\n",
    "        _R_dB = np.float64(np.zeros([units_array[i], 1]))\n",
    "        Weights.append(_W)\n",
    "        Bias.append(_B)\n",
    "        V_dW.append(_V_dW)\n",
    "        V_dB.append(_V_dB)\n",
    "        R_dW.append(_R_dW)\n",
    "        R_dB.append(_R_dB)\n",
    "    \n",
    "# output layer\n",
    "_W = np.float64(np.random.uniform(lower_bound, upper_bound, [1, units_array[i]]))\n",
    "_b = np.float64(np.random.uniform(lower_bound, upper_bound)) # b will be added in a broadcasting manner\n",
    "_V_dW = np.float64(np.zeros([1, units_array[i]]))\n",
    "_V_dB = np.float64(np.zeros(1))\n",
    "_R_dW = np.float64(np.zeros([1, units_array[i]]))\n",
    "_R_dB = np.float64(np.zeros(1))\n",
    "Weights.append(_W)\n",
    "Bias.append(_b)\n",
    "V_dW.append(_V_dW)\n",
    "V_dB.append(_V_dB)\n",
    "R_dW.append(_R_dW)\n",
    "R_dB.append(_R_dB)\n",
    "\n",
    "Weights = np.array(Weights)\n",
    "Bias = np.array(Bias)\n",
    "V_dW = np.array(V_dW)\n",
    "V_dB = np.array(V_dB)\n",
    "R_dW = np.array(R_dW)\n",
    "R_dB = np.array(R_dB)\n",
    "\n",
    "\n",
    "for index in range(0, len(Weights) - 1):\n",
    "    Weights[index] = np.where(Weights[index] != 0, Weights[index], np.random.uniform(lower_bound, upper_bound))\n",
    "\n",
    "#print(train_X.shape)\n",
    "#print(np.ravel(train_Y).shape)\n",
    "\n",
    "print('Weights Shape: ' + str(Weights[0].shape)) # matrix with a size of # of units X 784\n",
    "print('Bias Shape: ' + str(Bias[0].shape)) # vector with a size of the # of unit\n",
    "print('Velocity Weights Shape: ' + str(V_dW[0].shape)) # matrix with a size of # of units X 784\n",
    "print('Velocity Bias Shape: ' + str(V_dB[0].shape)) # vector with a size of the # of unit\n",
    "print('RMSProp Weights Shape: ' + str(R_dW[0].shape)) # matrix with a size of # of units X 784\n",
    "print('RMSProp Bias Shape: ' + str(R_dB[0].shape)) # vector with a size of the # of unit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we re-run our minibatch stochastic gradient descent algorithm with ADAM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Main Loop Epoch: 100\n",
      "Number Of Minibatches: 1000\n",
      "Cost: 0.0\n",
      "Main Loop Epoch: 200\n",
      "Number Of Minibatches: 1000\n",
      "Cost: 0.0\n",
      "Main Loop Epoch: 300\n",
      "Number Of Minibatches: 1000\n",
      "Cost: 0.0\n",
      "Main Loop Epoch: 400\n",
      "Number Of Minibatches: 1000\n",
      "Cost: 0.0\n",
      "\n",
      "Results:\n",
      "\n",
      "\n",
      "Run Time: 1049.1028203964233 seconds\n",
      "Cost: 0.0\n",
      "Accuracy: 98.94 %\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZMAAAEWCAYAAACjYXoKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xm8HGWd7/HPNwkQCFtIwpaEJEBGBVmNbG4ILigoqDjqRQUNcocZR9wBnVG86qjjAnjH64yODogM6rgQQVSQZWYYlBCUVcAQQIghJISEhC0L+d0/nqc8dfrU6XOq+/TpnJPv+/XqV1fX+jxV1f2tp6q6WxGBmZlZO8Z0uwBmZjbyOUzMzKxtDhMzM2ubw8TMzNrmMDEzs7Y5TMzMrG2jPkwk/VzSyd0uh3WfpJMl/Xyox7W+JO0n6cYhmtcrJD0wFPPqZ/5Nt7Wk6yWd0s+wvSWN2u9XSDpY0n8PZtyOhYmkByS9olPzH6yIeE1EXNiJeUvaXtJ5kh6U9ISke/PryZ1Y3lCSdI6k9bncxWPP0vADJd0s6an8fGBpmCR9QdKK/PhHSRqKaUvjnVQq19OSNpbL2kqdI+LCiHjNUI9bl6TFuU5PSFol6X8knVa1HvqZvu0PMElvkHSrpNWSHpX0K0l7DOGyPwN8sTRduc5LJX1L0oR26jAYkk6VdF1F/8WSjoTObut2lMvYLRHxW+BpSQOunxHdMpE0rovL3hK4GtgXOAbYHjgCWAEc0sL8ulGX70fEtqXHfbksWwLzgO8CE4ELgXm5P8BpwAnAAcD+wHHA/2532rKIuLgoF/AaYEm5rI3jd3NfaNFrcj1mkj50PwZ8YzgWLOk5wL8BZwA7ALOAfwY2DtH8pwEvBi5rGFTU+WDgUOCjQ7E864zSe+piKt6jfURERx7AA8Ar+hl2HHALsAq4Adi/NOwsYBGwBvg98IbSsFOA/wHOBR4jHf2cAlwPfAlYCdxP2mmLaa4DTi1N32zcWcB/5WX/Cvga8N1+6nAq8AiwbZN1EMDepdcXAJ/J3UcCi4EzgaXARcBdwHGl8ccBjwIH59eH5fW1CrgVOLKN7XNOk7q9CvgToFK/B4FjcvcNwGmlYXOB37Q7bZOyHgksrui/GPgIcDuwLvf7O+C+vA3vBF7fsM2uK63bIL1J7s37w1dbHHcscB7pQOI+4G+BaFKfxY3bDjic9GH+3Pz69aT3yJq8/v6+NO6SXJ4n8uOFwGzg2lyGR/P+tEM/y38rsKBJ+caQwm1Rntf3gIn9Lbti+ncDv2hWZ+ArwLzS6/G530Ok99X/A8bnYa8AHmjYFjNL034XOKfJ+/S6ZtugcRzSweE9wOPA+aTPnFNK2/rcvJ4XAe8tb2tgR1JQP5yX8X+AMaXl/GeeflXeV15VZz/J/ScBVwDL8754GTA1D3sbcGPD+GcCPxzses7bfinwb7n/DOBJYItm79Nhb5lIOhj4NumNOQn4F+CnkrbKoywCXkI6YvoU8F1Ju5VmcShpI+wMfLbU7x5gMvCPwLeanDJoNu6/A/Nzuc4B3tGkKq8gvWFaOuWS7QrsRNpYpwGXkHaGwquBRyPit5KmAj8jBehOwIeBH0ma0sbyXyfpMUl3Sjq91H9f4LbIe1J2W+5fDL+1NOzWhmGtTtuKt5JaLjvk138AXpRffxb4d0m7NJn+tcALgIOAtw9wara/cU8n7Q/7A3OAN9atRET8mvQGfknu9QTw9lyP1wFnSDouD3tpnqZoqd0EiLRv7AbsA+wJ/H0/i7sZ2E/SlyW9vOJ00weBY/NyppE+SL7aZNmN9iO9xypJmk76wL631PtLpIO5/UnBOBP4eH/z6BRJOwM/JB3UTiZ9oB9aGuV00gHTAaQzEH/ZMIvvAk8De5H2hWOBd5WGH0E6+JlECpVvtVDMMcA3gT1Inx3rSaEHcCnwHEmzS+O/nXRwAQOv52nAtnnefw0QEX8k7V/lefbVLGnaedBPywT4OvDphn73AC/rZz63AMfn7lOABxuGnwLcW3q9DenIZdf8+jp6t0wqx80rbwOwTcMRT39H71cBnx9gHQzUMllHPirI/fYmHYluk19fDHyidHRxUcP8fwmc3OL22QfYnXSkdQTpSOptedjfA99rGP9i8tEf8Cz5CDq/np3rqnambVLWI+m/ZfLOAep5B3Bs7q5qbRxWGvfHwIdbGPe/gLmlYcdQs2WS+y8Azuxnmn8CvljaT/qdfx7nROCmJsOPAP6D1PJ4hnSAV+x3Cym9H4HpwFrSh9hglv1vxX7eUOcn8v4dwJXkllOe7zPAjNL4LwEW5u52WyYbSC2B8mMjFS0TUqvq+tL0Y0jvjVNK2/rU0vDXFusDmEoKkq1Kw98BXFVazt2lYdvnukyus59UjDcHWF56/U3gU7n7wLyNtxjken4G2LJiGY8ARzQrRzeumcwAPpQvPK6StIq0s+4OIOmdkm4pDXs+6Qih8FDFPJcWHRHxVO7sc159gHF3Bx4r9etvWYUVpKPAdiyPiGdK5bmXdKrrdZK2IZ3q+Pc8eAbw5ob19uKqMjRcvK68SyUifh8RSyLi2Yi4gXRkc2Ie/ARpRy/bnvRBUDV8e+CJ/I5qZ9pW9NpGkk7JF5aLdfRceu8/jZaWup+i//2m2bi7N5Sj2X7TzFTS6VskHS7pOknLJT1O+iDqtx6SdpX0A0l/krSadODS7/gRcUNEvDkiJpNaG0cBZ+fBewCXldbh7aQPvZ0HWY+VwHYV/Y+LiO2Ao0mt0Z1y/12BrYDydru8xvIGcn1E7Fh+kE7XVem1LSNiI+lDvXI48MdS9wxSPR4p1eNrQLll3LgPQfN9rg9JEyT9q9KNP6uBa+i9rS8ETsrdbyddG13P4NbzIxGxrmKx25FCuF/dCJOHgM82bNxtIuISSTNIqfpeYFLe6HeQjngLrX7oDORhYKf8IV6Y3mT8XwGvHuCOlKdIrZ/Crg3Dq+pSnOo6Hvh9DhhI6+2ihvU2ISI+3ziDKF28jsHfpVK0LCBda9i/4VTh/rl/MfyA0rADGoa1Om0r/rwOle5G+zrpVESx/9xN7/2nEx4mnR4oNNtvKkk6jPShc33u9T3gR8D0iNgB+Fd66lG133yB1HrYLyK2J7XCB1XviJhPOj3y/NxrMfDKhn1tfEQs7WfZjW4D/qLJ8q4htSaKu70eIbXSn1Na3g653o3Tbsj1bPa+asfDlLafpDH03ra9hpOCt/AQ6T2/U6ke20fE/kNYPkg3LswCDsnb+qjywIi4Ppf9RaTPkuIU12DWc5/tmz+XIbVY+9XpMNlC0vjSYxwpLP5K0qFKJkg6VtJ2wARSZZYDSHoXPTt4R0U6L7gAOEfSlpIOJ52r7s9FpJ3nR5KeK2mMpEmSPibptXmcW4D/JWmspGOAlw2iKN8jnZM9nZ5WCaQ33+skvTrPb7ykI/OdM7VJOl7SxLwNDgHeR7oLC9KpwWeB90naStJ7c/9r8vN3gA9Kmippd+BDpCPhdqdt17b07D+SdCqpZdJpPwDeL2l3SRNJNwUMiqQdJBUt0Asi4q48aDtSS/mZHDRvLU22DAiVbuXO4z8JPJ6vSXy4yTJfpnTL7M759fNI+/pv8ij/DPyD8q3CknbOZexv2Y2uBF6onjv4qpwLvFbS8yPiWVJYnidpSt4np0l6VT/T3gqclN8Hx5Ja6EPlcuDA/P4YB3wAKF+XLLb1VEmTSKefAYiIh0gX2L+k9LWBMUq3Ur+0jfJsWfEZuh0ptFbmMnyiYrqLSAdWT0bEb3L56q7nwsuAX+XWTb86HSZXkM4hFo9zImIB8B7SOeCVpItwp0A69QJ8Gfg1KUX3I91JMVxOIt1Vs4J0MfP7pKOgPiJiLekc492k6yerSRfvJwPFl7XOIL1JV+V5XzpQASLiYVL9j8jLL/o/RGqtfIz0YfkQ6UOr1W34VtK6X0P6gP9C5O/j5GbuCcA7c9nfDZxQav7+C+kOkttJLcef5X5tTduuiLiNdKF4PukI8rn0bItO+jopRG8nXdz+GekIsJmfK31f5kHSxd4vkk5lFU4HPidpDWmb/6AYEBFrgM8BN+bTFXOAT5IuCD8O/JTUqunPSuANwB25DFfk+X85D/8K8Avg6rz8G0h3jPW37F4iYgnw3zQ5GMutnIvpuUngQ6RTRvNzHa6k/wu+78vlXwW8Odd3SETEI8BbSNtjBanlUd6Hvk76SsDtwE2ki/VlbycdFP+etJ7/g/ZaTr+k92fo35G2zw65fDcAVaeyv0M6EL+ooX+d9Vw4iXSA0ZRaP1U9+kn6PumC2Se7XRYbOSS9DjgvIvbqdlm6RdJ+wDcj4rBul2VzlE+/LwOeHxH3tzGfg4D/GxEDtv4cJiWSXki6AHo/6VTTpcDhEfG7rhbMNmn5jfsSUgt1N+AnwH9GRL+nmsw6SdJHSXfTDnQKa8iMtG8Nd9qupFs+J5EuQp7uILFBEOk7LT8kXbe4nPQdKbNhJ2kx6bsnxw/rct0yMTOzdo3o3+YyM7NNw4g4zTV58uSYOXNmt4thZjai3HzzzY9GRDs/uTRoIyJMZs6cyYIFC7pdDDOzEUXSHwcea2j4NJeZmbXNYWJmZm1zmJiZWdscJmZm1jaHiZmZtc1hYmZmbXOYmJlZ2xwmZmZl998PV17Z7VKMOCPiS4tmZsNm771h40bw7xbW4paJmVnZxo3dLsGI5DAxM7O2OUzMzKxtDhMzM2ubw8TMzNrmMDEzs7Y5TMzMrG0OEzMza5vDxMzM2uYwMTOztjlMzMysbR0PE0ljJf1O0uX59SxJN0paKOn7krbsdBnMzKyzhqNlcgZwV+n1F4BzI2I2sBKYOwxlMDOzDupomEiaBhwL/Gt+LeAo4Id5lAuBEzpZBjMz67xOt0zOAz4KFD/DOQlYFREb8uvFwNSqCSWdJmmBpAXLly/vcDHNzKwdHQsTSccByyLi5nLvilEr/zQgIr4REXMiYs6UKVM6UkYzMxsanfxzrBcBr5f0WmA8sD2ppbKjpHG5dTINWNLBMpiZ2TDoWMskIs6OiGkRMRN4K3BNRJwEXAucmEc7GZjXqTKYmdnw6Mb3TM4EPijpXtI1lG91oQxmZjaEhuU/4CPiOuC63H0fcMhwLNfMzIaHvwFvZmZtc5iYmVnbHCZmZtY2h4mZWZWo/Aqc9cNhYmZWxWFSi8PEzKyKw6QWh4mZmbXNYWJmVsUtk1ocJmZmVRwmtThMzMyqOExqcZiYmVVxmNTiMDEzs7Y5TMzMqrhlUovDxMysisOkFoeJmVkVh0ktDhMzsyoOk1ocJmZm1jaHiZlZFbdManGYmJlVcZjU4jAxM6viMKnFYWJmVsVhUovDxMzM2uYwMTOr4pZJLQ4TM7MqDpNaHCZmZlUcJrU4TMzMqjhManGYmJlZ2xwmZmZV3DKpxWFiZlbFYVKLw8TMrIrDpBaHiZlZFYdJLQ4TMzNrm8PEzKyKWya1OEzMzKo4TGpxmJiZVXGY1OIwMTOztjlMzMyquGVSS8fCRNJ4SfMl3SrpTkmfyv1nSbpR0kJJ35e0ZafKYGbWModJLZ1smawFjoqIA4ADgWMkHQZ8ATg3ImYDK4G5HSyDmVlrHCa1dCxMInkiv9wiPwI4Cvhh7n8hcEKnymBm1jKHSS0dvWYiaaykW4BlwFXAImBVRGzIoywGpvYz7WmSFkhasHz58k4W08zM2tTRMImIZyPiQGAacAjwvKrR+pn2GxExJyLmTJkypZPFNDPryy2TWoblbq6IWAVcBxwG7ChpXB40DVgyHGUwM6vFYVJLJ+/mmiJpx9y9NfAK4C7gWuDEPNrJwLxOlcHMrGUOk1rGDTxKy3YDLpQ0lhRaP4iIyyX9HviepM8AvwO+1cEymJm1xmFSS8fCJCJuAw6q6H8f6fqJmZmNEv4GvJlZFbdManGYmJlVcZjU4jAxM6viMKnFYWJmVsVhUovDxMzM2uYwMTOr4pZJLQ4TM7MqDpNaHCZmZlUcJrU4TMzMqjhManGYmJlZ2xwmZmZV3DKpxWFiZlbFYVKLw8TMrIrDpBaHiZmZtc1hYmZWxS2TWhwmZmZVHCa1OEzMzKo4TGpxmJiZVXGY1OIwMTOztjlMzMyquGVSi8PEzKyKw6QWh4mZWRWHSS0OEzOzKg6TWhwmZmbWtkGFiaSLBtPPzGzUcMuklsG2TPYtv5A0FnjB0BfHzGwT4TCppWmYSDpb0hpgf0mr82MNsAyYNywlNDPrBodJLU3DJCI+FxHbAV+MiO3zY7uImBQRZw9TGc3Mhp/DpJbBnua6XNIEAElvl/QVSTM6WC4zMxtBBhsmXweeknQA8FHgj8B3OlYqM7Nuc8uklsGGyYaICOB44PyIOB/YrnPFMjPrModJLeMGOd4aSWcD7wBeku/m2qJzxTIz6zKHSS2DbZm8BVgLvDsilgJTgS92rFRmZt3mMKllUGGSA+RiYAdJxwHPRISvmZiZGTD4b8D/JTAfeDPwl8CNkk7sZMHMzLrKLZNaBnvN5OPACyNiGYCkKcCvgB92qmBmZl3lMKllsNdMxhRBkq0YaFpJ0yVdK+kuSXdKOiP330nSVZIW5ueJLZbdzKxzHCa1DDZMfiHpl5JOkXQK8DPgigGm2QB8KCKeBxwG/I2kfYCzgKsjYjZwdX5tZmYjWNPTXJL2BnaJiI9IeiPwYkDAr0kX5PsVEQ8DD+fuNZLuIt0FdjxwZB7tQuA64MzWq2Bm1gFumdQyUMvkPGANQET8OCI+GBEfILVKzhvsQiTNBA4CbiSFUxEyDwM71y+2mVmHOUxqGShMZkbEbY09I2IBMHMwC5C0LfAj4P0RsXqwBZN0mqQFkhYsX758sJOZmQ0Nh0ktA4XJ+CbDth5o5pK2IAXJxRHx49z7EUm75eG7kX7Ovo+I+EZEzImIOVOmTBloUWZmQ8thUstAYXKTpPc09pQ0F7i52YSSBHwLuCsivlIa9FPg5Nx9Mv5fFDOzEW+g75m8H/iJpJPoCY85wJbAGwaY9kWk3/K6XdItud/HgM8DP8iB9CDpi5BmZpsWt0xqaRomEfEIcISklwPPz71/FhHXDDTjiLiedOdXlaNrldLMbLg5TGoZ1DfgI+Ja4NoOl8XMbNPhMKllsF9aNDPbvDhManGYmJlZ2xwmZmZV3DKpxWFiZlbFYVKLw8TMrIrDpBaHiZlZFYdJLQ4TMzNrm8PEzKyKWya1OEzMzKo4TGpxmJiZVXGY1OIwMTOr4jCpxWFiZmZtc5iYmVVxy6QWh4mZWRWHSS0OEzOzKg6TWhwmZmbWNoeJmVkVt0xqcZiYmVVxmNTiMDEzq+IwqcVhYmZWxWFSi8PEzMza5jAxM6vilkktDhMzsyoOk1ocJmZmhXKAOExqcZiYmRUcJi1zmJiZWdscJmZmBbdMWuYwMTMrOExa5jAxMys4TFrmMDEzKzhMWuYwMTOztjlMzMwKbpm0zGFiZlZwmLTMYWJmVnCYtMxhYmZWcJi0zGFiZmZt61iYSPq2pGWS7ij120nSVZIW5ueJnVq+mVltbpm0rJMtkwuAYxr6nQVcHRGzgavzazOzTYPDpGUdC5OI+C/gsYbexwMX5u4LgRM6tXwzs9ocJi0b7msmu0TEwwD5eef+RpR0mqQFkhYsX7582ApoZpsxh0nLNtkL8BHxjYiYExFzpkyZ0u3imNnmwAHSsuEOk0ck7QaQn5cN8/LNzAbHwVLLcIfJT4GTc/fJwLxhXr6ZWf98mqtlnbw1+BLg18BzJC2WNBf4PPBKSQuBV+bXZmabBodJy8Z1asYR8bZ+Bh3dqWWambXFAdKyTfYCvJnZsHPLpGUOEzOzKg6TWhwmZmYFt0xa5jAxMys4TFrmMDEzKzhAWuYwMTMruGXSMoeJmVkVh0ktDhMzs4JbJi1zmJiZFRwmLXOYmJkVHCAtc5iYmRXcMmmZw8TMrIrDpBaHiZlZwS2TljlMzMwKDpOWOUzMzAoOkJY5TMzMCm6ZtMxhYmZWxWFSi8PEzKzglknLHCZmZgWHScscJmZmBQdIyxwmZmYFt0xa5jAxM6viMKll8w6TefPg8su7XQoz21S4ZdKycd0uQFedcEJ69k5jZuDPgjZs3i0TM7Myt0xa5jAxMys4TFrmMDEzKzhMWuYwMTOr4jCpxWFiZlZwgLRs8w0T7zRm1sinuVq2+YbJM8/0dD/7bHp+8MGeboAVK2D16tS9cmV6VNmwAR56qPnyli+HJ55ovbybkmeegYcfbj7Oxo3wxz8OT3mqPPDA8H4YPPRQ733HRiaHScs23zB5+ume7meegT/9CWbMgE98oqf/5Mkwa1bqPukkeOc7q+f10Y/CHnvAo4/2v7ydd4Y5c9ov96bgTW+C3Xdv/mb74hdh5ky4555hK9af3XBD2m4XXDA8y3vkkbT9zzpreJZnneMwaZnDpOgujrSvuKL3eI89lp4feAB+97vqec2bl56bhQl054O1E4p1VF6Hja68Mj0/+GDny9PozjvT8/XXD8/yli9Pz/41hdHFYVLL5hsmTz3Vu/vJJ5uPv2JFar2Up2u0alV1/3Xr6pdvJFixYuBxNmzofDm67fHHu10CGyoOkJZtPmESkU5nbdzYNzyefDKdqijGg96hEdHTQrn33v6XsWJFOm++fn3vaQe6vgBpuqee6t7OvH59/XP+gwmTYr0Np6EI77Vr074yGINZDzYytHqaKyLtM1XK12dHsc0nTM48E7beGsaOhQkT4KCDeobtsw+85S2p+9Zb4T3v6f0huHp1zxH2woU9/ffYI41bWLEC9twTttwSpPQYMyZdOyhccEHqv2ZN7/LNmZPK9aY3Da4+q1b1LOPUU3v6R6R+n/nM4OYDsGwZ7LADzJ7d8waaObP3fD/wAdhpp97THXQQvO99aXnr18N3vpO6586Fa65J45xxRloH5Tfmu96V1lO5vJ/+NHzqU33HXbeup55vfGPqt2RJen3ZZfCGN8DBB/cuV7Ht+vsw2HPPNH1xPazR/Pkwfjy87nU9/S66KE1TtEKOPhpe/vLey7v7bnj3u3vP67LL0nRLllQvq/DhD8OOO/bud+KJcOCBzacbLrNmwTve0fM6AsaNg3POgc9+NtVx9er03Hit6iMfSfvXK18JL31p6lc17gtfCK9/ffNybNiQpjv33L7Drr8+DbvnnrQvSzB9evV8yuMWLrkEDjigdx0H6+ST0z5TnN4tfOIT6XNnuK7fdVNEbPKPF7zgBdG2ww6LSLtHhNTT3d/jllt6um+7raf7H/4hzW/dup5+e+6Znj/84YHnO2NGer7ppp6yPf54z/DJkwdXn+uv7z3fwp/+1LffQK68smeapUsj1q/vO4/i9ZNPVtfrD3+I2Guv/uu9dGnfeW3YELFkSfNx7767bz3nzUvdRx9dXdczzkj9Tjyxb103bOg9vw0b+o5z/vl95zt7dnr9m9/0rkNExJe+VL0tIiKOOir1mzev+TYopl2zpm+/jRubT9tp5XVWWLq073a79NL0PGtW7+kbx4uImD+/77iD2W8XLkzjjB3bd9jcuWnYuef2Xt769X3HPfXUNOz883v67btv7+nOPLN5WcqmTk3TfPKTvfsfeWTqP3fu4Oc1hIAF0cHP5vKjKy0TScdIukfSvZKG5xaYcoviyCMHHr986qI8bdH9wAM9/YrTQ7/5zcDzLY5sy6fLiu7DD08X8fu7Bbnsvvuq+//hDwNP26ixfs1u6e3vNN9Ay60a/tBD1f3L/RqHR6RrV9D7NFR5exXdVaefGutWdYNAeX00zmPRot6nLZ55pvlpLik9F2UeSLF+o3RU3O3TaFW3vZfXUeEXvxjc/J5+Oq3HsvI+3+z0YrP9rFhnd93Vu3/5vdqo8QxB1fwGsm5dz6nsxvVSlLeV9+UIM+xhImks8DXgNcA+wNsk7dOxBa5cmd6M5Tfk8ccPPF35DVTsIOPH93SXd5rFi9PzYMKkuEhfFVDHHtt3WH8axym+w1LuP9g3Q2NZyq/XrOl9DeiWW/qfR7MPgWKe5YvVjcvqrzxlS5ZUb4Nyd3Haqep6TeP8Brv84oBh4cLeQX7ffX2XU3w3CXquzTXbpuWbFIrxyqfFBrM/dFJ5+cX2qypTcZdfeT+ougFj0aKe6Ytxq95PA5WlUfGB/vOfDzxNsX7LB0flso4f3/9yGt1/f3U9nnyyZznd3obDQDHYD5yhWqB0OHBORLw6vz4bICI+1980c+bMiQULFtRf2Ny58O1vw6RJKUwOOCBdE1m0CPbaq/m0kyf33Oq7007pA+OII+Cmm9L52FWrqs+Db7VV/xfiyiZOhN12S90rVqQbABYsSNdOpk+H7bZrPv2SJb3vHtt773StZvnynltVn/e8niPjZhYvTtd/7r471XXcuJ667b13uo5RHFmV10vZpEkpuPsLlClT0mPt2p6j0t13T0FVlLdxXEjrpXwgMGtW7y+TFqZNg+23T93335+OfrfYIm2rssbttvvufa9VLFoE++4Lv/1tz3zvvjvVbeLE9Lpo4cyYkepdLk+xLSB9iKxfn6aZNq163WzY0LN+d901bYOnn071aKxbN5TX2V57pX380UfTtbYqY8bAc5+bust1K+yxRzpIWbmyZ9w1a3oO4GbNStcZqpT3h30ajkEXLap+71Vt4/vuS63KCRPSNoR0/aQ4aNh66/TYddfqcpQ9+WTaHw4+GG67Df7iL1L/detSWB18cNqXGss7WJdd1nONsSZJN0fEsHzBrRthciJwTEScml+/Azg0It7bMN5pwGkAe+yxxwv+2Mq3qb/85fTdkPXrYZtt0kW7devSFwg/+UlYujQdvU+fnjb4/PlpB3vkkbSDTJqUpl29Ok1z0klw/vk9H5iTJ6c3/dq1aYd98EE49NB08frWW9N469al6S65JL35nn467aCNpz322w8+/nH44AcHvlhbKC7sL1/e++60XXdNb9TBhFrhTW9Kb4gitCdNSuuimO/WW8O226ZljR+f1tnGjanOU6akMo8dm5b92GMpKKUUVBMnpnVdmDAhzaP4UCjKC33HhfThI/X+0J4+Pa3DCRNS+DWeGpw5M9Wnav/eZZee0C/u4iuT4K//Gi69tGc7jR2b6lQcNe+wQxqvCPQ990zLWras97YYMwZMSU90AAAG0ElEQVSmTh34FxK23jodQJQ/oLfbrrpu3TBpUlpf5V9x2GWXnvoX223q1LROy0f5Rd3Wru29zhrHnTgxfZg3Hig02m23FGblFjP0XHB/8MH0fh0/Pr3/GvenxnELY8em/ebFL07v3/5a4VUmTUoX4c87r/cB1YQJ8N73wle+0re8g3XeeWldtWC0h8mbgVc3hMkhEfG3/U3TcsvEzGwzNpxh0o0L8IuB8v1604BBHoqbmdmmqBthchMwW9IsSVsCbwV+2oVymJnZEBk33AuMiA2S3gv8EhgLfDsi7hzucpiZ2dAZ9jABiIgrgCsGHNHMzEaEzefnVMzMrGMcJmZm1jaHiZmZtc1hYmZmbRv2Ly22QtJyoJU/FJ8MDPD3h6PS5lhv13nz4DrXMyMipgxlYfozIsKkVZIWDNe3Pzclm2O9XefNg+u86fJpLjMza5vDxMzM2jbaw+Qb3S5Al2yO9XadNw+u8yZqVF8zMTOz4THaWyZmZjYMHCZmZta2URsmko6RdI+keyWd1e3yDBVJ35a0TNIdpX47SbpK0sL8PDH3l6Sv5nVwm6SDu1fy1kmaLulaSXdJulPSGbn/qK23pPGS5ku6Ndf5U7n/LEk35jp/P/+NA5K2yq/vzcNndrP87ZA0VtLvJF2eX4/qOkt6QNLtkm6RtCD3G3H79qgME0ljga8BrwH2Ad4mqcU/YN7kXAAc09DvLODqiJgNXJ1fQ6r/7Pw4Dfj6MJVxqG0APhQRzwMOA/4mb8/RXO+1wFERcQBwIHCMpMOALwDn5jqvBObm8ecCKyNib+DcPN5IdQZwV+n15lDnl0fEgaXvk4y8fTsiRt0DOBz4Zen12cDZ3S7XENZvJnBH6fU9wG65ezfgntz9L8DbqsYbyQ9gHvDKzaXewDbAb4FDSd+EHpf7/3k/J/0/0OG5e1weT90uewt1nUb68DwKuBzQZlDnB4DJDf1G3L49KlsmwFTgodLrxbnfaLVLRDwMkJ93zv1H3XrIpzIOAm5klNc7n+65BVgGXAUsAlZFxIY8Srlef65zHv44MGl4SzwkzgM+CmzMrycx+uscwJWSbpZ0Wu434vbtrvw51jBQRb/N8R7oUbUeJG0L/Ah4f0Sslqqql0at6Dfi6h0RzwIHStoR+AnwvKrR8vOIr7Ok44BlEXGzpCOL3hWjjpo6Zy+KiCWSdgauknR3k3E32TqP1pbJYmB66fU0YEmXyjIcHpG0G0B+Xpb7j5r1IGkLUpBcHBE/zr1Hfb0BImIVcB3petGOkoqDwHK9/lznPHwH4LHhLWnbXgS8XtIDwPdIp7rOY3TXmYhYkp+XkQ4aDmEE7tujNUxuAmbnu0C2BN4K/LTLZeqknwIn5+6TSdcUiv7vzHeAHAY8XjSdRxKlJsi3gLsi4iulQaO23pKm5BYJkrYGXkG6KH0tcGIerbHOxbo4Ebgm8kn1kSIizo6IaRExk/SevSYiTmIU11nSBEnbFd3Aq4A7GIn7drcv2nTwotZrgT+QzjN/vNvlGcJ6XQI8DKwnHaXMJZ0nvhpYmJ93yuOKdFfbIuB2YE63y99inV9MasrfBtySH68dzfUG9gd+l+t8B/CJ3H9PYD5wL/AfwFa5//j8+t48fM9u16HN+h8JXD7a65zrdmt+3Fl8Vo3Efds/p2JmZm0brae5zMxsGDlMzMysbQ4TMzNrm8PEzMza5jAxM7O2OUxssyXp2fxLrcVjyH5dWtJMlX7Z2Wy0G60/p2I2GE9HxIHdLoTZaOCWiVmD/P8SX8j/JzJf0t65/wxJV+f/kbha0h65/y6SfpL/e+RWSUfkWY2V9M38fyRX5m+yI+l9kn6f5/O9LlXTbEg5TGxztnXDaa63lIatjohDgH8i/T4Uufs7EbE/cDHw1dz/q8B/RvrvkYNJ32SG9J8TX4uIfYFVwJty/7OAg/J8/qpTlTMbTv4GvG22JD0REdtW9H+A9MdU9+UfmFwaEZMkPUr674j1uf/DETFZ0nJgWkSsLc1jJnBVpD83QtKZwBYR8RlJvwCeAC4FLo2IJzpcVbOOc8vErFr0093fOFXWlrqfpeca5bGk31d6AXBz6RdxzUYsh4lZtbeUnn+du28g/ZotwEnA9bn7auB0+PMfWm3f30wljQGmR8S1pD+B2hHo0zoyG2l8RGSbs63zPxkWfhERxe3BW0m6kXTA9bbc733AtyV9BFgOvCv3PwP4hqS5pBbI6aRfdq4yFviupB1IvwB7bqT/KzEb0XzNxKxBvmYyJyIe7XZZzEYKn+YyM7O2uWViZmZtc8vEzMza5jAxM7O2OUzMzKxtDhMzM2ubw8TMzNr2/wHmDAzh7h739wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaEAAAEWCAYAAADPZygPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJztnXncHdP9x9/f7E/2yJNERBYkraCExNZUUGtTlC72nVjqh7SW2lpUF1QptdfeKkXU1haxpdUWTRBCEFr7FiQiErJ9f398z7hz73OXuc9zb+48937fr9e8ZubMmZnvmTlzPud7zpkZUVUcx3EcpxZ0qLUBjuM4TuPiIuQ4juPUDBchx3Ecp2a4CDmO4zg1w0XIcRzHqRkuQo7jOE7NcBFyHCc1iPEfEVmnQsd7V0S+Volj5Tl2VxFZKCKrFdh+hIg8UGT/x0Rk32rYVmvCfXxKREaViltShETkVRFZHC72eyJyrYj0rIyplSHYuG2CeGuIyAoRuXRl2FUrRGSwiFwtIu+IyCci8oKInCkiPWptWylEZHcR+ZeILBKRR/JsHyMiM8L2GSIyJrZNROQcEfkwTOeKiFRi31i8fcKzsDA8Fyti6wvbkO61RWRZiThni8jScE+j+3qhiAws4zxtKvhEZISI3CEiH4jIxyLyjIjsXcFzfxd4S1WfD/tEaV4oIvNF5FERGdda+5MiIt1EREVk9Zzws0XkKgBV/VxVe6rq29W2pxziNtYKtRdQLwDOKBU3qSe0s6r2BDYCNgZOK9coEelU7j5VYH9gHrCniHRdmSdeWekXkVWAfwNNwOaq2gvYDugLrNWK463s+/YR8Bvg7Dy2dAHuBP4A9AOuB+4M4QCHAbsCGwDrAzsBh7d13ziqemMoeHoC3wDejtZDWLW5PtzT/sD3gBHAdBEZsBLODXAT8CIwFGgGDgI+qODxjwB+nxN2fbi2A4DHgD9V8HxOhYmVGbcD3xSR/kV3UNWiE/AqsG1s/VfAPWG5D3A18A7wFvAzoGPYdiDwT0wNPwJ+FsInAbOBT4DngY1C+GrAFGAu8D/gmNg5zwBuAW4I+z0HjAvbfg+sABYDC4ETi6TlFeBI4D3guznb1gWmBlvfA04J4R2BU8K+nwAzsAdwBKBAp9gxHgEOLZR+TAQeAj7EHtwbgb6x/YeGGzc3xLkY6Br2/0os3sCQ3gF50vgz4FmgQ4FrUK7dvwTmA+vF4g8I5x8Y1ncCng7x/gWsXypfJch3hwKP5IRtH/KZxMJeB3YMy/8CDottOwR4rK37FrFxK+DNPOFDMcH7APgvcERs23jgKWAB8C7wyxD+frgvC8O0YZ7jng1clRPWGXueoudrAPC3kIc+CnYMDtt+DSwHPgvn+HUIvwx4M9j0BLBZgfQKsBRYu8g12QJ4POSFJ4Hxxc6ds2/3cPzmQmnGKsIK9IqF7QY8E875D2Cd2LZ3ga+F5ZuB02LbdgReLpCObuE8qxe6B7lxsOfyr+E6/ht7dh6I7ftNYE6w83xMUPeNbT8cE/iPgL8AQ3LOMwkrh+YBFxS5By3ySWzbT7Dy9RNgFvDN2LVfAIyKxV0dWEQooxJc5+OxsnlRLPwfwB7FnqOy+oREZCgwEXuIwGqTy4CRwIbYg35obJdNsYdwIPBzEfkeJij7A72BXYAPRaQDcDcwExgCbANMFpEdYsfaBctEfYG7sAIaVd0PK0x2VquNnlvA9i2wi3ozJmj7x7b1Ah4A7sXEcCTwYNj8Q2CvkO7ewMHYjUlCVvqxh/iX4RyjscLqjGBDR+Ae4DVMKIYAN6vq58HmeDPGXljmnpvnnNsCt6vqioQ2lrL7p5gw7hXbvjswTVXfF5GNgGuwB6g/cAVwV5U8zXWBZzTk7sAzITzaPjO2bWbOttbum5hwH/+KidpqWEF3iohsGaJcDPxCVXsDo4A7QvgEYLlmvKqnSICqLsWenS1CUAfgcmAYsEYIuyDEPQ74D1bh6BnWwQrMr2D3707gVhHpnOdcignMFaHZNLepakRIz6nAKliLyR0i0q/IueOMBhaoal7PKuSp/bACb2EI2wy4FPPI+mOV0jtq1PJyJSYgg7DK7sHRBhFZFSt3jsMqCnOBcbHtewKTgZ3D/k9hXnucb2Dl7EbAQSKyVStsfBH4KuZAnAPcLCLNqroIuI3scmYf4C+qOj/hdd4Da3WJez6zsdaFwhRTqPC8vord8PlYAXkp1tQzCPgcaIrF3Qt4OCwfCLyec6z7gGPznGPTPHFPBq4Ny2eQXaNYB1icY+O2JdJxFXBHWN4cq3ENjNn9VIH9XgS+lSd8BKU9itdL2LRrdN5g09z48XKuzxsE7waYDuxe4JhziNW8K2E3Jmz/ja3/E9g/LF8GnJXnmm1ZKm+VuDb5PKEfY8IcD7sROCMsLydWS8cKecXEv9X7FrFxK3I8IWBLYE5O2JnAZWH5CayQ7p8TZ21gWYlrkreGixVezxbYZzPgndh6Vu07T3zBKllfLrC9GWsNmY21QEwneG3A6cDvcuJPI9SEE5x7G+DVPGn+HCt/lmMe4/jY9muBU3P2eQ3YNCy31RP6OJw7mj4jjycUllcAI2LHOJ9QbmHNvY/EtnUMadk3rD8M7BPb3hkrowbFzjMutv0uYHI5+aRA3BeAHWJ59+XYtmeBXcq4znvnOf6vgUuL2ZDUE9pVVfuq6nBV/b6qLgaGhwv1TugwnI/VguOdpG/kHGco5k7mMhxYLTpOONYp4QZEvBtbXgR0S1rbEZEmrP38RgBV/TfmPUUdqoXsKrWtFFnpF5GBInKziLwlIguwmk5z7DyvqWqLzmlVfRz4FNhSRNbGPLW7CpzzQ2BwK+3NazfWhNgkIpuKyHBgDPDnsG04cFzOvRuKeQFZiMgpsU78y1th10LMG43TG2tayLe9N7BQ7Wloy77lMBwYkXM9fgisGrYfgPU5vSQij+d4+61lCFYDR0R6icg1IvJ6yGP3k8ljeRGRk0XkRRH5GGvq6VZoH1X9QFVPUNXRIU0vYZ4yWNr3zUn7OPLkhQLMA3rlCf+9qvbF8vUrWP6LGI55mvFzDsCuSSVYN5R9fYMNvykQb1VMwOPPzmux5dXi21R1OdY8HDEcuDyWhrlYK1Pc28wtA8vugxSRQ8Jgkug8I8nc678DHUVkc7FBO4Oxpt3IvlLXObfcALuf84vZ1JYh2m9gNZTm2E3qrarxJozcB/gN8neOvwH8L36zVbWXqk5MaEupgmI3rFC5VGzI5rvYxYua5ArZVWzbp2HePRa2ak6cXLt+GcLWV2uO2RfLuNF5hhUR1utD/P2A21T1swLxHgB2C02c+SjbbrWmvVswj3FvrE8wKrzfAH6ec++6q+pNuSdW1V9oprnpiAL2FeM5YH2RrFFr64fwaHvc9d8gZ1tr9y2HN4AX8uTl3QBUdbaq7oFV1i4Cbg+DI8oVO+CLTuCdsLZ3gJOwgmvjkMe2J5PHyD2PiGwHHI09I32xZrTFOfvkRVXfx2r7I8RGXr6B1cDjae+hqhfkO3ceZgO9RKSQAL6PNfv+MhbnDeAnefLf7XkO8SnF831beBdL39BY2LDY8jvxbeH5zC3AD8xJR5OqzqiUgSLyJeC3mFe2ShDVlwn3OlS4biBTztys1twb2VfqOue7v6PJbuZuQatFSFXfwWpZvxaR3iLSQUTWirV95+Mq4HgRGSvGyFCzfgJYICI/EpEmEekoIuuJyMYJzXkPWLPI9gOwfouvYLWoMVgH8RgR+QrWF7OqiEwWG/vfS0Q2jdl8loiMCjavLyL91fpj3sJqfh1F5GBKjz7rRWjaFJEhwAmxbU9gGfVsEekhNkR0fGz777GCYl8soxTifExwrw/XFhEZIiLni8j6rbQb4I9Ym+8+YTnid8ARwUuSYPs3xfrZyibY1A3oBHQI1yHqn3gEa5I5Jtyn/wvhD4X5DcAPQ3pXw9rfr6vAvuXwaEjH5GB7p5BnNgrh+4f8sxxr6lGsGed9rBY6rOCRY4hIZxFZF6sc9MIEjbC8CMtjzbQcyZr7rPTCmn3mAl2wPsBuRc57noisE+5TH2w02yxV/RSrKH1PRLYJ25vCclTYF31OQwvLI1j/WKE4z2A19qhP6UrgaBEZF/JfTxHZRUS659n9aWAnEekbnr+jC52nXEKl8G7gzJDu9bFnJeIuYGMR2Snk5xMwwY+4HDhNRL4MICL9ROQ7bTCpY8h/0dQF85xWYPe6g4gcgXlCcW7A+nz3IrucKec6E9LQAytzHywUByh/dFzOtj5kRtZ8jHWm7Rm2HQg8mmefI7A+g4XY6IyoPXk1bPjnu5hb/lh0XqxP6A+xY4wg1q8BfAtrXpsPHJ9zviGYW/uVPLb8FTgvLK8XLta8YMNJmmm7PY3MiJL/kBkN840QPh9r+5xGdt/KoznnWxcbXbcQeyCOI9angNWc7iAzeu6inP0fCPejYD9F7FpeE9LxCdbuezrQvTV2x477Mtbs0yUnfMdwXeZjQnorsdFL5Uzh/JozXRfbvmG4houx0VcbxrYJcG6w8aOwLJXYt4CtW1F4dNwtWKE7D+tDmxC23RLu7SdYm/vE2H7nYAXEfGBMnuOejQnGJ1it/iWsZjs4Jw89GvLYC8D3ifU1Edr9g13nYk3qv8dGRr2F9S990Y+Sx4Yrw/4Lg613kj2ianw4/zxMWO8CVst37gLH/w7w55w0544I3DLYu0pY3yXc14+Bt7G+n6awLd4n1ANrOlyAlVXHU9nRcatig5sKjY7bOaS/0Oi4QzDvewHWlHd5IVvI6d/KY2PuM/Ry2HZeuP5zQ35r0U8X7t+LeY6b6DrH4u8H/LHUMy8hstMOEJFrsPdSyn5Py3HaA6G59HGsaer5WtvTiIjIH4HnVfVnbTiGYIK1p6q+VDSui1D7QGz469NY7f1/tbXGcZx6RERGYq0Eo1X1rVLxK4F/O64dICJnYU2Xv3IBchynGojIuVgz5U9XlgCBe0KO4zhODXFPyHEcx6kZafioaCKam5t1xIgRtTbDcRyn3TBjxowPVHVlfdy2VbQbERoxYgTTp0+vtRmO4zjtBhF5rXSs2uLNcY7jOE7NcBFyHMdxaoaLkOM4jlMzXIQcx3GcmuEi5DiO49QMFyHHcRynZrgIOY7jODXDRQhgxQq49lpYuLB651iwoHrHdhzHaac0rgjdfTf8+9+2fM01cPDBcMkl5R/ngw/gweL/bOLMM6FPH3ip6BfN4c03odxv+S1fDuedB2uvDaefDh9/DPfeW/o4N90Ev/hFeedyHMepMI0rQrvsAl/9qi1PmWLzl16CrbaC2bNhyRK4/364/HJYtAiWhr/cLlmSWVaFDTeEbbeFtdaCt94yUVi+3I71afiT9hln2PyBB+CKK+CEE+B3v7OwxYttPmcODB0Kv/612XPiibDHHjBpEsydm7F7xQo46ig46yxb/8tf7Hgvvgg//akd/xvfgIsuoiCXXAJ77w2nnpp/+yuvwMMPm7guWVL8Ok6dCq++2jL8s88y1ym6VosWZW+Ps3x55lq0lkWLYMAAOO44+PvfYexY2GsveO892/7aa3DXXZnzH3ggXH212bZ4McyYYdc3Hy++CB9+mJ2ezz9Pbtvy5fDCC61KVlksXWrncpz2Qmv+flmLaezYsVoxli5VtWLE1seNs+XVVsuE507f/rbqO+/Y8p572n7/+192nDXXtPn662fCDj9ctWPHzH7x+H/8o80PPVT1t7+15ZEj85//7rtVn3xSdffds/ffeOP88b/6VdWXXlLdbTfVE09U/eEPVfffX3XRoux4zc2q//yn6rHHqj76qMXJPdbaa6tOnqz6ne+o/utfZus++6guXmzbu3VTnTJFdeFCuy7PPqvav7/ZsP/+quuumznXsmWqZ52l2q+f6ssvW/y//U21Z0+bTjhB9bnnVC+8UPW991Tnz8/ct9/+VvWFF1TvvVf19NNV//pXO++CBaqnnFL43p1wgt2HaH36dNWhQzPrV1+dWb7pJjvX00+bnfvtZ9cPVCdMsHu+dKnqEUdY2C67mI0/+IHqrbeqfvSR6gcfqO6xh+qwYaqDB6tut53q9ttb/NtuUz34YNWjj1Y9/3zVd9+1ezJ3rp13xQo71kknqZ52muobb6h+//uqRx5p23/3O9VJk1TnzVP97DPVGTNUP//ctj3+uOqWW6oOH656//2qu+6qeu212Xn/rbdU11lH9e9/V5061ex/4omWz8i771o6G4mZM1Vff711+7Z2v2KsWNHmQwDTNQXld7Gp5gYknSoiQitWWAH56quZQkc1U0h26JBdeHXunL1+3HE2b2pSXb7cCmRQ7dGjcAEYnzbYIFm8+PTTn9r8oIPK3zff9PjjLcPWXjv5/r17Z5bvvTd72+67W+F4zjmF94+nY+21LX63bsXPqWoClxse7Veo8nDVVarbbJN/W7dumUrBiBGZ8DPOUH3kkeL2dO+evT5mTNvuiYjN81VABg3KLB92WGb51FMz+WmNNVT/8IfCx99qK9WddlKdPVv1kEMsbIstbIri/PrX9nycc07medh+e8vjl16qOnGiVWQmTFD98petUnLnnaq33GIiuGKFHX/JEtXzzjPRP+cc1eeftzw3fryFn3WWXe/111c95hjVf/wj+xldssSOE2fePBP6jz/ODv/Tn1S7djXhj/O//7WMG+eee6wCcO65qtddZ4IfVTCbmy0fP/SQ6n//a2lUtetw7rkm2Ndfr/rJJ1ahiraB6uWX2/W4/vrMuRYtsvRHgr54seXLY4/NiP9LL7VMw9VX271/5JHC6UiAi1AFpzaL0N//nilAo1pzhw62bY01Cj/Aw4aZpwCZAi2KHxVG06dbpipV2AwbVjpOfJoyxeybOFF1wIDy9o0XcNddl6ntT5li8wMPzI6XryCNPMRC0+mntww75BDVk0/OeH9g3sucOdnxjjnG5tdcY2L/9a9bQbbZZi2P+YMftPQiS0077mjXLt99aW42ryXXJlDt0iX/8f76V9Xf/CazvuGG5i2tumpxO6ZNa3nsSHRyp/XWyyw//HDGA4um1VdXHTIkc20LHef++81zP/vsloIZTU1NlqcGD86Edepk87gnH019+xZP55Zb2nz48OzwAQPMcyu27/HHW8F8zz2ZsNdft4L5uOOy89K226recEP2vbjkEvPwDjlE9bLLLGy11Uy4Zs1S/drXzDNVVX3llWT5Z5VVVEeNsuUddsgfZ+JEy7PxykE0jRypOnBgZn3rre06DBmSnUffe8/uBVgeX7HCxDqKM3ZsmzxSF6EKTm0SoWXLsjPIhAk2HzbMthcrSL75TdWnnrLlqOlrrbWy47zxhnlGxTJ1jx6qvXrZA3XkkcXjQrb9+Qr7pFOXLnaMqHZ/wQU2v/zy7HjRwxufSgnrdttlliNhXnttqwU3N6vuu6+lN2pWGDs2E//11+0h3XZbW7/oIouz007lpS9XPPfbLzNXVf3RjzLbdtgh0wSoajX4qCDfemurrRc6z6uv2j5RwfSHP9j6XnvZ+mabtbymYIVjtLz33jb/6ldbxtt5Zzvem29aflNVvfji7Di9elnTXOS5TZyoevPN2XEeeig770R5+ze/MS/vuusy3jWYB/Txx1aQdulizYzR/Xr7bavZz56t+uGHmXsc985KTZG3uuaaVhHafntrfo6aMwtNxx+fvb7nnlYZKXW+uKcukt3smiuQSabctOa2jowfn78loVAFATKV2T59Wm677z7b3qOHNaWDCfmSJa0q+tqDCDXGwIR33slef+YZmw8ZYvNiHeLdu0PHjrb88cc2X7YsO87AgdChg02FWH11+OQT6zQePDi57WCDHuLknmfyZOv0/tGPWu4bDSzo2tXmb7xh83XWgWOOycTbYIPs/W69FSZObHm8E0+Ef/zDlp94IhO+zjrwf/9ndlxxhY0GvO46G5whYnHiv+JYfXUb0PHAA7Y+bJjNe/Roec5i7Lcf/PKX2ccFG6AA0K1bZtstt2Rfyy5dMvGHD4dRozLbrr8++zxRvM03t/mmm9o8Ot7gwZaeXAYPhu22s+WttrJ5374t40X2DhkCY8bYcv/+2XE++QSamjLha61lg1firxaMHp29z/nn2/Xff38bPXnAAdCrV3b83r3tnn34IVx6aeZ+DR4M3/uejbxcZRUbPLNwIbz7ro2u7NIl+1xf+xrccQeMH2/rgwbBD35gy83NNgL1vvtsUM4ll8BBB7W8DhHnnZe9/sc/2qCdKB8fcACcdBJ897t27SZMsOPHX4VQtfx+0km2/lrOXw3uu88Gs8yfbwOVzjyzpR3/+Af8+Md27V55Bd5/365XxD//aXl+zz0zA50uuABmzoSLL7b1ESMsvaeearbeeCMcfXSmPAEbVARWNr3yCnzrW5njDRoEnTsXvlbtnMYQoVzRmD/f5lFhHh+1lUuPHtAp/HYpyjTxh75fv8zD2KnI75kiwQMrGG+/3Qq+fBx8cPZ6XLT22cdGruUe+8tfzhROQ4fClVdmx4kK49dft/nAgVYwxdMR55vfzH7YwNJ5zjlWEItkP0RNTfawRfTta+IdFRoRkyfDvvva/vFCOxKh7t1tPmZMZiRbMQYNskLm6KNh990z6cyd5y5HRPaNGJERmn79MqIQEVVELrnEhHPkSFtfYw2bq5ooTJ2aKYR32MHSeeedVngNHGjh+Ubg5Z4PrODPpakpc41WW83mPXrYKM5NNrHrEWevvex88fsbF/qhQzPLPXtmBCgfHTtm9t1zz5ajA4cPt8IzEt2ePS0fQWaEYkSHDrDZZoXPFefii80ukcwzu802Vvm49Va7ttOmmVDE6dfP8uEPf5gdvuGGcPbZdoymJqsw3XmniU18VGnnznZ/f/pTG+G65pp2vOg+brRRJm737rDqqrY8ejR85StWMQMrF77/ffjZz8zWQYNg3XVbXjuwisDrr1teivJjuZXWdkZjilBENJx66dLCNY3u3TPiEolXvPCNP/TFRCjKUGCF4W672bDfXD79tKWAxDPhDTdYrS9OVFhFhZMqHHKILUe1u6iwnTMnY3fPnplj9O0Ljz1m3tR3vmMPZ64IRdeoY8eWtfRu3ezBi8gVn4gLLoDf/96W4yIUCVhUyPXuDTvvXLxQjNIBVnj86U+Z4cmRaMSFJ989/va3rVA56KBMIbJihRVMEY8+mlnu2dMKr4iocI88zm23tfiq9r4W2LUcMADGjbP1I45oef2iexcnN050rGjoezxfHH44PP546esF2SIUT2dbia5flG86dDCPcaed4KqrWsafNMm86WuuKXzMAQPslYSIKH35/rIcr+idfjp89JG1guR6nuusY/k8yiPxYx99NFx4oa137Jj/mb7jDvNqpk/PeJVNTfDb31ola+utM+cBe55y+e53s9cHDrS8NWuW5b+RI2H77e1ZL/a6RR3Qbv6s2iYKidCyZZmmuGHDzA3OJe4JRTW/+PHiD0NuIde7d6Z5IFeEoGVzBuQvjOKFTVQTPP/8TA0vOk5UuKhaPNWW55w5096f6dMn2wPs08cKjKiZKbI/TtzegQPtRd348bff3mqTZ52V7J2foUOtoB40KFOYR2lI2iyXW/MvJkL5Cuizz7YadYcOmUJ0yZLswjnybPLxta9ZAXTiiaVtHTIkc0+23rq0ABQSoUjwWltDjlc+cu9xuXTu3FIU4+nq1MleDM+HCGy8cfZ7cHG6d2/p9Uf3MF8rQlxUdtzR5t26tXz+izWbg7UQ3H13pjksl3XXNa8GLJ9+8onZutpqVsmKGDTIvJrIY43Tv795PdE9bm62+3LPPba+wQZWuZw2rbitdUBjeUJx9zkKjwriqDkol3ifUD6+/OXMcm6tKd72nk+EktRaoWVTGVhb+1572XJUsMVFKJe4Z3LUUXbueGGUr6kqt4CKX4d8nhBkHtxiTZxxdtgh0wcCmTREgpdbYERCP2aM9Wnl9oFMmmR9GFGTZr50xYk38USF6Oef5++3yceAAZbWLbdMFj+id+/swjqf6OZrjuvePSNC+ZrwkhA/VzwPtIbnnsvck6hCUK6wFapwnHxyy2ar224zAY97PfmI58+kz1lE377WrJqvjyiXpiab56s8glW0CpUf8fsbiRBYJTG3j7aOaSwROuWUTFi3btlv6RcSobgnlI/4fsVEKLdPqBwKPURRM0XU4V1MhOLnjDJ7/OHPd45iBVRuwREdP6r1tfbrB7nHzbUrenBHjrRmk9xrPny4ffEi6uuICokkFGqOqxbRfRo7NrvJKSKefyKamjKVj3jFphzi97VYBSsJo0aZBxw/biRCSQv/3HseVSzy5aEdd4SHHiptdzEROvLIZHYlIbKjkAglpbk5c7833rhtx2pnNJYIxb2BPn2yPaHIvc8ttEqJ0D77ZJZzm+NKeULlcPnl8Oc/Z4eNH28FWVTgJvWEonilHuRizRa5D12Upqg2nK8ATUK8XysfkVeY9BqWc60jEYLyR+m1hXPPzW9nVHjGB5A0NdlAjPnzK+MJVYJohGDUYV+uCMXz0h57wH/+A4cdlj16MynRgJG4Jxu3Y7fdMiMcK0F07NaK0PHH27xHj8wxmpvbblc7ov5FaMGCzBDeuJj07duyTwha9jF07Vq4sL7ppuz4uYV2uSJUrAno8MNh110Lb4fyRahc4sfNfegi8R440IbX3nln686R1BMqNPAhl3JEqHt3a86bOjVz3kIeciUp5nGqZg8Xb2qyfNYWT63SInT22VZBippio/S0xhPq1s3Wr7ii5bOYhEcftW8elur3qTTleNxxfvUru8cimcpyg4lQ/Q9MOPJIGw4N2Z5K377w8suZj0oOGWIZoV+/7A9yLl2aLV5du2YGKJQqCOMi1L+/9XMsWZK/YHz6afjSlxInKy/FRCiehnIKob/8xaZLL80OL+QJidgHRFtLKduqKUKQPTLx2WezvaNKk9uXl4TWFnZx2toPlEtTU3YFKXrOkvarxdOf9L4WYtCgluIVF8Ny+4dK0VZPKE5UrrgI1RnxFyrjBXGfPjY6JWpO69nThkOOHAlPPZWJt2RJ9n7Nzfa1bGg5ui03g8dFqGNHE7j33stfMK6yStsLmFJNWRHxQmj99bNf0sxl4sTMOx7x9BUSobYwN9hYAAAb0klEQVQSXesoDYU8oWo0x+Wy3nqt3zcJtRKhajc1fulLNnpsv/3Kt6etIlSKNItQ9GV5F6E64+WXM8u5zXFxmprgkUfsRdQ77rCX1KZPt0wW32/AgIwIlXpgoiaTsWMz5ywkQpUoxKNjlBKh+EM/c2bp40Y122LNcZUuPEqJULU8oZVJlMZyPJNKiFAljlEMkcK/CclHt262j2r1RCg6fqVFKGr2cxFqNfXfJxQnLia5w0ijTNSzp73/8vDD1ml46KHZfULxYZX53vOJM3q0vVwavetQrFO9EgVDJKw//nHxeOXWhJO8z1SpNviokCgkpOX2haRZhKJ3ssopwCqRTypdELcVkcw1qNb9itKcZk8oao7L935YHVPfIrRiRfanQeIilFuw5mainj2t07CpKVuE4h5UqVpbx47WJBE1yxUToUo8fF27WuFdalRRa0WoWHNcpR7uaCBAVECvvXb29ug6Ffr5XC5pFqHbb4d//WvlixDYe1Y//3lljlUJomepmp5QfF5pKnFfoi8tNJgI1XdzXIcO9mWBaNROXIRyh10Xe/s8nnHjIlTKE8rN8MVEqNgw8EpTyu5c8n3uphI1v3xssIEN0oj6Y+67zz4nFHV8RxWCpH8PTbMI9emTfLhwx46W5kqJULzfMw1E+b/afUKVJnrG2/q+FdinjU47rfWvN7RTqu4JicixIjJLRJ4TkckhbIyIPCYiT4vIdBHZpGoGxAvQYiKUtGCOv5tRap/cJqpIhGr9oJVbG0zSHFfJGuYGG2Qe6kGD7KOYEfUkQuUQ5eN6SU8ukXfeXpvjKkFTU8svgDQAVRUhEVkPmARsAmwA7CQio4BzgTNVdQzwk7BeHeIFaKdO9q2yq65qfc0lLkKlxKQcTyjN5BOhUiMDq0Uk7PXQHFcOt9xiXlOlh1enhaiFob01x11yiVWa4p/vcsqi2m1Ao4HHVHURgIhMA3YDFIhGBvQB3q6aBbme0A472HL8Ez7lEHeVy/WEDjzQPmsT3++OO+Df/26dLSuLfM1xuSLe1g9hJqVcT6jcpse0svPONtUr0YCT9iZCEyZY87HTaqotQrOAn4tIf2AxMBGYDkwG7hOR8zBvLO/nakXkMOAwgGGtfXO9UHNcvBDdZZfkx4sXaqUemFwRWmutlt+t+ta3spub0ki+gjxK2w472Kfqq52GZ5+14e3R7y+SipCIvbMSVT6cdBKJULUqDdUemOC0mqqKkKrOFpFzgKnAQmAmsAw4EviBqk4Rkd2Bq4EWv6VU1SuBKwHGjRtX4uWXApTqE1pvPfsPTWuOV+7AhFrz4IPZP+RLSr73hCIR6trVPnVTbdZbz6bI4/r615PvW847K05tiESotR++ddotVR+SpapXYyKDiPwCeBP4JXBsiHIrkOePVxWilAgNG1Zev0FbPKFaU07BHSef2EbXrFqj5Aqx8cb2s7J8v7dw2i9Rn1D8h5GVxD2h1LIyRscNDPNhwLeBm7A+oOgHLF8H5lTNgFLNceUKRTmeUNpEqLXke09ou+3s75UXX7zy7XEBqj823NDmrfloaRJchFLLyng5ZUroE1oKHKWq80RkEnChiHQCPiP0+1SFUp5QuUKRO9quGPWS4fM1x3XsCGecURNznDrkgAPsdyrRv7EqjYtQalkZzXFb5Al7FBhb7XMDhUUjWi43U3bunPkOVSnqzRNynGohkvliQLWOH587qaG+v5gAlW+O69LFfv/w5JMtt+Vm8HoRoeha+QPstFdchFJL/YtQoa8ktLY5rnNn+1R9vn//5HpH9ZLhIxGK/pzpOI5TIeqkql6EuBDEBactnlBS6sUTWmUV+9PlvffW2hLHaR3uCaWW+veE4sQzYFs8oSTHz7fenjmsemNHHKfquAilljqpqreC1g5MKOYJ5TbH1Ysn5DjtHReh1NK4pWSUGSvpCeXiIuQ46cBFKLU0bikZeS2V7BOq5+Y4x3GcKuAiVElPyJvjHCeduCeUWhq3lIz+R+Oj4xyn/nERSi2NW0pGItSaLyYUwpvjHCeduAilFhehSnpCf/hD9qdH3BNynHTgIpRaGreUbK0IFfOExo+H++/PrLsIOY7jFKVxS8nWDkzI/a11LvHjea3LcdKBe0KppXFFqLWeUCkKfSbIcZza4SKUWhq3lGztwIRSxI/nGd5x0oGLUGppXBHadFOb77hj9c7hnpDjpAMXodTSGB8wffJJ+Oyz7LCNN4ZPP4Xu3ZMdo0sXWLKkvPO6CDlOOnARSi2NIULR/+tzSSpAAP/9L7z9dnnn9QzvOI5TlMYQoUowZIhN5eCekOOkA/eEUouXktXERchx0oGLUGrxUrKaeIZ3nHTgIpRaXISqiXtCjpMOXIRSi5eS1cQzvOM4TlFchKqJe0KOkw7cE0otXkpWExchx0kHLkKpxUvJauIZ3nHSgYtQanERqibuCTlOOnARSi1VLSVF5FgRmSUiz4nI5Fj40SLyYgg/t5o21BQXIcdJBy5CqSXRFxNEZApwDfA3VV2RcJ/1gEnAJsAS4F4R+QuwOvAtYH1V/VxEBrbK8vaAZ3jHcZyiJK2qXwbsDcwRkbNFZO0E+4wGHlPVRaq6DJgG7AYcCZytqp8DqOr7rbC7feCekOOkA/eEUkuiUlJVH1DVfYCNgFeBqSLyLxE5SEQK/e96FjBBRPqLSHdgIjAU+BKwhYg8LiLTRGTjticjpbgIOU46cBFKLYlLSRHpDxwIHAo8BVyIidLUfPFVdTZwTth+LzATWIY1AfYDNgNOAG4RyZ8zROQwEZkuItPnzp2b1NT04BnecdKBi1BqSSRCInI78A+gO7Czqu6iqn9S1aOBnoX2U9WrVXUjVZ0AfATMAd4EblfjCWAF0Fxg/ytVdZyqjhswYEB5KUsD7gk5TjpwEUotSX/lcLGqPpRvg6qOK7STiAxU1fdFZBjwbWBzTHS+DjwiIl8CugAflGd2O8EzvOM4TlGSVtVHi0jfaEVE+onI9xPsN0VEngfuBo5S1XnYKLs1RWQWcDNwgKpquYa3C9wTcpx04J5QaknqCU1S1UuiFVWdJyKTgEuL7aSqW+QJWwLsW5aV7RUXIcdJBy5CqSVpKdkhPnhARDpizWhOMTzDO046cBFKLUk9ofuwUWyXAwocgY14c4rhnpDjpAMXodSSVIR+BByOvWgqwP3AVdUyqm5wEXKcdOAilFoSiVD4VM9lYXKS4hnecRynKEm/HTcK+CWwDtAtClfVNatkV33gnpDjpAP3hFJL0lLyWswLWgZsDdwA/L5aRtUNLkKOky5chFJH0lKySVUfBERVX1PVM7AXTp1ieIZ3nHThz2TqSDow4TMR6YB9Rfv/gLeA+v0FQ6VwT8hx0oWLUOpIWkpOxr4bdwwwFnvZ9IBqGVU3eIZ3HMcpSklPKLyYuruqngAsBA6qulX1gntCjpMuvGKYOkqWkqq6HBhb6HcLThFchBwnXXgxljqS9gk9BdwpIrcCn0aBqnp7VayqFzzDO0668GcydSQVoVWAD8keEaeAi1Ax3BNynHThIpQ6kn4xwfuBWoOLkOOkCxeh1JH0iwnXYp5PFqp6cMUtqic8wzuO4xQlaXPcPbHlbsBuwNuVN6fOcE/IcdKFVwxTR9LmuCnxdRG5CXigKhbVEy5CjpMuXIRSR2tLyVHAsEoaUpd4hnecdOHPZOpI2if0Cdl9Qu9i/xhyHMdpP7gIpY6kzXG9qm2I4ziO03gkao4Tkd1EpE9sva+I7Fo9sxzHcaqAe0KpI2mf0Omq+nG0oqrzgdOrY5LjOE6VcBFKHUlFKF+8pMO7Hcdx0oGLUOpIKkLTReR8EVlLRNYUkQuAGdU0zHEcp+K4CKWOpCJ0NLAE+BNwC7AYOKpaRjmO41QFF6HUkXR03KfASVW2xXEcx2kwko6OmyoifWPr/UTkvuqZ5TiOUwXcE0odSZvjmsOIOABUdR4wsDomOY7jVAkXodSRVIRWiMgXn+kRkeHk+ap2PkTkWBGZJSLPicjknG3Hi4iKSHNykx3HcVqJi1DqSDrM+lTgURGZFtYnAIeX2klE1gMmAZtgAxvuFZG/qOocERkKbAe8Xr7ZjuM4rcBFKHUk8oRU9V5gIzKj48aGsFKMBh5T1UWqugyYhv0GAuAC4EQSelSO4zhO/ZH4K9qq+oGq3gM8DxwhIrMS7DYLmCAi/UWkOzARGCoiuwBvqerMYjuLyGEiMl1Eps+dOzepqY7jOPlxTyh1JB0dN1hEJovIE8BzQEdgr1L7qeps4BxgKnAvMBNYhjXv/STB/leq6jhVHTdgwIAkpjqO4xTGRSh1FBUhEZkkIg9hzWjNwKHAO6p6pqo+m+QEqnq1qm6kqhOAj4BXgTWAmSLyKrA68KSIrNr6ZKSMJ56A666rtRWO4+TiIpQ6Sg1MuAT4N7C3qk4HEJGy+nBEZKCqvh9G130b2FxVL4xtfxUYp6oflGV5mtl4Y5scx0kXLkKpo5QIrQZ8DzhfRAZhgxI6l3mOKSLSH1gKHBXeMXIcx1n5uAiljqIiFLyTy4DLRGR1YE/gfRGZDfxZVU8pdQJV3aLE9hHJzXUcx3HqiXJGx72pquep6lhgV+Dz6pnlOI5TBdwTSh2t+ieQqr4InFlhWxzHcaqLi1DqSOwJOY7jtHtchFKHi5DjOI2Di1DqSPqy6oNJwhzHcRynHIr2CYlIN6A70Cwi/YCoGtEbG77tOI7TfnBPKHWUGphwODAZE5wZZERoAfYiq+M4TvvBRSh1lHpP6ELgQhE5WlV/u5JschzHqQ4uQqkj0RBtVf2tiHwVGBHfR1VvqJJdjuM4lcdFKHUkEiER+T2wFvA0sDwEK+Ai5DiO47SapC+rjgPWUVX/AZ3jOO0X94RSR9L3hGYB9fOrBcdxGhMXodSR1BNqBp4PP7X74ptxqrpLVaxyHMepBi5CqSOpCJ1RTSMcx3FWCi5CqSPp6LhpIjIcGKWqD4hId+wX347jOO0HF6HUkfSzPZOA24ArQtAQ4I5qGeU4juM0BkkHJhwFjMe+lICqzgEGVssox3GcquCeUOpIKkKfq+qSaEVEOmHvCTmO47QfXIRSR1IRmiYipwBNIrIdcCtwd/XMchzHqQIuQqkjqQidBMwFnsU+avpX4LRqGeU4jlMVXIRSR9Ih2k3ANar6OwAR6RjCFlXLMMdxHKf+SeoJPYiJTkQT8EDlzXEcx6ki7gmljqQi1E1VF0YrYbl7dUxyHMepEi5CqSOpCH0qIhtFKyIyFlhcHZMcx3GqhItQ6kjaJ3QscKuIvB3WBwN7VMckx3GcKuEilDpKipCIdAC6AGsDX8Z+8f2Cqi6tsm2O4ziVxUUodZQUIVVdISK/VtXNsV86OI7jOE5FSNondL+IfEfEqxGO47RjvAhLHUlF6IfYVxKWiMgCEflERBaU2klEjhWRWSLynIhMDmG/EpEXROQZEfmziPRtg/2O4zjJcRFKHYlESFV7qWoHVe2sqr3Deu9i+4jIesAkYBNgA2AnERkFTAXWU9X1gZeAk9uWBMdxnIS4CKWOpL9yEBHZV0R+HNaHisgmJXYbDTymqotUdRkwDdhNVe8P6wCPAau31njHcZyycBFKHUmb4y4FNgf2DusLgUtK7DMLmCAi/cNP8CYCQ3PiHAz8rdABROQwEZkuItPnzp2b0FTHcRynvZBUhDZV1aOAzwBUdR42bLsgqjobOAdrfrsXmAlEHhAicmpYv7HIMa5U1XGqOm7AgAEJTXUcxymAe0KpI6kILQ0fLVUAERkArCi1k6peraobqeoE4CNgTtj/AGAnYB9V9f8SOY6zcnARSh1JRegi4M/AQBH5OfAo8ItSO4nIwDAfBnwbuElEdgR+BOyiqv4VbsdxVh4uQqkj0Wd7VPVGEZkBbIN9MWHX0NxWiiki0h9YChylqvNE5GKgKzA1vHb0mKoe0TrzHcdxysBFKHUUFSER6QYcAYzEfmh3RWxkW0lUdYs8YSPLNdJxHKciuAiljlLNcdcD4zAB+gZwXtUtchzHcRqGUs1x66jqVwBE5Grgieqb5DiOUyXcE0odpTyhL76UXU4znOM4TipxEUodpTyhDWLfiBOgKawLoKU+3eM4jpMqXIRSR1ERUtWOK8sQx3GcquMilDqSvifkOI7jOBXHRchxnMbBPaHU4SLkOE7j4CKUOlyEHMdpHFyEUoeLkOM4jYOLUOpwEXIcp3FwEUodLkKO4zhOzXARchyncXBPKHW4CDmO0zi4CKUOFyHHcRoHF6HU4SLkOE7j4CKUOlyEHMdxnJrhIuQ4TuPgnlDqcBFyHKdxcBFKHS5CjuM0Di5CqcNFyHGcxsFFKHW4CDmO0zi4CKUOFyHHcRynZrgIOY7TOLgnlDpchBzHaRxchFKHi5DjOI2Di1DqcBFyHKdxcBFKHVUXIRE5VkRmichzIjI5hK0iIlNFZE6Y96u2HY7jOE76qKoIich6wCRgE2ADYCcRGQWcBDyoqqOAB8O64zhOdXFPKHVU2xMaDTymqotUdRkwDdgN+BZwfYhzPbBrle1wHMdxEUoh1RahWcAEEekvIt2BicBQYJCqvgMQ5gOrbIfjOI6LUArpVM2Dq+psETkHmAosBGYCy5LuLyKHAYcBDBs2rCo2Oo7TQLgIpY6qD0xQ1atVdSNVnQB8BMwB3hORwQBh/n6Bfa9U1XGqOm7AgAHVNtVxHMdZyayM0XEDw3wY8G3gJuAu4IAQ5QDgzmrb4TiO46SPqjbHBaaISH9gKXCUqs4TkbOBW0TkEOB14HsrwQ7HcRwnZVRdhFR1izxhHwLbVPvcjuM4TrrxLyY4juM4NcNFyHEcx6kZLkKO4zhOzXARchzHcWqGi5DjOI5TM1yEHMdxnJrhIuQ4TuOgWmsLnBxchBzHcZya4SLkOE7j4B8wTR0uQo7jOE7NcBFyHMdxaoaLkOM4jlMzXIQcx3GcmuEi5DiO49QMFyHHcRynZrgIOY7jODXDRchxHMepGS5CjuM4Ts1wEXIcx3FqhouQ4ziOUzNchBzHcZya4SLkOI7j1AwXIcdx6p/u3W3uX9FOHZ1qbYDjOE7Vue02uOYaWHfdWlvi5OAi5DhO/TN0KJx+eq2tcPLgzXGO4zhOzXARchzHcWqGi5DjOI5TM6ouQiLyAxF5TkRmichNItJNRLYRkSdF5GkReVRERlbbDsdxHCd9VFWERGQIcAwwTlXXAzoCewKXAfuo6hjgj8Bp1bTDcRzHSScrozmuE9AkIp2A7sDbgAK9w/Y+IcxxHMdpMKo6RFtV3xKR84DXgcXA/ap6v4gcCvxVRBYDC4DN8u0vIocBhwEMGzasmqY6juM4NaDazXH9gG8BawCrAT1EZF/gB8BEVV0duBY4P9/+qnqlqo5T1XEDBgyopqmO4zhODaj2y6rbAv9T1bkAInI7MB7YQFUfD3H+BNxb6kAzZsz4QERea4UNzcAHrdivPeNpbhwaMd2e5uQMr7QhlabaIvQ6sJmIdMea47YBpgPfE5EvqepLwHbA7FIHUtVWuUIiMl1Vx7Vm3/aKp7lxaMR0e5rri2r3CT0uIrcBTwLLgKeAK4E3gSkisgKYBxxcTTscx3GcdFL1b8ep6ulA7keb/hwmx3Ecp4FphC8mXFlrA2qAp7lxaMR0e5rrCFHVWtvgOI7jNCiN4Ak5juM4KcVFyHEcx6kZdS1CIrKjiLwoIi+LyEm1tqdSiMg1IvK+iMyKha0iIlNFZE6Y9wvhIiIXhWvwjIhsVDvLW4+IDBWRh0Vkdvgg7rEhvG7THT72+4SIzAxpPjOEryEij4c0/0lEuoTwrmH95bB9RC3tbwsi0lFEnhKRe8J6XadZRF4VkWfDR52nh7C6zdtx6laERKQjcAnwDWAdYC8RWae2VlWM64Adc8JOAh5U1VHAg2EdLP2jwnQY9vHY9sgy4DhVHY195umocD/rOd2fA19X1Q2AMcCOIrIZcA5wQUjzPOCQEP8QYJ6qjgQuCPHaK8eS/f5gI6R5a1UdE3sfqJ7zdgZVrcsJ2By4L7Z+MnByre2qYPpGALNi6y8Cg8PyYODFsHwFsFe+eO15Au7EXnRuiHRjH/99EtgUe3O+Uwj/Ip8D9wGbh+VOIZ7U2vZWpHV1rND9OnAPIA2Q5leB5pywhsjbdesJAUOAN2Lrb4awemWQqr4DEOYDQ3jdXYfQ5LIh8Dh1nu7QLPU08D4wFXgFmK+qy0KUeLq+SHPY/jHQf+VaXBF+A5wIrAjr/an/NCtwv4jMCB9uhjrP2xFVf1m1hkiesEYcj15X10FEegJTgMmqukAkX/Isap6wdpduVV0OjBGRvtgL3qPzRQvzdp9mEdkJeF9VZ4jIVlFwnqh1k+bAeFV9W0QGAlNF5IUiceslzUAd9wlhtYOhsfXVqe//Fr0nIoMBwvz9EF4310FEOmMCdKOq3h6C6z7dAKo6H3gE6w/rG/7PBdnp+iLNYXsf4KOVa2mbGQ/sIiKvAjdjTXK/ob7TjKq+HebvY5WNTWiQvF3PIvQfYFQYVdMF+6PrXTW2qZrcBRwQlg/A+kyi8P3DiJrNgI8jF789IebyXA3MVtX4rz/qNt0iMiB4QIhIE/ZV+tnAw8B3Q7TcNEfX4rvAQxo6DdoLqnqyqq6uqiOwZ/YhVd2HOk6ziPQQkV7RMrA9MIs6zttZ1LpTqpoTMBF4CWtHP7XW9lQwXTcB7wBLsVrRIVg7+IPAnDBfJcQVbJTgK8Cz2K/Wa56GVqT5a1iTwzPA02GaWM/pBtbHPvr7DFYo/SSErwk8AbwM3Ap0DeHdwvrLYfuatU5DG9O/FXBPvac5pG1mmJ6Lyqp6ztvxyT/b4ziO49SMem6OcxzHcVKOi5DjOI5TM1yEHMdxnJrhIuQ4juPUDBchx3Ecp2a4CDkNjYgsD18ujqaKfW1dREZI7EvnjuO0pJ4/2+M4SVisqmNqbYTjNCruCTlOHsL/Xc4J//N5QkRGhvDhIvJg+I/LgyIyLIQPEpE/h3//zBSRr4ZDdRSR34X/Ad0fvnyAiBwjIs+H49xco2Q6Ts1xEXIanaac5rg9YtsWqOomwMXY98sIyzeo6vrAjcBFIfwiYJrav382wt58B/vnyyWqui4wH/hOCD8J2DAc54hqJc5x0o5/McFpaERkoar2zBP+KvZDuf+GD6e+q6r9ReQD7N8tS0P4O6raLCJzgdVV9fPYMUYAU9V+SoaI/AjorKo/E5F7gYXAHcAdqrqwykl1nFTinpDjFEYLLBeKk4/PY8vLyfTDfhP7/tdYYEbsC9GO01C4CDlOYfaIzf8dlv+Ffd0ZYB/g0bD8IHAkfPEjut6FDioiHYChqvow9vO2vkALb8xxGgGvfTmNTlP4c2nEvaoaDdPuKiKPY5W1vULYMcA1InICMBc4KIQfC1wpIodgHs+R2JfO89ER+IOI9MG+iHyB2v+CHKfh8D4hx8lD6BMap6of1NoWx6lnvDnOcRzHqRnuCTmO4zg1wz0hx3Ecp2a4CDmO4zg1w0XIcRzHqRkuQo7jOE7NcBFyHMdxasb/AzhtlKKBpp8dAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# gradient descent\n",
    "detailed_logger = False\n",
    "main_logger = True\n",
    "main_logger_output_epochs = 100\n",
    "L2 = False\n",
    "Dropout = False\n",
    "momentum = False\n",
    "adam = True\n",
    "hidden_layer_relu = True\n",
    "hidden_layer_tanh = False\n",
    "hidden_layer_sigmoid = False\n",
    "\n",
    "# hyber-parameters\n",
    "alpha = .01;\n",
    "epsilon = .85\n",
    "keep_prob = .9\n",
    "number_of_epochs = 500\n",
    "batch_size = 50\n",
    "momentum_coef = .5\n",
    "RMSProp_coef = .9\n",
    "epsilon = 1e-20\n",
    "t = 0\n",
    "\n",
    "# copy initalization\n",
    "W = Weights.copy()\n",
    "B = Bias.copy()\n",
    "\n",
    "# data arrays\n",
    "cost_array = []\n",
    "accuracy_array = []\n",
    "interation_array = []\n",
    "\n",
    "# rename\n",
    "X_train = np.float64(training_images).copy()\n",
    "Y_train = np.float64(training_labels).copy()\n",
    "\n",
    "X_test = np.float64(testing_images).copy()\n",
    "Y_test = np.float64(testing_labels).copy()\n",
    "\n",
    "#m = size\n",
    "m = number_of_training_images\n",
    "\n",
    "def model(W, B, A):\n",
    "    return np.dot(W, A) + B\n",
    "\n",
    "def activation_relu(Z):\n",
    "    Z = np.where(~np.isnan(Z), Z, 0)\n",
    "    Z = np.where(~np.isinf(Z), Z, 0)\n",
    "    return np.where(Z > 0, Z, 0)\n",
    "\n",
    "def activation_tanh(Z):\n",
    "    return np.tanh(Z)\n",
    "\n",
    "def activation_sigmoid(Z):\n",
    "     return 1/(1 + np.exp(-Z))\n",
    "\n",
    "def loss(A, Y):\n",
    "    epsilon = 1e-20\n",
    "    return np.where((Y == 1), np.multiply(-Y, np.log(A + epsilon)), -np.multiply((1 - Y), np.log(1 - A + epsilon)))\n",
    "    #return np.multiply(-Y, np.log(A)) - np.multiply((1 - Y), np.log(1 - A)) \n",
    "    \n",
    "def cost(L):\n",
    "    return np.multiply(1/L.shape[1], np.sum(L))\n",
    "\n",
    "def cost_L2(L, W, epsilon):\n",
    "    L2 = np.multiply(epsilon/(2*W.shape[1]), np.multiply(W[len(W)-3], W[len(W)-3]).sum() + np.multiply(W[len(W)-2], W[len(W)-2]).sum() + np.multiply(W[len(W)-1], W[len(W)-1]).sum())\n",
    "    J = cost(L)\n",
    "    return L2 + J\n",
    "\n",
    "def prediction(A):\n",
    "    return np.where(A >= 0.5, 1, 0)\n",
    "    \n",
    "def accuracy(prediction, Y):\n",
    "    return 100 - np.multiply(100/Y.shape[0], np.sum(np.absolute(Y - prediction)))  \n",
    "    \n",
    "def forward_propagation_return_layers(W, B, A, A_layers, Z_layers, layer, D, keep_prob):\n",
    "    if(layer < len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        Z_layers.append(Z)\n",
    "        if(hidden_layer_relu == True):\n",
    "            A = activation_relu(Z)\n",
    "        elif(hidden_layer_tanh == True):\n",
    "            A = activation_tanh(Z)\n",
    "        elif(hidden_layer_sigmoid == True): \n",
    "            A = activation_sigmoid(Z)\n",
    "        if(Dropout == True):\n",
    "            _D = np.float64(np.where(np.random.uniform(0, 1, A.shape) < keep_prob, 1, 0))\n",
    "            D.append(_D)\n",
    "            A = np.multiply(A, _D)\n",
    "        A_layers.append(A)\n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Training Data: ' + str(layer))\n",
    "        A_layers, Z_layers, D = forward_propagation_return_layers(W, B, A, A_layers, Z_layers, layer, D, keep_prob)\n",
    "    elif(layer == len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        Z_layers.append(Z)\n",
    "        A = activation_sigmoid(Z)\n",
    "        if(Dropout == True):\n",
    "            _D = np.float64(np.where(np.random.uniform(0, 1, A.shape) < keep_prob, 1, 0))\n",
    "            D.append(_D)\n",
    "            A = np.multiply(A, _D)\n",
    "        A_layers.append(A)\n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Training Data: ' + str(layer))\n",
    "            print('Forward Propagation Training Data Complete')\n",
    "    return A_layers, Z_layers, D\n",
    "\n",
    "def forward_propagation(W, B, A, layer):\n",
    "    if(layer < len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        if(hidden_layer_relu == True):\n",
    "            A = activation_relu(Z)\n",
    "        elif(hidden_layer_tanh == True):\n",
    "            A = activation_tanh(Z)\n",
    "        elif(hidden_layer_sigmoid == True): \n",
    "            A = activation_sigmoid(Z)\n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Testing Data: ' + str(layer))\n",
    "        A = forward_propagation(W, B, A, layer)\n",
    "    elif(layer == len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        A = activation_sigmoid(Z)  \n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Testing Data: ' + str(layer))\n",
    "            print('Forward Propagation Testing Data Complete')\n",
    "    return A\n",
    "\n",
    "def dZ(dZ, W, Z):\n",
    "    Z = np.where(~np.isnan(Z), Z, 0)\n",
    "    W = np.where(~np.isnan(W), W, 0)\n",
    "    dZ = np.where(~np.isnan(dZ), dZ, 0)\n",
    "    Z = np.where(~np.isinf(Z), Z, 0)\n",
    "    W = np.where(~np.isinf(W), W, 0)\n",
    "    dZ = np.where(~np.isinf(dZ), dZ, 0)\n",
    "    if(hidden_layer_relu == True):\n",
    "        return np.multiply(np.dot(np.transpose(W), dZ), np.where(Z > 0, 1, 0))\n",
    "    elif(hidden_layer_tanh == True):\n",
    "        A = activation_tanh(Z)\n",
    "        return np.multiply(np.dot(np.transpose(W), dZ), 1- np.multiply(A, A))\n",
    "    elif(hidden_layer_sigmoid == True): \n",
    "        A = activation_sigmoid(Z)\n",
    "        return np.multiply(np.dot(np.transpose(W), dZ), np.multiply(A, (1-A)))\n",
    "\n",
    "def dW(dZ, A):\n",
    "    return np.multiply(1/dZ.shape[1], np.dot(dZ, np.transpose(A)))\n",
    "\n",
    "def dW_L2(dZ, A, W, epsilon):\n",
    "    return np.multiply(epsilon/Z.shape[1], W) + dW(dZ, A)\n",
    "\n",
    "def dB(dZ):\n",
    "    return np.multiply(1/dZ.shape[1], np.sum(dZ))\n",
    "\n",
    "def backward_propagation(W, B, Y, A_layers, Z_layers, _dZ, alpha, epsilon, layer, D, V_dW, V_dB, R_dW, R_dB, t):\n",
    "    if(layer >= 0):\n",
    "        if(layer == len(W) - 1):\n",
    "            _dZ = A_layers[layer+1] - Y\n",
    "        elif(layer >= 0):\n",
    "            _dZ = dZ(_dZ, W[layer+1], Z_layers[layer])\n",
    "        if(Dropout == True):\n",
    "            _dZ = np.multiply(_dZ, D[layer])\n",
    "        if(L2 == True):\n",
    "            _dW = dW_L2(_dZ, A_layers[layer], W[layer], epsilon)\n",
    "        else:\n",
    "            _dW = dW(_dZ, A_layers[layer])\n",
    "        _dB = dB(_dZ)\n",
    "        if(adam == True):\n",
    "            epsilon = 1e-6\n",
    "\n",
    "            # ADAM - RMSProp + Momentum\n",
    "            V_dW[layer] = np.multiply(momentum_coef, V_dW[layer]) + np.multiply(1-momentum_coef, _dW)\n",
    "            V_dB[layer] = np.multiply(momentum_coef, V_dB[layer]) + np.multiply(1-momentum_coef, _dB)\n",
    "            R_dW[layer] = np.multiply(RMSProp_coef, R_dW[layer]) + np.multiply(1-RMSProp_coef, np.multiply(_dW, _dW))\n",
    "            R_dB[layer] = np.multiply(RMSProp_coef, R_dB[layer]) + np.multiply(1-RMSProp_coef, np.multiply(_dB, _dB))\n",
    "            \n",
    "            # index decay in bias correction\n",
    "            t = t + 1\n",
    "            \n",
    "            # correct bias for initial rounds\n",
    "            V_dW[layer] = np.multiply(V_dW[layer], 1/(1-np.power(momentum_coef, t)))\n",
    "            V_dB[layer] = np.multiply(V_dB[layer], 1/(1-np.power(momentum_coef, t)))\n",
    "            R_dW[layer] = np.multiply(R_dW[layer], 1/(1-np.power(RMSProp_coef, t)))\n",
    "            R_dB[layer] = np.multiply(R_dB[layer], 1/(1-np.power(RMSProp_coef, t)))\n",
    "            \n",
    "            val1 = 1/(np.sqrt(R_dW[layer])+ epsilon)\n",
    "            val2 = 1/(np.sqrt(R_dB[layer])+ epsilon)\n",
    "                      \n",
    "            W[layer] = W[layer] - np.multiply(alpha, np.multiply(V_dW[layer], val1 ))\n",
    "            B[layer] = B[layer] - np.multiply(alpha, np.multiply(V_dB[layer], val2 ))\n",
    "        elif(momentum == True):\n",
    "            V_dW[layer] = np.multiply(momentum_coef, V_dW[layer]) + np.multiply(alpha, _dW)\n",
    "            V_dB[layer] = np.multiply(momentum_coef, V_dB[layer]) + np.multiply(alpha, _dB)\n",
    "            W[layer] = W[layer] - V_dW[layer]\n",
    "            B[layer] = B[layer] - V_dB[layer]           \n",
    "        else:\n",
    "            W[layer] = W[layer] - np.multiply(alpha, _dW)\n",
    "            B[layer] = B[layer] - np.multiply(alpha, _dB)\n",
    "        if(detailed_logger == True):\n",
    "            print('Backward Layer: ' + str(layer))\n",
    "        layer = layer - 1\n",
    "        W, B, t = backward_propagation(W, B, Y, A_layers, Z_layers, _dZ, alpha, epsilon, layer, D, V_dW, V_dB, R_dW, R_dB, t)\n",
    "        if(detailed_logger == True):\n",
    "            print('Backward Propagation Complete')\n",
    "    return W, B, t\n",
    "    \n",
    "\n",
    "def shuffle(X, Y, number_of_training_images):\n",
    "    random_array = np.random.permutation(np.arange(number_of_training_images))\n",
    "    return X[:, random_array], Y[random_array]\n",
    "    \n",
    "start_time = time.time()   \n",
    "# main loop\n",
    "for epoch in range(1, number_of_epochs):\n",
    "    \n",
    "    # logger\n",
    "    if(main_logger == True and epoch % main_logger_output_epochs == 0):\n",
    "        print('Main Loop Epoch: ' + str(epoch))\n",
    "    \n",
    "    # saftey check\n",
    "    if(adam == True and momentum == True):\n",
    "        print(\"ERROR! Please Select Either Adam OR Momentum OR Neither, Not Both.\")\n",
    "        break\n",
    "\n",
    "    # saftey check\n",
    "    if(hidden_layer_relu + hidden_layer_tanh + hidden_layer_sigmoid != 1):\n",
    "        print(\"ERROR! Please Select Only 1 Hidden Layer Activation Function\")\n",
    "        break\n",
    "    \n",
    "    # shuffle data\n",
    "    X, Y = shuffle(X_train.copy(), Y_train.copy(), number_of_training_images)\n",
    "    number_of_batches = int(np.floor(number_of_training_images/batch_size))\n",
    "    split_index = number_of_batches*batch_size\n",
    "\n",
    "    # parse into minibatches\n",
    "    X_minibatches = np.split(X[:, 0:split_index], number_of_batches, axis=1)\n",
    "    if not(split_index == number_of_training_images):\n",
    "        X_left_over_portion  = X[:, split_index:number_of_training_images]\n",
    "        X_minibatches.append(X_left_over_portion)\n",
    "    \n",
    "    Y_minibatches = np.split(Y[0:split_index], number_of_batches, axis=0)\n",
    "    if not(split_index == number_of_training_images):\n",
    "        Y_left_over_portion  = Y[split_index:number_of_training_images]\n",
    "        Y_minibatches.append(Y_left_over_portion)\n",
    "    \n",
    "    number_of_minibatches = len(Y_minibatches)\n",
    "    \n",
    "    # logger\n",
    "    if(main_logger == True and epoch % main_logger_output_epochs == 0):\n",
    "        print('Number Of Minibatches: ' + str(number_of_minibatches))\n",
    "\n",
    "    for index in range(0, number_of_minibatches-1):\n",
    "        X_minibatch = X_minibatches[index]\n",
    "        Y_minibatch = Y_minibatches[index]\n",
    "\n",
    "        # forward propogation training data set\n",
    "        A_layers, Z_layers, D = forward_propagation_return_layers(W, B, X_minibatch, [X_minibatch], [], 0, [], keep_prob)\n",
    "        L = loss(A_layers[len(A_layers) - 1], Y_minibatch)\n",
    "        if(L2 == True):\n",
    "            C = cost_L2(L, W, epsilon) \n",
    "        else:\n",
    "            C = cost(L) \n",
    "\n",
    "        # backpropogation\n",
    "        W, B, t = backward_propagation(W, B, Y_minibatch, A_layers, Z_layers, 0, alpha, epsilon, len(W) - 1, D,  V_dW, V_dB, R_dW, R_dB, t)\n",
    "        \n",
    "    if(epoch % main_logger_output_epochs == 0):\n",
    "        print('Cost: ' + str(C))\n",
    "\n",
    "    # forward propogation test data set\n",
    "    A_test = forward_propagation(W, B, X_test, 0)\n",
    "\n",
    "    # accuracy\n",
    "    _prediction = prediction(A_test)    \n",
    "    _accuracy = accuracy(_prediction, Y_test)  \n",
    "\n",
    "    # storage for plotting\n",
    "    cost_array.append(C)\n",
    "    accuracy_array.append(_accuracy)\n",
    "    interation_array.append(epoch)\n",
    "\n",
    "\n",
    "end_time = time.time()\n",
    "run_time = end_time - start_time\n",
    "    \n",
    "print('')\n",
    "print('Results:')\n",
    "print('')\n",
    "    \n",
    "print('')\n",
    "print('Run Time: ' + str(run_time) + ' seconds')\n",
    "print('Cost: ' + str(C))    \n",
    "print('Accuracy: ' + str(_accuracy) + ' %')    \n",
    "print('')\n",
    "print('')\n",
    "\n",
    "\n",
    "pyplot.figure()\n",
    "pyplot.plot(interation_array, cost_array, 'red')\n",
    "pyplot.title('Learning Curve - ' + str(len(X[0])) + ' Training Data Set (Relu Hidden Layer)')\n",
    "pyplot.xlabel('Epochs')\n",
    "pyplot.ylabel('Cost')\n",
    "pyplot.show()\n",
    "\n",
    "# plot percent accuracy curve\n",
    "pyplot.figure()\n",
    "pyplot.plot(interation_array, accuracy_array, 'red')\n",
    "pyplot.title('Percent Accuracy Curve - ' + str(len(X_test[0])) + ' Test Data Set (Relu Hidden Layer)')\n",
    "pyplot.xlabel('Epochs')\n",
    "pyplot.ylabel('Percent Accuracy')\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As illustrated, after 500 epochs with minibatches of 50 the cost became approximately 0.0 and the test data accuracy reached 98.84%. These results are very good. The test accuracy is high because minibatch stochastic gradient descent inately provides a form of regularization, combined with the ADAM (momentum and RMSProp) which prevents us from getting stuck on local minima, and from focusing too much on specific features based on large gradients. It is important to note that having a cost of zero usually means we have overfit the training data; however, in this senario that doesn't appear to be the case since we still have a very high test accuracy.\n",
    "\n",
    "We now wish to explore the impact of adjusting the RMSProp hyper-parameter size for Adam. We will reset the momentum hyper-paramter to 0.9 re-run the algorithm with smaller RMSProp hyper-paramters of .5 see what the results we achieve.\n",
    "\n",
    "First we reinitialize our weights and bias's."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature Size: 784\n",
      "Weights Shape: (20, 784)\n",
      "Bias Shape: (20, 1)\n",
      "Velocity Weights Shape: (20, 784)\n",
      "Velocity Bias Shape: (20, 1)\n",
      "RMSProp Weights Shape: (20, 784)\n",
      "RMSProp Bias Shape: (20, 1)\n"
     ]
    }
   ],
   "source": [
    "# initialize weights & bias\n",
    "np.random.seed(10)\n",
    "print('Feature Size: ' + str(size))\n",
    "\n",
    "lower_bound = -.1\n",
    "upper_bound = .1\n",
    "\n",
    "#mean = 0.015\n",
    "#std = 0.005\n",
    "\n",
    "# hyper-parameters: hidden layers\n",
    "hidden_layers = 2\n",
    "units_array = [20, 10]\n",
    "Weights = []\n",
    "Bias = []\n",
    "V_dW = []\n",
    "V_dB = []\n",
    "R_dW = []\n",
    "R_dB = []\n",
    "for i in range(0, hidden_layers):\n",
    "    if(i == 0):\n",
    "        _W = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], size]))\n",
    "        _B = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], 1]))\n",
    "        _V_dW = np.float64(np.zeros([units_array[i], size]))\n",
    "        _V_dB = np.float64(np.zeros([units_array[i], 1]))\n",
    "        _R_dW = np.float64(np.zeros([units_array[i], size]))\n",
    "        _R_dB = np.float64(np.zeros([units_array[i], 1]))\n",
    "        Weights.append(_W)\n",
    "        Bias.append(_B)\n",
    "        V_dW.append(_V_dW)\n",
    "        V_dB.append(_V_dB)\n",
    "        R_dW.append(_R_dW)\n",
    "        R_dB.append(_R_dB)\n",
    "    else:\n",
    "        _W = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], units_array[i-1]]))\n",
    "        _B = np.float64(np.random.uniform(lower_bound, upper_bound, [units_array[i], 1]))\n",
    "        _V_dW = np.float64(np.zeros([units_array[i], units_array[i-1]]))\n",
    "        _V_dB = np.float64(np.zeros([units_array[i], 1]))\n",
    "        _R_dW = np.float64(np.zeros([units_array[i], units_array[i-1]]))\n",
    "        _R_dB = np.float64(np.zeros([units_array[i], 1]))\n",
    "        Weights.append(_W)\n",
    "        Bias.append(_B)\n",
    "        V_dW.append(_V_dW)\n",
    "        V_dB.append(_V_dB)\n",
    "        R_dW.append(_R_dW)\n",
    "        R_dB.append(_R_dB)\n",
    "    \n",
    "# output layer\n",
    "_W = np.float64(np.random.uniform(lower_bound, upper_bound, [1, units_array[i]]))\n",
    "_b = np.float64(np.random.uniform(lower_bound, upper_bound)) # b will be added in a broadcasting manner\n",
    "_V_dW = np.float64(np.zeros([1, units_array[i]]))\n",
    "_V_dB = np.float64(np.zeros(1))\n",
    "_R_dW = np.float64(np.zeros([1, units_array[i]]))\n",
    "_R_dB = np.float64(np.zeros(1))\n",
    "Weights.append(_W)\n",
    "Bias.append(_b)\n",
    "V_dW.append(_V_dW)\n",
    "V_dB.append(_V_dB)\n",
    "R_dW.append(_R_dW)\n",
    "R_dB.append(_R_dB)\n",
    "\n",
    "Weights = np.array(Weights)\n",
    "Bias = np.array(Bias)\n",
    "V_dW = np.array(V_dW)\n",
    "V_dB = np.array(V_dB)\n",
    "R_dW = np.array(R_dW)\n",
    "R_dB = np.array(R_dB)\n",
    "\n",
    "\n",
    "for index in range(0, len(Weights) - 1):\n",
    "    Weights[index] = np.where(Weights[index] != 0, Weights[index], np.random.uniform(lower_bound, upper_bound))\n",
    "\n",
    "#print(train_X.shape)\n",
    "#print(np.ravel(train_Y).shape)\n",
    "\n",
    "print('Weights Shape: ' + str(Weights[0].shape)) # matrix with a size of # of units X 784\n",
    "print('Bias Shape: ' + str(Bias[0].shape)) # vector with a size of the # of unit\n",
    "print('Velocity Weights Shape: ' + str(V_dW[0].shape)) # matrix with a size of # of units X 784\n",
    "print('Velocity Bias Shape: ' + str(V_dB[0].shape)) # vector with a size of the # of unit\n",
    "print('RMSProp Weights Shape: ' + str(R_dW[0].shape)) # matrix with a size of # of units X 784\n",
    "print('RMSProp Bias Shape: ' + str(R_dB[0].shape)) # vector with a size of the # of unit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we rerun our algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Main Loop Epoch: 100\n",
      "Number Of Minibatches: 1000\n",
      "Cost: 0.23567681362319903\n",
      "Main Loop Epoch: 200\n",
      "Number Of Minibatches: 1000\n",
      "Cost: 0.235604659004334\n",
      "Main Loop Epoch: 300\n",
      "Number Of Minibatches: 1000\n",
      "Cost: 0.4115037462206437\n",
      "Main Loop Epoch: 400\n",
      "Number Of Minibatches: 1000\n",
      "Cost: 0.23921075030401118\n",
      "\n",
      "Results:\n",
      "\n",
      "\n",
      "Run Time: 1448.7536845207214 seconds\n",
      "Cost: 0.23185894095927148\n",
      "Accuracy: 90.39 %\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZYAAAEWCAYAAABFSLFOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJztnXe4JUW1t991Js8ICAwoechKDkMUBRUJgiAqCmJAUT5RLiqIgHjRixETqCBXEFEBRUzkIBL0IiIzwACSZMABBiQNDBkm1fdHdblr16nqru7dvcM59T7PfvbeHaqru6vrV2ut6ipRSpFIJBKJRF0M9ToDiUQikRhZJGFJJBKJRK0kYUkkEolErSRhSSQSiUStJGFJJBKJRK0kYUkkEolErYwqYRGRy0Tkw73OR6L3iMiHReSyurdNDEdENhaRv9eU1s4iMqeOtALp595rEblORA4MrFtHREbs+xsisoWI/F/Mtl0RFhGZIyI7d+NYeSildldK/byJtEVkaRE5SUQeFJHnRWR29n9qE8erExH5sogszPJtPmtZ6zcTkZtE5MXsezNrnYjICSIyL/t8S0Skjn2t7Q6w8vWSiCyx81rlnJVSP1dK7V73tmURkbnZOT0vIvNF5K8icrDvOgT277gyE5F9RORWEXlWRJ4UkT+JyOo1HvurwLet/exzflREzhCRKZ2cQwwi8jERudazfK6I7ATN3utOsPPYK5RSNwMviUjh9RkxFouIjO3hsccDVwEbArsBSwPbA/OArSuk14tz+bVS6lXW5/4sL+OBC4CzgWWBnwMXZMsBDgbeCWwKbALsCfy/Tve1UUqdY/IF7A48YufV3b6XZaEiu2fnMQ1dAX8BOK0bBxaR9YEzgU8DywBrAv8LLKkp/VWBHYCLnFXmnLcAtgE+X8fxEs1gPVPn4HlGh6GUavwDzAF2DqzbE5gFzAeuBzax1h0N3Ac8B9wJ7GOtOxD4K3Ai8BS6VXQgcB3wHeBp4F/oAmz2uRb4mLV/3rZrAn/Jjv0n4BTg7MA5fAx4DHhVzjVQwDrW/58BX81+7wTMBY4CHgXOAu4C9rS2Hws8CWyR/d82u17zgVuBnTq4P1/OObddgIcBsZY9COyW/b4eONhadxBwQ6f75uR1J2CuZ/lc4EjgdmBBtuyLwP3ZPbwD2Mu5Z9da11ahH5jZWXn4QcVtxwAnoRsV9wP/Baic85nr3jtgO3TF/rrs/17oZ+S57Pr9t7XtI1l+ns8+WwHrAtdkeXgyK0/LBI6/HzAzJ39DaKG7L0vrXGDZ0LE9+38UuDzvnIHvARdY/ydmyx5CP1c/AiZm63YG5jj3Ypq179nAl3Oe02vz7oG7DbqheA/wDPB9dJ1zoHWvT8yu833Aofa9Bl6NFu1/Z8c4HhiyjvPnbP/5WVnZpUw5yZYvD1wKPJGVxYuAVbJ1+wN/d7Y/Cvht7HXO7v2jwJnZ8jWAF4Bxec9pTy0WEdkC+Cn6IV0e+DFwoYhMyDa5D3gjuiX1P8DZIrKSlcQ26BuyIvA1a9k9wFTgW8AZOW6FvG1/CdyY5evLwAdzTmVn9MNTyS2T8VpgOfSNOxj4FbpgGHYFnlRK3SwiqwCXoMV0OeBzwO9EZIUOjv8OEXlKRO4QkUOs5RsCt6msVGXcli0362+11t3qrKu6bxX2Q1s0y2T//wm8Ifv/NeCXIvKanP3fDmwJbA58oMB9G9r2EHR52ASYDryr7Ekopf6GfpjfmC16HvhAdh7vAD4tIntm696U7WMsuBmAoMvGSsAGwFrAfwcOdxOwsYh8V0Te7HFJHQ7skR1nVXSl8oOcY7tsjH7GvIjIaujKe7a1+Dvoht0maJGcBhwbSqMpRGRF4LfoBu5UdOW+jbXJIejG06Zoz8R7nSTOBl4C1kaXhT2Aj1jrt0c3hJZHC8wZFbI5BJwOrI6uOxaiBRDgfGB9EVnX2v4D6IYGFF/nVYFXZWl/EkAp9QC6fNlpDidPder6ELBYgFOBrzjL7gF2DKQzC9g7+30g8KCz/kBgtvV/MrpF89rs/7W0WyzebbMLuQiY7LSEQq36K4FvFlyDIotlAVlrIVu2DrqFOjn7fw5wnNXqOMtJ/wrgwxXvzwbAyugW2PboFtb+2br/Bs51tj+HrFUILCZrWWf/183OVTrZNyevOxG2WD5UcJ7/APbIfvuskG2tbX8PfK7Ctn8BDrLW7UZJiyVbPhM4KrDPycC3rXISTD/b5j3AjJz12wO/QVskL6Mbe6bc3Yv1PAKrAa+gK7SYY59pyrlzzs9n5VsBfySzqLJ0XwbWsLZ/I3Bv9rtTi2UR2kKwP0vwWCxoa+s6a/8h9LNxoHWvP2atf7u5HsAqaFGZYK3/IHCldZy7rXVLZ+cytUw58Ww3HXjC+n868D/Z782yezwu8jq/DIz3HOMxYPu8fPQ6xrIGcEQWtJwvIvPRBXdlABH5kIjMstZthG45GB7ypPmo+aGUejH7OcwPX7DtysBT1rLQsQzz0K3DTnhCKfWylZ/ZaHfYO0RkMtod8sts9RrAvs5128GXByfw7e3topS6Uyn1iFJqsVLqenSL5z3Z6ufRhd5maXSl4Fu/NPB89nR1sm8V2u6RiByYBaXNNXod7eXH5VHr94uEy03etis7+cgrN3msgnbxIiLbici1IvKEiDyDrpSC5yEirxWR80TkYRF5Ft2ICW6vlLpeKbWvUmoq2gp5C3BMtnp14CLrGt6OrgBXjDyPp4GlPMv3VEotBbwVbaUuly1/LTABsO/bxSWOV8R1SqlX2x+0S89H271USi1BV/De9cAD1u810OfxmHUepwC2xeyWIcgvc8MQkSki8hPRnYaeBa6m/V7/HDgg+/0BdCx1IXHX+TGl1ALPYZdCC3KQXgvLQ8DXnBs9WSn1KxFZA622hwLLZwXgH+iWsKFqBVTEv4HlsgrdsFrO9n8Cdi3o2fIi2ioyvNZZ7zsX4w7bG7gzExvQ1+0s57pNUUp9001AWYFvFd/bxVgcoGMTmzjuxE2y5Wb9pta6TZ11Vfetwn+uoehebaei3RWm/NxNe/lpgn+jXQiGvHLjRUS2RVdA12WLzgV+B6ymlFoG+Amt8/CVmxPQVsXGSqml0dZ51HkrpW5Eu1A2yhbNBd7mlLWJSqlHA8d2uQ1YL+d4V6OtDNNr7DG09b6+dbxlsvN2912UnWfec9UJ/8a6fyIyRPu9bVuPFmHDQ+hnfjnrPJZWSm1SY/5Ad3pYE9g6u9dvsVcqpa7L8v4GdF1i3GAx13nY/c3qZdCWbJBuCss4EZlofcaiheMTIrKNaKaIyB4ishQwBX1iTwCIyEdoFfZGUdqPOBP4soiMF5Ht0L7tEGehC9LvROR1IjIkIsuLyBdE5O3ZNrOA94vIGBHZDdgxIivnon24h9CyVkA/iO8QkV2z9CaKyE5ZD5zSiMjeIrJsdg+2Bg5D9+YC7T5cDBwmIhNE5NBs+dXZ9y+Aw0VkFRFZGTgC3ULudN9OeRWt8iMi8jG0xdI05wGfEZGVRWRZdIeCKERkGRExlunPlFJ3ZauWQlvQL2eis5+12+OAEqt7eLb9C8AzWQzjcznH3FF0N9wVs/+vR5f1G7JN/hf4umTdj0VkxSyPoWO7/BHYSlo9AX2cCLxdRDZSSi1GC+dJIrJCViZXFZFdAvveChyQPQd7oC33urgY2Cx7PsYCnwXsOKa516uIyPJoFzUASqmH0MH574h+FWFIdPfsN3WQn/GeOnQptIA9neXhOM9+Z6EbWS8opW7I8lf2Oht2BP6UWT1Buiksl6J9jubzZaXUTODjaJ/x0+gA3oGg3TPAd4G/odV1Y3SPjG5xALp3zjx0IPTX6NbRMJRSr6B9knej4y3PogP/UwHzYtin0Q/s/Czt84syoJT6N/r8t8+Ob5Y/hLZivoCuOB9CV2BV7+d+6Gv/HLqyP0Fl7/tkpvA7gQ9lef8o8E7LRP4xuifK7WiL8pJsWUf7dopS6jZ0kPlGdMvydbTuRZOcihbU29GB8UvQLcM8LhP9Ps6D6EDxt9HuLsMhwDdE5Dn0PT/PrFBKPQd8A/h75tKYDnwJHUx+BrgQbe2EeBrYB/hHlodLs/S/m63/HnA5cFV2/OvRPc9Cx25DKfUI8H/kNMwy6+ccWh0MjkC7lW7MzuGPhIPFh2X5nw/sm51vLSilHgPeh74f89AWiV2GTkW/ZnA7MAMd6Lf5ALqBfCf6Ov+GziyqK2ivQ7+Ivj/LZPm7HvC5u3+BbpSf5Swvc50NB6AbG7lIdXf26EJEfo0Otn2p13lJDA4i8g7gJKXU2r3OS68QkY2B05VS2/Y6L6ORzEX/OLCRUupfHaSzOfBDpVShVZiEJYCIbIUOnv4L7Y46H9hOKXVLTzOW6Guyh/iNaMt1JeAPwJ+VUkF3VCLRJCLyeXSv3CI3V20M2hvK3eS16G6ky6MDmIckUUlEIOh3Zn6LjnNcjH4HK5HoOiIyF/1uy95dPW6yWBKJRCJRJ73ubpxIJBKJEcbAucKmTp2qpk2b1utsJBKJxEBx0003PamU6mTYp2gGTlimTZvGzJkze52NRCKRGChE5IHireohucISiUQiUStJWBKJRCJRK0lYEolEIlErSVgSiUQiUStJWBKJRCJRK0lYEolEIlErSVgSiUQiUStJWJrm4Yfhoot6nYtEIpHoGgP3guTAscMOMGcOpDHZEonEKCFZLE0zZ47+XrKkp9lIJBKJbpGEpVssXtzrHCQSiURXSMLSLZKwJBKJUUISlqYZyi5xEpZEIjFKSMLSNE0JywMPpLhNIpHoS5KwNI2I/q5TWO6/H6ZNg+OPry/NRCKRqIkkLE3ThMUyd67+vuqq+tJMJBKJmkjC0jQpxpJIJEYZSViaJglLIpEYZSRhaZomhcXEbxKJRKKPSMLSNE0E7xOJRKKPScLSNMkVlkgkRhlJWJomCUsikRhlNCosIrKbiNwjIrNF5OjANu8VkTtF5A4R+WWT+ekJTQhLGik5kUj0MY0Nmy8iY4BTgLcBc4EZInKhUupOa5t1gWOANyilnhaRFZvKT89oMsaSgveJRKIPadJi2RqYrZS6Xym1ADgX2NvZ5uPAKUqppwGUUo83mJ/ekFxhiURilNGksKwCPGT9n5sts1kPWE9E/ioiN4jIbr6ERORgEZkpIjOfeOKJhrLbEKlXWCKRGGU0KSw+P40bHBgLrAvsBOwP/EREXj1sJ6VOU0pNV0pNX2GFFWrPaKMYiyUNGJlIJEYJTQrLXGA16/+qwCOebS5QSi1USv0LuActNCOHFLxPJBKjjCaFZQawroisKSLjgf2AC51tzgfeDCAiU9GusfsbzFP3STGWRCIxymhMWJRSi4BDgSuAu4DzlFJ3iMjxIrJXttkVwDwRuRO4BjhSKTWvqTz1hCQsiUR9XHcd7LwzLFrU65wkcmisuzGAUupS4FJn2XHWbwUcnn1GJqm7cSJRHx/8IMyZAw89BGuu2evcJAKkN++bJsVYEonEKCMJS9MkV1giUT+pcdXXJGFpmiQsiUR9JPfvQJCEpWnSfCyJRP0ki6WvScLSNOnN+0SiPlJjaiBIwtI0yRWWSNRPslj6miQsTZOEJZGoj2SxDARJWJomdTdOJOojlf2BIAlL0zQpLKn1lhjt3HYb3HFHr3ORcGj0zfsEzQTvU6stMVpxG1Obbqq/0zPRVySLpWmasFjSEPyJ0U4vheSMM7TAvfBC7/LQ5yRhaRqfsPzqV/D5z1dPM7XOEqOVfnD/fu1r+vvRR3ubjz5mdAnL738PN9zQ3WP6hOX974dvf7t6mlUslieegP33h+eeq37cxGAwdy4sXNjrXDRL3Y2rGTPgXe9KoybXxOgSlne/G7bbrrvHjHWFKRX/sFR5qI4/Hs49F848s/y+icHh2WdhtdXg0EN7nZNmaMpi2W8/+MMf4IEH9P9nnoENNoBZs7qXhxHE6BKWXhAbvF9rLYiddrmKsJh8JDfa4PN//6eHjH/++eHrjEV68cXdzVO3aaocm3SvvRbuugu+9KVmjjPCScLSNLEWy5w5MC9yjjPjCivTckrCMnI46ihdXm69tdc56R11l+NkhdRKEpam6ZcXJJOwjDxGY2VozrmpnpExz0d6hgpJwtI0/dLdOAlLYiTRlMWSno9aSMLSNMliSdRNuof1WyzJrVwrSViapkmLpczDYPKRHoaRw2h0hRmaDt7HbJNeVA6ShKVp+mVIl9TKSowEmoqxVBHpNGJ5kCQs3SIJS6IuUoC5txaLIVksQZKwNI0pqCl4n6gbXyt7tNzffgjeJ2EJkoSlaZoQlmSxJEJUib8NEk13Ny7znCZXWJAkLE3TbxZLamUNPnmNA3N/R3oDoimLpczzkZ6lIGk+lqaxheWFF+DOO+tLswzJYhl5jEZXWNMNJDfdvOuZLJYgyWJpGlNQFy/WowtvvXV9aaa+9/WzaNFgX6OR7goz96ape1RGLKqK2+zZ8OtfV9t3QGhUWERkNxG5R0Rmi8jRnvUHisgTIjIr+3ysyfw0wty5cNpp4fW2xTJjRj3H7LXFohQ8+WTn6ZThvvvgL3/J32bevM5bkVOm6Hd+vvWtztJpkhhX2EhnkF1hG2+sR1MewTQmLCIyBjgF2B3YANhfRDbwbPprpdRm2ecnTeUnlwsv1HO1GM45B5ZdNm5Oi912g//3//R8Jz5sYRka8q8rS697hZ1+uh6JuZtzja+zDuy4Y3j9c8/B1KlwxBGdHWfBAv191FGdpdMNkiusfuoM3j/wgH+g0JdfLpenAaRJi2VrYLZS6n6l1ALgXGDvBo9Xnb331nO1GD75SZg/3z8suYsRlFBBt4XFrQiqPhxVKo8637y/7DL9fc895ff917+aGZXXPKxnn11PelOn1pOOzd57w5veVF96vnvZC4tl3jz43Oe6O7nYIFgs06bBZpuF14/gRkCTwrIK8JD1f262zOXdInKbiPxWRFbzJSQiB4vITBGZ+UTIMqgT02qtw09t+4Rdi6WbwlK3KwyGn08Ma62V/7BVZWzWD+WllzpLZ/x4/f3a13aWjo8LL9RzqXRKXpyhF8Jy2GHw3e/CRRc1f6xuWSwx6XeahxE8W2WTwuKrld0n4SJgmlJqE+BPwM99CSmlTlNKTVdKTV8hdjKsTnjlFf1dx403hW/JkvqEpdfBe3P8KsLSFOa8OnUzmHMbhKl9ffeyF63gZ57R3+PGde+YTVssMc9mp/G8QShjFWmyZpgL2BbIqsAj9gZKqXlKqawW53RgywbzE0+d757YrcteusKaEJZ+6nlU18CAZv9BaE36zrUXFouxEidNav5Y3bJYYp79ZLEEaVJYZgDrisiaIjIe2A+40N5ARFay/u4F3NVgfspTt7DUbbGUYbRYLJ2mMQgWS56I9kJYjJU4cWL3jtlLi6WuRkw/l7EOaewFSaXUIhE5FLgCGAP8VCl1h4gcD8xUSl0IHCYiewGLgKeAA5vKTyXKCEuooNuF0G3hV304OrFY6qh4+lFY6jgv+7oOwkPfL64wY7GY+FQ3aOo87ffOiqja8BTR+R/BFkujb94rpS4FLnWWHWf9PgY4psk8dESZG1/UK6wJi6VXMRaTRj+6wjrBrigGQVjyLJZuCowRlm4es5fB+04baWPG6LplEMpYRfqoydmH1PEWbhPB+ypvHydXWDH2/ejn1mSeK6YXFotxhXXTDde0Kyzv2e/UFWZ6MPZzGeuQPqoZGqZKQaxDWEZid+NeCkuRy7ETBs1iyetu3E1r0lgs3RCWfupuXNUVNmaM/h6EMlaR0SMsVQpivwrLaHaFhe5JnYI5ZsxgPPT94gobbRaLoRNXGAxGGavI6BGWKgUxxlQtMovt9b20WOp8876XFks3hGXChP52U/SbK2wkWyx5jadOLZZ+LmMdMnqEpWmLpchFU+d7LKO5u3HoYazTFTZxoj7Hfh/QsV9cYWakitFisdQRvIdksYwIeuUKazJ43+1960yjKt2yWKD/H/x+cYW5x+4GTQ+b3+R7LCl4P4KILYh2Yak7xhKyWMo+JL2OsZQZ9qJumhQWk3a/C0u/ucIMeeXh2Wdhyy3h9ts7O0YTrjD72VyyBB55BD7/+eL9UvA+yOgRlpiC+PLL7SMa1ykseRaLfZzLLy8+VpXKo84JknrZKo4RlqoPrGux9HuLsl9cYe6xfVxzDdx8Mxx7rBaXmTM7O1adZW+PPWDWLP17yRL46EfjBjPt1BXW7+WrA5Kw2EyaBDvt1Ppv3/hzz4UDDiiffkyvMHvfPfcszmuVAm3n4/3v1w/TOeeUT8dOy83HAw/AMsvA3XeXS+/55+HFF1v/f/lL3cL1ERIWOy9VB6Js0hUWUxE+/7wuf//8Z1ya9jn/9Kd6RGZzffrNFWbcP888A5tsAlttVe0YTVgsZhoI0NfPxIyKqDvGcuedesqOJUt0Gf7jH6ul3weMHmGJfdBuuaX1267E9t9fV3gh6hKWxYuHV4x/+hO85jXwwgvtaZbBzsevfgWXXgof+ED5dOy03HM+7zwtCGecUS69pZZqDVN/661awD8WmEz0W98qHsqk6gPfpCsspnU6Zw78+c9w00352/mu/8c/Do891ttWcIywFM0CGktdwvnww+3/fZ6FEHW5wubPh498BN76Vjj1VLj3Xjj8cNh11/b6aIAYPcLSz8F79zh26x30bIaPPw53ZWN0Vnmo6nRfNeEKe+45/W1cEA8+6N/ue99ruS1s6hCWJl1hMSJlziFW0HzX3+zbb66wsTWPHuUeq2pZvMsZ99ad6TUv3bqC9yecAD/7GTz6aCtdY7V2ewrwmkjCkke3gvfuvq5/1w28V/GjNxFjaSJ4bx7qvGvvWzcSLBaT79jj+s7T7NuvrrC6UKree26nYyyKIupyhbnp2HmoY4T1HjB6hKWpFyQNdQXvYbjFUseoyHVaGXUNG+4j5oHyPfhNWCx1CktMWrHCknf9e9nTKO+61+2iq0tY3OfBtVjyqOsFSV8ekrAMCP1gscTEWCAsLJ1U6HWKQZO9wmIeKN+D3+8WS52uMJ/16e7bb66wuoTFDt7bx6vrZeMyMZa6LBb3OUoWywBRl7CEKtNOXGHucYpcYf3S3bhJiyUv7aYtFjNhVbdjLCPZFVa3JVWXxeLuZ1sLoXQ7baQVdTdOFssAUdfoxjHdXX3HzXOFFVksvrTK0kTwvuicO6EXFssgu8JM42Oku8Lqtlh81oJdvvLSrVrpm3hTnsUSE2vsY0aPsFQpeL6HwV1W1K/ertBjLZYiV1gV66NOV1id1o9LzJAaea3Ion1jjt108L5oXLlOeoX1a3fjpi2WqmWxyBXmq9jrHissL8bShFegCyRhyaNpiyVU2cfGWMqcU53uq1Badfj1Q2Lr2ya0rN+7Gxc1QupwhfWCbgbvm4qxuMH7OiwWVziKgvcLFyZX2MDQa1dYXvC+aoylisVSR0FtMnhv8jfSuhvXKSx51qd5a7zfgvdNWCzdcIWVbeDEbFdksSRhGSDqslhCLa9u9grrF1dYE2Z6jGXVlCusWzGWovHO6nCFpeB9HN0I3ieLZQRT5Nf24RMR90YXFbI632Nxj1Wm8ohxMZVNyz3nTnqruWn3wmLplrA04QorK0pNMIjB+yoxlph1ecdwg/cuSVgGiKKH2UcZV1hRN+Q63rx3j1XmYQq5wup82bJM5WH2dffpp+B9nXEBO60id2odL0iOBldYN16QLFsOY45h0k8WywigaWEZFIvFHbm1ygMZqtjKVB6hSnQ0Wyx5Vsf558O8ef7tYXh34265wmKu+0svwSWX1HO8frBYynaECVn2KcYyAij7YiP0X4xl0SI9iu0VV7SnHYPZ1hWWKgU3lG/zoMSkGRKWJiyWW26BVVYZXjG72FMTA9xwA3z/+3Diifn7xRATYwldkyeegH320R/f9qHjNMX99+vrvWBBnLB85jPF8wwtXAhf+lJrBO8imoyxFFksbuPn+9+HU05prV+4UN8ze/ujj4a11mrfPwnLCCBU8PJu3Kc/PXyY7+ee8xf+GIvFdU+YIbuLeoUZvvhF+MlPWrPw5QmLefjdfIQshDKYfe6+G370o9Zyk/bChXr4/K23Hj56bGx+zDW58UY9ZYFNkcVy7bXwv//b+v/Vr+pZAa+5JnhKbcc2FstJJ+lK8fDD/dtvvTWcfHJ+moY8i+WHP9SiERKWV17R3/ffr79di/H554fvW5cr7De/aZ8b55prYO214Q1v0NfJHnZ+yRI9dcLll+spAF5+WT9DRdMAAPz853D88fDlL7eW/fnP+jwefFCL2K67ttJye4X5noUf/lDPEQS6DK2zTvtEfjfeqPNnc9ZZ7WnleS3M8T/zGTj00Nb6gw6CFVdsz9sJJ8C//gXXX68bLADf/rY+v9//vj39RYtacZgkLH1OFVcY6HlLbDbeGF71quHbnX6635rJs1j2318/mLEWy9/+Fpf3mTP1w28qvYcf1q4UCFssL7+sH+73vCcsBu45ff3r8KlPtZab81+4EK68EmbM0GLoI9Zi2WcfPcmabxtfnkA/5IccMvxYRaPWmnTHj8/fzjBjBvzXf8Vtm2ex3HKLrnBCYlvU8WTq1OHHqcMVdscd8N736orSYCZxM2Vx9uzWuiVL4H3vg913hzXX1EPB/+AHccJiyqVd8Z92mv7+y1/guuvaJ74qcoU9+SQcdpgWI6V0GbrvPj39hGGbbYZbsQ8+CGeeGU7XXhZ6/n71K//2oAX5mWfa1z/ySPv/ZLEMEKEHrejGxT6g55/vd5nY8RDfUCSPPlp+dOOivJm5HK6/Xn9vv71+qKD1AK+2Wnv+jjwSDjwQfvc77W7Lo8hnvHBhy50Ums0xNsbiO/ciV5iLSatocEGTbt1DvEN7o8PN/6JF+jqErok5t1DnD2PR+PbtBGOpzJ3bWmasOTdvdn5C/wE23NB/LN+b5nZs0T5Hsyzv2OZ6z5/fvq5sh4w8iyU23lrWK2ALSy97+XVAo8IiIruJyD0iMltEjs7Z7j0iokRkemOZqWqxlME3KU+eKyyUh06Fxa2I7EmzjLCYVrl5ONyZ9PIIPThlhCXUOo+JseQFVPPyGyss48blb1cFO88+v/6iRf3XK8xUwral51pzeVaDz0JOHG5zAAAgAElEQVQM3QOz3Ccs4J83Je/Y5r9I+75lLYAqFov7XPqeU1egbZKwhBGRMcApwO7ABsD+IrKBZ7ulgMOAvzeVF6BajKUOYl5mjH3zPpR2aLlvP1NQTeVpP4BF6YbWVxGWblosZV1hTVss7v1evFhfhyJXWMy8PHW6wkw+7etWxmLxiUiRsNjXxrZYXEsjNnjfqbDkNWJiXz3w5W3KlPAxk7DksjUwWyl1v1JqAXAusLdnu68A3wICNVBNVOkVlrdf2eMuWRKfh1iLpSjvvv1CFotN0TnHWCxGuFwXRmgfg+tm8J1D2TyXdYX1wmLJc4WFrkXTvcJ8QltGWHxCHirLRa6wshZLKABfh8USY1UXpeGL0xryGhkDQpPCsgrwkPV/brbsP4jI5sBqSqmL8xISkYNFZKaIzHzC7sZXhn6wWEIt7aIYS1HascthuLBUsVhihMWkEerhFtsrbDRYLIsW6WVmeZGw5LnCmnip075uoYFUffnp1GKxj+F7sTcmviPSfk3qiLEUucJcfGWzSFhi3aJ9SpPC4mua/OcKi8gQcCJwRFFCSqnTlFLTlVLTV1hhhWq5KRNjKVPJxh7XfRAMtm/d0KQrLGSxNCUsVV1hPsEzjESLBVr3JtZiyXOF1YHPFZY3ckNZi8Xe3tcLqsgVlmexmO2biLHEDJRalEaRK2w0CIuInBWzzGEusJr1f1XA7le3FLARcK2IzAG2BS5sLIBfpleYXbF0emOLXGELF9bfKyxvvzpiLDHCYpaVFRb3WtRpsXTSKywmIJtHXuUWKywuvXCF+VxSofwUWSz29nnB+xhXmG9oFpNGUzGWTlxhSVgAaOsjmAXmtyzYZwawroisKSLjgf2AC81KpdQzSqmpSqlpSqlpwA3AXkqpmdG5L0MZi8V+kGLdUiGKXGF2IQods2yMpYrF4ts/REzwvkhYYl/Y7MRicV1GReeV5wrLq1BjKOpuDK14VCeusCYmJ6vTYgkNl1LUK6xs8L4uYanSK8zFV+7y3K3289PLids6IFdYROQYEXkO2EREns0+zwGPAxfk7auUWgQcClwB3AWcp5S6Q0SOF5G9asp/PGWExbZY6hKWMhbLSy+1b9ukK8zXmq8qLPYLknVZLHn75uXJ3i422JrnCnMf8LLCEmOxxAqLoR9cYXnnVeRm9rnCYuOQRcF72xVWd4ylDldYXnfwEWCx5EYplVLfAL4hIt9QSh1TNnGl1KXApc6y4wLb7lQ2/ZKZ8S8vcoV1w2JxW3aml5Cp/KsKiw/XFZYXJA/RpCusTotl8WJ9bYu6h7rp+lqTdhdqXz6LyLNYBtkVZp9XTL5DrjBzn0MxlrLB+6YslpguzkVpQL5bdhR1N75YRKYAiMgHROR7IrJGg/mqn6oWS6hXkyE2HhEK3vssFggLmp23Oi0W3/4hyghLkbuurhhLTJC1E4sldpiVEHVaLL10hbktfvt47rF9eQm5wsw5+YTFd9wiiyV0vTuNseR1wgjhKysj3GKJFZZTgRdFZFPg88ADwC8ay1UTlOlu3E1X2IIF/rzZx7ULof3mc5UYiznfTnqFlYmxhOiGxWIqo1ifeJHFkpfPIvIqpKIYi+3WsemWKyzWYnHHofO9wxRyhfnEvx8tFl+ey6Zh8hWizCgMfUqssCxSSin0C47fV0p9H92ra3Ao84KkT1iKehQVHTfPFeYroLalZBdC+wW1Kr3CDE2/x1JU8cZaLDHHt9PzpRXrCsvrFdapsBS9eQ+tijk0+VlVi+WVV4ZX+jHEBO/zLBafGzRksfjEv0hYOomxxIpCmV5wIaq4wkaJsDwnIscAHwQuyXqFNdDZv0Hcrol5AThfr7CYGMQ114QnY8oL3pexWGKEJc9iMdgxlvnz27e95RZ497vbt7/5Zr3dCSeEK9kqFot9nq57I3QO9j27+WY47rh8YSnrCvMJyz/+0T4Yo53W1Ve3b+dzn9b5HkueBeariJZeWs9HU5aY4H1ejKVIWHyt/xdf9HftLnKFxXY3vuceOPXU8FQILnkutuQKCxIrLO8DXgE+qpR6FP0G/bcby1UT2IXg5JN15fHEE3EWy/XXD3+gXve64fvNmAG77da+rMhiufpqPV+IS8gFZ7vC7ryzfYBJ95gxFsvOO8Oyy+phyW3sOSIWL4Ytt9TbHe0ZS7RIWL72NS1UP/hBa9mNN8K997aPpLxkyXC/e1GMZaut4Ctfieu9424zbx685S3wpz/Bu97VEgRfN9ndd2+NCO3m4a1v1UPDi+hpFSZP1oH+N75RD4kuoqcRMHzve/r7qafgoov8rrBXXoG3vQ2OOaY937Nnt+ZlOecc2MvpYOmriBYs8A+Qmsevfw0HH6x/n3UW/PWv+rdbwXfiCvNV0n/+M+yxh57n5/TT9bI6XWFHHgmf/GR7Wcwjz2L5zW9gKctxc8MN8THBoSG47DL/aM+2e3wkC0smJucAy4jInsDLSqnBirHYhfBnP9PfDz4YjrFMmqR/L16s51Bwuecef9fF227zH9dYLCusAB/8YGv92WfDrFnD0wm5wtzRZdfw9KGICSybdMwQOWZCJB9Fhdt9CBYvbn+YrrpKC5U9qdJ73wvrrdeejrvfkiXt524mT/JVSL48FsVYzjxTW5lvexv84Q+tOUaKhn7xpfWb37T/f+UVLZymQjaTOwH89rd6/1/8QgvDU0/p5bbF8vjjWvC++c32EYvXXbeVzr/+NXxmxroqov32a/1esAB22EH/7sQVdsQRxRYL6ArXnuen0+7GZeMqhk03LXbV+uaPcQnFWHbbDd7//uHrXn453n3bp8S+ef9e4EZgX+C9wN9F5D1NZqx27Jtrj0sUslhefBE++tF8c9fn8ggFtk0La/31/RaKS4wrLEQZiyWGor7/RcJSVNkZN41bWSxe3H4O++7bfjwbXwzBdYEVudnMzKB25bfVVv48u2n57suCBeF7sHhxa74TM/GTbbHYFUpefCQU6O+EmOFxfMc3v3fZRX/bFsvqq8N3vhMWlrznLDSkS6zFEromxx7b/n/99VsNtenT4R3vKPf+kt0d3c2riykXvh6IL74Y3+GkT4kdbe9YYCul1OMAIrIC8Cfgt01lrHbsG2S/jOW76eZmDw3lP2QxPcZcV5hIXEeAToQlpjCWGQ8r1mIxD6E7/lnR/mZAvsWLh1sj9rnnTdcaIyxFrT9zzfOGibfzZuPbbuHCsLAsWtQ63nPP6W9TES9Z0l6hmeUxcb46LJY8IYvpFWYqWFtYzDUtcoXFHrdqd2MbN5amVPuzP25cy1qKmdExJCx5wfsRKiyxMZYhIyoZ80rs2x/YAhFjsYB+CPJubBlhMSI2NJQvLKaSjeluHCLmpccyFkunrrBYYfFZLDbu+GY2vsrQ7QFU9JD6LJbQdXLT8lUqeee9eHHrHpsyYp+DXSmb5d0SFnMdfOTFWMyxjRvZdoWZijnGFebSSYxlaKiasIwZ0/ptX9MqFkted+MRKiyxFsvlInIFYCZzfh/OG/V9j89icV0thiYtljFj8oVlyhTtt43pbhxiUIWlKYslVliMv9y+P01ZLLawGGwxsSvlbgtLXrmOcYXlWSxF3Y191Dm6sU2MxQLtoy7UJSx5FstLL41sYRGRdYDXKKWOFJF3ATugh8P/GzqYPzj4Yiyhm2YK3NBQZxaL26IyvZyKhCUv7SYslilT8lupnQpLkd/f9KwpirGY+1I1xuJr9dp04grzVSohV6vJS56w+H4PmrD4LJaiFyR9VAne29uHyp8rLEuWFFssVVxheb3jRqjFUuTOOgl4DkAp9Xul1OFKqc+irZWTms5crdgPuG2xNOkKs49pTPciV9jkycPTLmuxhN7UtrELdN4Q3lBeWNwYS9HLeSGLxRUWd3wzmyrBe/eB97nCOrFYIHzt7BiLwT4Hn8USQx29iMoIiy/GYlxh9vW1G2uG2OC977iuKyz2PRabMhZLTD5DwuK7fyNcWIpcYdOUUre5C5VSM0VkWiM5aopQ8N5HXa4wt+DHBO8nTND5q8MVlodtsZiKIESnFktoIEpDyGJx709ZV5jb3djdz71/nVgsoXsaai2XsVjKuMJC+StDtyyWTlxhdXQ3zrNYQsJSxWLxlc2i4H2s+7ZPKRKWwJUCoKA26jNC3Y3zYiwxFkve+E0+V1iRxTI0pCv6bgbvjZUUIkZYzEdkuOsiNO+9oazF4rsnvjwWxVjcCrST4H2onNQpLGVoSlhigvexMZZuBO/zuhvXHbwPvf9U1hW2eHF7D8EBpMgVNkNEPu4uFJGDgJuayVJDhHqFhV6QNNvV6QqLsVhEdEUfM7pxCLu1FrK46hYWt1Kx9ymyWEK9wtzgfV6XzyrBe/cam/VVXGGhirEOV1iZGIuh7OjLNqF4m6+Cz3OFFfUKK2Ox+IQlJsZS1hVmu+zKCktoXVmLBVodSQZUWIosls8AfxCRA2gJyXRgPLBPkxmrnZDF4qvkmxCW2OC9EZbQcP0xb4XbFWmotWYX6DqExRxn4kSd91DXWR+xFsvQUPv8KjZ1uMJAHy/G9VhFWGyh91ksoWtmfpeJn5j8VRGYULlesKDzXmFVg/c+V1iVIV1sOgneT5gwvFyHjlPWYoGRLSxKqceA7UXkzej56QEuUUpd3XjO6sYXvF+40D/goO0Ky6tU3ZkeXexCERu897nC7HTypjQ12MISyn83LZai/WNjLOba1RW891Wg7r2JFZY8y8Qwblwrn1W6GxfNDeTLX5U38UPCYvv+DbExFl/wvkmLpe7uxu5xyghLXvA+5Go1VuNIFBaDUuoa4JqG89IsvuD9woX+St5+CPJa23bvDR8hV1ieS8PnCvPlPQ/7xcAYYSkK3scM6ZInLEWYXml1WyxF3Y19Fah7fWOFJeTus6/d+PH5wlL0gmQZYTFlr25hiRmE0tcrrMhiKXIxdaO7cYywmHQnTmwNyWMIlfkyrjAjWANusQzW2/Od4HOFheZCsQtXHmWEpawrzH64fdZWHr20WEylEhtwnjixPSif9x5LVYslNsZijmFjX6dQ4BnCDZDQcl+MJWSxmOVlJp1ze+rZy4ooY7GUDd778gj5rjCfS7fq6MY2nQTvfQ2OkICVcYWZRlYSlgEh5Aoreo8lj4ceag0gaFi0yF+RlXGFTZ6sTeEXX2x/C9fOu83ChTptu7uv+bZHX7WJFZbnnmuNZRVi/vxWRVjWYpk0qXU9nn02P3jvWiz2/CixQ7q89FJreVmLZexYve+8ecPLzcyZ/vOzBcKdcMq1QEK9wkwZq+IK87mq3PmInnyyvfUdCt534grzldu5c+Gxx2DOHF2GQjz22PB5jtxn10wlAPra2iNCP/44XmItllde0ec9e3arnPmEJfSslREWE290hSXvBeY+JHZIl8GnCYvlF4GZA17/evjnP2GttdqPH9MrbPp0PRz6jBmt1ostAr4HdPx42HtvuOACPe+HXaEedpj/OHaBznOFLb10eJ3h7W9v/e5EWLbeGv7rv1rr3HtjWyxXXaXnkjHEusImT4addtLD5fsqavfe2PNtLFjQum4XXti+nTtsvsGuXJdZpvXfVwmFeoX9NhvrtYx70TdD58KFukI88kj47nd1HkxF9upXw9NP698vvqjvi3t9XnghX1hieoXZvPe9cedi5q+x+fGP2xtERx0Fn/+8/m2X7Vmz/NNSwHBh2XLL1rNmWyxveUtrGzN9gO+dlf/5H/9xyrjCTHmzy+6vf62nMbjtNj3fzwAweiyWUIzFPDw/+hG87336d6zFYgqZy7336m8z1wm0WyxjxsBdd8Emm7Tvd+ONeoZG14KwC2YoeH/BBfr7kkvaC2WoFWUX6JgOAbGUFZbx49srHddisSt622J59NH2dPKExZ3K99pr9bfPdeGK7JQp+oFee+325WaCtWWWGZ6Gjd1aXW+91kRpPivQvmZ5sb0f/ACmTm1f9vGP66HpDXkWy4kn6m/7GtoWw4IFWoDmzBmevxhXmBEr35v3Pt74xtbviy4Kb+dyaYfDFdp5+tvfdAVu7r9tsdg89JD+zntR+Uc/av8f6n0IYYvFsGRJ65qEBLIPGT3C4nuPZeHClom5776tCb1iLJb11vPPIunDVIa2a+d1r2tvDYOe+2PcuHzXVFGMxe4SaoLhPqvDLtAxw/jHUjbGMm5c+/HtyigvxuIKV94LknYXXxufG9S99kNDupX41rf6037b24afj43bM2qnnfRvN/Drkvfuz4YbDi87O+4Iyy3X+p8nLIZQLGXhQn0ea6wBr3lNa/mCBcPF2E7TrJswYbiQ5JXbLbZo/Y6xkA2m3Oy6q/4u27XazuO22+pK3dz/kLCY+iL0lj3APs6bGL7GXZHFYkgxlj4nZLGYh2vy5NbN9nWNdBkaKh5jy97WWCy+QRVd8lxTRcLyyivt728sXuzv0mgX6JgOAbGYllxsb6Rx49qPbz+EPleYEWm3kox5j6VoNkAYfk9NGXCvUSiImycsY8a00imKW+VZLL4RsseObS83PleYe09C1qwRFt/yPIvFrBs7drhA51n/ttVXppFjrqUp32UrYd/zZ/Jtv9NiY65ZnsXiPm958ZEYiyVm4r4+Y3QKi7lBtrDYvv4YV5gJssdgC4v94IQq9DotlkWL/A9BqLeTTZWX68q6wlyLxa7s3JfgTHzKZ7GEXGF27yG3Io21WMD/Mh0Mv7budq7F4gpLqIzlWSxjxgzfb8yY9rz73mNxr5nb8cTezjwD9vX3CYvvPrt5gXyhtK2UToSlqMy5z06esNjxNBtzX/KExd3PJyzmuiaLZcDxVZLGFTZ5sn5QTcGLcYWZbsEx+FxhZrmPuoTFWCy+B8R+qEJpVhmCvYorLM9icTtdlLFY3PdiXCvBZ7GEhCVksbitU/dau11uzXU3eVl++eF5cPdz8VksIWEJ9QqDOGFxl+dZLHZeXKs7rzxUtVjMtqaSLyqvrhj4yr25hq+8kj98UhlhCVmGvm19Fovvd58zeoTFd4OMxWL7VaEZYfFZLFVcYUWBdtM1EloVq1v52SIK4fMs896EoVOLxa783Rfj8mIsIVdYnrCUsVjcSsge2sMmz2KxXWEmxuIG4Q15wjJ2bDVXWKzFEmqtL1yor6l9bN999rnC8sY765bFUnSvoJXvl1/OF5a8dW66ZSwWNz5lW+1FY+71EY0Ki4jsJiL3iMhsETnas/4TInK7iMwSketEZIPGMmNXIqaVZYTF+NVdi6XIp5knALaFZOYn6ZbFYscWFi0aLiym5V+UZpn3JgxVhKVbFovbcvRZLO49DbnCQu8zlHWFhYSlU1cYtL/bBOVcYabM+FxhduA61mIxwuLzHMS4ZX24wuI2JPKOY+9vEyssefl00y0TY3Fd7Hb5r/I89ojGhEVExgCnALsDGwD7e4Tjl0qpjZVSmwHfAjwd1mvCLtCm8C1a1HKFQTmLBfKFxe066gveVxGWoqB4jMUCw7vx+uiFxeLGWDqxWEyMyRBjsbjXImSxmIrSFRZ3/6LgfRWLJcYVBsMF2C07rrDYVk6eK8w+55CwlImxVO2h6HOF5ZU7XwPLJdYVltfAc0W/jMXic2mabas8jz2iSYtla2C2Uup+pdQC4Fxgb3sDpZTd53IK0MFY3wXkWSzmRpa1WPK6HLpjfdXlCiuKXfhiLEWusDotlrIxFvc9lm7EWMz1iBktuKywuC1y910ON8ZS1WLxucJiLRZTrl1hMeuLgvd2ua/DFVZVWMw9t11heQ2vMsJSZLGU6UlZRlh8Fot5BgbIYmnyzftVgIes/3OBbdyNRORTwOHoofjf4q7PtjkYOBhgdfslsDL4xiUywXvjCivT3VgkXljsPHRqsRQNQW93NzYt9kG1WFxhMMLiWiIQjrH4LBYjBjHB0LKuMJcmYiwhV5jbIHEFuMgV9uKLuqzkWSyLFhULS17w3ucK61RYYi0W93nrprCYcuvi3kefsBhBGSBhadJi8TX3h5UqpdQpSqm1gaOAL/oSUkqdppSarpSavsIKK1TLjc8V1knwHvKFxddKced/qSIsVSwW9wFp0mIpKyx23AGGD6XuWixDQ36LJfSCpM9iMRVR7IMOYYulaEbPsq4wI2BlLZaQK6xMd2PTkChyhdmiUYcrzBbtMsLi3gN7jDAfZYSlyBVWJp/20DmGUFd+V5SXLGndl+QKA7SFspr1f1XgkZztzwXe2Vhuuh289xUC903ykCusLmEZhBhL3gOaZ7F0EmMxeSxjscS6wlzygvcisOyy7dube1WlV1inwXtzvxcs8AfvTdmKCd53I8biCks3LZYy+XzhheHvp+SlmyyWXGYA64rImiIyHtgPaBu5T0TWtf7uAdzbWG7yXGFVuxs35QrLE6wiV5jdKyzkCuuGxRIbYxEJb1vGYonpbmzOJ89icQm5wsx9cK+d2xhxLRaTzrPP6nLnvulvRKNKr7AiV9iiRe0TZLnDysRaLLaYhmIsJi++IfRduuUKc+9hkcXiPidF+4ZYsmS4sORZLK6wmPsyQMLSWIxFKbVIRA4FrgDGAD9VSt0hIscDM5VSFwKHisjOwELgaeDDTeVnWPdfKA7ed+IKC1ksTbvC7F5heUO6xOSjylDdVSyWUKVTt8ViqBJjCVksReLkBu/tGMvSS/vfmxk3Lr5X2Lhx+lrE9ApbuNA/HL/B3O8iYbHFsOjN+6WW0iI5KBaLOTcj7OPG+e9x2fH1YoXFZ7EkV1g7SqlLlVLrKaXWVkp9LVt2XCYqKKU+rZTaUCm1mVLqzUqpOxrLjF2JmFFCQ8H7GFeYyHC/qY2vEMybF+cKyzPBYywW1xXmi7HEuMJOOCH/WD5i34K2j50nLD6LJVZYbrwxPC3CVVe1WqU2bmOhU2EJxVheeUVXIL4xtcaNi4+xmAorxhV20klwzz2t/z5X2K23wp13+nuF3Xcf3HRTe3nyvQtjW0/mGclrEFUVlqee0t91xljcXo2hZ7Hs+Hp5dYWbbnKFDRBbbdWqyE2BXLCg3WIxhchUjm7BO+QQ2G03/VtED6X+wx+2b3P44fo75AqLsRRCfvuNN27tb+aecHnsMbj++tbxQhbLmDH6XC66KJyPOyrovEmrjCssVBn43mNZbTU9l8qtt7Zv66vgzz8f/v3v4cuffRZ22UX/Nvd8ueX0vXPn/ihyhW21Fay8cmu52xK1hdHtqDB5cmv0YNMpZWhIi1teJWILizn2MssMj9cccgh89KOt/9deC5tt1vrvusJ23rm13leh/vKX+tueDsI+F5M3W+SMUH/1q/5z2WmnuOD9JpvAm97kX2eOVYcrzJz317+uv0PPRtF0CSec0F6WXDelXU6++c32PI0AV9joEZaddoJvf7t92VNP6RtsbuSb36znujDDeLut2bFjYc8925cdeig88ID+PPFEa2KtF15oFaYjjmhtHxNjWWUVOPts+NWvWsve+U49L8iJJ+r0vvY12H///HPOi7GIwGWX6fNxH7BPfzo/3RCHHgqvfa3+XcZi2XNP+MQndH4uvxzOPLOV/yVLYLvttJDusAN8+cu6Up81S089MGsWrLRSOP0nnxy+bP78VoVvKpKxY/XkV24vrZheYWecEXeukya1V26TJ8NGG2kLwcxJIqK3yxsA1LYI9tgDbr8dpk3TFe8FF7QmnDJCAO339L//WzeK8u5RqKW+xRZwyilwxRXwl7+0lp90kn6+Lr64vXI0wxl94Qut8wNYf30938uVV8ZZLFddBcccM3z5RhvBmmvq33nC8qUvFb/MalCqdayQV+FDH4LvfMe/DnR53dt6be/d7x5+DMNRR8GKK7aOZwuLUq1GWnKF9SmuUJgpS40rbOJEPYOhqUTcgudaHIbVV9efqVNbaZlpXI86qlXZunnIG/frgAP0rHGrrtq+34or6gLt6xnkpp9nsdi4leZqWWe+LbYItzR9HH1065zKCMv48XDqqdqC2nVXPTeGnf8tt9QtwClT2lvl48bBppv6Y11mEjXf8PT2MpPfPJ+3vZ3BCIs7PE4ekycPt1hAzzhqx/V8MTZ7Zk278hk7VleuZt+99vK/G2MPdvmJTxS3uH29wkBbQdttpy0+21Jbe2343OdaFr09YZaNSe/YY/V8L2PHxgmL/b7TppvqGS8BDj64fW760AuSn/1seBy4PEICO358e4PRh/0sfuhD+dsai3vcuOHdjY2wJIulTwkJSyhY7nsoirogm7SMsLg9eGJcYTZ5HQl8y+wgYd4glHnpmEp0woRiUXLzas4pVlh819Pkx1gsvsrYzqfvOhqftm9CLbvyyYtn2fkLWSxDQ/EzcIaExc7H0JC/V6Bbbsw2vnP39VSzhWTy5Pyeh3Z+XOzAfV6czu0Q42Lf97LCAq3KdvLkdmEJlTtfr7lOhCVm37xn3RXsvLlskrD0OW4lZkzL0IRd7vb2kCwhgTGtZzM/+Nix7dvGuMJsOhGW0CCULm4+zP8JE4orXjevZWMsvnMwaRhhtLfxVca+it0IS9GEWkXnF7rfVSwWnyvMzUdo1Gy3e7gpZ7HCYleqkyYVj8wdui52ENq+L+71CVksZrvQ0EYxwmJ3SpgypbV/XvC+qrDYeSv77JbpiGBbLL5OGJBcYX1LSAxiLZaQK8zdZ9Kk1tAk7lvSsa4wQ14PtRiLxTcTXpHFYh6aiROrC0snL0iaZeY9FvshnjBh+HzhvofcNBZihaXIFRYaA6wOV5idj1iLxcQLfGXIXaZUe5rjx8cLi3vediPMPhe3TIUsFt9siDEWi+36tYUl1mIZOzbc4y8PO29le6/leTfc62qs6LypyZPF0qeUFRafxRIzPejkya0KzbVYmnaF+d4xKBIHNx+2K6yssJSNsfiuZ57FYre+u2mxuO+8hIQlr3zECouvPLrlxlihMRaLKywxcwmVdYWFhKVs5R3a3u4ib1fKZVxhMYNQ5uWtrLDkbRPqPRgSlvHjk7D0LaGHPuQKy0So/QIAABpOSURBVLNYiioQ49vPs1iaEBY7feOOKno7PMZiKRq6BNq709bhCjMWS8h/n2exlBWWIoslT1hiYyyTJhULi+u2Mtj7DQ2Fu8S729rHtikSltDLo7HCEooBdTK6sU9YpkyJExb7msUcy5c3+3dTrjBfjAV0jKzoPZ0+IgkLlAvexxQWV1hCFksTrjCbkLC4hB5+W1hC4uumY9IqmjfGkCeYvuA9DBcW33U0+fUF723qtljyuglPnlwcLyqyWEx5ynsR1Xe/3TSLhCU0GnHIFRYS/1CvMJu8TgC+7VyLpSjGYtZXsVjssuWLBeU9w+4zax8/zxXma1iYzhcDYrUkYYF4V5g71leIKVNaFZrrCrOJafXkDeHvW+YOGminYSiyWOwBFk0eY6dhjm29h/IC7RaL6wqz81KnK6xbMRb7fMvEWMwxzLeppHyWoe96lLVYQi3jsq6wmGsTani5VHWFudcs5liGIoslb1gnN/0Yqz/kCkvC0sd06gqL6RUG7TEW1xVmvyHeqSvM5MFXYKdMqW6x2MJiWlKxwlJ2qIs8wTTHLrJYYoL3oftV1WIxFVhZV5iNT1h829n5MN95oyDX4Qoz55dnsXTSKyxEE66wuoTFZ7GUERZ721BDJiQsSy+tv5Ow9CHd6BVm0gsF720XUV2uMF/hftWr4mMsIWGZOLG9900MdQiL27OsjMVi9xibMGH4rJEusS9IhmIOZS2W0H9zLosX+1v7boPGNCaaEpZQjMxOJ6ZXWJk4Q9H2MRaLzwVr9qsSY7HLli8WlGeFuNfE3jZU3uwYi319jcUyIF2OR5ewhApSmfdYYnuFhYL3dVosecKy1FLhFn8oHYMZAHHixFYFExNjiTmWS54rzAhLGYvFfk9i0qTWfQgJS1VXmL2+TmFZssTf2nePYSopnwDUabG4hOIhocE862houOvKxlhM3gbNFWaLWnKF9TFl3SGdWCyhGIvdoipTsPOExec6sV9kK4qxuA+/EZYJE9rfcI6hbIwl77xCFos5X3Msn8ViguDmPoTucVVXmL2+zAuSNp1YLHmusDpiLHlTCbt5cn+b/xMnNmOx2IwfX+wKC7mMY/Jmi5HPYskTlioWSxKWAcS+0euv718e2h70YIexMRa7tW1v647WW0SMK8wusL6HqKzFYrvCTDpmFN4i7GPFiEyeK2z+/OFpQrzF8sgjrRdVO7VY8oQl9Ha2S4zFsmhRvsXi3t9Yi6Vsr7AYy7zo/R33vZ0YylosZvnQ0HBhMfelVxaL3buyaFv7GD5hMTGW5ArrQ+zCP3GiHlX2978Pb28XvLPO0gMyxhRGt0umvY9tscQ8vFVdYWut1Z4HmyKL5ZBD9KCa+++vh13/+tfhuOPatwnNLxHqnhnC5x4w52WGHXd7dsXEWNzrVSQsIUw673hHeH1s5enmoZsxFreM5AnLVlvpUbRtNtxQDz5pk+cKAy2QTcVY1lmnfd24ccOFxVyjTiwWu2ztsMPwfWPEwjyLsRaLa5FDslj6Gnfk39e/HvbZJ7y9XfA+8IH24UTycAdKDLnCYsh7Ec5XuM2ovh/7WHse1llHDzPvw017nXX0NAArr6wL+jHHDBeSD34wnNa73qV/501WBbDjjv6h0N2Kcfr09v/mmphRo0MWi02nFsvaa+ttNtigfYTlMu+xuGVn3XVbv+33UvIsFpP+1lvr7333HX4c9/p97GPDjx0qC6BFxQzjbo538smteX4MRcLis1jyro+bZmidUjBzpi6jhrFjtcjaZc4VlqLBWH0ceqj+vuwyPa2Cm5ef/xw+/GE9rYXL0kvr9Vdeqf9/4hOtdXnB+3Hj9AjjZpRxGDhhaWxq4r7EFZYy2xtiXWEG1xVW1mIxacX2Cjv5ZC2Cb35zex7uvRcuuUTPfZJnsZgRn13clv3HP67njfniF/V/u0W/9975lqDhxBOHT9nq5ueuu4ZXgocdphsFZhIru2U3frx2Dw0Nwf33t1qLoZZlrMViuOMOmD27JQohi+XQQ/VApGeeqYext+cu+e53tXivskprmbnPL73UbrFMnqzn8jBiY1rkRuh82Nfjwgvh7W/Xv9/4Rnj/+/XvddbRx3roIT0hmj1x3KabFl8Hd5mvfO6zjx4aP5Z11/Wnc/PN7cdTSle09ojNEybA97/f+n/AAbDeenoeFsMb3tCebszzt8MO+WK49trws5/BDTfo6+hiD5f/oQ/pcvi+94XTM+Xxn//UXpKDD9b/kyusjykrLEUPUwjXYrGPVVZY3GmTfXmxK81Jk9pFBVoVXyjv9nIzk6GLWwGPH6/n1DBceGHrd2ygP5Qf+7r4hGf99XXF7fNF291czVw20HowXWItFhv7/ELvseyyS6uFvNxyWggNhx+u503xpelaLC+8AL/4ResexwyV43ZTNf//8pf2VvPEiboyP/LI1rLbb/e7OfPeowqtP+EE+OQnw/vYvPgi/OMfer27zeabtx/Dd6/se7LMMnqiPFOWzTO30kpw993+43dKWZdfnisM9L2xy9WrX63rgiKLr08YvRZL0VzlUPwwhXAtFltM7OOWScsnhLF+Xvft/aIYiw9XWPIepDrfefEJS146dqVs5zmUTlmLBYYLS2i/vBc4Xey4nC/GUjR/io29X5lBRH3bm4qsqOKMKct2ei72+Q0N+Z/PPGHxXT+zzHaPlRWAWGLjbOY6FQkLtOd1lVVaHVEGgNFrscQIS12uMPtYVV1hvnhFrLCUsVhCNCEsZTtChAhZLDZNCouvUrGFpayV64uxdEtYQh0u6hKWGIrKaZGwmN/m2w7oNyUssekWXafQ3DRl72OPGb3C0k1XmC0mVV1hvqCdr7uxjTu0eicWi1vh5F2H2Eow5rhltwkJS52uMLdV6auMh4byh7Z38bW4qwpL6G3xGKrOmBgrLDHb1S0sMWl3Stlu1WUtljIzufYBSVhit89b5uJ2N7YtlhhLySbGYgkJizsZVCcWi0jcTH8w3GKp2gqOJcZiMcLiVgBVLBY3tuCrVEQ6t1jqcIWVfWG1qiuszgo7JvbmYj9zxoLvprDUZbGEhCVZLH1MryyWOlxheRZLqNAZYanDYoHqwhLKX9lWXohQjMXGCIt77lUsFne9r3eUG+Mpok6Lpc4Yi6EfXGGGouC9YSRZLElY+hj7RlUN3leJsdQRvM8TlpCZ7AqLOV5VYYmdkMmtBKtWVrHYgmdaq6EYi9ugqGKxuOvt62m38O2XHouwK8E6LZZ+C97HUGSx+CplX5yxny2WmK7iyRU2INT5HksedQbvfT1b3LyE3B0hV1jRfCwh7EqnzNAl3bJYRFoVsPsiakhYYkc3DhG6DmWFxRYO8yLuSIyxxFBFWHzi3o8WS3KFjUB60d240+B9nsViCD0sIVeYS90WS6yw1G2xiLRarq4Qh3qXdWqx5O1nj1hchDt6gDuNcXKFFbf2Db0QlrreYwm9G5QslhYispuI3CMis0XkaM/6w0XkThG5TUSuEpESr+lWylDrd69cYXUKiymcoXRiYyxVLJa8fdzKqRcWi3u9Qh0cOo2xhLB7hZXtsAHDpzHuVXfjXgTviwaF9d0re9kgWyw2yWIZjoiMAU4Bdgc2APYXkQ2czW4BpiulNgF+C3yrqfxkmWr97pYrLC94H0Ned2NzDkXCUlQ51G2x2Pmxe0e5FF3LqVPj8hVjsXQ6bH5ZhoZa5cAM3V+GXlksea69KvtVoYqw2Jj1vlZ+v1ssoTTrvL5doEmLZWtgtlLqfqXUAuBcYG97A6XUNUopM/jNDcCqNEkdvcKqvHkf6m7cLYvFnp/dt33dFoubdhWLZfZsuOeeuGPYreyQxdILYdl2W/37scfK7++ODFym23AnMZYQRWW1GxVfWWHx0S8WS1lhGTCaHNJlFeAh6/9cYJuc7Q8CLvOtEJGDgYMBVl999Xpy1+l7LHkPkl3IJkzoLHhvKkpf8L7IEgkJS15+84jtbuymXaVyW3vt+G1N/pcsGW6xXHMNXH11sbDU7QoT0YMgVsV1hZWpuDuxWFx60Sus6BidjJfVa4ulqitswGhSWHxX0FsiROQDwHRgR996pdRpwGkA06dPr16qbDFpMngPcMYZutW8zjrtYnLKKa3fG25YnI6xWD772eHriiwWU8k2abGMGdMaMdeHz2JZaik9x0rdwfslS2DaNP3bDEC40076c+21/n1DFe+ll8Kpp5avMO33WES0qMXM2QHwjW+0BoCcNCk8PXARvnlcOqWfhMXHSiu1fh9ySHi7XlsspmFspj3IIwmLl7mANaEAqwKPuBuJyM7AscCOSinPrEU1YgtLVVM0trVkhnSHViV39tl61FvD5pvrWQ5XXnn4kN6GMWPCxyyKsZx6qo5T7Lqr/t+pxWL7rE1aRTGjiRP1HCa33KLnrDj/fLj4Yp13e16TTrDz/8EP6riUOz+Giddstpkewdc0LEIWy+6760+I9dbTQ5sbPvUp2GYb+MIX9H9zfdyRpvM42urfMmVK9UEHi2Z2LIMtlHnEVoKf+Qycd16562JYcUXdwPrIR4av+8pXWlMp2Oe8777t89748hk7th3o8pzX9b8IU/42cMPNHpKweJkBrCsiawIPA/sBbU1bEdkc+DGwm1IqMBFIjdhWynXXFW+f91CWeWCPPFIP4e5r2a+0Ejz4ICy/fHx6hiKLZZVV4Kc/bf3v1GI59thWhV20z7XX6ofw2Wf1fCqnn65dc5dfrucFqbOF684g+e53D99mo43gootaFdrKK+u8VXUV3XCDbhQYTj5Zf7vCUpUvfck/O2QMZYdxAV0xX3LJ8OV1u8K22aZaLzlzDDOrqMukSXDQQcOXn3de+3/3PO6+u1wDZ+5c/5woZXo4brRR3HZJWIajlFokIocCVwBjgJ8qpe4QkeOBmUqpC4FvA68CfiO6YD6olNormGinmBb++98PW25ZvH1dN3bcuPYJf1zsmeLKYAvLrFntEx/56NRi2dvqe1F0bXb0ejVbk07VSWz+99yz9dtUhEUxlhDLLptfIXVaduxpcA1nnw2veU3xvlW6cX/xi61J23z0gyusDtzzWH/9cvsvv7y/Edjp/X744eG9B5Ow+FFKXQpc6iw7zvq9c5PHH0aR68il3x8WuzXpm/XPpVOLpdN9mqJKC93Q1PsBTVyfAw7o3bFHqrDURafvZK28sv7Y9NMzVpLBzXkVYs16Qycxlm5QVig7tVhi0uoFVfJvrlnRkC5V6afr0wn91CusDnrdK6xKmgNYlgYvx51gfLudCIuhHx6kohiLS8hiGXRhqWKxuK6wuumn61MH3Xzzvkn61WLxYfIaGjWijxmQ0lATpoXfRJ/zXlBVWGKX59FP1yYJS3OMNIulqXw2abEM2DhhMFqFJbZ10e+usKquvZFmsXTSWqwavC9iUCraWEaKsDRFslja6KPaoQuUtVj63RVWV2eE0Ri8N/lPFkscSVjyaeL8k8UyINThCusni6Xq+dTRK6yfKpJOgvdNWSwjRVhGmitskEgWy4BQR/C+bFyjScrmJWYOiEFktHQ37iUjJXg/SCRhGRDKxljqevO+KcrGWAz9kPc6SRZL8ySLpfuY+iq5wvqcOmMs/UDZGEs/ufHqpJNeYU0EXe30B53kCusdCxbo72Sx9Dl1CItZ1on7pS6quuVGWiVQpUXnDq44UkW3LpKwdB8jLMli6XPqCN5vvz0cfjj8/Of15asqdcVYDHvs0Vl+ekWZ2RVd+t0q7Re6KSxNWZGDhhmENFksfU4dFsuYMfDd7+rRintNWWFZcUX9fcQRw9fNnw9/+EM9+eo2VYQlZsK2ThgpFlBsGatToM86a/i0B3Vy/vntUx7Uxcknw9//Xl96Sy+tv1//+vrS7BJ94M/pIsakjJ1/od9bs2aeCTO5VRFTpoQrvKKRkfuZKsJyxBHwuc+15mnJm6xsUCk7cq+P3/1ON6SKrIg6BXqXXfRHJDxKdifsvXfxNlX41KfqTW/bbfVUBjt3d6zeOhhdwnLIITBvHhx1VNz2/e43/tSnYOON9QyJ3WLWLJg5s3vHi6GqsBjL7cknW63Duuh12XnppXoaRu98Z5z10MT5PvmkbgyNZpqYZqILjC5hmTBBT2gUS79bLEND3RUV0MPzxwzR3006ibFAtUnW+p3Y6ZDroglhGYn3ZZTQ5zVnj+l1qzMRR7cr0TzWW09/D2BPno7o90ZYoquMLoulLOlhGQw6tVjq5Le/hb/+NW6mx5FEaoQlLFLNmUcSlsGgqWFZqrDssu1TII8WkrAkLFLNmUd6WBKJONKzkrBIwpJHslgSiTiSsCQsUs2ZRxKWRCKOJCwJi1Rz5pEelkQijvSsJCySsOQRmso3kUgkEkGSsOSRBCWRSCRKk4Qlj2SxJBKJRGmSsOSRhCWRSCRKk4Qlj6aHVk/URz9MvJZIJICGhUVEdhORe0Rktogc7Vn/JhG5WUQWich7msxLJZKwDA6PPQYPP9zrXCQSCRoUFhEZA5wC7A5sAOwvIhs4mz0IHAj8sql8dEQSlsFhueVg5ZV7nYtEIkGzFsvWwGyl1P1KqQXAuUDbDDtKqTlKqduAJQ3mozrjxum5W/72t17nJJFIJAaGJh3TqwAPWf/nAttUSUhEDgYOBlh99dU7z1n8geGb3+ze8RKJRGIE0KSw+PxHlSYCV0qdBpwGMH369BEymXgiMQK4+WY9TUAiYdGksMwFVrP+rwo80uDxEolEt9l8c/1JJCyajLHMANYVkTVFZDywH3Bhg8dLJBKJRB/QmLAopRYBhwJXAHcB5yml7hCR40VkLwAR2UpE5gL7Aj8WkTuayk8ikUgkukOjb5UppS4FLnWWHWf9noF2kSUSiURihJDevE8kEolErSRhSSQSiUStJGFJJBKJRK0kYUkkEolErSRhSSQSiUStiFKD9SK7iDwBPFBx96nAkzVmZxBI5zw6SOc8OujknNdQSq1QZ2ZCDJywdIKIzFRKTe91PrpJOufRQTrn0cGgnHNyhSUSiUSiVpKwJBKJRKJWRpuwnNbrDPSAdM6jg3TOo4OBOOdRFWNJJBKJRPOMNoslkUgkEg2ThCWRSCQStTIqhEVEdhORe0Rktogc3ev81IWI/FREHheRf1jLlhORK0Xk3ux72Wy5iMgPsmtwm4hs0bucV0dEVhORa0TkLhG5Q0Q+nS0fsectIhNF5EYRuTU75//Jlq8pIn/PzvnX2bxHiMiE7P/sbP20Xua/E0RkjIjcIiIXZ/9HwznPEZHbRWSWiMzMlg1U+R7xwiIiY4BTgN2BDYD9RWSD3uaqNn4G7OYsOxq4Sim1LnBV9h/0+a+bfQ4GTu1SHutmEXCEUur1wLbAp7L7OZLP+xXgLUqpTYHNgN1EZFvgBODE7JyfBg7Ktj8IeFoptQ5wYrbdoPJp9HxOhtFwzgBvVkptZr2zMljlWyk1oj/AdsAV1v9jgGN6na8az28a8A/r/z3AStnvlYB7st8/Bvb3bTfIH+AC4G2j5byBycDNwDboN7DHZsv/U87Rk+ttl/0em20nvc57hXNdFV2JvgW4GJCRfs5Z/ucAU51lA1W+R7zFAqwCPGT9n5stG6m8Rin1b4Dse8Vs+Yi7Dpm7Y3Pg74zw885cQrOAx4ErgfuA+UrP1Art5/Wfc87WPwMs390c18JJwOeBJdn/5Rn55wyggD+KyE0icnC2bKDKd6MzSPYJ4lk2GvtYj6jrICKvAn4HfEYp9ayI7/T0pp5lA3feSqnFwGYi8mrgD8DrfZtl3wN/ziKyJ/C4UuomEdnJLPZsOmLO2eINSqlHRGRF4EoRuTtn274879FgscwFVrP+rwo80qO8dIPHRGQlgOz78Wz5iLkOIjIOLSrnKKV+ny0e8ecNoJSaD1yLji+9WkRM49A+r/+cc7Z+GeCp7ua0Y94A7CUic4Bz0e6wkxjZ5wyAUuqR7PtxdCNiawasfI8GYZkBrJv1JhkP7Adc2OM8NcmFwIez3x9GxyDM8g9lvUi2BZ4xpvUgIdo0OQO4Syn1PWvViD1vEVkhs1QQkUnAzuiA9jXAe7LN3HM21+I9wNUqc8APCkqpY5RSqyqlpqGf2auVUgcwgs8ZQESmiMhS5jewC/APBq189zrI06Vg2NuBf6L90sf2Oj81ntevgH8DC9Etl4PQfuWrgHuz7+WybQXdO+4+4HZgeq/zX/Gcd0Cb+rcBs7LP20fyeQObALdk5/wP4Lhs+VrAjcBs4DfAhGz5xOz/7Gz9Wr0+hw7Pfyfg4tFwztn53Zp97jD11aCV7zSkSyKRSCRqZTS4whKJRCLRRZKwJBKJRKJWkrAkEolEolaSsCQSiUSiVpKwJBKJRKJWkrAkEhkisjgbUdZ8ahsJW0SmiTUKdSIxkhkNQ7okErG8pJTarNeZSCQGnWSxJBIFZPNjnJDNiXKjiKyTLV9DRK7K5sG4SkRWz5a/RkT+kM2fcquIbJ8lNUZETs/mVPlj9hY9InKYiNyZpXNuj04zkaiNJCyJRItJjivsfda6Z5VSWwMno8esIvv9C6XUJsA5wA+y5T8A/qz0/ClboN+gBj1nxilKqQ2B+cC7s+VHA5tn6XyiqZNLJLpFevM+kcgQkeeVUq/yLJ+Dnmjr/mwAzEeVUsuLyJPouS8WZsv/rZSaKiJPAKsqpV6x0pgGXKn0RE2IyFHAOKXUV0XkcuB54HzgfKXU8w2faiLRKMliSSTiUIHfoW18vGL9XkwrxrkHerynLYGbrNF7E4mBJAlLIhHH+6zvv2W/r0ePvAtwAHBd9vsq4BD4zwRdS4cSFZEhYDWl1DXoSa1eDQyzmhKJQSK1jBKJFpOyWRoNlyulTJfjCSLyd3RjbP9s2WHAT0XkSOAJ4CPZ8k8Dp4nIQWjL5BD0KNQ+xgBni8gy6JFqT1R6zpVEYmBJMZZEooAsxjJdKfVkr/OSSAwCyRWWSCQSiVpJFksikUgkaiVZLIlEIpGolSQsiUQikaiVJCyJRCKRqJUkLIlEIpGolSQsiUQikaiV/w+AzC2Gyeu41wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaEAAAEWCAYAAADPZygPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XmcHFW5//HPlwRIAoFAFtaEQcmVNcQwsogsV9CLgAhurILKIuoF4g7KveB1ARSBiwoaBQREEGSV6wJE4SdXARM2A2FT2YMJkACByE3I8/vjnCaVpmemZzKVmnR/36/XvKbr1Pac6up66pyqrlZEYGZmVoWVqg7AzMzal5OQmZlVxknIzMwq4yRkZmaVcRIyM7PKOAmZmVllnITMbMBQ8mdJm/fT8p6R9I7+WFaDZa8qab6k9bsYf7Skm7qZ/zZJh5QRW9Xy+3iXpPE9TdtjEpL0qKQFeWP/Q9IFklbvn1D7R45x9yam21jSYknnLI+4qiJpPUnnSZol6SVJD0j6qqTVqo6tJ5I+LOmPkl6RdHOD8RMlTc/jp0uaWBgnSadJei7/fUuS+mPewnQH58/C/Py5WFwYnr8M9d5U0qIepjlV0sL8ntbe1/+WNKYX61mmA5+kDknXSHpW0guS7pV0UD+u+4PAUxFxf56nVuf5kuZJulVSZ1/jb5akIZJC0oZ15adK+jFARLwaEatHxNNlx9MbxRirEukLqGcCJ/c0bbMtofdGxOrAJOBtwIm9DUrS4N7OU4JDgbnAAZJWXZ4rXl71l7Q28CdgKLBDRAwH3gWMAN7ch+Ut7/fteeAs4NQGsawCXAv8FFgLuBC4NpcDHAXsC2wNTAD2Bj6xrPMWRcQl+cCzOvAe4OnacC4r24X5PR0JfAjoAKZJGr0c1g1wKfAgMBYYBXwMeLYfl380cHFd2YV5244GbgN+3o/rs35WOGZcBewlaWS3M0REt3/Ao8DuheFvA9fn12sC5wGzgKeArwOD8riPAv9LyobPA1/P5UcCM4GXgPuBSbl8feBKYA7wd+DYwjpPBi4HLsrz3Qd05nEXA4uBBcB84Ivd1OWvwCeBfwAfrBu3BXBjjvUfwJdz+SDgy3nel4DppA9gBxDA4MIybgaO6Kr+pCTwO+A50gf3EmBEYf6x+Y2bk6f5HrBqnn+rwnRjcn1HN6jj14G/ACt1sQ16G/cpwDxgy8L0o/P6x+ThvYG783R/BCb0tF81sd8dAdxcV/buvJ+pUPY4sEd+/UfgqMK4w4HblnXebmLcFXiyQflYUsJ7FvgbcHRh3I7AXcCLwDPAKbl8dn5f5ue/tzZY7qnAj+vKViZ9nmqfr9HAr/M+9HyOY7087jvAa8A/8zq+k8vPBZ7MMd0BbN9FfQUsBDbtZpvsBNye94U7gR27W3fdvMPy8kd1VWfSiXAAwwtl+wH35nX+Adi8MO4Z4B359WXAiYVxewCPdFGPIXk9G3b1HtRPQ/pc/ipvxz+RPjs3FebdC3g4x3kGKaEeUhj/CVKCfx74H2CDuvUcSToOzQXO7OY9eMN+Uhj3n6Tj60vADGCvwrZ/ERhfmHZD4BXyMaqJ7fx50rH5lUL5H4D9u/sc9eqakKSxwJ6kDxGks8lFwCbAW0kf9CMKs2xH+hCOAb4h6UOkhHIosAawD/CcpJWAXwL3ABsAuwGTJf1bYVn7kHaiEcB1pAM0EfER0sHkvZHORr/VRew7kTbqZaSEdmhh3HDgJuA3pGS4CTA1j/4scGCu9xrAx0lvTDOWqj/pQ3xKXsdmpIPVyTmGQcD1wGOkRLEBcFlEvJpjLnZjHEjauec0WOfuwFURsbjJGHuK+79IifHAwvgPA7dExGxJk4DzSR+gkcAPgetKamluAdwbee/O7s3ltfH3FMbdUzeur/M2Lb+PvyIltfVJB7ovS9olT/I94JsRsQYwHrgml+8MvBZLWlV30YSIWEj67OyUi1YCfgCMAzbOZWfmaT8H/Jl0wrF6HoZ0wNyK9P5dC1whaeUG6wpSgvlh7jat76rqyPX5CrA2qcfkGklrdbPuos2AFyOiYcsq71MfIR3w5uey7YFzSC2ykaST0msq6nmZQkog65BOdj9eGyFpXdJx53OkE4U5QGdh/AHAZOC9ef67SK32oveQjrOTgI9J2rUPMT4IvJ3UgDgNuEzSqIh4BfgFSx9nDgb+JyLmNbmd9yf1uhRbPjNJvQtd6y5D5c/ro6Q3fB7pAHkOqatnHeBVYGhh2gOB3+fXHwUer1vWb4HjGqxjuwbTngBckF+fzNJnFJsDC+pi3L2HevwYuCa/3oF0xjWmEPddXcz3IPC+BuUd9NyieLyHmPatrTfHNKe4vLrt8wS5dQNMAz7cxTIfpnDm3R9xkxLb3wrD/wscml+fC3ytwTbbpad9q4dt06gl9B+kxFwsuwQ4Ob9+jcJZOukgH6Tk3+d5u4lxV+paQsAuwMN1ZV8Fzs2v7yAdpEfWTbMpsKiHbdLwDJd08PpLF/NsD8wqDC919t1gepFOst7SxfhRpN6QmaQeiGnkVhtwEvCjuulvIZ8JN7Hu3YBHG9T5VdLx5zVSi3HHwvgLgK/UzfMYsF1+vawtoRfyumt//6RBSyi/Xgx0FJZxBvm4ReruvbkwblCuyyF5+PfAwYXxK5OOUesU1tNZGH8dMLk3+0kX0z4A/Fth332kMO4vwD692M4HNVj+d4Bzuouh2ZbQvhExIiI2iohPRcQCYKO8oWblC4bzSGfBxYukT9QtZyypOVlvI2D92nLysr6c34CaZwqvXwGGNHu2I2koqf/8EoCI+BOp9VS7oNpVXD2N68lS9Zc0RtJlkp6S9CLpTGdUYT2PRcQbLk5HxO3Ay8AukjYltdSu62KdzwHr9THehnGTuhCHStpO0kbARODqPG4j4HN1791YUitgKZK+XLiI/4M+xDWf1BotWoPUtdBo/BrA/EifhmWZtzc2AjrqtsdngXXz+MNI15weknR7XWu/rzYgnYEjabik8yU9nvexG1iyjzUk6QRJD0p6gdTVM6SreSLi2Yj4QkRsluv0EKmlDKnuh9TVvZMG+0IX5gLDG5RfHBEjSPv1X0n7X81GpJZmcZ2jSdukP2yRj30jcgxndTHduqQEXvzsPFZ4vX5xXES8RuoertkI+EGhDnNIvUzF1mb9MbDX1yAlHZ5vJqmtZxOWvNf/DxgkaQelm3bWI3Xt1uLraTvXHzcgvZ/zuotpWW7RfoJ0hjKq8CatERHFLoz6D/ATNL44/gTw9+KbHRHDI2LPJmPp6UCxH+mgco7SLZvPkDZerUuuq7i6G/dy/j+sULZu3TT1cZ2SyyZE6o45hLTj1tYzrpvEemGe/iPALyLin11MdxOwX+7ibKTXcUfq2ruc1GI8iHRNsHbwfgL4Rt17NywiLq1fcUR8M5Z0Nx3dRXzduQ+YIC1119qEXF4bX2z6b103rq/z9sYTwAMN9uX9ACJiZkTsTzpZOxu4Kt8c0dtkB7x+EXhvUt87wPGkA9fb8j72bpbsY9SvR9K7gGNIn5ERpG60BXXzNBQRs0ln+x1Kd14+QToDL9Z9tYg4s9G6G5gJDJfUVQKcTer2PaUwzRPAfzbY/65qsIiX6X6/XxbPkOo3tlA2rvB6VnFc/nzWH8A/WlePoRExvb8ClPQvwHdJrbK1c1J9hPxe5xOui1hynLksUndvLb6etnOj93czlu7mfoM+J6GImEU6y/qOpDUkrSTpzYW+70Z+DHxe0jZKNsln1ncAL0r6kqShkgZJ2lLS25oM5x/Am7oZfxjpusVWpLOoiaQLxBMlbUW6FrOupMlK9/4Pl7RdIeavSRqfY54gaWSk6zFPkc78Bkn6OD3ffTac3LUpaQPgC4Vxd5B21FMlraZ0i+iOhfEXkw4Uh5B2lK6cQUq4F+Zti6QNJJ0haUIf4wb4GanP9+D8uuZHwNG5laQc+15K19l6Lcc0BBgMrJS3Q+36xM2kLplj8/v077n8d/n/RcBnc33XJ/W//6Qf5u2NW3M9JufYB+d9ZlIuPzTvP6+RunqC1I0zm3QWOq7LJRdIWlnSFqSTg+GkhEZ+/QppHxvFG+9krf+sDCd1+8wBViFdAxzSzXpPl7R5fp/WJN3NNiMiXiadKH1I0m55/ND8unaw7/ZzmntYbiZdH+tqmntJZ+y1a0pTgGMkdeb9b3VJ+0ga1mD2u4G9JY3In79julpPb+WTwl8CX831nkD6rNRcB7xN0t55f/4CKeHX/AA4UdJbACStJekDyxDSoLz/1f5WIbWcFpPe65UkHU1qCRVdRLrmeyBLH2d6s53JdViNdMyd2tU0QO/vjqsbtyZL7qx5gXQx7YA87qPArQ3mOZp0zWA+6e6MWn/y+qTbP58hNctvq62XdE3op4VldFC4rgG8j9S9Ng/4fN36NiA1a7dqEMuvgNPz6y3zxpqbYzg+lvTdnsiSO0r+zJK7Yd6Ty+eR+j5vYelrK7fWrW8L0t1180kfiM9RuKZAOnO6hiV3z51dN/9N+f3o8jpFYVuen+vxEqnf9yRgWF/iLiz3EVK3zyp15Xvk7TKPlEivoHD3Um/+8vqj7u8nhfFvzdtwAenuq7cWxgn4Vo7x+fxa/TFvF7HuStd3x11OOujOJV1D2zmPuzy/ty+R+tz3LMx3GukAMQ+Y2GC5p5ISxkuks/qHSGe269XtQ7fmfewB4FMUrjWR+/1zXN8idalfTLoz6inS9aXXr6M0iGFKnn9+jvValr6jase8/rmkxHodsH6jdXex/A8AV9fVuf6OwF1yvGvn4X3y+/oC8DTp2s/QPK54TWg1Utfhi6Rj1efp37vj1iXd3NTV3XHvzfXv6u64w0mt7xdJXXk/6CoW6q5vNYix/jP0SB53et7+c/L+9obrdPn9e7DBcpvazoXpPwL8rKfPvPLEtgKQdD7peym9/p6W2Yogd5feTuqaur/qeNqRpJ8B90fE15dhGSIlrAMi4qFup3USWjEo3f56N+ns/e/VRmNmrUjSJqRegs0i4qmepu8PfnbcCkDS10hdl992AjKzMkj6Fqmb8r+WVwICt4TMzKxCbgmZmVllBsJDRZsyatSo6OjoqDoMM7MVxvTp05+NiOX1cNs+WWGSUEdHB9OmTas6DDOzFYakx3qeqlrujjMzs8o4CZmZWWWchMzMrDJOQmZmVhknITMzq4yTkJmZVcZJyMzMKuMkZGZmlXESMjOzyjgJmZlZZZyEzMysMk5CZmZWGSchMzOrjJOQmZlVxknIzMwq4yRkZmaVcRIyM7PKOAmZmVllnITMzKwyTkJmZlYZJyEzM6uMk5CZmVXGScjMzCrjJGRmZpVxEjIzs8qUnoQkHSdphqT7JE2uG/d5SSFpVNlxmJnZwFNqEpK0JXAksC2wNbC3pPF53FjgXcDjZcZgZmYDV9ktoc2A2yLilYhYBNwC7JfHnQl8EYiSYzAzswGq7CQ0A9hZ0khJw4A9gbGS9gGeioh7uptZ0lGSpkmaNmfOnJJDNTOz5W1wmQuPiJmSTgNuBOYD9wCLgK8A725i/inAFIDOzk63mMzMWkzpNyZExHkRMSkidgaeBx4FNgbukfQosCFwp6R1y47FzMwGluVxd9yY/H8c8H7googYExEdEdEBPAlMiohnyo7FzMwGllK747IrJY0EFgKfjoi5y2GdZma2Aig9CUXETj2M7yg7BjMzG5j8xAQzM6uMk5CZmVXGScjMzCrjJGRmZpVxEjIzs8o4CZmZWWWchMzMrDJOQmZmVhknITMzq4yTkJmZVcZJyMzMKuMkZGZmlXESMjOzyjgJmZlZZZyEzMysMk5CZmZWGSchMzOrjJOQmZlVxknIzMwq4yRkZmaVcRIyM7PKOAmZmVllnITMzKwyTkJmZlYZJyEzM6uMk5CZmVXGScjMzCrjJGRmZpVxEjIzs8o4CZmZWWWchMzMrDJNJSFJV0raS5KTlpmZ9Ztmk8q5wEHAw5JOlbRpiTGZmVmbaCoJRcRNEXEwMAl4FLhR0h8lfUzSymUGaGZmravp7jVJI4GPAkcAdwH/TUpKN5YSmZmZtbxmrwldBfwBGAa8NyL2iYifR8QxwOrdzHecpBmS7pM0OZd9W9IDku6VdLWkEf1RETMzW/E02xL6XkRsHhGnRMSs4oiI6Gw0g6QtgSOBbYGtgb0ljSe1nLaMiAnAQ8AJfY7ezMxWaM0moc2KLRZJa0n6VE/zALdFxCsRsQi4BdgvIm7IwwC3ARv2OmozM2sJzSahIyNiXm0gIuaSWjndmQHsLGmkpGHAnsDYumk+Dvy6qwVIOkrSNEnT5syZ02SoZma2omg2Ca0kSbUBSYOAVbqbISJmAqeRut9+A9wD1FpASPpKHr6km2VMiYjOiOgcPXp0k6GamdmKotkk9Fvgckm7SXoncCkpsXQrIs6LiEkRsTPwPPAwgKTDgL2BgyMi+ha6mZmt6AY3Od2XgE8AnwQE3AD8uKeZJI2JiNmSxgHvB3aQtEde3i4R8UrfwjYzs1bQVBKKiMWkpyac28vlX5m/X7QQ+HREzJX0PWBV0hdeId28cHQvl2tmZi2gqSSUb60+BdgcGFIrj4g3dTdfROzUoGyTXsZoZmYtqtlrQheQWkGLgH8FLgIuLisoMzNrD80moaERMRVQRDwWEScD7ywvLDMzawfN3pjwz/wzDg9L+nfgKWBMeWGZmVk7aLYlNJn03LhjgW2AQ4DDygrKzMzaQ48tofzF1A9HxBeA+cDHSo/KzMzaQo8toYh4Ddim+MQEMzOz/tDsNaG7gGslXQG8XCuMiKtKicrMzNpCs0lobeA5lr4jLgAnITMz67Nmn5jg60BmZtbvmn1iwgWkls9SIuLj/R6RmZm1jWa7464vvB4C7Ac83f/hmJlZO2m2O+7K4rCkS4GbSonIzMzaRrNfVq03HhjXn4GYmVn7afaa0EssfU3oGdJvApmZmfVZs91xw8sOxMzM2k9T3XGS9pO0ZmF4hKR9ywvLzMzaQbPXhE6KiBdqAxExDzipnJDMzKxdNJuEGk3X7O3dZmZmDTWbhKZJOkPSmyW9SdKZwPQyAzMzs9bXbBI6Bvg/4OfA5cAC4NNlBWVmZu2h2bvjXgaOLzkWMzNrM83eHXejpBGF4bUk/ba8sMzMrB002x03Kt8RB0BEzAXGlBOSmZm1i2aT0GJJrz+mR9JGNHiqtpmZWW80e5v1V4BbJd2Sh3cGPlFOSGZm1i6avTHhN5ImAdsDAj4TEc+WGpmZmbW8pp+iHRHPRsT1wP3A0ZJmlBeWmZm1g2bvjltP0mRJdwD3AYOAA0uNzMzMWl633XGSjiQlmw1JX1I9Arg2Ir66HGLrH5Mnw913Vx2FmVnfTJwIZ51VdRSl6ema0PeBPwEHRcQ0AEm+K87MzPpFT0lofeBDwBmS1iG1hlYuPar+1MJnEGZmK7purwnlmxHOjYidgd2AF4DZkmZK+uZyidDMzFpWb+6OezIiTo+IbYB9gVfLC8vMzNpBn34TKCIeBFacmxPMzGxAarol1FeSjpM0Q9J9kibnsrXzQ1Efzv/XKjsOMzMbeEpNQpK2BI4EtgW2BvaWNJ70sxBTI2I8MBX/TISZWVtq9suqU5spa2Az4LaIeCUiFgG3APsB7wMuzNNcSLrGZGZmbabbJCRpiKS1gVH5N4TWzn8dpNu3ezID2FnSSEnDgD2BscA6ETELIP/3z0KYmbWhnm5M+AQwmZRwppMeXgrwIumLrN2KiJmSTgNuBOYD9wCLmg1O0lHAUQDjxo3rYWozM1vRKKLnByBIOiYivrvMK0vfLXoSOA7YNSJmSVoPuDki3tLdvJ2dnTFt2rRlDcHMrG1Imh4RnVXH0Z1mf8rhu5LeDnQU54mIi3qaV9KYiJidfxTv/cAOwMbAYcCp+f+1vQ/dzMxWdE0lIUkXA28G7gZey8UB9JiEgCsljQQWAp+OiLmSTgUul3Q48Djp0UBmZtZmmv2yaieweTTTd1cnInZqUPYc6TFAZmbWxpr9ntAMYN0yAzEzs/bTbEtoFHB//lG7158ZFxH7lBKVmZm1hWaT0MllBmFmZu2p2bvjbpG0ETA+Im7KXzwdVG5oZmbW6pp9bM+RwC+AH+aiDYBrygrKzMzaQ7M3Jnwa2JH0pAQi4mH8qB0zM1tGzSahVyPi/2oDkgaTvidkZmbWZ80moVskfRkYKuldwBXAL8sLy8zM2kGzSeh4YA7wF9JDTX8FnFhWUGZm1h6avUV7KHB+RPwIQNKgXPZKWYGZmVnra7YlNJWUdGqGAjf1fzhmZtZOmk1CQyJifm0gvx5WTkhmZtYumk1CL0uaVBuQtA2woJyQzMysXTR7Teg44ApJT+fh9YD9ywnJzMzaRY9JSNJKwCrApsBbSD/x/UBELCw5NjMza3E9JqGIWCzpOxGxA+knHczMzPpFs9eEbpD0AUkqNRozM2srzV4T+iywGvCapAWkLrmIiDVKi8zMzFpesz/lMLzsQMzMrP00+1MOknSIpP/Iw2MlbVtuaGZm1uqavSZ0DrADcFAeng98v5SIzMysbTR7TWi7iJgk6S6AiJgraZUS4zIzszbQbEtoYX5oaQBIGg0sLi0qMzNrC80mobOBq4Exkr4B3Ap8s7SozMysLTR7d9wlkqYDu5Fuz943ImaWGpmZmbW8bpOQpCHA0cAmpB+0+2FELFoegZmZWevrqTvuQqCTlIDeA5xeekRmZtY2euqO2zwitgKQdB5wR/khmZlZu+ipJfT6k7LdDWdmZv2tp5bQ1pJezK8FDM3DfnacmZkts26TUEQMWl6BmJlZ+2n2e0JmZmb9zknIzMwq4yRkZmaVcRIyM7PKlJ6EJH1G0n2SZki6VNIQSbtJulPS3ZJulbRJ2XGYmdnAU2oSkrQBcCzQGRFbAoOAA4BzgYMjYiLwM+DEMuMwM7OBaXl0xw0mfb9oMDAMeJr0kxC17xitmcvMzKzNNPujdn0SEU9JOh14HFgA3BARN0g6AviVpAXAi8D2jeaXdBRwFMC4cePKDNXMzCpQdnfcWsD7gI2B9YHVJB0CfAbYMyI2BC4Azmg0f0RMiYjOiOgcPXp0maGamVkFyu6O2x34e0TMiYiFwFXAjsDWEXF7nubnwNtLjsPMzAagspPQ48D2koZJEulH8e4H1pT0L3madwH+gTwzszZU9jWh2yX9ArgTWATcBUwBngSulLQYmAt8vMw4zMxsYCo1CQFExEnASXXFV+c/MzNrY35igpmZVcZJyMzMKuMkZGZmlXESMjOzyjgJmZlZZZyEzMysMk5CZmZWGSchMzOrjJOQmZlVxknIzMwq4yRkZmaVcRIyM7PKOAmZmVllnITMzKwyTkJmZlYZJyEzM6uMk5CZmVXGScjMzCrjJGRmZpVxEjIzs8o4CZmZWWWchMzMrDJOQmZmVhknITMzq4yTkJmZVcZJyMzMKuMkZGZmlXESMjOzyjgJmZlZZZyEzMysMk5CZmZWGSchMzOrjJOQmZlVxknIzMwq4yRkZmaVKT0JSfqMpPskzZB0qaQhSr4h6SFJMyUdW3YcZmY28Awuc+GSNgCOBTaPiAWSLgcOAASMBTaNiMWSxpQZh5mZDUylJqHCOoZKWggMA54Gvg4cFBGLASJi9nKIw8zMBphSu+Mi4ingdOBxYBbwQkTcALwZ2F/SNEm/ljS+0fySjsrTTJszZ06ZoZqZWQVKTUKS1gLeB2wMrA+sJukQYFXgnxHRCfwIOL/R/BExJSI6I6Jz9OjRZYZqZmYVKPvGhN2Bv0fEnIhYCFwFvB14ErgyT3M1MKHkOMzMbAAqOwk9DmwvaZgkAbsBM4FrgHfmaXYBHio5DjMzG4BKvTEhIm6X9AvgTmARcBcwBRgKXCLpM8B84Igy4zAzs4Gp9LvjIuIk4KS64leBvcpet5mZDWx+YoKZmVXGScjMzCrjJGRmZpVxEjIzs8o4CZmZWWWchMzMrDJOQmZmVhlFRNUxNEXSHOCxPsw6Cni2n8MZ6Fzn9tGO9Xadm7dRRAzoB2+uMEmoryRNyw9KbRuuc/tox3q7zq3F3XFmZlYZJyEzM6tMOyShKVUHUAHXuX20Y71d5xbS8teEzMxs4GqHlpCZmQ1QTkJmZlaZlk5CkvaQ9KCkRyQdX3U8/UXS+ZJmS5pRKFtb0o2SHs7/18rlknR23gb3SppUXeR9J2mspN9LminpPknH5fKWrbekIZLukHRPrvNXc/nGkm7Pdf65pFVy+ap5+JE8vqPK+JeFpEGS7pJ0fR5u6TpLelTSXyTdLWlaLmvZfbuoZZOQpEHA94H3AJsDB0ravNqo+s1PgD3qyo4HpkbEeGBqHoZU//H57yjg3OUUY39bBHwuIjYDtgc+nd/PVq73q8A7I2JrYCKwh6TtgdOAM3Od5wKH5+kPB+ZGxCbAmXm6FdVxwMzCcDvU+V8jYmLh+0CtvG8vEREt+QfsAPy2MHwCcELVcfVj/TqAGYXhB4H18uv1gAfz6x8CBzaabkX+A64F3tUu9QaGAXcC25G+OT84l7++nwO/BXbIrwfn6VR17H2o64akg+47gesBtUGdHwVG1ZW1xb7dsi0hYAPgicLwk7msVa0TEbMA8v8xubzltkPucnkrcDstXu/cLXU3MBu4EfgrMC8iFuVJivV6vc55/AvAyOUbcb84C/gisDgPj6T16xzADZKmSzoql7X0vl0zuOoASqQGZe14P3pLbQdJqwNXApMj4kWpUfXSpA3KVrh6R8RrwERJI4Crgc0aTZb/r/B1lrQ3MDsipkvatVbcYNKWqXO2Y0Q8LWkMcKOkB7qZtlXqDLTwNSHS2cHYwvCGwNMVxbI8/EPSegD5/+xc3jLbQdLKpAR0SURclYtbvt4AETEPuJl0PWyEpNoJZLFer9c5j18TeH75RrrMdgT2kfQocBmpS+4sWrvORMTT+f9s0snGtrTJvt3KSejPwPh8V80qwAHAdRXHVKbrgMPy68NI10xq5YfmO2q2B16oNfFXJEpNnvOAmRFxRmFUy9Zb0ujcAkLSUGB30sX63wMfzJPV17m2LT4I/C7yRYMVRUScEBEbRkQH6TP7u4g4mBaus6TVJA2vvQbeDcyghfftpVR9UarMP2BP4CFSP/pXqo6nH+vu6j47AAACXklEQVR1KTALWEg6Kzqc1A8+FXg4/187TyvSXYJ/Bf4CdFYdfx/r/A5Sl8O9wN35b89WrjcwAbgr13kG8J+5/E3AHcAjwBXAqrl8SB5+JI9/U9V1WMb67wpc3+p1znW7J//dVztWtfK+XfzzY3vMzKwyrdwdZ2ZmA5yTkJmZVcZJyMzMKuMkZGZmlXESMjOzyjgJWVuT9Fp+cnHtr9+eti6pQ4UnnZvZG7XyY3vMmrEgIiZWHYRZu3JLyKyB/Psup+Xf87lD0ia5fCNJU/PvuEyVNC6XryPp6vzbP/dIente1CBJP8q/B3RDfvIBko6VdH9ezmUVVdOsck5C1u6G1nXH7V8Y92JEbAt8j/T8MvLriyJiAnAJcHYuPxu4JdJv/0wiffMd0m++fD8itgDmAR/I5ccDb83LObqsypkNdH5igrU1SfMjYvUG5Y+SflDub/nBqc9ExEhJz5J+u2VhLp8VEaMkzQE2jIhXC8voAG6M9KNkSPoSsHJEfF3Sb4D5wDXANRExv+Sqmg1IbgmZdS26eN3VNI28Wnj9Gkuuw+5Fev7XNsD0whOizdqKk5BZ1/Yv/P9Tfv1H0tOdAQ4Gbs2vpwKfhNd/iG6NrhYqaSVgbET8nvTjbSOAN7TGzNqBz76s3Q3Nv1xa85uIqN2mvaqk20knawfmsmOB8yV9AZgDfCyXHwdMkXQ4qcXzSdKTzhsZBPxU0pqkJyKfGen3gszajq8JmTWQrwl1RsSzVcdi1srcHWdmZpVxS8jMzCrjlpCZmVXGScjMzCrjJGRmZpVxEjIzs8o4CZmZWWX+P7rMRhwfmYVBAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# gradient descent\n",
    "detailed_logger = False\n",
    "main_logger = True\n",
    "main_logger_output_epochs = 100\n",
    "L2 = False\n",
    "Dropout = False\n",
    "momentum = False\n",
    "adam = True\n",
    "hidden_layer_relu = True\n",
    "hidden_layer_tanh = False\n",
    "hidden_layer_sigmoid = False\n",
    "\n",
    "# hyber-parameters\n",
    "alpha = .01;\n",
    "epsilon = .85\n",
    "keep_prob = .9\n",
    "number_of_epochs = 500\n",
    "batch_size = 50\n",
    "momentum_coef = .9\n",
    "RMSProp_coef = .5\n",
    "epsilon = 1e-20\n",
    "t = 0\n",
    "\n",
    "# copy initalization\n",
    "W = Weights.copy()\n",
    "B = Bias.copy()\n",
    "\n",
    "# data arrays\n",
    "cost_array = []\n",
    "accuracy_array = []\n",
    "interation_array = []\n",
    "\n",
    "# rename\n",
    "X_train = np.float64(training_images).copy()\n",
    "Y_train = np.float64(training_labels).copy()\n",
    "\n",
    "X_test = np.float64(testing_images).copy()\n",
    "Y_test = np.float64(testing_labels).copy()\n",
    "\n",
    "#m = size\n",
    "m = number_of_training_images\n",
    "\n",
    "def model(W, B, A):\n",
    "    return np.dot(W, A) + B\n",
    "\n",
    "def activation_relu(Z):\n",
    "    Z = np.where(~np.isnan(Z), Z, 0)\n",
    "    Z = np.where(~np.isinf(Z), Z, 0)\n",
    "    return np.where(Z > 0, Z, 0)\n",
    "\n",
    "def activation_tanh(Z):\n",
    "    return np.tanh(Z)\n",
    "\n",
    "def activation_sigmoid(Z):\n",
    "     return 1/(1 + np.exp(-Z))\n",
    "\n",
    "def loss(A, Y):\n",
    "    epsilon = 1e-20\n",
    "    return np.where((Y == 1), np.multiply(-Y, np.log(A + epsilon)), -np.multiply((1 - Y), np.log(1 - A + epsilon)))\n",
    "    #return np.multiply(-Y, np.log(A)) - np.multiply((1 - Y), np.log(1 - A)) \n",
    "    \n",
    "def cost(L):\n",
    "    return np.multiply(1/L.shape[1], np.sum(L))\n",
    "\n",
    "def cost_L2(L, W, epsilon):\n",
    "    L2 = np.multiply(epsilon/(2*W.shape[1]), np.multiply(W[len(W)-3], W[len(W)-3]).sum() + np.multiply(W[len(W)-2], W[len(W)-2]).sum() + np.multiply(W[len(W)-1], W[len(W)-1]).sum())\n",
    "    J = cost(L)\n",
    "    return L2 + J\n",
    "\n",
    "def prediction(A):\n",
    "    return np.where(A >= 0.5, 1, 0)\n",
    "    \n",
    "def accuracy(prediction, Y):\n",
    "    return 100 - np.multiply(100/Y.shape[0], np.sum(np.absolute(Y - prediction)))  \n",
    "    \n",
    "def forward_propagation_return_layers(W, B, A, A_layers, Z_layers, layer, D, keep_prob):\n",
    "    if(layer < len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        Z_layers.append(Z)\n",
    "        if(hidden_layer_relu == True):\n",
    "            A = activation_relu(Z)\n",
    "        elif(hidden_layer_tanh == True):\n",
    "            A = activation_tanh(Z)\n",
    "        elif(hidden_layer_sigmoid == True): \n",
    "            A = activation_sigmoid(Z)\n",
    "        if(Dropout == True):\n",
    "            _D = np.float64(np.where(np.random.uniform(0, 1, A.shape) < keep_prob, 1, 0))\n",
    "            D.append(_D)\n",
    "            A = np.multiply(A, _D)\n",
    "        A_layers.append(A)\n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Training Data: ' + str(layer))\n",
    "        A_layers, Z_layers, D = forward_propagation_return_layers(W, B, A, A_layers, Z_layers, layer, D, keep_prob)\n",
    "    elif(layer == len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        Z_layers.append(Z)\n",
    "        A = activation_sigmoid(Z)\n",
    "        if(Dropout == True):\n",
    "            _D = np.float64(np.where(np.random.uniform(0, 1, A.shape) < keep_prob, 1, 0))\n",
    "            D.append(_D)\n",
    "            A = np.multiply(A, _D)\n",
    "        A_layers.append(A)\n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Training Data: ' + str(layer))\n",
    "            print('Forward Propagation Training Data Complete')\n",
    "    return A_layers, Z_layers, D\n",
    "\n",
    "def forward_propagation(W, B, A, layer):\n",
    "    if(layer < len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        if(hidden_layer_relu == True):\n",
    "            A = activation_relu(Z)\n",
    "        elif(hidden_layer_tanh == True):\n",
    "            A = activation_tanh(Z)\n",
    "        elif(hidden_layer_sigmoid == True): \n",
    "            A = activation_sigmoid(Z)\n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Testing Data: ' + str(layer))\n",
    "        A = forward_propagation(W, B, A, layer)\n",
    "    elif(layer == len(W) - 1):\n",
    "        Z = model(W[layer], B[layer], A)\n",
    "        A = activation_sigmoid(Z)  \n",
    "        layer = layer + 1\n",
    "        if(detailed_logger == True):\n",
    "            print('Forward Layer Testing Data: ' + str(layer))\n",
    "            print('Forward Propagation Testing Data Complete')\n",
    "    return A\n",
    "\n",
    "def dZ(dZ, W, Z):\n",
    "    Z = np.where(~np.isnan(Z), Z, 0)\n",
    "    W = np.where(~np.isnan(W), W, 0)\n",
    "    dZ = np.where(~np.isnan(dZ), dZ, 0)\n",
    "    Z = np.where(~np.isinf(Z), Z, 0)\n",
    "    W = np.where(~np.isinf(W), W, 0)\n",
    "    dZ = np.where(~np.isinf(dZ), dZ, 0)\n",
    "    if(hidden_layer_relu == True):\n",
    "        return np.multiply(np.dot(np.transpose(W), dZ), np.where(Z > 0, 1, 0))\n",
    "    elif(hidden_layer_tanh == True):\n",
    "        A = activation_tanh(Z)\n",
    "        return np.multiply(np.dot(np.transpose(W), dZ), 1- np.multiply(A, A))\n",
    "    elif(hidden_layer_sigmoid == True): \n",
    "        A = activation_sigmoid(Z)\n",
    "        return np.multiply(np.dot(np.transpose(W), dZ), np.multiply(A, (1-A)))\n",
    "\n",
    "def dW(dZ, A):\n",
    "    return np.multiply(1/dZ.shape[1], np.dot(dZ, np.transpose(A)))\n",
    "\n",
    "def dW_L2(dZ, A, W, epsilon):\n",
    "    return np.multiply(epsilon/Z.shape[1], W) + dW(dZ, A)\n",
    "\n",
    "def dB(dZ):\n",
    "    return np.multiply(1/dZ.shape[1], np.sum(dZ))\n",
    "\n",
    "def backward_propagation(W, B, Y, A_layers, Z_layers, _dZ, alpha, epsilon, layer, D, V_dW, V_dB, R_dW, R_dB, t):\n",
    "    if(layer >= 0):\n",
    "        if(layer == len(W) - 1):\n",
    "            _dZ = A_layers[layer+1] - Y\n",
    "        elif(layer >= 0):\n",
    "            _dZ = dZ(_dZ, W[layer+1], Z_layers[layer])\n",
    "        if(Dropout == True):\n",
    "            _dZ = np.multiply(_dZ, D[layer])\n",
    "        if(L2 == True):\n",
    "            _dW = dW_L2(_dZ, A_layers[layer], W[layer], epsilon)\n",
    "        else:\n",
    "            _dW = dW(_dZ, A_layers[layer])\n",
    "        _dB = dB(_dZ)\n",
    "        if(adam == True):\n",
    "            epsilon = 1e-6\n",
    "\n",
    "            # ADAM - RMSProp + Momentum\n",
    "            V_dW[layer] = np.multiply(momentum_coef, V_dW[layer]) + np.multiply(1-momentum_coef, _dW)\n",
    "            V_dB[layer] = np.multiply(momentum_coef, V_dB[layer]) + np.multiply(1-momentum_coef, _dB)\n",
    "            R_dW[layer] = np.multiply(RMSProp_coef, R_dW[layer]) + np.multiply(1-RMSProp_coef, np.multiply(_dW, _dW))\n",
    "            R_dB[layer] = np.multiply(RMSProp_coef, R_dB[layer]) + np.multiply(1-RMSProp_coef, np.multiply(_dB, _dB))\n",
    "            \n",
    "            # index decay in bias correction\n",
    "            t = t + 1\n",
    "            \n",
    "            # correct bias for initial rounds\n",
    "            V_dW[layer] = np.multiply(V_dW[layer], 1/(1-np.power(momentum_coef, t)))\n",
    "            V_dB[layer] = np.multiply(V_dB[layer], 1/(1-np.power(momentum_coef, t)))\n",
    "            R_dW[layer] = np.multiply(R_dW[layer], 1/(1-np.power(RMSProp_coef, t)))\n",
    "            R_dB[layer] = np.multiply(R_dB[layer], 1/(1-np.power(RMSProp_coef, t)))\n",
    "            \n",
    "            val1 = 1/(np.sqrt(R_dW[layer])+ epsilon)\n",
    "            val2 = 1/(np.sqrt(R_dB[layer])+ epsilon)\n",
    "                      \n",
    "            W[layer] = W[layer] - np.multiply(alpha, np.multiply(V_dW[layer], val1 ))\n",
    "            B[layer] = B[layer] - np.multiply(alpha, np.multiply(V_dB[layer], val2 ))\n",
    "        elif(momentum == True):\n",
    "            V_dW[layer] = np.multiply(momentum_coef, V_dW[layer]) + np.multiply(alpha, _dW)\n",
    "            V_dB[layer] = np.multiply(momentum_coef, V_dB[layer]) + np.multiply(alpha, _dB)\n",
    "            W[layer] = W[layer] - V_dW[layer]\n",
    "            B[layer] = B[layer] - V_dB[layer]           \n",
    "        else:\n",
    "            W[layer] = W[layer] - np.multiply(alpha, _dW)\n",
    "            B[layer] = B[layer] - np.multiply(alpha, _dB)\n",
    "        if(detailed_logger == True):\n",
    "            print('Backward Layer: ' + str(layer))\n",
    "        layer = layer - 1\n",
    "        W, B, t = backward_propagation(W, B, Y, A_layers, Z_layers, _dZ, alpha, epsilon, layer, D, V_dW, V_dB, R_dW, R_dB, t)\n",
    "        if(detailed_logger == True):\n",
    "            print('Backward Propagation Complete')\n",
    "    return W, B, t\n",
    "    \n",
    "\n",
    "def shuffle(X, Y, number_of_training_images):\n",
    "    random_array = np.random.permutation(np.arange(number_of_training_images))\n",
    "    return X[:, random_array], Y[random_array]\n",
    "    \n",
    "start_time = time.time()   \n",
    "# main loop\n",
    "for epoch in range(1, number_of_epochs):\n",
    "    \n",
    "    # logger\n",
    "    if(main_logger == True and epoch % main_logger_output_epochs == 0):\n",
    "        print('Main Loop Epoch: ' + str(epoch))\n",
    "    \n",
    "    # saftey check\n",
    "    if(adam == True and momentum == True):\n",
    "        print(\"ERROR! Please Select Either Adam OR Momentum OR Neither, Not Both.\")\n",
    "        break\n",
    "\n",
    "    # saftey check\n",
    "    if(hidden_layer_relu + hidden_layer_tanh + hidden_layer_sigmoid != 1):\n",
    "        print(\"ERROR! Please Select Only 1 Hidden Layer Activation Function\")\n",
    "        break\n",
    "    \n",
    "    # shuffle data\n",
    "    X, Y = shuffle(X_train.copy(), Y_train.copy(), number_of_training_images)\n",
    "    number_of_batches = int(np.floor(number_of_training_images/batch_size))\n",
    "    split_index = number_of_batches*batch_size\n",
    "\n",
    "    # parse into minibatches\n",
    "    X_minibatches = np.split(X[:, 0:split_index], number_of_batches, axis=1)\n",
    "    if not(split_index == number_of_training_images):\n",
    "        X_left_over_portion  = X[:, split_index:number_of_training_images]\n",
    "        X_minibatches.append(X_left_over_portion)\n",
    "    \n",
    "    Y_minibatches = np.split(Y[0:split_index], number_of_batches, axis=0)\n",
    "    if not(split_index == number_of_training_images):\n",
    "        Y_left_over_portion  = Y[split_index:number_of_training_images]\n",
    "        Y_minibatches.append(Y_left_over_portion)\n",
    "    \n",
    "    number_of_minibatches = len(Y_minibatches)\n",
    "    \n",
    "    # logger\n",
    "    if(main_logger == True and epoch % main_logger_output_epochs == 0):\n",
    "        print('Number Of Minibatches: ' + str(number_of_minibatches))\n",
    "\n",
    "    for index in range(0, number_of_minibatches-1):\n",
    "        X_minibatch = X_minibatches[index]\n",
    "        Y_minibatch = Y_minibatches[index]\n",
    "\n",
    "        # forward propogation training data set\n",
    "        A_layers, Z_layers, D = forward_propagation_return_layers(W, B, X_minibatch, [X_minibatch], [], 0, [], keep_prob)\n",
    "        L = loss(A_layers[len(A_layers) - 1], Y_minibatch)\n",
    "        if(L2 == True):\n",
    "            C = cost_L2(L, W, epsilon) \n",
    "        else:\n",
    "            C = cost(L) \n",
    "\n",
    "        # backpropogation\n",
    "        W, B, t = backward_propagation(W, B, Y_minibatch, A_layers, Z_layers, 0, alpha, epsilon, len(W) - 1, D,  V_dW, V_dB, R_dW, R_dB, t)\n",
    "        \n",
    "    if(epoch % main_logger_output_epochs == 0):\n",
    "        print('Cost: ' + str(C))\n",
    "\n",
    "    # forward propogation test data set\n",
    "    A_test = forward_propagation(W, B, X_test, 0)\n",
    "\n",
    "    # accuracy\n",
    "    _prediction = prediction(A_test)    \n",
    "    _accuracy = accuracy(_prediction, Y_test)  \n",
    "\n",
    "    # storage for plotting\n",
    "    cost_array.append(C)\n",
    "    accuracy_array.append(_accuracy)\n",
    "    interation_array.append(epoch)\n",
    "\n",
    "\n",
    "end_time = time.time()\n",
    "run_time = end_time - start_time\n",
    "    \n",
    "print('')\n",
    "print('Results:')\n",
    "print('')\n",
    "    \n",
    "print('')\n",
    "print('Run Time: ' + str(run_time) + ' seconds')\n",
    "print('Cost: ' + str(C))    \n",
    "print('Accuracy: ' + str(_accuracy) + ' %')    \n",
    "print('')\n",
    "print('')\n",
    "\n",
    "\n",
    "pyplot.figure()\n",
    "pyplot.plot(interation_array, cost_array, 'red')\n",
    "pyplot.title('Learning Curve - ' + str(len(X[0])) + ' Training Data Set (Relu Hidden Layer)')\n",
    "pyplot.xlabel('Epochs')\n",
    "pyplot.ylabel('Cost')\n",
    "pyplot.show()\n",
    "\n",
    "# plot percent accuracy curve\n",
    "pyplot.figure()\n",
    "pyplot.plot(interation_array, accuracy_array, 'red')\n",
    "pyplot.title('Percent Accuracy Curve - ' + str(len(X_test[0])) + ' Test Data Set (Relu Hidden Layer)')\n",
    "pyplot.xlabel('Epochs')\n",
    "pyplot.ylabel('Percent Accuracy')\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As illustrated, after 500 epochs with minibatches of 50 we reached a cost of approximately 0.23; however, our test data accuracy is stuck at approximately 90%. This indicates that we overfit the training data, since no 9s were detected in the test data. This lines up with what we would intuitively expect. Because the momentum hyper-paramter was high and the RMSProp was low, the weights with consistent large gradients were focused on and not penalized as much as when the RMSProp hyper-parameter was high. This has likely caused the network to over focus on specific features from the training data, and therefore, overfit the training data and perform poorly on the test data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have illustrated how the minibatch stochastic gradient descent with ADAM provides many regularization and convergence benefits for training neural networks. When the momentum and RMSProp hyper-paramters are high, this technique provides a good form of regularization by preventing the network from getting stuck local minima while also preventing the network from over focusing on specific features of the training data. The networks also converge faster. We have explored how specific variations in the ADAM (momentum and RMSProp) hyper-paramters can alter the regularization effect of network."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}