8-convolutional-neural-networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment8\n",
    "## CS-5891-01 Special Topics Deep Learning\n",
    "## Ronald Picard\n",
    "\n",
    "In this notebook we will walk through the design, training, and testing of convolutional neural networks (CNNs). These neural networks will be used for image classification. Classification will be performed on images of handwritten single numerical digits (0-9). \n",
    "\n",
    "The data set we will be using is the MNIST data set. This is a very popular data set amoung the machine learning community. The data set contains 60,000 images, and each image contains a handwritten numerical digit. Each of the images have been provided with a truth label that corresponds to the handwritten digit within the image from the set {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}.\n",
    "\n",
    "In order to do this we will be utilize Keras, which is a high-level API for Tensorflow, which is Google's machine learning library.\n",
    "\n",
    "Convolutional neural networks differ from traditional feed-forward networks in that they replace the matrix multiplication operator with a convolution operator. This provides sparse connections and a large reduction in the number of paramters that need to be trained. In CNNs, the weights that need to be trained are kernel elements.\n",
    "\n",
    "We wish to explore the impact of adjustments in the CNN architecture, optimization algorithms, regularization\n",
    "techniques, and hyperparameters. Therefore, we will start with a base network and adjust from there. \n",
    "\n",
    "We will start with the following base network:\n",
    "\n",
    "1. CNN architecture: \n",
    "    a. Convolution layer (kernel 3X3) then MaxPooling (3X3), Relu\n",
    "    b. Convolution layer (kernel 2X2) then MaxPooling (2X2), Relu, Dropout (Keep Probability 0.25)\n",
    "    c. Dense layer (units 128), Relu, Dropout (Keep Probability 0.5)\n",
    "    d. Output layer (units 128), Softmax\n",
    "2. Dropout: \n",
    "    a. As specified in 1\n",
    "3. Optimization algorithm: \n",
    "    a. AdaDelta, which is a more robust extension of Adagrad that adapts learning rates based on a moving window of gradient updates.\n",
    "4. Minibatches: \n",
    "    a. 256\n",
    "5. Epochs:\n",
    "    a. 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10000, 28, 28, 1)\n",
      "(5000, 10)\n",
      "Train on 10000 samples, validate on 5000 samples\n",
      "Epoch 1/20\n",
      " - 2s - loss: 0.9949 - acc: 0.6815 - val_loss: 0.4530 - val_acc: 0.8600\n",
      "Epoch 2/20\n",
      " - 1s - loss: 0.3271 - acc: 0.9011 - val_loss: 0.2929 - val_acc: 0.8978\n",
      "Epoch 3/20\n",
      " - 1s - loss: 0.2172 - acc: 0.9331 - val_loss: 0.1478 - val_acc: 0.9540\n",
      "Epoch 4/20\n",
      " - 1s - loss: 0.1755 - acc: 0.9486 - val_loss: 0.1114 - val_acc: 0.9662\n",
      "Epoch 5/20\n",
      " - 1s - loss: 0.1371 - acc: 0.9579 - val_loss: 0.1619 - val_acc: 0.9518\n",
      "Epoch 6/20\n",
      " - 1s - loss: 0.1235 - acc: 0.9617 - val_loss: 0.0977 - val_acc: 0.9654\n",
      "Epoch 7/20\n",
      " - 1s - loss: 0.1080 - acc: 0.9685 - val_loss: 0.0807 - val_acc: 0.9752\n",
      "Epoch 8/20\n",
      " - 1s - loss: 0.0914 - acc: 0.9726 - val_loss: 0.0703 - val_acc: 0.9780\n",
      "Epoch 9/20\n",
      " - 1s - loss: 0.0887 - acc: 0.9731 - val_loss: 0.0822 - val_acc: 0.9748\n",
      "Epoch 10/20\n",
      " - 1s - loss: 0.0717 - acc: 0.9771 - val_loss: 0.0659 - val_acc: 0.9796\n",
      "Epoch 11/20\n",
      " - 2s - loss: 0.0635 - acc: 0.9811 - val_loss: 0.0811 - val_acc: 0.9772\n",
      "Epoch 12/20\n",
      " - 2s - loss: 0.0629 - acc: 0.9801 - val_loss: 0.0867 - val_acc: 0.9714\n",
      "Epoch 13/20\n",
      " - 2s - loss: 0.0564 - acc: 0.9836 - val_loss: 0.0653 - val_acc: 0.9778\n",
      "Epoch 14/20\n",
      " - 1s - loss: 0.0487 - acc: 0.9846 - val_loss: 0.0589 - val_acc: 0.9794\n",
      "Epoch 15/20\n",
      " - 1s - loss: 0.0426 - acc: 0.9873 - val_loss: 0.0803 - val_acc: 0.9766\n",
      "Epoch 16/20\n",
      " - 1s - loss: 0.0452 - acc: 0.9865 - val_loss: 0.0616 - val_acc: 0.9818\n",
      "Epoch 17/20\n",
      " - 1s - loss: 0.0365 - acc: 0.9893 - val_loss: 0.0594 - val_acc: 0.9826\n",
      "Epoch 18/20\n",
      " - 2s - loss: 0.0328 - acc: 0.9902 - val_loss: 0.0580 - val_acc: 0.9824\n",
      "Epoch 19/20\n",
      " - 1s - loss: 0.0338 - acc: 0.9884 - val_loss: 0.0542 - val_acc: 0.9836\n",
      "Epoch 20/20\n",
      " - 1s - loss: 0.0317 - acc: 0.9896 - val_loss: 0.0589 - val_acc: 0.9808\n",
      "5000/5000 [==============================] - 1s 206us/step\n",
      "[0.05887098443075374, 0.9808]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xl8VNX9//HXJ5OEhH1HVgOIAq4IIrjWfW391tatVqut2lrb2n77rdXaWu2itj9r/bq07n7rWhWXuuCKC4uIJCDIpoQ9bCFA9nWS8/vj3plMkkkygZDJTN7PxyMPZu7cmfvJwnvOnHvuOeacQ0REkktKvAsQEZH2p3AXEUlCCncRkSSkcBcRSUIKdxGRJKRwFxFJQgp3kb1gZrea2dMdcJwsM3NmlrqvjyXJQeEuHcLM1pvZqfGuozMxsyvMbG6865DkpHAXiZFazZJIFO4Sd2Z2tZnlmtkuM3vNzIb5283M/m5m+WZWZGZLzewQ/7GzzWyFmZWY2WYz+59mXvsKM5tnZvf5r7HKzE6JeLyPmT1mZlv91/mTmQUaPffvZrYLuLWZbyHDzJ73a1lkZodHvP6NZrbGf2yFmX3T3z4BeBCYbmalZlbob880s7+Z2Qa/3rlmlhlxrEvNbKOZFZjZzXv6M5fkp3CXuDKzk4E7gAuBocAG4N/+w6cDJwAHAn2Bi4Cd/mOPAT90zvUCDgE+aOEwRwNrgYHA74GXzay//9i/gCBwADDJP+ZVUZ47GPhzM69/HvAi0B94FnjVzNL8x9YAxwN9gNuAp81sqHNuJfAjYL5zrqdzrq+//13AZOAY//VuAOoijnUccBBwCnCL/yYh0oTCXeLtUuBx59wi51wVcBNeazYLqAF6AeMBc86tdM5t9Z9XA0w0s97Oud3OuUUtHCMfuMc5V+Ocex74EjjHzIYAZwE/d86VOefygb8DF0c8d4tz7j7nXNA5V9HM6+c452Y452qAu4EMYBqAc+5F59wW51ydf+zVwNRoL2JmKcD3geudc5udc7XOuU/8n0vIbc65CufcEmAJcHi01xJRuEu8DcNrrQPgnCvFa50Pd859ANwPPABsN7OHzay3v+u3gLOBDWb2sZlNb+EYm13DGfI2+MfdH0gDtppZod818hBeKz1kUwzfQ3gf51wdkOe/PmZ2uZl9HvH6h+B9gohmIN4bw5oWjrUt4nY50DOG+qQLUrhLvG3BC1kAzKwHMADYDOCcu9c5Nxk4GK975lf+9oXOufPwgvhV4IUWjjHczCzi/ij/uJuAKmCgc66v/9XbOXdwxL6xTJs6MqL+FGAEsMXM9gceAX4CDPC7XpYBoVoav3YBUAmMjeGYIi1SuEtHSjOzjIivVLw+6ivN7Agz6wbcDixwzq03s6PM7Gi//7oML/hqzSzdzC41sz5+V0gxUNvCcQcDPzOzNDO7AJgAzPS7eN4F/mZmvc0sxczGmtmJbfy+JpvZ+f7383O8N4xPgR54Ab4DwMyuxGu5h2wHRphZOoRb/Y8Dd5vZMDMLmNl0/+ci0iYKd+lIM4GKiK9bnXOzgN8BLwFb8VqtoT7v3ngt3914XSk78U44AlwGrDezYrwTk99t4bgLgHF4LeM/A992zoVOzF4OpAMr/OPMwDux2xb/wTvZu9uv63y/f38F8DdgPl6QHwrMi3jeB8ByYJuZFfjb/gf4AlgI7AL+gv6fyh4wLdYhyczMrgCucs4dF+9aRDqSWgQiIklI4S4ikoTULSMikoTUchcRSUJxmwhp4MCBLisrK16HFxFJSDk5OQXOuUGt7Re3cM/KyiI7OztehxcRSUhmtqH1vdQtIyKSlBTuIiJJSOEuIpKEFO4iIklI4S4ikoQU7iIiSUjhLiKShBIu3L/cVsLf3v2SnaVVre8sItJFJVy4r91Ryn0f5JJfonAXEWlOwoV7RnoAgMqalhbeERHp2hIv3FO9cK9QuIuINCvhwj1TLXcRkVYlXrin+S336ro4VyIi0nklXLhnpHklq+UuItK8hAv3cMtd4S4i0qyEC3eNlhERaV3ChXuo5a5wFxFpXsKFe1oghUCKqVtGRKQFCRfu4LXeNVpGRKR5CRnuGWkBtdxFRFqQkOGemZ5ClcJdRKRZCRnuGalquYuItCQhwz0zXeEuItKShAz3jLQAFdUKdxGR5iRsuFcGNVpGRKQ5CRnumWkpVKrlLiLSrAQNd/W5i4i0JDHDPT2g6QdERFqQkOHeTUMhRURalJDhrpa7iEjLEjPc0wLU1DqCtRoxIyISTavhbmYjzexDM1tpZsvN7Poo+5iZ3WtmuWa21MyO3DflesKrMWk4pIhIVLG03IPAL51zE4BpwHVmNrHRPmcB4/yva4B/tmuVjdSvo6quGRGRaFoNd+fcVufcIv92CbASGN5ot/OAJ53nU6CvmQ1t92p9GVqwQ0SkRW3qczezLGASsKDRQ8OBTRH382j6BtBuMrXUnohIi2IOdzPrCbwE/Nw5V9z44ShPcVFe4xozyzaz7B07drSt0ggZqVokW0SkJTGFu5ml4QX7M865l6PskgeMjLg/AtjSeCfn3MPOuSnOuSmDBg3ak3qB+pa7+txFRKKLZbSMAY8BK51zdzez22vA5f6omWlAkXNuazvW2UCoz10tdxGR6FJj2OdY4DLgCzP73N/2G2AUgHPuQWAmcDaQC5QDV7Z/qfUywydUNRRSRCSaVsPdOTeX6H3qkfs44Lr2Kqo1oXHuK7cWc+Yh+3XUYUVEEkZiXqHq97n/76zV7CqrjnM1IiKdT2KGu98tA+p3FxGJJiHDPSMi3EVEpKmEDPduqfVla/IwEZGmEjLcvdGZnhqFu4hIEwkZ7pFqaptcCCsi0uUlQbir5S4i0lgShLta7iIijSVBuKvlLiLSWMKG+4s/mg5AUC13EZEmEjbcU1O8ETNquYuINJWw4Z4W8EpXuIuINJUE4a5uGRGRxhI43L1umWCdWu4iIo0lcLh7pVcHFe4iIo0lbLinhlvu6pYREWksYcNdJ1RFRJqXuOGeohOqIiLNSdxwT9U4dxGR5iRsuKf6LXfN5y4i0lTChntoKGS1umVERJpI2HA3M1JTTC13EZEoEjbcwRsxo6GQIiJNJXS4pwZMFzGJiESR0OGeHkjR9AMiIlEkdLinBoyaoLplREQaS+hwTwukUKOWu4hIE4kf7hoKKSLSRIKHu4ZCiohEk9DhnpqSoukHRESiSOhwT0tVt4yISDSJHe4pppa7iEgUiR3ugRSCarmLiDSR0OGeGjCq1XIXEWmi1XA3s8fNLN/MljXz+NfMrMjMPve/bmn/MqNL0xWqIiJRpcawz/8B9wNPtrDPHOfcue1SURuk6QpVEZGoWm25O+dmA7s6oJY2S9UVqiIiUbVXn/t0M1tiZm+Z2cHN7WRm15hZtpll79ixY68Pmh7QOHcRkWjaI9wXAfs75w4H7gNebW5H59zDzrkpzrkpgwYN2usDe4t1qFtGRKSxvQ5351yxc67Uvz0TSDOzgXtdWQy8i5jUchcRaWyvw93M9jMz829P9V9z596+biy8i5jUchcRaazV0TJm9hzwNWCgmeUBvwfSAJxzDwLfBq41syBQAVzsnOuQxPUuYlLLXUSksVbD3Tl3SSuP3483VLLDpWrKXxGRqBL6CtX0gFFTV0cHfVAQEUkYCR3uqYEUnIMdpVXxLkVEpFNJ6HBPC3jln3LXx3GuRESkc0nwcDcASqqCca5ERKRzSehwD6RYvEsQEemUEjrcdXWqiEh0CR3umstdRCS6hA53TT0gIhKdwl1EJAkleLirz11EJJqEDvce6bEsJCUi0vUkdLj/8MQxAAztkxHnSkREOpeEDveMtADfOXqUumdERBpJ6HAH6JaaQlWwNt5liIh0KkkQ7gGqgho1IyISKeHDPT01heqgpv0VEYmU8OHeLdX7FtR6FxGpp3AXEUlCiR/uaQEAnVQVEYmQ+OHut9yr1XIXEQlLmnBXt4yISL0kCHe/W6ZG4S4iEpL44Z7mfQu7yqrjXImISOeR+OHuL5L93ccWxLkSEZHOI/HDPS3hvwURkXaX8MmYHgjEuwQRkU4n4cO9vDoY7xJERDqdhA/3CcN6x7sEEZFOJ+HDvXdGGt+bvj99MtPiXYqISKeR8OEOkJEeoLJG0w+IiIQkR7j7c7rX1WnaXxERSJJwz0z3RsxUavIwEREgWcLdnxmyolrhLiICSRbuy7YUx7kSEZHOodVwN7PHzSzfzJY187iZ2b1mlmtmS83syPYvs2UZfrfM9x7/jJlfbO3ow4uIdDqxtNz/DzizhcfPAsb5X9cA/9z7stom1HIHWJNf2tGHFxHpdFoNd+fcbGBXC7ucBzzpPJ8Cfc1saHsVGIuMiPllMtI0HYGISHv0uQ8HNkXcz/O3NWFm15hZtpll79ixox0O7YlsuWdoIjERkXYJd4uyLeqAc+fcw865Kc65KYMGDWqHQ3siW+vd1HIXEWmXcM8DRkbcHwFsaYfXjVlonDvUL7snItKVtUcSvgZc7o+amQYUOec6dMhKZLdMra5SFREhtbUdzOw54GvAQDPLA34PpAE45x4EZgJnA7lAOXDlviq2OZHdMsFahbuISKvh7py7pJXHHXBdu1W0ByJb7kG13EVEkuMK1ch+9mBdXRwrERHpHJIi3FNSjE9uPBlQt4yICCRJuAP09hfrUMtdRCSJwj01xRturz53EZEkDPdadcuIiCRPuAf8cK+urcMbwCMi0nUlTbibGakpxn0f5HLcXz6MdzkiInGVNOEO9a33zYUVca5ERCS+kirc1RsjIuJJqnCvrtUwSBERSLJwFxERT9KGe41a8SLShSVtuJdX18a7BBGRuEnicA/GuwQRkbhJqnC/6azx4dtlVWq5i0jXlVTh/sMTx/LY96YAarmLSNeWVOEO0D3dW39ELXcR6cqSLtx7dvPCXS13EenKki7cu3fzltwrrVK4i0jXlXTh3iM91HJXt4yIdF1JF+6hlnuZWu4i0oUlX7ineeG+o7QqzpWIiMRP0oV7asD7lh76eC0ffpkf52pEROIj6cI90urtJawrKKOwvDrepYiIdKikDPcrjskCYECPbpx010ecc+/c+BYkItLBkjLcf/y1sQBUBr0RM1qZSUS6mqQM926p3knVLQp1EemikjPc07xv64EP18S5EhGR+EjKcE8PJOW3JSISs6RMwZQUi3cJIiJxlZTh3pyiihqybnyTGTl58S5FRGSf6lLhvnm3d4L1kdlr2/zcdQVlZN34Jqu3l7R3WSIi7a5LhXtqwOuuCda1ffHsN5duAeDlxZvbtSYRkX2hS4V7innhXufa/lwLP3cPniwi0sFiCnczO9PMvjSzXDO7McrjV5jZDjP73P+6qv1L3Tuzv9oRbrHvScs9EDpJq2wXkQTQaribWQB4ADgLmAhcYmYTo+z6vHPuCP/r0Xaus83m33Qyr/z4mPD9yx//jJqgl8y1tW1P6ND4G7XcRSQRxNJynwrkOufWOueqgX8D5+3bsvbe0D6ZTBrVj6lZ/cPbasIt97YHdKhLR9kuIokglnAfDmyKuJ/nb2vsW2a21MxmmNnIaC9kZteYWbaZZe/YsWMPym273REzQtYEvXCv3YNw97N9j/rrRUQ6WizhHu2KoMYR9zqQ5Zw7DHgf+Fe0F3LOPeycm+KcmzJo0KC2VbqHIldkKqyoAfas5a4TqiKSSGIJ9zwgsiU+AtgSuYNzbqdzLrT00SPA5PYpb++VRayl+sOncoA9a7nrolcRSSSxhPtCYJyZjTazdOBi4LXIHcxsaMTdbwAr26/EvRNtLdU9GS2Topa7iCSQ1NZ2cM4FzewnwDtAAHjcObfczP4AZDvnXgN+ZmbfAILALuCKfVhzm6SkWJOO8r3rc1e4i0jn12q4AzjnZgIzG227JeL2TcBN7Vta+3j52mM4976GKzHV1DoqqmvJTA/E/Dqm0TIikkCS/grVQ4b34c2fHddk+4Rb3uY3r3xBXYyt+BSNlhGRBJL04Q5w8LA+HDq8T5Ptzy7YSEGZdx64rs4RrG2+Lz7UYncJ1nR/dM5aDr/t3XiXISIdrEuEO0RMH9BIhT+a5sfPLOKwFkIw1E+fYNnOn95cSVFFTcyfUEQkOXT5cC+rqsU5x9vLt1FeXRt1dA3Uh3uinlCtqKltfScRSRpdJtybG6deURNkwbpd4fu5+aVR9wuFeqI2gMurFe4iXUlMo2WSQUst968iFuDYWVYVdb/QVa0uwaaFTA+kUF1bF+5+EpGuocuEe9/M9Ab3zzh4CO8s387lj3/WYHu0Fu47y7fx3ortQOL1uacFjOpaKK+J3t0kIsmpy3TL9OvRMNyH9+0edb81+WX8/N+LqaxpOG1BzobdQOL1uaeler9idcuIdC1dpuU+ol8mAMP7ZrK5sILJ+/fj8Xnrmuz39/e/AmDC0N4cOrwPO0obdtPUtDBcsiXbiirZVVbNxGG99+j5eyot4IW7umVEupYuE+4/OG40wVrHNSeMYW1BKVkDejR4/A/nHcwt/1kevn/HW6uivk6RP7NkW51010dU1NSy/s5z9uj5eyo9oJa7SFfUZbplMtICXH/qODLTAxw8rA89uqVyzmHefGdXHJPF5dOzYnqdwvI9C/fQUMSOvggqtCh4ebX63EW6ki4T7tGM7Of1uw/smd7KnvViDfeNO8uprXPMWrm9QaAXV3RsyAb8OXGqg3vWnSQiianLdMtE08OfOGxQr24xP6cwYmWnxj5bt4u5uQUcOaovVzyxkAOH9OSr7aU88J0jw/sUlFXRp3vanhfdVv4I0Oo9PFcgIompS4f71SeMoWdGKt+e7K1FMnV0fz6LuKApmrLqWqqDdaSnplBeHSQjNeBNKwxc+ND8Bvt+td27ICpvd3l4287SasbGsAhVZU0tGWmxz1rZnNDo/hq13EW6lC7dLZORFuDKY0eHL3D61RkHhR8bM6hHc0/jhexNLNtcxMRb3uHSRxewubCixb70yC6R4hhOyK7cWsz4373N28u2xvJttCg0VfGetty3FFbsdQ0i0vG6dLg31iO9/oPMmz89vtn9fvvqsvAc8fPX7uSkuz5qcTRKZLAWV7Ye7l/kFQHw3or8VvdtTejC3Jratp/I/c/nmznmzg/4dO3Ova5DRDqWwj1C5OIdmemBqPPAR1MdrGNXWfN98VURLff/fmFJ1JObJZU13DBjCUUVNeFVnxpPdVAdrGNbUWVMNbVUQ6wW+RdurdxavEfHFJH4UbhH2K93Br0yUrnvkkkA9M6I/cTnQ7PXNPtYQUlVOLCBqAH91KcbeCE7j0dmr61fr7XRLGU3zFjCtDtmtelCqtCcOHty8VXoXMKeLEsoIvGlcI+QmR7gi1vP4OuHDwNo06iWpz/d2OxjG3eVN5iTJnSC9ZkFG8JBb/6pz5raOlL830rjTH318y0AlFfVdwEt21zEwy28sQT97pg9GQqZoqUFRRKWwr0FPdMbDia64pis8O22jI3P9rs3QvJ2V5BfXMnNryxj2h2z+PObK3jLP3laW+eYs7oAoNn5JyMnATv3vrncPnNVs4txhFaX2pOWe+hEc6LNpyMiCvcWpTSaJjhrQP1kYwtvPpXbv3lok+f079F66G/YVRaeiAzgkTnrWOqfRA3WOV5etBmA15dsYdHG3SzbXMSFD9YPs/zdq8uahHV5M4tx1NTF1nLPL6lsMPUxEO5KqlW4iyQchXsrHvveFF780XRuOXciJx40OLzdzPjO0aP45MaTG+zf0gRdvz1nAgAPfLiGa59ZFHWfWau2N7h//j8+4fevLeez9fXj799fmc+4m9/i2qdzwtsiV5C6462VzMstwDkXHnr574WbmHb7LHI27I4a9Mfd+SGn/312g22BZvr+K2tq+eMbK1q8oEtE4kvh3opTJgzhqKz+fP+40fTJ9PrgL5oyMvz4sL6ZDfaPXM5u+pgBXD59//D9Q4b3oXdGy9eNbdrVdFx5TqNunZC3lm0L3y6pDLK5sIJHZq/loY/XcumjCyitCjYYJbOtuJJv/fMTbnxpaZPXijYOPhA+odpw+4vZm3hs7jr++dEaSquCrC8o46aXv4hpmGc0zjnumLmSJZsK9+j54H3y0BQLIvUU7m3Qv0c6s355Iref37Q7BmDODSeFb+/XO4PnrpnGDWeOD2/rlppC78x9M/VAaVWQJ+au488zV4a3PTqn6ZTGAC8v3tzgfuQngGBEkodOqG7aXc7EW94OvynsKPVa7HNWF3DI79/hL2+v4rnPNvLAB7ks2ribKX96Pzw0dNOu8lYvxqqsqeOh2Wv59oOfxPrtNlBb55j651n88sUle/R8kWSkcG+jsYN6Nlmy74tbT2fZbWcwsn99n/xLPz4GgJ7dUrnhTO/K18G9M8JT8B53wMB2rau0MsjORmPt/3fW6laft7O0qsEngNKqIPfNWs0nuQXhbTNy8iivruXfCzcB9VfZrtzmjX+f5++7u7ya8//xCQWlVeEx8hc8OJ8fPb2IoooafvfqMkqitO5L/S6lxhda/erFJVz1r2zumLmSHSX18+rX1TleWLiJ215fTm5+SfjT0utLtrT6/Yp0FQr3dtArI42e3bzulvsumcSd5x/K8IjummtPHMtnN5/C8L6ZrC0oA2B7cSUnHth0kpm3rm/+ytiWlFYFm51rfkCUk7yhFvrMiGAH2FVWzd/e+4rvPLqg2SkLQuHe3Z/7prjSC+cNO+vn0AktS7it2Bvqec/7X/HUpxt4bG7TTxOR5wtCauscL+bk8f7K7Tw0ey2/e3VZ+LFnP9vIDS8t5Yl567nmqZwOm874pZy8VuceEuksFO7t7OuHD+PiqaMabDMzBvfKAOAOv0vn12eO5xenHdjk+ROG9ubYAwYAcP0p45o8PriZGSwXb9zNl9tKoj72g+NHN9m2YVc5zy/cGA7N0MyVkQuWPPdZ07H71cE6KoNeS7ms0cnjyE8Oz2dv4uS/fRS+n5sfmkSt4Tw8NbV13PzqF+H73/zHPDbuLGd3o5O1kf35v40I+to61+Qk9q6y6hb73389YynvLN/W7OPN+eWLS5pMDteePsktIDff+x2+lJPHh1/u/fQT0nV16Vkh4+GSqaO4pFH4A/zi1AM5daI3GueJK6ZSGawlPZBCj24Bbp9ZvyrU7BtOYvzv3g7fP3R4H3aWVvHQ7LXNHvO0CUP469tfNtx298cNLpLar4/35jM3ojsm2tz1ZVXBZuek317c8MrbtTvKwrdDY/dn5OQxdlBPrv3aWAA+XJXPvNz6uWsWbyzk6/fP5Ucnjm3wWqGhn5WNhnymB1IazOtz8ytf8MyCjZxx8BAeumwK4J2wLSyvoV+PdOrqHM9nb+L57E1NVsWqrXOUVQejXpnc3HUE7ek7jy4AYP2d54TPH8SycpdzjnveX80ZB+/X4cs4gvd7TzFr09TZsu+p5R5nr153LD8/dRzXnzqOg4f1ASA9NYXeGWlkpAW45oSGIZeRFuCU8d6bwO/Onci/vj+V3547kRSDo7L6RT1G1sCmM1w2zqrDR/SJqd5Jf3yvwRtApJLK2LpHXlmcB3it0/+JchK0qKKGv7zdcJnDUH98466n2jrH8i31c988s8D7tPHO8u3kbNhFzoZd3PP+aib98T027iyntFEXzootxTwxbx3B2jpue305h936Lss2FzWpqbCZLq+iihq+++gCnpq/nt2NznkUllc3eMPbWVpF1o1v8vqSLZRU1rAgYkK2yOsW2rpaV3FlkP+dtZpLH/20Tc8D703rkdlrWx3pFGzhIrijb5/FUX9+P3z/mQUbmv0U2V5yNuzm6iezW6yrq1PLPc6OGNmXI0b2jWnfP3/zEADuvugICkqrGDuoJwBnHzqUNbefjZlx+G3vNgnAtEAKS245ncP/8G7U1x03uCepgRTm33Qy0+/4IOo+k0b1ZfHG2IYqvvuLE/j3Z5uiLkAO3jz3P31ucZtOgBZX1PD/3lnV5E1pbUFZ1DcIgG/9s2EXyutLt3DM2AHh+1uLKjj73jmAN6w01J9+7n1zeeaqoxnVvzvD+2aSkmLsKqs/obu5sILhfTPZWVrFw3PWMje3gLm5Bby0aDOvXndseL9T7/6YgtJq1t95DtuLK8MXp/30ucXhfW77xsFMGzOAqmD9p49on4yW5hUyemAP0lNTmP1VAadNHIJzjq1FleETypU1TYOuKlhLUUUN6YEUSiqDDU76A8xevYM/z1zJ6vwS/vrtw6P+HGf4b8K/PWcCVx0/Juo+Ic45bn7F6zabc8NJlFUHGb9f858mqoN1lFYFY7r4L9JPnl3E1qJKthRWMmpA99af0AUp3BPIpUd7Y+b7ZKaFx9yHhOZtH943k6KKGo4Y2Zf7LpkUnumyT/e08AVXx9zZMMBnXOuN7BnaJ5OHL5tMWXWQT9fs4vnsTeF9jh07MGq4n3vYUMYN7sXf3/8qvK1/j3Ru+frEZsMd2j6yZW1BGQ982HAOnUG9ujUYRdOa//dOw66pc+6dG77d+ETppX4XyXEHDGREv0zOOGS/8GMXPjifeTeezBn3zKGgtP74n28q5B8f5TKkVwb/NWk4Bf6Q0cqaWi58aH6DE84hv39teZNt+SX1rf3y6iB1Dr5x/zxOGT+YcUN68eDHa3ju6mkUV9bww6dyONdfCzgt0HAU1+0zV/Kw3103pHc3thdXse6Os9lSVEnernKOHjMgfL5idwvLR4bePF9furXVcI/sIjv+rx8CXtfSl9tK+OWLn/PI5VMY2qd+sMHPnlvM28u3se6Os8N/w+D9zCb/8T3u/NZh4bmeIoWG6e4sq2q3cM8vriQzPUCviG652jrXZHRcolC4J4BfnnYg3dJi60E7/8jhrHizmLsuOKxJK21Y38wmH/mH9slo8EZx+sFeiJ172LAG4X759P25/8Pc8P1rThjDqP7d+e407w2nvCbIQx97QRIaOTSwZ7cG4ffsVUezpqCswciXkDd+ehzn/+OTNi0q8qMTx3L7zJV7PGtlS9M0h4S6oCLPP2wurKC2zjX43kJC5zZyNtZfeBZ5jiQWM3Lywrd/9eJSLpgyAvDePEIXpW0rrgi/sb2x1LuOoLgySM6G3Uzevx+vLt4cDnaA7cWVFL8XAAAOIklEQVRV/vMqOe/+eRSUVvGLUw9kkV9naWUwapBFdnss2VTIz55bTK+MVOqcNzig8TQY0RZ3+ft7X4WH5T7z6Ub+J2JRnLf9E9tl1bXhvxuA/OIqyqpruX3myibhvq2okmCdd9z8kip+PWMpA3ul86szxtOS91ds55qnsvm/K6dywoGDWLm1mNEDe4RXPJt6+yy6paZw8vjB3H3hESxYt5MrnljIOz8/gYP269XgtT5YtZ0+melM3r9pV+hf317FhKG9o74pvZi9ifTUFM47YniLtbYHa2v/XnuZMmWKy87Ojsuxk5lzji1FlQ2GYjaWs2E3fTLTOPXujznpoEE8ceXUqPtd+3QO76/czp3nH8a3Jo8gN7+EU++ezdA+Gcy/6ZQG+9bVOXaUVvHlthJO8Id4FlXUUB2sY0ZOHn95exXzbjyZnt1SeXTOWu77wHujGD2wB1cfP4bvHD2Kq/6Vzfsrt5P921OZkZPHnW/V97ufcOAgZn+1g3MPG8obS7cyol8mc399MvPX7OSSR1rvaw61XBu78tgsnpi3vtXnxyItYDEvinLPRUfwpzdXhFv3AI9cPoWrn9y3/ycunDKCF7Lzoj52zmFDue/iSazaVsJ/v/A5q7aVcOnRo8LnMRp76LLJpJiFaz7/yOHhrqfmjN+vFzedPYE/vL6csYN68q4/ZPaJK47iFy98zqs/Ppb3VmynT2YaN/gXza2/8xw+/DKfv779JcUVNWxuZnWw1k4+X/NkNu+u2M71p4wjIy0QPq+z8OZT2VZUydfvr/8kd9cFh/NJbgEvL97Md6eN4spjRzOiXybdUr03gqwb3wTgawcN4p6LjqDEv85k/H69wm/m910yKRzwxZU1zPmqgOueXRRTrS0xsxzn3JRW91O4d11zVu/g8JF92zRvvXOOOkebPqo659hZVs3AnvWjKc65dw4rtxbz+e9PDx+/orqWzYXlHDC4V/h5y7cUM3/NTq48Nota58L/uSJf+8n5GzjnsKHMyy3g+n9/HrWGMw4ewjvL6+ft+eN5B5NfUsUJBw7iggejD2+cNqY/n65teVz7TWeN59gDBhJIMYoqarj44dhOav7r+1N5Y8kWXoxopYdC7MonFgLw2k+O5Rv3z4vp9WKRHkjptAulHzmqL4uaOadz1XGjeTTK9RGN3XvJJPJ2l7OhoJzzjhjGra8v59dnjueEAwexZkcp5947N7y+QSzOOWwoby6tv7p6zKAeDO7VjTMP3o9bX18R3n7xUSPDF/j9/aLD+cXzXjfWhVNG8NtzJ5KWksLp93zcYGqRv3zrUC46qumouVi0a7ib2ZnA/wIB4FHn3J2NHu8GPAlMBnYCFznn1rf0mgr3rq06WMfu8mqG9M5o99feVlTJ5sJyXliYx+r8EiaN6scPTxzD859t4oDBPTlm7MDwXP3rCso46a6POGX8YGatyueQ4b1ZttkbfRP6j3r8uIHMWV3Ak9+fyubCCm562RuXf9rEITxyef3/sYrqWi5/fAHD+mbyH3/u/fu/M4mfPLuYq48fzcnjh4Q/Zbz+k+PYf2B3Hp2zjntnrWZ430zm+edEVm0rZldpNcccMJCcDbv51j+9aRmyBnRnvd9vf/L4wXywyhsH/9Blkzkqqz+rt5dwUaM3l0umjuKKY7J4IXsTR47qF245ttWIfpnk7d779XQPH9l3r+YQCjl/0vAm02i0pl/3NFLMmlzJHQ+/OuMgrjvpgD16bruFu5kFgK+A04A8YCFwiXNuRcQ+PwYOc879yMwuBr7pnLuopddVuEtn8dX2EsYN7omZUVhezT3vr+aCKSOYOLQ3ufmljB3Uk+0lleETgTW1dTwxbx0XTx0V9VNPqPvquaunMX3sADYXVjCsTwZm5nVP5Rbwwg+nh/t6y6uDGNZgmcdImwsrCJixX58MTv7bR1TV1DHvxpN5YeEmdpRWhUNiV1k1R/7xvQbPXfS708IjUerqHNc+k8M7y7czsGc6BaXVfP3wYRyV1a/BxWtpAWPSyH58/7gsfvT0InpnpPLI5VP44dM5XDB5BE9/upHZN5zExl1lTUYkhUwY2puVW4vplZFKSWWQy6btzx//yxvtFWtXWmMPfncyf31nFRXVtcz79cmM+c3M8GP9uqe1eFIY4OHLJvP28m0tdh3dfPYEHvgoN3yOJVR/yLjBPVntX5AXsv+A7lFPlkdz6oQhvL9yOy9de0zU/vpYtGe4Twdudc6d4d+/CcA5d0fEPu/4+8w3s1RgGzDItfDiCneRtgudPG6uW6w6WEdJZQ3biiuZOLR3gxEokXaXVTN/7U7OPtQbaVNQWkVxRQ27yqo5bERf0lNTwsdrfCznXPh1V28voay6lg07yzgqqz+BFOP/PlnPL087kMWbCpkwtDevLN7MNycNb3DC9NkFG/nNK/VXJt91weEszSvkyfkbGN43k82FFfRID/Dfpx/E5P37cciw3qQGUqgO1lHnHBlpgXC/95Pfn8qIfpk8v3AT5x0xnC+3F3PAoF5MGNqLix7+lI27yrn6+NFcc8JYyqqCzP5qB8u2FPGfz7dwyLA+FJRWcfM5EwjWOY7K6g94/fMfrMpn9g0nsa24khcWbuI350wgPZAS7lM/65D9+NN/HUL/HunMX7uT655ZxO7yGsYM8s4jLc0rYuXWYn5w3Gg2F1Zw/qThDOzZjdwdpRw4pOEJ2rZoz3D/NnCmc+4q//5lwNHOuZ9E7LPM3yfPv7/G36eg0WtdA1wDMGrUqMkbNmxo23clIknlyfnrOWR4H44c5bViK8Nj9mtJC6TQo1vzA/q+yCuiuLKGY9t5Er7WlFTW8FJOHhceNZLujVZrC9Uf+lS2L8Qa7rEMhYz21t/4HSGWfXDOPQw8DF7LPYZji0gSu3x6VoP7oVCMJRwPjfGq6vbWKyONK45tOl8T7NtQb6tYBk/nASMj7o8AGl+BEt7H75bpA2j6PBGROIkl3BcC48xstJmlAxcDrzXa5zXge/7tbwMftNTfLiIi+1ar3TLOuaCZ/QR4B28o5OPOueVm9gcg2zn3GvAY8JSZ5eK12C/el0WLiEjLYpp+wDk3E5jZaNstEbcrgQvatzQREdlTmvJXRCQJKdxFRJKQwl1EJAkp3EVEklDcZoU0sx3Anl6iOhCIvtZb/HXW2lRX26iutlFdbbM3de3vnBvU2k5xC/e9YWbZsVx+Gw+dtTbV1Taqq21UV9t0RF3qlhERSUIKdxGRJJSo4f5wvAtoQWetTXW1jepqG9XVNvu8roTscxcRkZYlastdRERaoHAXEUlCCRfuZnammX1pZrlmdmMHH/txM8v3V54KbetvZu+Z2Wr/337+djOze/06l5rZkfuwrpFm9qGZrTSz5WZ2fWeozcwyzOwzM1vi13Wbv320mS3w63ren0oaM+vm38/1H8/aF3VF1Bcws8Vm9kZnqcvM1pvZF2b2uZll+9s6w99YXzObYWar/L+z6fGuy8wO8n9Ooa9iM/t5vOvyj/UL/29+mZk95/9f6Ni/L+dcwnzhTTm8BhgDpANLgIkdePwTgCOBZRHb/grc6N++EfiLf/ts4C28VaqmAQv2YV1DgSP9273wFjSfGO/a/Nfv6d9OAxb4x3sBuNjf/iBwrX/7x8CD/u2Lgef38e/zv4FngTf8+3GvC1gPDGy0rTP8jf0LuMq/nQ707Qx1RdQXwFu7ef941wUMB9YBmRF/V1d09N/XPv2B74Mf2nTgnYj7NwE3dXANWTQM9y+Bof7tocCX/u2HgEui7dcBNf4HOK0z1QZ0BxYBR+NdmZfa+HeKt2bAdP92qr+f7aN6RgCzgJOBN/z/8J2hrvU0Dfe4/h6B3n5YWWeqq1EtpwPzOkNdeOG+Cejv/728AZzR0X9fidYtE/qhheT52+JpiHNuK4D/72B/e1xq9T/STcJrJce9Nr/r43MgH3gP75NXoXMuGOXY4br8x4uAAfuiLuAe4Aagzr8/oJPU5YB3zSzHvAXlIf6/xzHADuAJvxvrUTPr0QnqinQx8Jx/O651Oec2A3cBG4GteH8vOXTw31eihXtMC3F3Eh1eq5n1BF4Cfu6cK25p1yjb9kltzrla59wReC3lqcCEFo7dIXWZ2blAvnMuJ3JzvOvyHeucOxI4C7jOzE5oYd+OqisVrzvyn865SUAZXndHvOvyDub1XX8DeLG1XaNs2xd/X/2A84DRwDCgB97vs7lj75O6Ei3cY1msu6NtN7OhAP6/+f72Dq3VzNLwgv0Z59zLnak2AOdcIfARXl9nX/MWUm987I5aaP1Y4Btmth74N17XzD2doC6cc1v8f/OBV/DeEOP9e8wD8pxzC/z7M/DCPt51hZwFLHLObffvx7uuU4F1zrkdzrka4GXgGDr47yvRwj2Wxbo7WuTi4N/D6+8Obb/cP0M/DSgKfVRsb2ZmeOvYrnTO3d1ZajOzQWbW17+difdHvxL4EG8h9Wh17fOF1p1zNznnRjjnsvD+hj5wzl0a77rMrIeZ9QrdxutHXkacf4/OuW3AJjM7yN90CrAi3nVFuIT6LpnQ8eNZ10Zgmpl19/9vhn5eHfv3tS9PcuyLL7wz3l/h9d3e3MHHfg6vD60G7932B3h9Y7OA1f6//f19DXjAr/MLYMo+rOs4vI9xS4HP/a+z410bcBiw2K9rGXCLv30M8BmQi/dRupu/PcO/n+s/PqYDfqdfo360TFzr8o+/xP9aHvr7jvfv0T/WEUC2/7t8FejXSerqDuwE+kRs6wx13Qas8v/unwK6dfTfl6YfEBFJQonWLSMiIjFQuIuIJCGFu4hIElK4i4gkIYW7iEgSUriLiCQhhbuISBL6/7eCUEyJUgYZAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#import sys\n",
    "#print(sys.path)\n",
    "#print(sys.executable)\n",
    "\n",
    "import cnn_utils\n",
    "from matplotlib import pyplot as plt\n",
    "import keras\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Flatten\n",
    "from keras.layers import Conv2D, MaxPooling2D\n",
    "import keras.callbacks as cb\n",
    "\n",
    "class LossHistory(cb.Callback):\n",
    "    def on_train_begin(self, logs={}):\n",
    "        self.losses = []\n",
    "\n",
    "    def on_batch_end(self, batch, logs={}):\n",
    "        batch_loss = logs.get('loss')\n",
    "        self.losses.append(batch_loss)\n",
    "\n",
    "def init_model():\n",
    "    model = Sequential()\n",
    "    model.add(Conv2D(32, kernel_size=(3, 3),\n",
    "                    activation='relu',\n",
    "                    input_shape=(28, 28, 1)))\n",
    "    model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "    model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "    model.add(Dropout(0.25))\n",
    "    model.add(Flatten())\n",
    "    model.add(Dense(128, activation='relu'))\n",
    "    model.add(Dropout(0.5))\n",
    "    model.add(Dense(10, activation='softmax'))\n",
    "\n",
    "    model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer=keras.optimizers.Adadelta(),\n",
    "              metrics=['accuracy'])\n",
    "    return model\n",
    "\n",
    "def plot_losses(losses):\n",
    "    plt.plot(losses)\n",
    "    plt.title('Loss per batch')\n",
    "    plt.show()\n",
    "    \n",
    "filePath = 'C:/Users/computer/Desktop/Assignment8/'   # the training set is stored in this directory\n",
    "mnistData= cnn_utils.loadData(filePath)\n",
    "mnistData_preprocessed = cnn_utils.preprocessData_CNN(mnistData)\n",
    "(X_train, y_train), (X_test, y_test) = mnistData_preprocessed\n",
    "\n",
    "(X_train_mini_batch, y_train_mini_batch) = cnn_utils.mini_batch(X_train, y_train, 10000, 0)\n",
    "(X_test_mini_batch, y_test_mini_batch) = cnn_utils.mini_batch(X_test, y_test, 5000, 0)\n",
    "print(X_train_mini_batch.shape)\n",
    "print(y_test_mini_batch.shape)\n",
    "\n",
    "model = init_model()\n",
    "history = LossHistory()\n",
    "model.fit(X_train_mini_batch, y_train_mini_batch, epochs=20, batch_size=256,\n",
    "                callbacks=[history],\n",
    "                validation_data=(X_test_mini_batch, y_test_mini_batch), verbose=2)\n",
    "\n",
    "score = model.evaluate(X_test_mini_batch, y_test_mini_batch, batch_size=16)\n",
    "print(score)\n",
    "plot_losses(history.losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As illustrated after 20 epochs we reached a cost of approximately 0.05887 and a test accuracy of 0.9808. The results are very good and reflect upon the power of CNNs.\n",
    "\n",
    "We will now investigate the impact of the CNN architecture by adding a duplicated third convolution layer (kernel 2X2) after the second convolution layer with the same dropout rate and followed by max pooling; essentially deepending the network one extra layer.\n",
    "\n",
    "We will use the following:\n",
    "\n",
    "1. CNN architecture: \n",
    "    a. Convolution layer (kernel 3X3) then MaxPooling (3X3), Relu\n",
    "    b. Convolution layer (kernel 2X2) then MaxPooling (2X2), Relu, Dropout (Keep Probability 0.25)\n",
    "    c. Convolution layer (kernel 2X2) then MaxPooling (2X2), Relu, Dropout (Keep Probability 0.25)\n",
    "    d. Dense layer (units 128), Relu, Dropout (Keep Probability 0.5)\n",
    "    e. Output layer (units 128), Softmax\n",
    "2. Dropout: \n",
    "    a. As specified in 1\n",
    "3. Optimization algorithm: \n",
    "    a. AdaDelta, which is a more robust extension of Adagrad that adapts learning rates based on a moving window of gradient updates.\n",
    "4. Minibatches: \n",
    "    a. 256\n",
    "5. Epochs:\n",
    "    a. 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10000, 28, 28, 1)\n",
      "(5000, 10)\n",
      "Train on 10000 samples, validate on 5000 samples\n",
      "Epoch 1/20\n",
      " - 2s - loss: 1.3108 - acc: 0.5607 - val_loss: 0.8883 - val_acc: 0.6878\n",
      "Epoch 2/20\n",
      " - 1s - loss: 0.4110 - acc: 0.8771 - val_loss: 0.2178 - val_acc: 0.9314\n",
      "Epoch 3/20\n",
      " - 1s - loss: 0.2746 - acc: 0.9209 - val_loss: 0.1359 - val_acc: 0.9602\n",
      "Epoch 4/20\n",
      " - 1s - loss: 0.2049 - acc: 0.9388 - val_loss: 0.0881 - val_acc: 0.9744\n",
      "Epoch 5/20\n",
      " - 1s - loss: 0.1769 - acc: 0.9438 - val_loss: 0.1166 - val_acc: 0.9644\n",
      "Epoch 6/20\n",
      " - 1s - loss: 0.1577 - acc: 0.9526 - val_loss: 0.1537 - val_acc: 0.9508\n",
      "Epoch 7/20\n",
      " - 1s - loss: 0.1467 - acc: 0.9546 - val_loss: 0.0779 - val_acc: 0.9754\n",
      "Epoch 8/20\n",
      " - 1s - loss: 0.1251 - acc: 0.9622 - val_loss: 0.0578 - val_acc: 0.9826\n",
      "Epoch 9/20\n",
      " - 1s - loss: 0.1077 - acc: 0.9686 - val_loss: 0.0608 - val_acc: 0.9826\n",
      "Epoch 10/20\n",
      " - 1s - loss: 0.1045 - acc: 0.9671 - val_loss: 0.0515 - val_acc: 0.9842\n",
      "Epoch 11/20\n",
      " - 1s - loss: 0.0909 - acc: 0.9705 - val_loss: 0.0474 - val_acc: 0.9850\n",
      "Epoch 12/20\n",
      " - 1s - loss: 0.0842 - acc: 0.9744 - val_loss: 0.0520 - val_acc: 0.9852\n",
      "Epoch 13/20\n",
      " - 1s - loss: 0.0828 - acc: 0.9764 - val_loss: 0.0579 - val_acc: 0.9820\n",
      "Epoch 14/20\n",
      " - 1s - loss: 0.0801 - acc: 0.9753 - val_loss: 0.0586 - val_acc: 0.9808\n",
      "Epoch 15/20\n",
      " - 1s - loss: 0.0732 - acc: 0.9789 - val_loss: 0.0700 - val_acc: 0.9790\n",
      "Epoch 16/20\n",
      " - 1s - loss: 0.0715 - acc: 0.9790 - val_loss: 0.0425 - val_acc: 0.9876\n",
      "Epoch 17/20\n",
      " - 1s - loss: 0.0616 - acc: 0.9803 - val_loss: 0.0436 - val_acc: 0.9870\n",
      "Epoch 18/20\n",
      " - 1s - loss: 0.0601 - acc: 0.9822 - val_loss: 0.0438 - val_acc: 0.9852\n",
      "Epoch 19/20\n",
      " - 1s - loss: 0.0580 - acc: 0.9817 - val_loss: 0.0410 - val_acc: 0.9872\n",
      "Epoch 20/20\n",
      " - 1s - loss: 0.0526 - acc: 0.9832 - val_loss: 0.0449 - val_acc: 0.9864\n",
      "5000/5000 [==============================] - 1s 143us/step\n",
      "[0.04494083561436273, 0.9864]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xd4VGX6//H3nUwagdBBCCU0RUQQiRQVxbqIbe26ru27LuoWy89dRV3Xtuq66rqKu7ruigU7tlVEBUUUEJFQpfcSCBASSEghbZ7fH+fM5ExJMoFkJjm5X9eVKzNnzpzzDBk+88zTjhhjUEop5S5xsS6AUkqphqfhrpRSLqThrpRSLqThrpRSLqThrpRSLqThrpRSLqThrtRhEJEHReSNKJwnQ0SMiHga+1zKHTTcVVSIyBYROTPW5WhKROR6EZkb63Iod9JwVypCWmtWzYmGu4o5Efm1iGwQkXwR+UREutvbRUSeEZE9IlIgIstFZLD92HgRWSUiB0Rkh4j8oYZjXy8i80Rkkn2MNSJyhuPxtiLysojk2Mf5i4jEBz33GRHJBx6s4SUki8i7dlkWi8hQx/EnishG+7FVInKRvf1o4EVgtIgUich+e3uKiDwtIlvt8s4VkRTHua4WkW0isldE7jvUf3PlfhruKqZE5HTgceByoBuwFXjHfvhs4BTgSKAdcAWQZz/2MnCTMaYNMBiYVctpRgKbgE7AA8CHItLBfuw1oBLoDwyzz3ljmOd2AR6t4fgXAlOBDsBbwMcikmA/thEYA7QFHgLeEJFuxpjVwM3AfGNMa2NMO3v/p4DhwIn28e4CvI5znQwcBZwB/Nn+kFAqhIa7irWrgcnGmMXGmDLgHqzabAZQAbQBBgJijFltjMmxn1cBDBKRNGPMPmPM4lrOsQf4hzGmwhjzLrAWOFdEugLnALcbY4qNMXuAZ4ArHc/daYyZZIypNMaU1nD8RcaY940xFcDfgWRgFIAxZqoxZqcxxmufez0wItxBRCQO+D/gNmPMDmNMlTHme/vfxechY0ypMWYZsAwYGu5YSmm4q1jrjlVbB8AYU4RVO083xswCngf+CewWkZdEJM3e9RJgPLBVRL4VkdG1nGOHCVwhb6t93t5AApAjIvvtppF/Y9XSfbZH8Br8+xhjvEC2fXxE5FoRWeo4/mCsbxDhdML6YNhYy7l2OW6XAK0jKJ9qgTTcVaztxApZAEQkFegI7AAwxjxnjBkOHIPVPPNHe/tCY8yFWEH8MfBeLedIFxFx3O9ln3c7UAZ0Msa0s3/SjDHHOPaNZNnUno7yxwE9gJ0i0hv4D/A7oKPd9LIC8JUl+Nh7gYNAvwjOqVStNNxVNCWISLLjx4PVRn2DiBwnIknAY8ACY8wWETlBREba7dfFWMFXJSKJInK1iLS1m0IKgapaztsFuFVEEkTkMuBoYLrdxDMDeFpE0kQkTkT6icip9Xxdw0XkYvv13I71gfEDkIoV4LkAInIDVs3dZzfQQ0QSwV/rnwz8XUS6i0i8iIy2/12UqhcNdxVN04FSx8+DxpivgfuBD4AcrFqrr807Davmuw+rKSUPq8MR4Bpgi4gUYnVM/rKW8y4ABmDVjB8FLjXG+DpmrwUSgVX2ed7H6titj/9hdfbus8t1sd2+vwp4GpiPFeTHAvMcz5sFrAR2ichee9sfgJ+AhUA+8AT6/1QdAtGLdSg3E5HrgRuNMSfHuixKRZPWCJRSyoU03JVSyoW0WUYppVxIa+5KKeVCMVsIqVOnTiYjIyNWp1dKqWZp0aJFe40xnevaL2bhnpGRQVZWVqxOr5RSzZKIbK17L22WUUopV9JwV0opF9JwV0opF9JwV0opF9JwV0opF9JwV0opF9JwV0opF2p24b521wGe+GINBaUVsS6KUko1Wc0u3Lfll/DC7I1syi2KdVGUUqrJanbh3qdTKwC25BXHuCRKKdV0Nbtw79mhFXECCzblx7ooSinVZDW7cE/yxHP6wK58vHQHulyxUkqF1+zCHeDk/h05WOFlb1F5rIuilFJNUrMM9/T2Vrv7jv2lMS6JUko1Tc0z3NulALBjn4a7UkqF0zzDvb0d7vtLYlwSpZRqmppluLdNSaBNkkdr7kopVYNmGe4A3dulsLPgYKyLoZRSTVKzDff2qQkUlOgSBEopFU6zDfd2KYnsL9WhkEopFU6zDfe2KQm6eJhSStWg2YZ7u1YJ7NdmGaWUCqvZhnvbVgmUVXo5WFEV66IopVST02zDvXWSB4DissoYl0QppZqeZhvuCfFW0SuqdPEwpZQK1mzDPdEO9/JKb4xLopRSTU+zDfcEjx3uVRruSikVrNmGe2K8AFpzV0qpcJpvuHt8be5eSsur+GbNnhiXSCmlmo5mG+6+DtUL/zmPW99Zwg2vLmR1TmGMS6WUUk1Dsw13X4cqwMxVuwF0xqpSStmabbj7OlSd9JKqSillqTPcRaSniHwjIqtFZKWI3BZmHxGR50Rkg4gsF5HjG6e41Zw1dx+DprtSSgF4ItinErjTGLNYRNoAi0RkpjFmlWOfc4AB9s9I4AX7d6NJDFNzV0opZakzIY0xOcaYxfbtA8BqID1otwuB143lB6CdiHRr8NI6JISpuWvFXSmlLPWq/opIBjAMWBD0UDqw3XE/m9APAERkgohkiUhWbm5u/UoaJFzNXbNdKaUsEYe7iLQGPgBuN8YEjzmUME8JyVpjzEvGmExjTGbnzp3rV9IgCfGhp9QOVaWUskQU7iKSgBXsbxpjPgyzSzbQ03G/B7Dz8ItXs6T4+JBt2qGqlFKWSEbLCPAysNoY8/cadvsEuNYeNTMKKDDG5DRgOUMkJehQSKWUqkkko2VOAq4BfhKRpfa2e4FeAMaYF4HpwHhgA1AC3NDwRQ2UnBBac1dKKWWpM9yNMXMJ36bu3McAv22oQh0qrbgrpZTFVYPFjbbLKKUU4LZwj3UBlFKqiXBVuGu6K6WUpVmH++e3jQm4X+XVdFdKKWjm4X50tzRO6t/Rf79K29yVUgpo5uEO4ImrfglerbkrpRTggnB3LkOgNXellLI0+3B31ty1zV0ppSzNP9wdNXev1tyVUgpwQbg713Wv8sawIEop1YQ0+3D3xDlq7toso5RSgBvC3Vlz12YZpZQCXBDuAaNltOaulFKAC8I9YJy71tyVUgpwQbhrzV0ppUI1+3D3aLgrpVSI5h/u2iyjlFIhmn24BzbLxLAgSinVhDT7cHcOhdSau1JKWZp/uMdpm7tSSgVr9uEeuPyAhrtSSoELwl0XDlNKqVDNPtwTdMlfpZQK0ezD3aMX61BKqRAuCHe9zJ5SSgVr9uGeEKfj3JVSKlizD3cd566UUqFcEO46zl0ppYI1+3APGC2jNXellAJcEO4B49y15q6UUoALwl3Xc1dKqVDNPtw92iyjlFIhmn+4a7OMUkqFaPbhHrBwmGa7UkoBEYS7iEwWkT0isqKGx8eKSIGILLV//tzwxaxZoiPc9xWXkzHxM97L2h7NIiilVJMTSc39VWBcHfvMMcYcZ/88fPjFilxGp1Re/OVwendsxea9xQBMmb81mkVQSqkmp85wN8Z8B+RHoSyHbNzgI0hJiKessgoIbIdXSqmWqKHa3EeLyDIR+VxEjqlpJxGZICJZIpKVm5vbQKe2xMcJZRXW4jLOqzMppVRL1BDhvhjobYwZCkwCPq5pR2PMS8aYTGNMZufOnRvg1NXi44SyKl+4N/t+YqWUOiyHnYLGmEJjTJF9ezqQICKdDrtk9RQnQnmlHe7aLKOUauEOO9xF5AgREfv2CPuYeYd73PqKdzTFaLOMUqql89S1g4i8DYwFOolINvAAkABgjHkRuBS4RUQqgVLgSmOiP1U0XhzhHq/NMkqplq3OcDfGXFXH488DzzdYiQ6Rs5lda+5KqZbONVXcgGYZrbkrpVo416RgnKNZJkFr7kqpFs414e6sucdruCulWjj3hLuj5u6sxSulVEvkmnCP09q6Ukr5uSbcnTX3Sl3XXSnVwrkn3OOcl9vzxrAkSikVe64Jd2ezjNbclVItnWvC3bmcjF4oWynV0rkm3LXmrpRS1VwT7s7L7WnNXSnV0rkm3JMT4v23teaulGrpXBnuOlpGKdXSuSjcq19KZZXW3JVSLZuLwt1Zc9dwV0q1bK4J9xRtc1dKKT/XhLuzWUZr7kqpls5F4a41d6WU8nFNuCd5IhstM2PlLorKKqNRJKWUihnXhHtKYt019817i5kwZRF/eG9ZtIqllFIx4Z5wj2C0TEm5VWPfml8SlTIppVSsuCbc2yR7/LdrGucuWOvPGKNt8kopd3NNuLdNSfDfrqnmHme/Ws12pZTbuTLca2pz99fc0XRXSrmba8K9VUCHau1ry2jNXSnldq4Jd3FcQ7WqhjZ335Lvmu1KKbdzTbgD9O2cCtTSLGOHu1er7kopl3NVuM+6cyw3n9qvluUHtOqulGoZXBXuAAnxUnebe5TKopRSseK6cI+PE7wGvGFr79Y2HeeulHI714W7x+41rQoT4L5Nuq6YUsrtXBfu8fZMpXDt7hrqSqmWwnXh7qu5hxsx4xslo5OYlFJuV2e4i8hkEdkjIitqeFxE5DkR2SAiy0Xk+IYvZuTifc0yYca6+5tl9PrZSimXi6Tm/iowrpbHzwEG2D8TgBcOv1iHzhPvq7mHJriOb1dKtRR1hrsx5jsgv5ZdLgReN5YfgHYi0q2hClhf/pp7LQ3sOlpGKeV2DdHmng5sd9zPtreFEJEJIpIlIlm5ubkNcOpQvjb3qYuyyZj4WcBVl6rb3JVSyt0aItwlzLaw+WmMeckYk2mMyezcuXMDnDqUb7TMi99uBGBzbrHj/NZvbZ5RSrldQ4R7NtDTcb8HsLMBjntIfDX3tGRrCeD9peX+x/w1d812pZTLNUS4fwJca4+aGQUUGGNyGuC4h8TX5p6aZC0BvK+kwv+Yrxles10p5XaeunYQkbeBsUAnEckGHgASAIwxLwLTgfHABqAEuKGxChsJ37ruSR7r9/6ScsejWnNXSrUMdYa7MeaqOh43wG8brESHqXWS9ZJ8k5iKy6r8j1UPoNF0V0q5m+tmqLa2L5RdVGY1x1Q5xrvr2jJKqZbCdeHeJsnqSN2eXwpAhWOmanWHqqa7UsrdXBfuvpq7T2WYmrtGu1LK7VwX7r5RMj6Vjpq70aGQSqkWwnXh7hsl4xPYLGP91mYZpZTbuS7cAS4Y2t1/O6BZBl1+QCnVMrgy3K8a0ct/27muu/+mprtSyuVcGe5pKdWdqpVVzg5VK9V1bRmllNu5MtxTE53h7uxQtX9Hu0BKKRVlrgx33xIEABXecOPco14kpZSKKleGe4oj3AObZezfWndXSrmcK8O9laNZJvwM1agXSSmlosqV4e5b9hegvMpbPXnJ3qbhrpRyO1eGu9N363K5acoiAP9vHS2jlHI714c7wIxVuwPua7grpdyuRYR7MF3yVynldq4N92evPC7WRVBKqZhxbbhfeFw6J/fvFOtiKKVUTLg23AGqtP1FKdVCuTrcpXpEpAa9UqpFcXW4J3qqX16FY6aqUkq5nbvDPb765eUVl8ewJEopFV3uDndHzf1PH/0Uw5IopVR0tZhwzy0qi2FJlFIqulwd7kmOcPdqk7tSqgVxdbiffcwR/turcgpjWBKllIouV4f7aUd1wbFAZACvDo1USrmYq8Mdal5HpkoXD1NKuZjrw70mOqlJKeVmrg/3Y7qnhd2uk5qUUm7m+nB/88aRYbdnbd0XtTIYY8iY+Bkvz90ctXMqpVo214d7u1aJYbff8MrCqJXB1wL0yLRVUTunUqplc324NwWVOsheKRVlLSLczx/anYuGpcfs/JrtSqloiyjcRWSciKwVkQ0iMjHM49eLSK6ILLV/bmz4oh66SVcN45krYndlJh12qZSKNk9dO4hIPPBP4CwgG1goIp8YY4IbkN81xvyuEcrYYEQgFjmrwy6VUtEWSc19BLDBGLPJGFMOvANc2LjFahxxUsN01UbmC/cYnV4p1QJFEu7pwHbH/Wx7W7BLRGS5iLwvIj3DHUhEJohIlohk5ebmHkJxD0/wUgTXvLyA295Z0ujn9YW7ts4opaIlknAPV98MjqlPgQxjzBDgK+C1cAcyxrxkjMk0xmR27ty5fiVtAL6a+2/G9gNgzvq9/G/pzkY/r1dTXSkVZZGEezbgrIn3AAIS0RiTZ4zxLZj+H2B4wxSvYV2eab2Ma0dnRPW8ldrmrpSKskjCfSEwQET6iEgicCXwiXMHEenmuHsBsLrhithwHrzgGFY89DPapyYEbM+Y+Blz1lc3Ez0/az2Xvfg9+xro0nxebXNXSkVZneFujKkEfgd8iRXa7xljVorIwyJygb3brSKyUkSWAbcC1zdWgQ9HfJzQOslDkic+5LEXv93ov/3UjHUs3LKP79Y3TL+AjpZRSkVbnUMhAYwx04HpQdv+7Lh9D3BPwxYtusoqrJlGxtE+nlfUMDV3HeeulIq2iMK9JSiv8nL/xyvo36W1f1t+AzXLaM1dKRVtLTbcO6YmkucI77IKL1N+2Bqwj+/xyiov6/cUcXS38MsH10XDXSkVbS1ibZlwFt1/FteN7u2/n72vJGSft3/cxqKt+3jyy7Wc8+wcNuUWHdK5NNyVUtHWYsMd4IoTevlvF5dXhd1nwutZLNm+H4DcA2Vh96mLhrtSKtpadLinJIaOmglWXun1z+K64qUfGP3415RX1m+ZR1+Hqo6EVEpFS4sO9+SEml9+qi/4JXBNmpyCgyzP3l+v8/jGuWv9XSkVLS073MOMd/d5/JIhDDyiDa2TPMQF/SutyikM2X9q1nY+WbaTPYUHOW/SnIBL6mmzjFIq2lp2uCfUHO5pyR5G9ulASXlVyGqShaUVVHkNuwsPAlBWWcUf31/OrW8v4Y/vL2fFjkIembbKfxHuWIb77LV7uPO9ZTE7v1IqNlp0uCd5an75iZ44UhI9lAaFe3JCHE/NWMcV/57PyMe+Zl9xOQWlFf7Hv11XPas1e18pENs29+tfWcgHi7MDJmcppdyvRYd7XPAawA5JnjhSE+Mpr/IGrOqYlmytS5O1dR8ACzbnMenrDWGP8dlya321ptAsU1bPTmCfPfa3E6VU89Kiw702ifHx/tE0c9bv9W9vFTTC5uY3FodMfvLJ2roPY0xA+3usHKwIP9SzNtN/ymHEY1+zYFNeI5RIKdWYWny4d2mTxHlDuoVsT0qIIylMm3xiLU05wUrKqpi6KDvgwyHYFyt2ccsbiyI+5qEqPYRwX2aP7/d9S1FKNR8tdvkBnx/vO5PySi/TlufQv0trNuyxZqGmJnk4cLAiZP+E+HqEe0Uld72/PGT7D5vyWLxtH78Z25+b7WCv8hq8xtTr+PVRWsMkrdq0TrLeHsVllQ1dHKVUI2vx4Q5WbfzVG07gmO5tOeHRrwAr2CoqA9vK+3ZOrVf47isO/HDwGhj52FfsLrRmuo7q29H/WL97rUU3t/z13EN6DXWpqeZ+xtOzOW9Id+4468iQx1I13JVqtlp8s4zP2KO60LlNkv9+6yQPvxrTJ2CfdyaMIiE+sjEvl2f2YMf+0pDtvmAHuPhf30d0rF0FB3nqy7WH1TFbU5v7xtxinv16fdjHfH0ORWX1r/UrpWJLw70Gvgt7DOvVzr+tS5tkPMEzmhzm33O6//bBikMbnQLWlaEe+N8KDlZU4fUaHvlsFc9/s4Fb3ljE/I2H1rlZWh5anrqGR/ouD1hUFto8pZRq2jTc67Bu14GA+wm1dKh2TK2u+e8vPbRA9AXua/O3csKjX3Hj61m0sjt2Z6zazVX/+QGAwoMVzF67p9ZjrdhR4L8drlmmvKr2DyDfGjrFddTcT396NudNmlPrPkqp6NJwr8NVI6yVI0/IaA9AYi3NMs6RNAUlh3ahj+351U05Bw5WMmvNHjxhznn/xyu4/pWFbM0rrvFY9328wn+7pDy03fzx6Wv8t78J80Hhm2FbVEeb+6bcYlbsCF2SYdHWfXy1anetz20u9hWXc+NrC8krOrSVQZWKNg33IIlBHab3jj+aNY+MY+rNJwIwok+HiI4zqHvbQzr/KU9+E7LNGfg+vqUP/j5zHV6vYcWOAj7/KSdgnxxHm3/hwdCAfvX7Lf7bN7yykIyJn7Flr/Vh8cYPW/lkqTUJy9ehWlnl5Y9Tl/HQpysjmvF6yQvfc+PrWXXu52OMaRITvsJ544etfLV6T5OYs6BUJDTcg2TdfybL/ny2/35cnASsQfPrMX0jOs4D5w9qsDLtDaotGmPo1jYFgP8t3cm8jXs5b9JcbnlzMQD7S8pZt/sAHVIT/c/ZH+ElA+dssMbk/+njFf4F0goPVnDTlCye/Xo9Uxdl88q8Lcxe2zAXD3d6asZa+t07nco6motiIcleQbS+yz07GWN4dd5mrf2rqNBwD5KWnEDbVgk1Pi4S2WiZ5IR41v5lHKse/hnv3TSaMwZ28T82/57TuWZU71qebenZwQrw4HDvc8909hyoXhbAOVTxgufnctzDMzn7me9Y4+gveHrmOmatqW4iGf3412HPGW5Fht2FZXy5cjeTZlUvs3CgHsMja6vlPzNzHf3tYaCvfW/N9D0Q5ltGrPm+0R3qMg4AK3cW8uCnq8LOfVCqoWm4N7AF957BvInWqJkkTzytEj2M6NOBMwd19e/TKtHjv6qT84Lcwc482nrO3qLQWve8DXnE20n8nznVTQXLswsC9vvlqOqrTU1bnsM3a/dQWl5FTkH4NWPu+2hFREsV+NeoDwru8kov363LZfPe6r4AX5v9p8t2cvrTs/F6DftLyvlixS6e/Xo9lV5DRZXXv5BbQVBndFllFVe+NJ/zJ83lrQXb+O+cTXWWry7Ls/fzu7cWR/wtIdFeHvpwau4l9kSywjCT45RqaBruh+A3Y/vRNiWBOXedxvRbxwQ81jUtmfR2KSHPuTyzJz3aW9tTE+PZZ3e4Xja8R43n+dkxR9Rajt4dWgFWx2VN2iRXfwtZvHUfN7yykDunLq31uFtq6aT18YWcsyb7yLRVPP75aq6d/COnPTXbv90X1ne+t4xNucXM2bCX37+9xD87F+DzFbv8FyT3hV/ugTK255fw0Ker+GFTPj/tKODej37iL5+trrVsS7fvJ6+ojBkrd7G/ho7tW99ewrTlOWzJq7527p4DB8POSgbw2B+kdY0wqo2vgzqSiXAHDlZE9CH75oKtLN4W2fIQXq/h6RlrySkI7cM5HKt2FrJu94G6d1RRpTNUD8Fd4wZy17iB9XpOfJww9+7qcfCPXnQsby3YxvUnZZC9r5Qbx/Th0hfnB1yndXB67Z2ykXQ9tkn28OSlQ/jj+8v9QTb9p12h+yV5/E0tby/Y5t/euU1S2GvHfrc+l7SUBD5cnO3fVlNn4/6SCnq0B2OX+LrJP9LL/mDyufXtJf7b103+kScvHervjK3t202wyiovP//nPP/9EX068N5NowFraGj71ETS26X4+1H2FpXRv0tr1uwqZNw/5pDeLsX/zcvJF+pllYc+ocv3geipI9z3l5Rz3MMzuXhYOn+/4rha973vI2tEVE0zm3/cnM/stXu4a9xAVuUUMmnWBhZszvf/mzSE8c/NqbUMKja05h4j/bu05s/nDyLJE88jPx9M746pPHzBMf7HE+LFv7aLz+bHxwfcLy2v4s0bR4Yce8yATozIsEb19Ovcmssye3LTqbV3BD968bG8cPXxgDXG3qemi4JPW57DzW8sYkYNQx07Ojpzd9qjdiIdCLOvpIJnvlrnv1+fdXF8a+j7rHSM9T9v0lzG2qORfLNvcwpK2by3mHH/sAIq3KxiqA7mSJtlHv50Ff/8JnApaF+zTEItS00DrLfXN/pwyY6Qx9btPsDKnQUh252MMXyxYhcVVV4u//d8/jV7I8YYfN1FhYc4B+NwnfPsHB7+dBVfrNjF9vySup+gDouGexPibEKZdJUVtLPuPNW/TUS47YwB/vvxccJJ/TsxtEdgDf/qkb0Y3c9at+a4ntYM2y5tkms9d0bHVvQMqk0fqkRPHNeOzvDf/2ZtLlvzigPa57fV8Z+7naNTO1zg1hSywcsvl1RU8emynWRM/AyAiirDgk15LNlmrXi5c/9BvlsXOPKnvNIb0pdQFqYZCqylITImfsbMoA+5yfM28+SXa3lzwVa8XsOm3CJmrbHmEuwtKmPhlnz/vit3FrDW0fld01o++4rLOfuZ7zj3ubn2awn/b/Dmgm3c/MYiPlhU/a2qvMqL7yXlF5fHJFxX5xQyed5mbn5jERf9a17dT1CHRcO9CWmVVD3k0hdufTsHNkn8/vT+/PMXxzM4PY1/2jVtgkbwDE5vy+9P78+Ce8+ga5oV6s51c3yuGtHTfzujU2pAoB6OieMG0q1t9YfJ2z9u49QnZ9frAuFFdYyY+X7j3oAx8QUlFdz+zpKQpiFjCKlBX/HSD/7bT365NmRJ5iP/9Dnjn5sbEJ7h+hjACiyAX7+exejHv2ZTblHA4/d9tIJPl+/k9Ke/5QO7CWtZdgGXvTgfYwyrcwo597m5/Owf3/mfU+L4prJ42z6e/Wo9eUVlDHtkZsCxS2r4RjPPHs662nGt39LyKn/fzJ4DZYz52zd8umxn2OeD1bw1ee5mHpu+2t95br3+Kmau2u3/8HN+yE7N2s7wR2ayOqeQyiqvfy5GOOEGCUTLgk15Id9I1+wqDDvRLxKLt+2r8YM2lrTNvQFcf2JGrZfsi9SQ9LZckdmT/l1aB6wY6eSJj+PcId04N8wa9D7p7VIQEX+wAwFh6zOgSxvenTCKmat2k5acQFpyAu/dNJrL/z3fv8/MO05hza4D/N5uE+/eNpkqYwIWQOvUOtH/n3XNI+NITogPuNygT7gRkamJ8RSHCall2bU3PVz/ykJG9OnAL0f15mfHdGXowzNq3Leu4atfrQ5tWlqdU8jstbmcZY9y8tW0l27fT2WVF098HHsKD3LDqwv9z8kpsL4FpLcP7FDfV8Mcg8Xb9nHJC/NDtjtnBPsWl8vamh+wz2/fXFxnU5uzea2kvIoHPlkZ8Pjv315C7oEyBnZrw4n9OgU89tj0NUyeZ31QnjWoKyfYzXx/n7GOf3+3iUQfK1jgAAAScUlEQVRPHIO6pTHpqmH+59z30QrKq7yc8+wcTuzXke835rH4/rMC5luEk1dURsfWoZUPsCbrOd/H4Vzz8gLi44RXbxhR635gdSpf8dIP9O7Yim//eBpgfWD5muWSPHF8cMuJdfZ3+Xy5chc3TVnEoxcN5uqRdQ9vjiatuTeABy84hnvGH33Yx/HEx/HEpUP49SmB/2n/dskQnrliaI3PaxPUNh8uzI7pnhaybV9JOSP7duRP51VPuBrRpwN9OqUCcN/4o+nfpTXnD+3OELvpJykhnsuG9ww4zjsTRvlv+zoqO9bxH9qnd8fUkG3hQmt0mA+7Hzfnc+vbS/jFfxbUeg5nDbY+fv16FhkTP+Ou95cx164Nl1d6Ofe5uUz5YStfhulvWLGzkGte/jFgW03DTj9dFjij+MbXFpJfXE5JmGaZ4G8Xn/2UwwXPVzdt+C6HOGd9Lp+vCO0wv/SF8CuQPjxtFb/4zwIyJn7Ghc/P9dfEnR94m/cWU1RWyRNfrPFPbCuv9LJ0+37G/K16RnW8oy/he3uBu432N5lwM4/3FpXx8KerGP6Xr3jNMVv6oyXZ3P3+cj5dtpORj33NIvuDraisMmxz3Jz1e5m9Ntc/u9rnmpcXhMzn8I1S2+oYJbW/pLoPoqzSy90fWPMQHvxkJTdNCZxhXVnlDejz+Gy59Tcssddf8noN65vIyCGtuTcDl5/Qs9bHn7xsCK99v5XB6Wl0DzMME6yx9RNO6cuRXdvg9Rru+mC5v1Mx2Me/OYlcexSJz0MXHMO9H63g/vOO5rie7aio8pJfXE5+cbm/ZuX8z+0rR0K8UFFVc4PMf6/LpMpriI8TTvzrLAA6B9Xijk1vy6RfDGPME9/giZOQCVS1DQWtzYg+HeiYmhg2DJ3ey8oOuL929wHud6zb47S78CA/bg6sZf/7u/Dj8p3LPwB8tXoPU+ZvDZigFqmJH/7E5OtPCPlg8dlZwweM07LsAl6fv4Ubx/QNWM/olXlbmDJ/Kz/tqP3blCdMR3HWln0ke+Lp1yX0QzzzL1/5bz/wyUoe+GQlr95wAne8uwyAJdutv+vS7QV8syaX57/ZwPhjj+BfVw8Pe/6xT81m4jkDeeKLNQHfEnMPlPmbJXPDzA7OD/pmtSm32N9HE+zT5Tu5491l/OXng/mT4z1Q4bU+dP41ewNPzVjHF7ePYUCXNsSJVbs/bWAX9haVk7UlnwuPSw977Iam4e4C3dqmMPGcuodm3mt/uzDGkOiJY/yx4Zt22rYKnaU7rFd7Pr+teky/85uKMYZbzxjA2Y6JWh1SE5l156mkt0+hoKSCEY+Fzoh95oqhAR9GbZI8nHpUZ/9/xCPSkvl+4un+C5mvfOhnbMkr5vSnv63ztdamX+dUNuYW06N9Cif371RnuNfH5r11zxEIlt4uxd9p7BwlFE6vDq3Cdkav2FHAr+uxjk9NNu8tZtaa3SQ4lraO9JtPfJgF7p74Yg1PAOOPrX3Ohs/v36oeErtut1XrLyit4Hm732T6T7vILy5n4ZZ8kjxxId9o/vr5GoKdN2kOf714CEN6tA1oa1+4JZ9B3dJCrhEcvILq1rxi4uOEHu1bkWc3PwZ/MH+4eAeXDu/hX5bD18wz5VcjuPmNxVw3ujfzN+WxbncRpw/sEjB4orFIJAtANYbMzEyTlXX4b0bV9Blj+O1bi7lqRC/apiSweW8x8zbs5W+Xhm9qmrlqN79+PYvje7Xjw9+cFPCYbwx4sIuHpXPjmL4M6p5WY60LrCak9XuKuP/jFVw2vAfnD+3OtZOt2u7Zg7qyq/Agy7MLeOyiYzlvaDemZmXzyLRVNR5vULc03vr1yLBlcjprUFduO2MA502aG7B9ZJ8OXJ7ZkzunLgt5Tve2yQE17mtH96ZXh1Z1TuICGNKjbchs5YbSt3MqE8b0Ja+4nCe/XNugxxYJ7Zvp2SGFTq2T/COcwhndtyPz67iQ+5FdW5OZ0YG3HPM46uOZK4by9eo9TFueE/bxjqmJGEK/CQR76ZrhnF3HBMXaiMgiY0xmXftpm7tqdCLCv64ezpgBnRnSox0XHpdeY7CDtbzyCRnteeKSISGPOWs8fTtVf9V/7OJjGRTUr3D3uIHccWbg5QNH9e3IMHt46GkDu3Biv47cdsYAFt9/Fi9dm+lvWuiQmkhacgK/OrkPD54/iDEDOrHmkXEc2dVqqvqHPbno4uPTadcqkScvHcLFx4f/ur3h0XP4z7WZYTvp3rhxJMN7tw/YduFx3fn75UP5/p4zArYP7dEu7OznYJ444ZLja575XJPuYTrdL88MPc6sO8dy5Yhe/GZsv1qP5/z7OP0tzN8V4MoTeobtdN+eX1prsAMMTg/tUwq2bncRby3YFtB8WB93vLusxmAHyLObKesyYcoituU1/lBUDXfV5LRrlcjUm09kQNc2IY/FxwkXDUvnuauG8fntY3jl+hO4PLNHwMqdPreM7cdtZw5gzSPjmHHHKbz2f9ZoisHpbVn+4NmMP7Ybnvg47jjrSP+IDt/SAMkJ1f81rj+pD1N+NZLkhHh/QHdvl8LKh37Gr062LsV4WWZPhvWqDumlfz4LsOYZOGekfmeP0PBJiI+jd8fA+QXPXjmMi+1wfvm6TH8H8+h+HRk3+Ah+vPcMnrtqGG/8aiSnHtk55HVveGw8152Y4b+fluzh/ZtHB1xVLCFe+NO5VtNap9bWa7/ltP4hx3r84iEMtT8M7xp3FN/8Yaz/MRFhZNAS2B/cUj3z9YvbTwk5HsB5Q7sx7fcnM+33J/u3Zf3pTP7y88Gc1D+w43zqzaNpbzcRHtm1dY0fbn06VfcPnTukG0vuP8v/QRzso9+c6L/9zBVDefjCY7gis/Z+rUj1aF/3hy/Q4EtAhBNRm7uIjAOeBeKB/xpj/hr0eBLwOjAcyAOuMMZsadiiKmV5xjEl/7SBXTjNseJmOMkJ8RzZtQ1HOj4s0mpo8/SFe03DJx+5cDDDelnfLIL38XUED+3RlnatEpl8fSaZGYHh18sR5L4lAESEOXedFjDyxOeMo7tyxtFd+cPZR/nL1iUtmQuGdgdgeO/2LNicxwuzN1JW6eUSx7eHpy8byq7Cg/zWDu23bhxF9r4SznrmOy7P7MmvTu7D2KM60zE1iU+X7+SXI3vRp2MqlV4vf5i6jOG92xMfJ1x/Ym/ueHc/V2T2DBmy+M6EUdw5dRkfLrZm0x5vf8C1TUnwX7zmvCHd/DXegUe0oVWih8HpbQPWzulkH3fK/43096u0SfZwQkYHpt06hpP+Oov4uDiG9GjNjv2l3HbGgIBr/2Z0bMXR3dJYnVPIkV3a0D41kRl3WBMA/+/Vhcxas4cTMtrzp3MHMaRHu7BLJdx//iC255dwzcsL2FtUzou/HO5f/+g/12ZSUFrBH8I0n517bDdmrt5NeaWXk/t3YlfhQWavzaVNsoeRfToGjDz6xchevLVgW3TG+Rtjav3BCvSNQF8gEVgGDAra5zfAi/btK4F36zru8OHDjVKN4a6py0zvu6cd0nOvm7zA9L57mpm5cle9n/vj5jzT++5p5vxJc2rdb+WOArO/uDxke++7px1yueujtLzSVFV5a93H6639cae9Bw6aF2ZvMPlFZcYYY2as3GW25xcbY4ypqvIar9drVu4oML3vnmY25xYFnKOm15yzv9R/PK/Xa176dqNZv/uA2bGvxNzz4XJTWl7pf+7qnAL/89btKjTllVUBxyooLTcfLt4e8evJLyoz63YVGmOMeWz6KnP8wzP8/x7Ltu8zD/xvhel99zRzwaQ55uYpWaawtNz8lL3f9L57mlm8Nd+s333A9L57mvl27R5TUVllissqzOS5m8y9Hy43ewoPmt53TzOvztsccXmCAVmmjnw1xkRUcx8BbDDGbAIQkXeACwFnL9OFwIP27feB50VE7IIoFVVPXDqEJy4N365bl2O6pzF7bS4dWkc2Tt+pbYr1baBPDW3NPsF9A9EWrgkrWKTXLQDo2DqJm0+tbn8/yzFqyjfSaVD3tJDasu8cncJMYDrC0f4vIgFzPx676NiAfY9yfCML15SXlpzARcMi74Non5pIe7uZbuK4gUwcN9Bf1iE92lHpNbz6/RauGZ3BpfaqroPT2wa8vs2Pj/c/xxMfxw0nWc13VV5DnNS8ZlNDiiTc04HtjvvZQPBqVf59jDGVIlIAdAQCximJyARgAkCvXr1Qqqm548wjOWVAZ3/zQn0c2bUNL/7yeE4eENoOHol3JowKu0yEm02/dQxd0g7tNb96wwnkFZXX64OovsId+/he7Zk38fRaO7drKlN8nPD/zjoyoH+msUQS7uFKGVwjj2QfjDEvAS+BNRQygnMrFVWe+DhG1rD0QyTGDa55WYi61LTkhJsdzreYsUfV3tfSmCIZtVST350+oO6dGkAko2WyAWdXcg8geMUh/z4i4gHaAvkopZSKiUjCfSEwQET6iEgiVofpJ0H7fAJcZ9++FJil7e1KKRU7dTbL2G3ovwO+xBo5M9kYs1JEHsbqtf0EeBmYIiIbsGrsVzZmoZVSStUuonHuxpjpwPSgbX923D4IXNawRVNKKXWodIaqUkq5kIa7Ukq5kIa7Ukq5kIa7Ukq5UMzWcxeRXGBrnTuG14mg2a9NSFMtm5arfrRc9aPlqp/DKVdvY0yd06BjFu6HQ0SyTASL1cdCUy2blqt+tFz1o+Wqn2iUS5tllFLKhTTclVLKhZpruL8U6wLUoqmWTctVP1qu+tFy1U+jl6tZtrkrpZSqXXOtuSullKqFhrtSSrlQswt3ERknImtFZIOITIzyuSeLyB4RWeHY1kFEZorIevt3e3u7iMhzdjmXi8jxjViuniLyjYisFpGVInJbUyibiCSLyI8isswu10P29j4issAu17v2UtKISJJ9f4P9eEZjlMtRvngRWSIi05pKuURki4j8JCJLRSTL3tYU3mPtROR9EVljv89Gx7pcInKU/e/k+ykUkdtjXS77XHfY7/kVIvK2/X8huu+vSC602lR+iOBi3Y18/lOA44EVjm1/AybatycCT9i3xwOfY12lahSwoBHL1Q043r7dBlgHDIp12ezjt7ZvJwAL7PO9B1xpb38RuMW+Xe8LrR9m+f4f8BYwzb4f83IBW4BOQduawnvsNeBG+3Yi0K4plMtRvnhgF9A71uXCuuzoZiDF8b66Ptrvr0b9B2+Ef7TRwJeO+/cA90S5DBkEhvtaoJt9uxuw1r79b+CqcPtFoYz/A85qSmUDWgGLsa6/uxfwBP9Nsa4ZMNq+7bH3k0YqTw/ga+B0YJr9H74plGsLoeEe078jkGaHlTSlcgWV5WxgXlMoF9XXlO5gv1+mAT+L9vuruTXLhLtYd3qMyuLT1RiTA2D/9l3YMSZltb/SDcOqJce8bHbTx1JgDzAT65vXfmNMZZhzB1xoHfBdaL0x/AO4C/Da9zs2kXIZYIaILBLrgvIQ+79jXyAXeMVuxvqviKQ2gXI5XQm8bd+OabmMMTuAp4BtQA7W+2URUX5/Nbdwj+hC3E1E1MsqIq2BD4DbjTGFte0aZlujlM0YU2WMOQ6rpjwCOLqWc0elXCJyHrDHGLPIuTnW5bKdZIw5HjgH+K2InFLLvtEqlwerOfIFY8wwoBiruSPW5bJOZrVdXwBMrWvXMNsa4/3VHrgQ6AN0B1Kx/p41nbtRytXcwj2Si3VH224R6QZg/95jb49qWUUkASvY3zTGfNiUygZgjNkPzMZq62wn1oXUg88drQutnwRcICJbgHewmmb+0QTKhTFmp/17D/AR1gdirP+O2UC2MWaBff99rLCPdbl8zgEWG2N22/djXa4zgc3GmFxjTAXwIXAiUX5/Nbdwj+Ri3dHmvDj4dVjt3b7t19o99KOAAt9XxYYmIoJ1HdvVxpi/N5WyiUhnEWln307BetOvBr7BupB6uHI1+oXWjTH3GGN6GGMysN5Ds4wxV8e6XCKSKiJtfLex2pFXEOO/ozFmF7BdRI6yN50BrIp1uRyuorpJxnf+WJZrGzBKRFrZ/zd9/17RfX81ZidHY/xg9Xivw2q7vS/K534bqw2tAuvT9ldYbWNfA+vt3x3sfQX4p13On4DMRizXyVhf45YDS+2f8bEuGzAEWGKXawXwZ3t7X+BHYAPWV+kke3uyfX+D/XjfKPxNx1I9Wiam5bLPv8z+Wel7f8f672if6zggy/5bfgy0byLlagXkAW0d25pCuR4C1tjv+ylAUrTfX7r8gFJKuVBza5ZRSikVAQ13pZRyIQ13pZRyIQ13pZRyIQ13pZRyIQ13pZRyIQ13pZRyof8Ptk2DOARxJc4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#import sys\n",
    "#print(sys.path)\n",
    "#print(sys.executable)\n",
    "\n",
    "import cnn_utils\n",
    "from matplotlib import pyplot as plt\n",
    "import keras\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Flatten\n",
    "from keras.layers import Conv2D, MaxPooling2D\n",
    "import keras.callbacks as cb\n",
    "\n",
    "class LossHistory(cb.Callback):\n",
    "    def on_train_begin(self, logs={}):\n",
    "        self.losses = []\n",
    "\n",
    "    def on_batch_end(self, batch, logs={}):\n",
    "        batch_loss = logs.get('loss')\n",
    "        self.losses.append(batch_loss)\n",
    "\n",
    "def init_model():\n",
    "    model = Sequential()\n",
    "    model.add(Conv2D(32, kernel_size=(3, 3),\n",
    "                    activation='relu',\n",
    "                    input_shape=(28, 28, 1)))\n",
    "    model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "    model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "    model.add(Dropout(0.25))\n",
    "    model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "    model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "    model.add(Dropout(0.25))\n",
    "    model.add(Flatten())\n",
    "    model.add(Dense(128, activation='relu'))\n",
    "    model.add(Dropout(0.5))\n",
    "    model.add(Dense(10, activation='softmax'))\n",
    "\n",
    "    model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer=keras.optimizers.Adadelta(),\n",
    "              metrics=['accuracy'])\n",
    "    return model\n",
    "\n",
    "def plot_losses(losses):\n",
    "    plt.plot(losses)\n",
    "    plt.title('Loss per batch')\n",
    "    plt.show()\n",
    "    \n",
    "filePath = 'C:/Users/computer/Desktop/Assignment8/'   # the training set is stored in this directory\n",
    "mnistData= cnn_utils.loadData(filePath)\n",
    "mnistData_preprocessed = cnn_utils.preprocessData_CNN(mnistData)\n",
    "(X_train, y_train), (X_test, y_test) = mnistData_preprocessed\n",
    "\n",
    "(X_train_mini_batch, y_train_mini_batch) = cnn_utils.mini_batch(X_train, y_train, 10000, 0)\n",
    "(X_test_mini_batch, y_test_mini_batch) = cnn_utils.mini_batch(X_test, y_test, 5000, 0)\n",
    "print(X_train_mini_batch.shape)\n",
    "print(y_test_mini_batch.shape)\n",
    "\n",
    "model = init_model()\n",
    "history = LossHistory()\n",
    "model.fit(X_train_mini_batch, y_train_mini_batch, epochs=20, batch_size=256,\n",
    "                callbacks=[history],\n",
    "                validation_data=(X_test_mini_batch, y_test_mini_batch), verbose=2)\n",
    "\n",
    "score = model.evaluate(X_test_mini_batch, y_test_mini_batch, batch_size=16)\n",
    "print(score)\n",
    "plot_losses(history.losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As illustrated after 20 epochs we reached a cost of approximately 0.0449 and a test accuracy of 0.9864. The results are very good and reflect upon the power of CNNs. These results are slightly higher than the nextwork with less layers. This may line up with the intuition that if we have enough data, we can learn additional features by deepending the network which increases our accuracy. \n",
    "\n",
    "We will now investigate the impact of the optimaization algorithm used by swithing from AdaDelta to ADAM. We will also use the first version of the network for comparison.\n",
    "\n",
    "We will use the following:\n",
    "\n",
    "1. CNN architecture: \n",
    "    a. Convolution layer (kernel 3X3) then MaxPooling (3X3), Relu\n",
    "    b. Convolution layer (kernel 2X2) then MaxPooling (2X2), Relu, Dropout (Keep Probability 0.25)\n",
    "    c. Dense layer (units 128), Relu, Dropout (Keep Probability 0.5)\n",
    "    d. Output layer (units 128), Softmax\n",
    "2. Dropout: \n",
    "    a. As specified in 1\n",
    "3. Optimization algorithm: \n",
    "    a. Adam\n",
    "4. Minibatches: \n",
    "    a. 256\n",
    "5. Epochs:\n",
    "    a. 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10000, 28, 28, 1)\n",
      "(5000, 10)\n",
      "Train on 10000 samples, validate on 5000 samples\n",
      "Epoch 1/20\n",
      " - 2s - loss: 0.8476 - acc: 0.7388 - val_loss: 0.2533 - val_acc: 0.9256\n",
      "Epoch 2/20\n",
      " - 1s - loss: 0.3085 - acc: 0.9096 - val_loss: 0.1384 - val_acc: 0.9568\n",
      "Epoch 3/20\n",
      " - 1s - loss: 0.2061 - acc: 0.9390 - val_loss: 0.1049 - val_acc: 0.9664\n",
      "Epoch 4/20\n",
      " - 1s - loss: 0.1512 - acc: 0.9535 - val_loss: 0.0878 - val_acc: 0.9724\n",
      "Epoch 5/20\n",
      " - 1s - loss: 0.1176 - acc: 0.9648 - val_loss: 0.0970 - val_acc: 0.9714\n",
      "Epoch 6/20\n",
      " - 1s - loss: 0.1141 - acc: 0.9647 - val_loss: 0.0684 - val_acc: 0.9776\n",
      "Epoch 7/20\n",
      " - 1s - loss: 0.0904 - acc: 0.9714 - val_loss: 0.0632 - val_acc: 0.9794\n",
      "Epoch 8/20\n",
      " - 1s - loss: 0.0712 - acc: 0.9796 - val_loss: 0.0618 - val_acc: 0.9800\n",
      "Epoch 9/20\n",
      " - 1s - loss: 0.0668 - acc: 0.9802 - val_loss: 0.0575 - val_acc: 0.9826\n",
      "Epoch 10/20\n",
      " - 1s - loss: 0.0593 - acc: 0.9806 - val_loss: 0.0588 - val_acc: 0.9820\n",
      "Epoch 11/20\n",
      " - 1s - loss: 0.0487 - acc: 0.9849 - val_loss: 0.0520 - val_acc: 0.9828\n",
      "Epoch 12/20\n",
      " - 1s - loss: 0.0417 - acc: 0.9873 - val_loss: 0.0556 - val_acc: 0.9818\n",
      "Epoch 13/20\n",
      " - 1s - loss: 0.0525 - acc: 0.9821 - val_loss: 0.0509 - val_acc: 0.9828\n",
      "Epoch 14/20\n",
      " - 1s - loss: 0.0393 - acc: 0.9866 - val_loss: 0.0541 - val_acc: 0.9836\n",
      "Epoch 15/20\n",
      " - 1s - loss: 0.0385 - acc: 0.9872 - val_loss: 0.0526 - val_acc: 0.9830\n",
      "Epoch 16/20\n",
      " - 1s - loss: 0.0437 - acc: 0.9852 - val_loss: 0.0504 - val_acc: 0.9848\n",
      "Epoch 17/20\n",
      " - 1s - loss: 0.0298 - acc: 0.9915 - val_loss: 0.0545 - val_acc: 0.9836\n",
      "Epoch 18/20\n",
      " - 1s - loss: 0.0262 - acc: 0.9925 - val_loss: 0.0514 - val_acc: 0.9840\n",
      "Epoch 19/20\n",
      " - 1s - loss: 0.0256 - acc: 0.9911 - val_loss: 0.0569 - val_acc: 0.9836\n",
      "Epoch 20/20\n",
      " - 1s - loss: 0.0242 - acc: 0.9917 - val_loss: 0.0529 - val_acc: 0.9844\n",
      "5000/5000 [==============================] - 1s 146us/step\n",
      "[0.052882575372658176, 0.9844]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xl4VNX9x/H3N5ksbGELm4BsKuCGIop7XVvUtlprq63V2lZtbW1rd1xL/dlqW7XWpdpFa63aqsUdBEVxAQXZl7BvgbAGyAJkmyTn98fcGWYyM8kEkkxm+LyeJ09m7ty595tk8pkz5557rjnnEBGR9JKR7AJERKTlKdxFRNKQwl1EJA0p3EVE0pDCXUQkDSncRUTSkMJd5CCY2QQze7YN9jPYzJyZ+Vp7X5IeFO7SJsxsg5ldkOw62hMzu87MZiS7DklPCneRBKnVLKlE4S5JZ2Y3mNkaM9ttZq+b2WHecjOzP5nZDjMrM7PFZnas99jFZrbMzPaY2WYz+3mcbV9nZjPN7BFvGyvM7Pywx7ua2ZNmttXbzj1mltnguX8ys93AhDg/Qq6ZveDVMt/MRoVtf7yZrfUeW2ZmX/KWjwSeAE4zs71mVuot72BmD5hZoVfvDDPrELavq81so5ntNLPbD/R3LulP4S5JZWbnAfcCXwX6AYXAf72HPwucDRwFdAOuBHZ5jz0JfNc51wU4Fnivkd2MBdYB+cCvgZfNrIf32L+AWuAI4ERvn9fHeG5v4Ldxtn8p8BLQA3geeNXMsrzH1gJnAV2B3wDPmlk/59xy4HvAJ865zs65bt769wMnAad72/slUB+2rzOB4cD5wF3em4RIFIW7JNvVwFPOufnOuWrgVgKt2cGAH+gCjADMObfcObfVe54fONrM8pxzJc65+Y3sYwfwkHPO75x7AVgJXGJmfYCLgFucc/ucczuAPwFXhT13i3PuEedcrXOuMs725znn/uec8wMPArnAqQDOuZecc1ucc/XevlcDp8TaiJllAN8Gfuyc2+ycq3POfez9XoJ+45yrdM4tAhYBo2JtS0ThLsl2GIHWOgDOub0EWuf9nXPvAY8CjwHbzexvZpbnrfpl4GKg0Mw+MLPTGtnHZhc5Q16ht99BQBaw1cxKva6RvxJopQdtSuBnCK3jnKsHirztY2bXmtnCsO0fS+ATRCz5BN4Y1jayr21htyuAzgnUJ4cghbsk2xYCIQuAmXUCegKbAZxzDzvnTgKOIdA98wtv+Rzn3KUEgvhV4MVG9tHfzCzs/uHefjcB1UC+c66b95XnnDsmbN1Epk0dGFZ/BjAA2GJmg4C/AzcDPb2ul6VAsJaG294JVAHDEtinSKMU7tKWsswsN+zLR6CP+ltmdoKZ5QC/A2Y75zaY2clmNtbrv95HIPjqzCzbzK42s65eV0g5UNfIfnsDPzKzLDP7CjASmOx18bwNPGBmeWaWYWbDzOwzzfy5TjKzy72f5xYCbxizgE4EArwYwMy+RaDlHrQdGGBm2RBq9T8FPGhmh5lZppmd5v1eRJpF4S5taTJQGfY1wTn3LnAnMBHYSqDVGuzzziPQ8i0h0JWyi8ABR4BrgA1mVk7gwOQ3GtnvbOBIAi3j3wJXOOeCB2avBbKBZd5+/kfgwG5zvEbgYG+JV9flXv/+MuAB4BMCQX4cMDPsee8BBcA2M9vpLfs5sASYA+wGfo/+T+UAmC7WIenMzK4DrnfOnZnsWkTakloEIiJpSOEuIpKG1C0jIpKG1HIXEUlDSZsIKT8/3w0ePDhZuxcRSUnz5s3b6Zzr1dR6SQv3wYMHM3fu3GTtXkQkJZlZYdNrqVtGRCQtKdxFRNKQwl1EJA0p3EVE0pDCXUQkDSncRUTSkMJdRCQNpVy4r9y2hwfeXsmuvdVNrywicohKuXBfW7yXR95bw449CncRkXhSLtw7ZGUCUOVv7MI7IiKHtpQL95ysQMlV/vokVyIi0n6lXLir5S4i0rSUC/dchbuISJNSLtxDLfdahbuISDwpF+7BlntljfrcRUTiSblwV5+7iEjTUi7cQ6Nl1C0jIhJX6oW7LwMzqKpRuIuIxJNy4W5m5PoyqVS3jIhIXCkX7gCdcnzsrVa4i4jEk5Lh3iXXx97q2mSXISLSbqVkuHfO8bG3yp/sMkRE2q3UDXe13EVE4krNcM/1sadK4S4iEk9KhnsXtdxFRBqVkuHeSeEuItKolAz3HF8G/lrNLSMiEk9KhnuWLwN/nUt2GSIi7VZqhntmBjV19TingBcRiSU1wz3DAKitV7iLiMSSmuHuC5Ttr1O/u4hILE2Gu5kNNLPpZrbczArM7Mcx1jEze9jM1pjZYjMb3TrlBmRleuFeq5a7iEgsvgTWqQV+5pybb2ZdgHlm9o5zblnYOhcBR3pfY4HHve+tIjsz0C3jr1fLXUQkliZb7s65rc65+d7tPcByoH+D1S4FnnEBs4BuZtavxav1hFru6pYREYmpWX3uZjYYOBGY3eCh/sCmsPtFRL8BYGY3mtlcM5tbXFzcvErDqFtGRKRxCYe7mXUGJgK3OOfKGz4c4ylRyeuc+5tzboxzbkyvXr2aV2mY4AHVGrXcRURiSijczSyLQLA/55x7OcYqRcDAsPsDgC0HX15swaGQ6pYREYktkdEyBjwJLHfOPRhntdeBa71RM6cCZc65rS1YZwT1uYuINC6R0TJnANcAS8xsobfsNuBwAOfcE8Bk4GJgDVABfKvlS91P49xFRBrXZLg752YQu089fB0H/KClimpKVnAopOaXERGJKSXPUM1Wt4yISKNSMtzV5y4i0riUDvcajXMXEYkpRcNdQyFFRBqTouGubhkRkcakZrhrKKSISKNSM9w1FFJEpFEpGe4aCiki0riUDHf1uYuINC7Fw13dMiIisaRouAf63Gtq1XIXEYklJcPdzPBlmLplRETiSMlwh0DXjMJdRCS2FA53U5+7iEgcKRvu2T613EVE4knZcFe3jIhIfCkb7j51y4iIxJWy4Z6VmUGNWu4iIjGlbLhnZ2bg1zh3EZGYUjbc1ecuIhJfCoe7UVuvPncRkVhSONwzNP2AiEgcKRvuGucuIhJfyoZ7YG4ZdcuIiMSSsuGuA6oiIvGlbrj7NM5dRCSelA33bLXcRUTiStlwz8o0atXnLiISUwqHu1ruIiLxpHS4a5y7iEhsKRzuGgopIhJPCoe7umVEROJJ6XCvrXfUa34ZEZEoKRvu2b5A6f56td5FRBpqMtzN7Ckz22FmS+M8fo6ZlZnZQu/rrpYvM1pWpgFoOKSISAy+BNZ5GngUeKaRdT5yzn2+RSpKUFam13JXv7uISJQmW+7OuQ+B3W1QS7MEw11TEIiIRGupPvfTzGyRmb1lZsfEW8nMbjSzuWY2t7i4+KB2GOyW0XBIEZFoLRHu84FBzrlRwCPAq/FWdM79zTk3xjk3plevXge101C3jE5kEhGJctDh7pwrd87t9W5PBrLMLP+gK2uC+txFROI76HA3s75mZt7tU7xt7jrY7TZlf7irW0ZEpKEmR8uY2X+Ac4B8MysCfg1kATjnngCuAG4ys1qgErjKOdfqiZvtC/a5q+UuItJQk+HunPtaE48/SmCoZJtSt4yISHwpe4aqhkKKiMSXwuGuoZAiIvGkcLhrKKSISDypH+7qlhERiZL64a4pf0VEoqRsuGerW0ZEJK6UDfcsjXMXEYkrdcNdfe4iInGlbrhnBMe5q89dRKShlA33nKxA6VX+uiRXIiLS/qRuuPsyyDCorFG4i4g0lLLhbmZ0zPZRoXAXEYmSsuEO0DE7k4qa2mSXISLS7qR0uHfK8bFPLXcRkSgpHe4dsjKpVMtdRCRKSod7p5xM9lWr5S4i0lBKh3vggKpa7iIiDaV4uGdqtIyISAwpHe45vgyqNXGYiEiUFA/3TKpr1XIXEWkopcM925dBjVruIiJRFO4iImkopcNdfe4iIrGldLhn+zKorXfU6VJ7IiIRUjrcc3yZAOqaERFpIKXDPdvnXbBD4S4iEiGlwz3HC3cNhxQRiZTS4Z4dCne13EVEwqV0uAdb7jW6SLaISIS0CPdqv8JdRCRciod7YLRMlfrcRUQipHS453XwAbCnStP+ioiES+lw79ohC4DSipokVyIi0r6keLhnA1Be6U9yJSIi7UuT4W5mT5nZDjNbGudxM7OHzWyNmS02s9EtX2Zs+1vuCncRkXCJtNyfBsY18vhFwJHe143A4wdfVmKyfRl0zM6kVC13EZEITYa7c+5DYHcjq1wKPOMCZgHdzKxfSxXYlCH5nZhXWNJWuxMRSQkt0efeH9gUdr/IWxbFzG40s7lmNre4uLgFdg1nHdmLxUWlLbItEZF00RLhbjGWxZyD1zn3N+fcGOfcmF69erXArgMXya53UKuzVEVEQloi3IuAgWH3BwBbWmC7CcnWFAQiIlFaItxfB671Rs2cCpQ557a2wHYTkp2paX9FRBryNbWCmf0HOAfIN7Mi4NdAFoBz7glgMnAxsAaoAL7VWsXGojndRUSiNRnuzrmvNfG4A37QYhU1k6b9FRGJltJnqIKm/RURiSXlw1197iIi0VI/3NUtIyISJeXDPTinu1ruIiL7pXy4B1vukxa32dB6EZF2L23C/V+fFCa5EhGR9iPlwz0rM9bsByIih7aUD/cqXRxbRCRKyof7CQO7AdAnLyfJlYiItB8pH+6ZGcbnjulDN++SeyIikgbhDuDLzMBfr+4ZEZGgtAj3rAyjti7mFPIiIoektAh3X2aGLtYhIhImLcI9K9Pw16vlLiISlBbh7stQy11EJFx6hHum+txFRMKlRbhnabSMiEiEtAh3X4ZR5a/n/qkrqaypS3Y5IiJJlx7h7l2w49Hpa3hq5vokVyMiknzpEe4Z+ycP8+vAqohIeoR7lX9/V0xWZlr8SCIiByUtknBPVW3odmaGpgAWEUmTcPeHbvsU7iIi6RLu+1vu6pYREUmTcO+dlxu6naGWu4hIeoT7nZ8fuf/2q0sZPH4S//l0YxIrEhFJrrQI947Zvqhli4vKWnQfzjkWbCzBOU1zICLtX1qEO8ApQ3pE3A8/yNoSJi/Zxpf+8jEvz9/cotsVEWkNaRPuz18/NnT7uP5dKfcOshbvqaaipjbe0xK2fudeANZ530VE2rO0CXdf2CiZbh2zQi33k387jcv/8vFBb1+9MSKSStIm3MPl5WZRXrm/W2bFtj0ttm1Do3FEpP1Lz3Dv4GN7eTVH3zUl2aWIiCRFeoZ7bhZ7q2upCJv+t7SiJokViYi0rbQM9y650UMjb391aRIqERFJjoTC3czGmdlKM1tjZuNjPH6dmRWb2ULv6/qWLzVxeR2yopap5S4ih5LoJm4DZpYJPAZcCBQBc8zsdefcsgarvuCcu7kVakzYhUf34ZjD8sjLjQ53XYVPRA4lTYY7cAqwxjm3DsDM/gtcCjQM96T7+7VjAHh3+faox+oOciyjRkKKSCpJpFumP7Ap7H6Rt6yhL5vZYjP7n5kNjLUhM7vRzOaa2dzi4uIDKDcxsbplPl2/m49WH/w+TSMhRSQFJBLuseKsYUP2DWCwc+54YBrwr1gbcs79zTk3xjk3plevXs2rtBl6d8mJufzbT89ptX2KiLQniYR7ERDeEh8AbAlfwTm3yzlX7d39O3BSy5R3YAZ27xhzeYeszAPeps5QFZFUkki4zwGONLMhZpYNXAW8Hr6CmfULu/tFYHnLldh8GRnGj847Imp5p5zAIYZNuyuYsXrnAW1bvTIikgqaDHfnXC1wMzCVQGi/6JwrMLO7zeyL3mo/MrMCM1sE/Ai4rrUKTtRPPzucN394ZsSyDtmBlvt1//yUbzw5m399vKHZ21UDXkRSQSKjZXDOTQYmN1h2V9jtW4FbW7a0g3ds/64R9zt64b57X2DM+69fL+Cbpw9u67JERFpdWp6hGk/fvA4AjBrYDYCj+nROZjkiIq3mkAn3o/p0pqikgj1VfvZVB+Z3r/TXNfGsaOpzF5FUkFC3TDrIy81ibmEJx014O7SstKJlr9YkItJeHDIt9+BImXB7qmqprUtsXgKnQ6kikkLSPtwn3nQ6N597BNm+2D9qWaWfmtp6fvLCQj5clcAZrDpFVURSQNqH+0mDuvPzzw2nrj6y5T24Z+BEp5++uIij7niLVxZs5tqnPk1GiSIiLS7twz2e/t0DI2c+aNBa37S7IhnliIi0qEMm3Pt2zY2437NT7PlnFheVxVwemn5A8xCISAo4ZML9shMiJ7K85Ph+Mdf7wfPzmfB6Af6wA60T5xXx6sLNANQ72FPlj3hcRKS9OWTC/ZQhPZh6y9kAXHxcX/p36xB33ac/3sC7y7fz5Iz13P3GMn720iIKdwW6a+qd47gJb/PD5xc0a/8z1+xk197qpldsYeVVflZu29Pm+xWR5DpkxrkDDO/bhcevHs2ZR+ZTvKfxoP3es/NjLq/1DsxOKdiW8H7r6h1X/2M2w/t0YepPzk684BZwzT9ms6iojA33XdKm+xWR5DpkWu5BFx3Xjy65WXTMjnxf69kpO6Hn76lq/olPwS6cldvbvgW9yDuG0HC0kIikt0Mu3IM65eyf2/2CkX2YcktiLeqyyvjhXlbpZ/D4SUxavJWFm0qp9wK1ujb5/fNVBzDVgoikrkM23LvkZvHgV0cBcPslI8nvnFjLvbyyNu5ja3bsBQIHZS97bCZPzVwPQI3CXUTa2CEb7gCXjx7AhvsuYUh+JyzBM08bdst8++k5PDurEIDqBgH6wpxNDB4/iRXbylum4INQ1Q7eYESk7RzS4d6YJ74xmt9+6dio5eVV+1vuNz07j/dW7OCOV5cC0d0vq72W/MR5Ra1YaWLUchc5tCjc4xh3bD86x5hsrDysz/2tpZEjZgq2xD4BaktpVeh2skJW4S5yaFG4h+nfrQMnDOzGlFvOAogaUQOwy7uKU0OfrN3F/W+vivnY1vLK0O0X5mxqso6de6s54773WNWCo2uq/OqWETmUKNzDzBx/Hq/+4AxG9M0DoGuHrISfe+9b8a8JXrpvf2s/3uyU4d5dvp3NpZX87cN1Ta77wNsrGTx+UpPrPTljHT95YWGT64lIejikTmJqrh5xxr7n+DI4vEfHUJ86xJ+TBmBP9f5++g5ZmVGPT1+xg289PYexQ3owdmjP0LVe6xvMY1NbV095VW1EXY+8tyawbr0jIyP+QeHJSwJdSH+68oS464hI+lDLvRHxTmzqnOPjl+NG0KNTdsx++caE930/N7uQrWWVPDQt0J0ze/1uHn53Nfe9tSKwQoPzju58bSmj/++dmEMrq2oT61OfvnIHU5YmfnatiKQmtdwb0bBb5qRB3ZlXWELHnEwuPLoPFx59IWuL93L+Ax8kvM3xLy9hb3Ut01fuYOaaXXTvmEVJnMv9vbxgM8cP6Mp1ZwwBYOK8wORl+6pryfZFvvFU1tTFPEbQ0Lf+OQdA0xGIpDm13BvRsJvjG6ceDsAXRx0WWjasV+dmb/eeScuZuWYXQNxgD183KNhNs6cq+kSqipoDHw2zdHMZD01bxZ+nrY5Yvml3BZc+NpN5hSX87MVFmglTpAVMLdjWooMl4lG4N+HDX5zL7RePBOD0Yfl8PP48fnbh8Ih1jujdmX5dc/nol+dGPb+RbvCE1NY7JrxewITXC0KTlj33aSGuQX98sLsnvNunsTNjq8O6cT7/yAwemraaP02LHO3z1w/XsmhTKV9+/GMmzi9ifmHJwf0w0qjLHpvZ6IF5SQ8/fH4BryzY3Or7Ubg34fCeHbn+rCGs+e1F9MnL5bBuHaJa9NN++hk+ufV8BvboGPX8zjk+Tjy8G5kHkfJPf7yBpz/eELr/1w/WMXH+ZraW7R9iWemvY/X2PYy4cwpvLt4CQGll7GGbADvKq5lXWMIbi7YkXEfwzaXKX8e9by2nLOxTR1mFP+GLjTfX5CVbY47T37m3Oq0mRFu4qZS/ftD0CKn2Zu6G3a32t0839fWOmrp6sjNbP3rV554AM8OXeWDhfP7IPvzxiuPZV1PHqN+83WI1/fylRRH3K2rqeHPxVgBufn4B64r3Me7YvnGf/8epK3l72bao8e81tfWh4ZpG5M8c7Pp55pMN/PWDdXTO9vHD84+ksqaOUXe/zQ1nDeGYw7pSvKea688akvCUDo2ZvW4X339uPtedPpgJXzwmtLy8ys+Ye6bxnTOHcOfnjz7o/ciBWbSplCue+IQfnncEP/vs8KafcIir8d4Ec7IU7iln1ICuLCoqY/rPz6Guvp6BPTriy8yga4fIP+bzN4zlpblFLfbxrKzSz4qwi3I8+M4qHnwn9klVAK/HabFvLatk1fa9EZ8KgkoqAp8Elm0JzJXTtWMWe6r8odE3f/9ofWjdkf3yOPPIfP43r4h/zyrktR+c0fwfiv3HJIpKIusp8U4mm7J0W5PhvnRzGfdMWsbT3zqF3KxMauvq8dc5OmRHD0uV5tnpXYCmYEvy509KBdVeYyrH1/qvPXXLtLDXbj4zNBnZEb27RPwRbzx7KBAIvtOH5Tc65rxvXm7cx2L57r/n8WGDi30fiM/88X1ueGYud71WEHWQttQL973VgeWz1u3iuAlv84v/LQYIjc8H+MaTs4HAJ4xFm0p5asZ6bnhmbrPnww8eW2jYqxU8qJzlfaJ68J1VXOPt858z1/P1v88KrXv7q0uZtW53KICuf2YuI++a0qw6Wlt9ffQxlOMmTA11sbVXwS7K2rD6n51V2C4my2uPqusC/zs5CZzMeLAU7m3ototHsuG+S3jrx2eFlgUDf8avzuXVsNbtrNvOD43OCbfgzgtbpbb7vzIqatmSzaUR9z9dX8L3/j0vNIfOp+sjD7A2fDO4f+rK0O2731zGO8u285f310btxznHTc/OY+itk7jnzWURB4vrQuEeme4frd4JQJbXd/nwu6v5aPVO6usdv3ljGR+v3RXVT19UErhU4vsri0P7bai+3sVc3toanqewvbyKPVW1/OzFRawt3svvJi8PHV+oqIk/7XSb835VwTenunrHHa8u5QuPzEhiUe3X/pa7wj3tBQN/QPeOnDCwW8RjV48dFPUi6N7IFaN+/+XjYp4BG7TudxfHfHMYmt8JX4wDvqu27424P235dqYUbGNrWWAitJ1NXBP20elropY9/v5a3lqyNXS/uraObeVVvLV0G/UO/jFjPUNuncwLczYyr7AkNEd++AHphZtK+f2UwIlepZX+iGvT7gi7fOJfpq/h6ZnrqasP/EP9+L+R0y9U+utYv3MfrywIzNo5Y/VOht42mcsemxnz5ynYUsaW0ujuqnjmFe5m2rLtMbcT/BQU1PDYR6nXHVVdW8/5D3zA3z5cx6KiUl5ZUMTRd01l/c59CdfREjbuqoh5UDv4hh584wlOrOeva903yOdmFzJ4/CQqD2IIcDIEZ47NaeT/tKWoz72dyc3KCP2jj+yXx8p7LgrNHfPXa04C4KtjBvDi3CI+ufU8npqxnitPHkiOL5OBPTryx6krqfTXce/lx3Hry0sitp2RYXTrGHli1oi+XXjhu6eFzrStqavn3snL2VNVG/FRe3ifLi12mcBfTVzMlrIqHn1vddxx/r+aGFm7WeAganZmRkSXS/Geak66Z1ro/oPv7P+08PB70W8u4cNDyyr9XPbYTMoq/Xzh+MOY8EYBELg04bVPfcrGXfu478vHc+rQnuytruWShwOt0Znjz6N/tw6UVtRQWuHnggc/4Jlvn8LpR+RH7OvLj38CRJ4w5pzjkodnMLJfXsQnuPDgrPLXcdOz86Jqr6iuC00jsXxrOUPyO0Wt09BbS7YyrHdnjurTpcl1Y3HO8fynG7n9laV8dcwA/nBF5Ce8Sn9kuO8Oe9N6fvZGzhvRm73Vtfxl+hpuOHsoI/vlHVAdDT3q/W137KliUM/o34Nzjr9/tI5Ljj+M/t06tMg+W8L28kDDSKNlDkFzbr+AeKP7PndMYPTLfZcfz72XH09mhnH7JZEHE/t368DOvTUM6tGRqbecTXmVn6888UnocTPjjktGkplh/OaNZcD+M3EvO7E/AF85aQCbSys58/fTQ8/7+7VjuOap2RTuqoiq66wj88nNyuQdr5X65DfHMKxXZ865//2YP0d5VS3/9+ayBH4b+/nr6jl+QtOjjV6c2/jc+Ufd8Vbo9q8mLgldNnHCGwWhTwlA6PjFzc8v4L83jo04OD3uoQ/5ePx5nHD3O6Flz3xSGBXu4Spr6nj8g7U8/G7gRLHlW8tZUlTGcQO64pwLfRoCeHPxVraE3Q/aubeaTK976vvPzWfhXRdyz6TlZBgM7N6RbF8GZx6Zz43PzOPGs4dy7WmDuOm5wIXel989judmF3LNaYOiDub9e1YhG3buY/xFI0LdXEGrtu/l9lcC1yuYvX43z84q5JLj+rGppIKnZ27g+AFdAaj1Ph2VhM2aetsrkW/Q768qxl9Xz4NfPYELj+4T8Vh1bR1Pz9zAdWcMZvgdU/j2GUO46wvxD5QHu+lKKvwM6hn9eFFJJb+bvIJJi7fy2s1nhpZv3FVBXgcfq7bv5Z8z1/Pw106M+Jmr/HVMX7GDccf2bZHRXg1d/Y/AcSGNljkEdclteibKxiYIu/XikXzn6TmM6JcXmmDsa6ccHjrwCHD9WUOp8tfx9w/X8YvPRQ9fMzMGdI8cs987L4fJPzqL5VvLueKJT8j2ZYRawf/+zlgAfvHSIl6aV8TZR/UiKzODST86k80llfxx6srQJGt3X3oMd71WELXP2y4ewe8mr4j7c00tiO7eOFjhB6CfnbUx5jo791ZzwYMfRizbU1Ub+icNmlKwjW/981MuHz2AL4SdwQzw3ortPDZ9LfManAT2hUdn8Ox3xoYOPgc1HOYadPebyyIGp4a/uYR4712/fr2Ac4b3Ci2+5JGPWFe8j7zcLC46ri/byqp4e9l2/hh2XOSUIT04YWA3fv/WCvI6ZHH56P7sDgvr2rpAf/ofpqwIXbRmqXf8ZUtpFYW79jU6k2lwWzc8MxeAkwd3p3+3Dtz/lVE8N2sj9761gn3eJHtPzVzPzecdQfeOWZgZ9fWO6St3cO7w3mRkWOhs7csem8nCuy6kW8fI7srgm3Zp2PUXVmwrZ9xDH0Vh388uAAAMqklEQVSsd//bKxk/bkQoyO9+cxnPz97IK98/nf7dO/Dnaau545Kj6ZCdSV294763lvPN0wczoHtHXpyzifW79vGrcSOAwKeFunqHL4FWeVv0uVsyDh4BjBkzxs2dOzcp+041V/71E3KzMvnXt09p0/2edu+7bC2ronvHLBbc9dnQ8t37auic42PZ1nJKKmo4d3hvIDBrZVmln56dcyK284+P1nHPpOXMvu18+uTl8uyswtDVq04Z0oNjDsvj1184hhF3vkWVv56JN53Olx//GIDLTjiMVxdGjhi5+Li+/Pj8o1i6uYw3Fm/h/ZXF/Pj8I/nzu/unT7jjkpERUzf88LwjQjNotqRuHbMY2TePT9btilg+ZlB35h7kGb0//+xRPDRtdUT3WGvK8WXQo1N2xKeIoL55uWwrj17eEr77maF0yvbFHbr7pytHsbe6jjtfXcopQ3pw41lDuf6Z/dlx8uDu3HTOMM4b0Ye5G3Zz52sF3HDWEH764iKG9erExJtO593lO/hZnDfNUQO68uR1J7O5pJJLveMtj319NG8s2sKUgm088Y2TGHdsXxZsLOFLf/mYM47oycNXnRjqDvzvjafy+Ptr2bm3moIt5bz/83M4/8EPeOAro7j0hMNYsW0PQ3t1wpeRwbDbJgMw8abTOWlQ9wP6fZnZPOfcmCbXSyTczWwc8GcgE/iHc+6+Bo/nAM8AJwG7gCudcxsa26bCvf2rrKlje3kV/brlHtS4XOcce6trQ59KNuzcxzn3v8+VYwby+yuOD6138/PzeXPxVlbeM47hdwSGKm647xIemraKzSWVXHh0H2at282dnx8Z8ZF59rpdjB7UnUffW8Oj09cw41fn0q9rB+Zs2M2SojLyu+RwzvBeEd06d33+aPx19dzrzcD5i88Nj2jFDu3ViXXFTR+0XH/vxeytruWDVcV0yc3im099GrXO0PxOXHnywNC+ErXmtxeRYcayreW8smAzT85Y3/STYjiyd+eI6akTccHI3kxbviN0f+yQHsxev/uA9j/llrP49yeFPDc79qejRIw7pi9TChqfzXTO7Rdw4Z8+CB2MPhjnDO/Fhp372LCrgqP6dObwHp2Ytrz5nx6/f86wmCPEJt50GicN6nFAtbVYuJtZJrAKuBAoAuYAX3POLQtb5/vA8c6575nZVcCXnHNXNrZdhbs0VOWvY0d5NYf37Mjg8ZPom5fLrNvOb9Y2nHNx+0rLKv1MX7GDC47uEzqAvLm0kq4dsuiUncnHa3fx/OyNTFqylT9fdQJ98nL52YuL+MvVo6moqeOFORsjPkXcdM6w0EfyYP0j7tw/fv6Nm89k+dZyLh/dH19mBjNW7wx1waz4v3F8un43Q/I7MbVgGzm+DDIzMrjtlSWcfVQvPlxVHHEgtr7e8fay7Zx+RE+MwJW/Kv11jBrQjV37qsnMyCAv18d5YTOULpnwWXJ8mZRU1PDGoi2cPiyfix/+iPzO2ZwypEfo4GxDL33vNE4e3CPi53nkayfyw/8sAOD568fy9X/M5vIT+/Pygs0RrfofnX8kx/fvynkjevPr1wu4+tTDGdE3j+raOh54exUDuneI2S3Xnnxx1GFxT/JrKf+54VROGxbjYEECWjLcTwMmOOc+592/FcA5d2/YOlO9dT4xMx+wDejlGtm4wl0aU1bpJyvTEprGuCW9sqCIn7ywiGk//QxH9I6c8bOmtp6SihqWbi4jr0MWYwZ1j3ojqa6t44OVxVx4dJ+YbzL3Tl5OUWklj319dNwaDuYM2g0791GwpZyP1+7kt186Lupxf1196ACic46CLeU4B68t3Mzby7bzl6tHc2z/rqH1/zBlBSce3p0Lj+5DRU0thbsqGNkvjw079zGoZ0cKd1XQo3M2+6pr6dU5J6H+5nmFJTw3q5CXw87Ofu76sRSVVFC4qwJfhvHwe2sY0bcLZx2ZHzrz+XufGca3zxjMnA0ljBrYNXTAv09eDtvLA0Ngxw7pwWUn9ue1hZtDb2yLi8oY1qsTE754DCUVfqr8dZw7vDfZvgwKtpSR3zmHipo6FheVYmZcc+ognptdyO2vLOWsI/Opqa2P+6nl+evHsmtfDbe9siTmbK3h8nJ9XHHSQPK7ZPO9s4c1euysMS0Z7lcA45xz13v3rwHGOuduDltnqbdOkXd/rbfOzgbbuhG4EeDwww8/qbCwsHk/lUgb2FddS6dmXoRFDsz/5hUxom+XiDcUgL3VtfgyjNyswIHMPVV+uuRmRZzvsGJbOXuqajl5cA827a5g974aRjU4V+RghL8R1tc7zAJDhX0ZGUycX8SlJxwW0V25Yec+unfKxl9XzzMfb+C0Yfl075RFbZ1j0pKt/Pyzww9qAsGglgz3rwCfaxDupzjnfhi2ToG3Tni4n+Kc2xVrm6CWu4jIgUg03BMZj1MEDAy7PwBo2CEVWsfrlukKHNjRFxEROWiJhPsc4EgzG2Jm2cBVwOsN1nkd+KZ3+wrgvcb620VEpHU12bHonKs1s5uBqQSGQj7lnCsws7uBuc6514EngX+b2RoCLfarWrNoERFpXEJHjZxzk4HJDZbdFXa7CvhKy5YmIiIHSrNCioikIYW7iEgaUriLiKQhhbuISBpK2qyQZlYMHOgpqvnAzibXSo72Wpvqah7V1Tyqq3kOpq5BzrleTa2UtHA/GGY2N5EztJKhvdamuppHdTWP6mqetqhL3TIiImlI4S4ikoZSNdz/luwCGtFea1NdzaO6mkd1NU+r15WSfe4iItK4VG25i4hIIxTuIiJpKOXC3czGmdlKM1tjZuPbeN9PmdkO78pTwWU9zOwdM1vtfe/uLTcze9irc7GZxb+u2sHXNdDMppvZcjMrMLMft4fazCzXzD41s0VeXb/xlg8xs9leXS94U0ljZjne/TXe44Nbo66w+jLNbIGZvdle6jKzDWa2xMwWmtlcb1l7eI11M7P/mdkK73V2WrLrMrPh3u8p+FVuZrckuy5vXz/xXvNLzew/3v9C276+nHMp80VgyuG1wFAgG1gEHN2G+z8bGA0sDVv2B2C8d3s88Hvv9sXAW4ABpwKzW7GufsBo73YXAhc0PzrZtXnb7+zdzgJme/t7EbjKW/4EcJN3+/vAE97tq4AXWvnv+VPgeeBN737S6wI2APkNlrWH19i/gOu929lAt/ZQV1h9mQSu3Two2XUB/YH1QIew19V1bf36atVfeCv80k4DpobdvxW4tY1rGExkuK8E+nm3+wErvdt/Bb4Wa702qPE14ML2VBvQEZgPjCVwZp6v4d+UwDUDTvNu+7z1rJXqGQC8C5wHvOn9w7eHujYQHe5J/TsCeV5YWXuqq0EtnwVmtoe6CIT7JqCH93p5E/hcW7++Uq1bJvhLCyryliVTH+fcVgDve29veVJq9T7SnUiglZz02ryuj4XADuAdAp+8Sp1zwUvFh+87VJf3eBnQszXqAh4CfgnUe/d7tpO6HPC2mc2zwAXlIfl/x6FAMfBPrxvrH2bWqR3UFe4q4D/e7aTW5ZzbDNwPbAS2Eni9zKONX1+pFu6xLh3eXsdytnmtZtYZmAjc4pwrb2zVGMtapTbnXJ1z7gQCLeVTgJGN7LtN6jKzzwM7nHPzwhcnuy7PGc650cBFwA/M7OxG1m2runwEuiMfd86dCOwj0N2R7LoCOwv0XX8ReKmpVWMsa43XV3fgUmAIcBjQicDfM96+W6WuVAv3RC7W3da2m1k/AO/7Dm95m9ZqZlkEgv0559zL7ak2AOdcKfA+gb7Obha4kHrDfbfVhdbPAL5oZhuA/xLomnmoHdSFc26L930H8AqBN8Rk/x2LgCLn3Gzv/v8IhH2y6wq6CJjvnNvu3U92XRcA651zxc45P/AycDpt/PpKtXBP5GLdbS384uDfJNDfHVx+rXeE/lSgLPhRsaWZmRG4ju1y59yD7aU2M+tlZt282x0IvOiXA9MJXEg9Vl2tfqF159ytzrkBzrnBBF5D7znnrk52XWbWycy6BG8T6EdeSpL/js65bcAmMxvuLTofWJbsusJ8jf1dMsH9J7OujcCpZtbR+98M/r7a9vXVmgc5WuOLwBHvVQT6bm9v433/h0Afmp/Au+13CPSNvQus9r738NY14DGvziXAmFas60wCH+MWAwu9r4uTXRtwPLDAq2spcJe3fCjwKbCGwEfpHG95rnd/jff40Db4m57D/tEySa3L2/8i76sg+PpO9t/R29cJwFzvb/kq0L2d1NUR2AV0DVvWHur6DbDCe93/G8hp69eXph8QEUlDqdYtIyIiCVC4i4ikIYW7iEgaUriLiKQhhbuISBpSuIuIpCGFu4hIGvp/V/9aw5lp7dwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#import sys\n",
    "#print(sys.path)\n",
    "#print(sys.executable)\n",
    "\n",
    "import cnn_utils\n",
    "from matplotlib import pyplot as plt\n",
    "import keras\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Flatten\n",
    "from keras.layers import Conv2D, MaxPooling2D\n",
    "import keras.callbacks as cb\n",
    "\n",
    "class LossHistory(cb.Callback):\n",
    "    def on_train_begin(self, logs={}):\n",
    "        self.losses = []\n",
    "\n",
    "    def on_batch_end(self, batch, logs={}):\n",
    "        batch_loss = logs.get('loss')\n",
    "        self.losses.append(batch_loss)\n",
    "\n",
    "def init_model():\n",
    "    model = Sequential()\n",
    "    model.add(Conv2D(32, kernel_size=(3, 3),\n",
    "                    activation='relu',\n",
    "                    input_shape=(28, 28, 1)))\n",
    "    model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "    model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "    model.add(Dropout(0.25))\n",
    "    model.add(Flatten())\n",
    "    model.add(Dense(128, activation='relu'))\n",
    "    model.add(Dropout(0.5))\n",
    "    model.add(Dense(10, activation='softmax'))\n",
    "\n",
    "    model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer=keras.optimizers.Adam(),\n",
    "              metrics=['accuracy'])\n",
    "    return model\n",
    "\n",
    "def plot_losses(losses):\n",
    "    plt.plot(losses)\n",
    "    plt.title('Loss per batch')\n",
    "    plt.show()\n",
    "    \n",
    "filePath = 'C:/Users/computer/Desktop/Assignment8/'   # the training set is stored in this directory\n",
    "mnistData= cnn_utils.loadData(filePath)\n",
    "mnistData_preprocessed = cnn_utils.preprocessData_CNN(mnistData)\n",
    "(X_train, y_train), (X_test, y_test) = mnistData_preprocessed\n",
    "\n",
    "(X_train_mini_batch, y_train_mini_batch) = cnn_utils.mini_batch(X_train, y_train, 10000, 0)\n",
    "(X_test_mini_batch, y_test_mini_batch) = cnn_utils.mini_batch(X_test, y_test, 5000, 0)\n",
    "print(X_train_mini_batch.shape)\n",
    "print(y_test_mini_batch.shape)\n",
    "\n",
    "model = init_model()\n",
    "history = LossHistory()\n",
    "model.fit(X_train_mini_batch, y_train_mini_batch, epochs=20, batch_size=256,\n",
    "                callbacks=[history],\n",
    "                validation_data=(X_test_mini_batch, y_test_mini_batch), verbose=2)\n",
    "\n",
    "score = model.evaluate(X_test_mini_batch, y_test_mini_batch, batch_size=16)\n",
    "print(score)\n",
    "plot_losses(history.losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As illustrated after 20 epochs we reached a cost of approximately 0.05288 and a test accuracy of 0.9844. The results are very good and reflect upon the power of CNNs. These results are slightly higher than the first network that utilized AdaDelta. This may line up with the intuition that, since Adam utilizes the exponentially weighted moving average of the gradient and the square of the gradient, rather than just the square the gradient alone to adjust the learning rate in each dimension, the velocity of the gradient helps to push the network to the minima faster, thus, reaching a lower cost and higher accuracy in the same number of epochs. \n",
    "\n",
    "Now we with to investigate the impact of the regularization techniques used on the network. We will add L2 Normalization with a hyper-parameter of 0.85 to the first dense layer. We will reset the optimization algorithm to AdaDelta for comparison with the first network.\n",
    "\n",
    "We will use the following:\n",
    "\n",
    "1. CNN architecture: \n",
    "    a. Convolution layer (kernel 3X3) then MaxPooling (3X3), Relu\n",
    "    b. Convolution layer (kernel 2X2) then MaxPooling (2X2), Relu, Dropout (Keep Probability 0.25)\n",
    "    c. Dense layer (units 128), Relu, Dropout (Keep Probability 0.5) & L2 Regularization (Hyper-Paramters 0.85)\n",
    "    d. Output layer (units 128), Softmax\n",
    "2. Dropout: \n",
    "    a. As specified in 1\n",
    "3. Optimization algorithm: \n",
    "    a. AdaDelta, which is a more robust extension of Adagrad that adapts learning rates based on a moving window of gradient updates.\n",
    "4. Minibatches: \n",
    "    a. 256\n",
    "5. Epochs:\n",
    "    a. 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10000, 28, 28, 1)\n",
      "(5000, 10)\n",
      "Train on 10000 samples, validate on 5000 samples\n",
      "Epoch 1/20\n",
      " - 2s - loss: 36.4693 - acc: 0.3377 - val_loss: 2.6179 - val_acc: 0.2822\n",
      "Epoch 2/20\n",
      " - 1s - loss: 2.1179 - acc: 0.5021 - val_loss: 2.5394 - val_acc: 0.4400\n",
      "Epoch 3/20\n",
      " - 1s - loss: 1.8228 - acc: 0.6050 - val_loss: 1.8938 - val_acc: 0.6556\n",
      "Epoch 4/20\n",
      " - 1s - loss: 1.5863 - acc: 0.6803 - val_loss: 1.5547 - val_acc: 0.6830\n",
      "Epoch 5/20\n",
      " - 1s - loss: 1.4296 - acc: 0.7192 - val_loss: 1.7215 - val_acc: 0.6682\n",
      "Epoch 6/20\n",
      " - 1s - loss: 1.3064 - acc: 0.7400 - val_loss: 1.2562 - val_acc: 0.7658\n",
      "Epoch 7/20\n",
      " - 1s - loss: 1.1981 - acc: 0.7648 - val_loss: 1.0298 - val_acc: 0.8656\n",
      "Epoch 8/20\n",
      " - 1s - loss: 1.1338 - acc: 0.7872 - val_loss: 1.4741 - val_acc: 0.6896\n",
      "Epoch 9/20\n",
      " - 1s - loss: 1.0622 - acc: 0.8015 - val_loss: 1.2671 - val_acc: 0.7646\n",
      "Epoch 10/20\n",
      " - 1s - loss: 0.9862 - acc: 0.8169 - val_loss: 0.9725 - val_acc: 0.8292\n",
      "Epoch 11/20\n",
      " - 1s - loss: 0.9470 - acc: 0.8194 - val_loss: 1.0506 - val_acc: 0.7860\n",
      "Epoch 12/20\n",
      " - 1s - loss: 0.8900 - acc: 0.8333 - val_loss: 0.9123 - val_acc: 0.8220\n",
      "Epoch 13/20\n",
      " - 1s - loss: 0.8598 - acc: 0.8419 - val_loss: 0.7844 - val_acc: 0.8898\n",
      "Epoch 14/20\n",
      " - 1s - loss: 0.8264 - acc: 0.8454 - val_loss: 0.6657 - val_acc: 0.9234\n",
      "Epoch 15/20\n",
      " - 1s - loss: 0.7868 - acc: 0.8561 - val_loss: 0.9156 - val_acc: 0.8064\n",
      "Epoch 16/20\n",
      " - 1s - loss: 0.7843 - acc: 0.8568 - val_loss: 0.5514 - val_acc: 0.9154\n",
      "Epoch 17/20\n",
      " - 1s - loss: 0.7607 - acc: 0.8558 - val_loss: 0.6644 - val_acc: 0.8810\n",
      "Epoch 18/20\n",
      " - 1s - loss: 0.7586 - acc: 0.8587 - val_loss: 0.6239 - val_acc: 0.9206\n",
      "Epoch 19/20\n",
      " - 1s - loss: 0.6905 - acc: 0.8791 - val_loss: 0.5955 - val_acc: 0.9050\n",
      "Epoch 20/20\n",
      " - 1s - loss: 0.7563 - acc: 0.8603 - val_loss: 0.5558 - val_acc: 0.9312\n",
      "5000/5000 [==============================] - 1s 173us/step\n",
      "[0.5557979704856872, 0.9312]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEICAYAAABRSj9aAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAH9xJREFUeJzt3XlwpHd95/H39+lb10gaSWPN5fExGJvDxp6AHY5ATAB7KUwSnJgi2LCwZhPYhewmKXtTG6BqSWArHIFlIU7wxkm4zGkXZRKITUJBjPGMsc3YZvDYM+PRXNLoPlt9fPeP55GsmeluaUYjtfrx51XV1d2/frqfb7ceffrXv9/TT5u7IyIi8RXUuwAREVlZCnoRkZhT0IuIxJyCXkQk5hT0IiIxp6AXEYk5Bb3IWWJmHzKzf1yF9WwzMzez5EqvS+JBQS+rzsz2m9lr613HWmJm7zCzH9W7DoknBb3IGVBvWhqJgl7WFDP7T2a218yGzOxuM9sYtZuZfdLM+s1s1MweNbMXRrdda2aPm9m4mR0ysz+q8tjvMLMfm9lnosf4hZldveD2dWb2BTM7Ej3O/zKzxEn3/aSZDQEfqvIUsmb21aiWh8zs0gWPf4uZPRXd9riZ/WbUfjHweeAqM5sws5GoPWdmHzezA1G9PzKz3IJ1vc3MnjGz42b2p2f6mkv8KehlzTCzXwf+AvgdoBc4AHwluvl1wKuA5wHtwO8Cg9FtXwDe4+6twAuB+2qs5mXA00AX8EHgm2bWGd12B1AELgReEq3z3RXu2wN8pMrjXwd8DegEvgR828xS0W1PAa8E1gEfBv7RzHrd/QngPwP3u3uLu7dHy/8lcAXwq9Hj/QlQXrCuVwAXAVcDfxa9YYicQkEva8nbgNvd/SF3zwO3EvZytwEFoBV4PmDu/oS7H4nuVwAuMbM2dx9294dqrKMf+JS7F9z9q8Ae4D+Y2QbgGuAD7j7p7v3AJ4EbFtz3sLt/xt2L7j5d5fF3ufvX3b0AfALIAlcCuPvX3P2wu5ejdT8JvLTSg5hZAPxH4P3ufsjdS+7+79HrMufD7j7t7o8AjwCXVnosEQW9rCUbCXvxALj7BGGvfZO73wf8H+CzwDEzu83M2qJFfxu4FjhgZv9mZlfVWMchP/FIfgei9Z4LpIAjZjYSDZ/8NWHvfc7BJTyH+WXcvQz0RY+Pmd1oZg8vePwXEn6yqKSL8E3iqRrrOrrg8hTQsoT65DlIQS9ryWHCwAXAzJqB9cAhAHf/tLtfAbyAcAjnj6P2B939OsJQ/jZwZ411bDIzW3B9a7Teg0Ae6HL39ujU5u4vWLDsUg71umVB/QGwGThsZucCfwO8D1gfDc/sBuZqOfmxjwMzwAVLWKdITQp6qZeUmWUXnJKEY9rvNLPLzCwD/DnwgLvvN7NfMbOXRePdk4QhWDKztJm9zczWRcMlY0Cpxnp7gP9qZikzux64GLgnGgb6HvBxM2szs8DMLjCzXzvN53WFmf1W9Hw+QPjm8ROgmTDMBwDM7J2EPfo5x4DNZpaG+U8DtwOfMLONZpYws6ui10XktCjopV7uAaYXnD7k7vcC/xP4BnCEsDc7N0beRtgjHiYcbhkknKwEeDuw38zGCCc1f6/Geh8AthP2mD8CvMXd5yZ1bwTSwOPRer5OOCl8Ou4inCgejur6rWg+4HHg48D9hKH+IuDHC+53H/AYcNTMjkdtfwT8HHgQGAI+hv5n5QyYfnhEnivM7B3Au939FfWuRWQ1qXcgIhJzCnoRkZjT0I2ISMypRy8iEnNr4sBMXV1dvm3btnqXISLSUHbt2nXc3bsXW25NBP22bdvYuXNnvcsQEWkoZnZg8aU0dCMiEnsKehGRmFPQi4jEnIJeRCTmFPQiIjGnoBcRiTkFvYhIzDV00O85Os7Hv7eHwYn84guLiDxHNXTQPzUwwWfu28vg5Gy9SxERWbMaOuiD6BfhiiUdmE1EpJqGDvpEEAZ9WUfgFBGpqqGDPhkFfamsoBcRqaahgz6Igr6ooBcRqaqhgz5hGroREVlMYwd9oMlYEZHFxCLo1aMXEakuFkGvyVgRkeoU9CIiMdfYQW8KehGRxTR20Gv3ShGRRS0a9Ga2xcx+YGZPmNljZvb+qL3TzL5vZk9G5x1Ru5nZp81sr5k9amaXr1TxmowVEVncUnr0ReC/u/vFwJXAe83sEuAW4F533w7cG10HuAbYHp1uBj531quOaIxeRGRxiwa9ux9x94eiy+PAE8Am4DrgjmixO4A3R5evA/7eQz8B2s2s96xXjoJeRGQpTmuM3sy2AS8BHgA2uPsRCN8MgJ5osU3AwQV364vazjpNxoqILG7JQW9mLcA3gA+4+1itRSu0nZLEZnazme00s50DAwNLLeMEiYSCXkRkMUsKejNLEYb8F939m1Hzsbkhmei8P2rvA7YsuPtm4PDJj+nut7n7Dnff0d3dfUbFz/foNRkrIlLVUva6MeALwBPu/okFN90N3BRdvgm4a0H7jdHeN1cCo3NDPGdbEFWvHr2ISHXJJSzzcuDtwM/N7OGo7X8AHwXuNLN3Ac8A10e33QNcC+wFpoB3ntWKF0hGSa+gFxGpbtGgd/cfUXncHeDqCss78N5l1rUkmowVEVlcY38zVpOxIiKLauyg12SsiMiiGjroNRkrIrK4hg56TcaKiCyuoYM+OgKCgl5EpIaGDnozIzAFvYhILQ0d9BAO32gyVkSkuoYP+iCAsnr0IiJVNXzQJ4NAvzAlIlJDwwe9xuhFRGpr+KBPBKagFxGpIQZBr8lYEZFaYhD0mowVEaml8YPeTJOxIiI1NH7QJ0w9ehGRGho/6NWjFxGpqfGDPjBNxoqI1BCLoNfQjYhIdQ0f9IGGbkREamr4oE9qMlZEpKaGD3pNxoqI1Nb4QR8YZU3GiohUFYug17FuRESqa/ig12SsiEhtDR/0mowVEamt4YNePXoRkdoaPug1GSsiUlvDB31Sk7EiIjU1fNAHpqAXEaml4YM+mVDQi4jU0vBBrx69iEhtDR/0OkyxiEht8Qh69ehFRKpq/KDX0I2ISE2NH/Tq0YuI1KSgFxGJuXgEvSZjRUSqWjTozex2M+s3s90L2j5kZofM7OHodO2C2241s71mtsfMXr9Shc9JBgGlkoJeRKSapfTo/w54Q4X2T7r7ZdHpHgAzuwS4AXhBdJ//a2aJs1VsJcmEUSiXV3IVIiINbdGgd/cfAkNLfLzrgK+4e97d9wF7gZcuo75FJQOjqB69iEhVyxmjf5+ZPRoN7XREbZuAgwuW6YvaTmFmN5vZTjPbOTAwcMZFJBMBxbLjGqcXEanoTIP+c8AFwGXAEeDjUbtVWLZiArv7be6+w913dHd3n2EZYY8e0J43IiJVnFHQu/sxdy+5exn4G54dnukDtixYdDNweHkl1pZMhEGvHx8REansjILezHoXXP1NYG6PnLuBG8wsY2bnAduBny6vxNpSQfgUCiVNyIqIVJJcbAEz+zLwaqDLzPqADwKvNrPLCIdl9gPvAXD3x8zsTuBxoAi8191LK1N6aL5HrwlZEZGKFg16d39rheYv1Fj+I8BHllPU6Zgbo9fQjYhIZQ3/zdhkInwKRe1LLyJSUeMHfaChGxGRWho+6FPzPXoFvYhIJQ0f9In5Hr2GbkREKmn4oE9Fe90UNHQjIlJRwwd9MtBkrIhILY0f9PpmrIhITY0f9HM9eg3diIhU1PhBn9BkrIhILQ0f9POTsRq6ERGpqOGDPhEN3ZQ0GSsiUlHDB/3cN2O1e6WISGUNH/Tz34xV0IuIVNTwQf/s7pUauhERqaTxg14HNRMRqanxg16HKRYRqanhgz6lyVgRkZoaPujnevQl7UcvIlJRwwd9Yr5Hr6EbEZFKGj7oUzqomYhITQ0f9M8e1Ew9ehGRSmIQ9OrRi4jU0vBBHwRGYNqPXkSkmoYPegj3vCloP3oRkYpiEfSpwNSjFxGpIhZBnwhM+9GLiFQRi6BPJQLtRy8iUkUsgj6Z0NCNiEg18Qj6QJOxIiLVxCPoExqjFxGpJh5Br71uRESqikXQazJWRKQ6Bb2ISMzFJOhNPzwiIlJFLII+nQyYLapHLyJSSUyCPkFeQzciIhXFI+gT6tGLiFSzaNCb2e1m1m9muxe0dZrZ983syei8I2o3M/u0me01s0fN7PKVLH5OOmmajBURqWIpPfq/A95wUtstwL3uvh24N7oOcA2wPTrdDHzu7JRZm3r0IiLVLRr07v5DYOik5uuAO6LLdwBvXtD+9x76CdBuZr1nq9hqNBkrIlLdmY7Rb3D3IwDReU/Uvgk4uGC5vqjtFGZ2s5ntNLOdAwMDZ1hGKJ0MmNXQjYhIRWd7MtYqtFXcwd3db3P3He6+o7u7e1krTSUCCurRi4hUdKZBf2xuSCY674/a+4AtC5bbDBw+8/KWJp0MtHuliEgVZxr0dwM3RZdvAu5a0H5jtPfNlcDo3BDPSspEk7Hu+nasiMjJkostYGZfBl4NdJlZH/BB4KPAnWb2LuAZ4Ppo8XuAa4G9wBTwzhWo+RTpZPh+VSg56WSl0SMRkeeuRYPe3d9a5aarKyzrwHuXW9TpSiXmgr48H/oiIhKKRSrOhbt2sRQROVW8gl4TsiIip4hH0CfUoxcRqSYeQa8evYhIVfEIevXoRUSqikfQazJWRKSqeAW9hm5ERE4Ri6Cf349ePXoRkVPEIujnevQ63o2IyKniEfSajBURqSoWQZ/RZKyISFWxCPqFx7oREZETxSLotXuliEh18Qp69ehFRE4Ri6BPaTJWRKSqWAR9NhXtXqmgFxE5RSyCPp0ICAymZ0v1LkVEZM2JRdCbGdlUgpmCgl5E5GSxCHqAXCrBtIJeROQUsQn6rIJeRKSiGAV9QL6gyVgRkZPFJuhzafXoRUQqiU/QazJWRKSi2AS9xuhFRCqLV9BrP3oRkVPEKuj1zVgRkVPFJuhzqUA9ehGRCmIU9BqjFxGpJDZBr0MgiIhUFqugzxfLlMte71JERNaUWAU96FDFIiIni03Q56Jj0mucXkTkRPEJ+nTYo1fQi4icKDZBPzd0owlZEZETxS7otS+9iMiJYhP0TdHQzZSCXkTkBMnl3NnM9gPjQAkouvsOM+sEvgpsA/YDv+Puw8src3EtmfCpTOaLK70qEZGGcjZ69K9x98vcfUd0/RbgXnffDtwbXV9xrdkw6McV9CIiJ1iJoZvrgDuiy3cAb16BdZyiWT16EZGKlhv0DnzPzHaZ2c1R2wZ3PwIQnfcscx1LMjd0MzGjoBcRWWhZY/TAy939sJn1AN83s18s9Y7RG8PNAFu3bl1mGdCc1tCNiEgly+rRu/vh6Lwf+BbwUuCYmfUCROf9Ve57m7vvcPcd3d3dyykDgCAwWjJJ9ehFRE5yxkFvZs1m1jp3GXgdsBu4G7gpWuwm4K7lFrlULZmkxuhFRE6ynKGbDcC3zGzucb7k7v9kZg8Cd5rZu4BngOuXX+bSNGcSTCjoRUROcMZB7+5PA5dWaB8Erl5OUWeqJZvSGL2IyEli881YgNZMkomZQr3LEBFZU2IV9C2ZpIZuREROEq+gzyaZzOtYNyIiC8Ur6DNJxjV0IyJyglgFfWs2HLrR78aKiDwrVkHf0ZSm7DCmXr2IyLx4BX1zCoDhKQW9iMiceAV9UxqAocnZOlciIrJ2xCroO5vDoB9W0IuIzItV0M/16IenFPQiInPiFfTNCnoRkZPFKuib0wnSiYChSU3GiojMiVXQmxntTSlG1KMXEZkXq6CHcEJWe92IiDwrdkG/viXNwES+3mWIiKwZsQv6c9pyHB2dqXcZIiJrRuyCvnddlv7xPMVSud6liIisCfEL+vYspbJr+EZEJBK/oF+XBeCIhm9ERIBYBn0OgCMjCnoREYhh0G+cC/rR6TpXIiKyNsQu6NtySVoySQ4OTdW7FBGRNSF2QW9mbOtqYt+ggl5EBGIY9ADndbWw7/hEvcsQEVkT4hn065s4NDxNvliqdykiInUXz6DvbqbsaJxeRISYBv0F3S0APHlMwzciIrEM+udtaCURGI8dHqt3KSIidRfLoM+mEjz/nFYe3D9U71JEROoulkEP8IrtXTz0zDCT+WK9SxERqavYBv0rL+ymUHIe2DdY71JEROoqtkG/Y1sH2VTAD395vN6liIjUVWyDPptK8PILuvin3Ue1P72IPKfFNugBfu+qczk6NsPHvrun3qWIiNRNrIP+NRf18JYrNvPFBw7QP67DFovIc1Osgx7gfa+5kEKpzG9/7t/ZfWi03uWIiKy62Af9tq5mbrxqGweHpnnjZ37EdZ/9Mfc/tfieOKWy4+5Vb58t6jdpRaQxWK0wW9YDm70B+CsgAfytu3+02rI7duzwnTt3rkgdAO7Og/uH+YMv7uL4xCwAuVSC6UKJ1168AYBMKgCHp49P0pJJcGR0hu09LXS1ZHjxlnY2t+c4v7uZbz50iLZcio9+9wl+/9UXsqEtw788foxrXtjLxb1tNGcSNKWT7B+c5EWb1pFLJegbnmZgYoYjozN0tWQou5NJJnjhpjaeHphke08L+WKZ5kxyvt4H9g2xLpfi4t42Do1M09uWJQiMctn53uNHueLcTrpbMyc8RzNb9LV47PAoz9vQSipR+T2+XHaCYPHHWQmzxTKphJ3yPMpl5/6nB7lsS/v8a7RU07MlBifzbO5oOpuliqwJZrbL3XcsutxKBL2ZJYBfAr8B9AEPAm9198crLb/SQT9nMl9k3/FJ/vjrj3J0dJpS2RmbWdkvVLVkkkxU+dLW3G3pREChXOaiDa2U3Tk4NM10IdxTaO4NaWtnE7lUgj3HxgHoaEqxqSPHtvXNPHxwhNlimUu3tDM6VSCXTlAslxmbLrIul+LJ/nF6WrM8MzTF6HSBSzevo70pzUS+yIs2rWP3oVESgdGUTvDjvYO86bKNTBdK5FIJmtIJ9g9Osak9xxNHxtja2cSmjhypRMC/7eknm0rw2os3cP/Tg2RTAZf0tvHwwVE2d+Qou9OUTpJNBYxNFzk2PkNLOskFPc3c9fBhetdleeOLN5JOBgxOzvJX//JLLuhu4Xd/ZQtHRmc4OjrDsbHw9EjfKC/evI53veI8kkHAyPQsO/cPc15XM1s7myiVnaZ0gr39E2zuzLEul+LwyAx37jzIo32j/OFrn8fze1splpz+8Zn5N8m5zb8tl+LY2AyDE7Ns6sgxmS/SlE6w+9AoG9tzbGrP0d6UZteBYaZni5yzLkcyMDa2h79otm9wkr6hKc7raubCnhYKJeexw6McHZ3hqgvW05xJUiw5xXKZUtkxg8CMpwcmSSUDLuhqplB2iqUyhZIzPDXLwaEpLt3STnsuRWs2xcj0LHv7J+hsTvOSrR1M5otM5It0NWdoziQ4ODzNgcFJpmZLjEwVeP45rWRSAdlUgnyhjFn4ZrqxPXx9JvNFmjIJhicLFMtl9vZPcGBwio7mNOd3NbOhLUtzJrzvTLGEO2SSAYWSMzpdoFgq096UJp0MCKLns+fYOAcGJ7ni3A7SiQTNmQTFsjMTbc/ntGV5sn+CfcfDZdpyKXKpBIVSmWwywYGhSfqGp2nLpuhsTtPVkqZYdvb2T7B9Qwvu0D+eZ2//BFs7m9jSmaNUdsplGJjIUyqHr3FPa4ZyVG9Pa5bZUpnxmQKDE7Okk+Frkk0FDE7M8sC+IS7f2s6Gtiz5YpmmdIJkYBRKPv/c3GFytsiR0RnacynacilSiYBEYByfyPMP9x+gLZvk+h1baM4kac4kGJ0ucHx8ls7mNLsPjXLxxjayyYDhqQIdTSnyxTL7j0/S05bhwp7WM8qXegf9VcCH3P310fVbAdz9Lyotv1pBX8nYTIHmdJKfPTNMT2uWkjulcpnJfIl9xyfpac0wXSiRTSXYdWCYrZ1NjE4XeN0LNnDng30cG5/h157XzUMHhmnJJHFgeGqWrZ1NDE3OcnxilkwyoG94mgt7WkgljFQi4MH9Q6QTAd2tGVKJgKnZEqPTBWZLZWYKJRJm9LZnmcqXMIORqQJm8MzQFCNTBV6ytZ2yO4dHZuhsTtPTmmHf8UmyqXADmy2GjzNdKLG9p4WSO10tGf51zwBN6QRbO5toSif42cERzutqBuDwyDQXndPG7kOjnNOWZWq2yPBUgXW5FIVSmelCifZciuGpAgAv3rwuCuI8mWRAV0uGQyPP/oRjUzr8By6UwmALHzN8nuub0zgwNDk7v/y565s4NjbDTOHZYbH2phQj0frWN6cZXLC82bNBvZoSgVEqn7riwKBC8xmr1/OT1fWeV53PrddefEb3rXfQvwV4g7u/O7r+duBl7v6+BcvcDNwMsHXr1isOHDhw1uuIq6UO07g7ZQ+DqZrZYpl0MjjhcUtlJxEY7s54vkhbNnXCfQYn8gRmdDSnKZbKjM2EnxwSgTEyNRv1gsI3NAOKZScwoseEvuFpetrCHnXf8BTFspNLJdjc0cTUbJGnBibZ0JZhQ2s4XHVwaGq+B763f4J0MqBYcja15xiczFMsO8nAGJspsm19+AY7NlMkkwyiN9eAwYk8R0ZnCMzoakkzHL1xGmE4D07m6WnN0pZNMjpdoCWbZDJfxMwwYCJfZGhylgt7WuhoSjMwnmciX2R8pkixXKZ3XZbNHU08NTDBkZEZmjIJ2nNpzutq5ueHRimWy/M9wGRgTM2GPeTNHTkGJ2eZKZRIBkYyEZCMPl31rsvxi6NjjM8UmcwXKbmz49xO9g9Osu/4JC1Rz3FgPM/0bIne9hzndTUzUyhxTlv4CW5ytsT0bJHWbAoDUsmAvf0TFEtlcunwOa5vSZMMAja0ZcgkE/SPz5BJJjg6Ns1MoUw2FZBJhj3zUrlMYEZbLkU2mWBkapbZUvjGPJEvsqk9x+aOHAeHwt+DmC6USCUCssnwU2b/eJ6N63Ksb0lzcHiamdkSU7NFimWnb3iazuY0r9zexWyxHHaUojf2ruY0R8dmSCcD1uVSbGzP0Tc8zWS+SMKMILD53ndTOuzsJIOAmUKJ/vE82VQi6oykKTvkiyVmCmUSgXFJb9i5yRfLtGTCv79DtJ2FHZxsMkEqGXBOW5bpQomx6UL06SHc9i7f2kGxXObJ/gmm8kVGpgu0ZVP0tGU4NDzNhrYsQ5OzBAbtTWGHxYCLzmlle08LPW3Z04mAefUO+uuB158U9C919/9Safl69uhFRBrVUoN+pfa66QO2LLi+GTi8QusSEZEaViroHwS2m9l5ZpYGbgDuXqF1iYhIDae3r9oSuXvRzN4H/DPh7pW3u/tjK7EuERGpbUWCHsDd7wHuWanHFxGRpYn9N2NFRJ7rFPQiIjGnoBcRiTkFvYhIzK3YQc1OqwizAeBMvxrbBazV3wtcq7WprtOjuk6P6jo9y6nrXHfvXmyhNRH0y2FmO5fyzbB6WKu1qa7To7pOj+o6PatRl4ZuRERiTkEvIhJzcQj62+pdQA1rtTbVdXpU1+lRXadnxetq+DF6ERGpLQ49ehERqUFBLyIScw0d9Gb2BjPbY2Z7zeyWVV737WbWb2a7F7R1mtn3zezJ6Lwjajcz+3RU56NmdvkK1rXFzH5gZk+Y2WNm9v61UJuZZc3sp2b2SFTXh6P288zsgaiur0aHtcbMMtH1vdHt21airgX1JczsZ2b2nbVSl5ntN7Ofm9nDZrYzalsL21i7mX3dzH4RbWdX1bsuM7soep3mTmNm9oF61xWt6w+jbX63mX05+l9Y3e3L3RvyRHj446eA84E08AhwySqu/1XA5cDuBW3/G7glunwL8LHo8rXAdwl/ue5K4IEVrKsXuDy63Er4I+2X1Lu26PFbossp4IFofXcCN0Ttnwd+P7r8B8Dno8s3AF9d4b/nfwO+BHwnul73uoD9QNdJbWthG7sDeHd0OQ20r4W6FtSXAI4C59a7LmATsA/ILdiu3rHa29eKvuAr/Me8CvjnBddvBW5d5Rq2cWLQ7wF6o8u9wJ7o8l8Db6203CrUeBfwG2upNqAJeAh4GeE3ApMn/00Jf8vgquhyMlrOVqiezcC9wK8D34n++ddCXfs5Nejr+ncE2qLgsrVU10m1vA748VqoizDoDwKd0fbyHeD1q719NfLQzdwLOKcvaqunDe5+BCA674na61Jr9LHvJYS957rXFg2PPAz0A98n/EQ24u7FCuueryu6fRRYvxJ1AZ8C/gQoR9fXr5G6HPieme0ys5ujtnr/Hc8HBoD/Fw11/a2ZNa+Buha6AfhydLmudbn7IeAvgWeAI4Tbyy5Weftq5KC3Cm1rdV/RVa/VzFqAbwAfcPexWotWaFuR2ty95O6XEfagXwpcXGPdq1KXmb0R6Hf3XQub611X5OXufjlwDfBeM3tVjWVXq64k4ZDl59z9JcAk4ZBIvesKVxaOdb8J+Npii1ZoW4ntqwO4DjgP2Ag0E/49q617Repq5KBfiz9AfszMegGi8/6ofVVrNbMUYch/0d2/uZZqA3D3EeBfCcdG281s7pfOFq57vq7o9nXA0AqU83LgTWa2H/gK4fDNp9ZAXbj74ei8H/gW4Ztjvf+OfUCfuz8QXf86YfDXu6451wAPufux6Hq963otsM/dB9y9AHwT+FVWeftq5KBfiz9AfjdwU3T5JsLx8bn2G6OZ/iuB0bmPk2ebmRnwBeAJd//EWqnNzLrNrD26nCP8B3gC+AHwlip1zdX7FuA+jwYuzyZ3v9XdN7v7NsJt6D53f1u96zKzZjNrnbtMOO68mzr/Hd39KHDQzC6Kmq4GHq93XQu8lWeHbebWX8+6ngGuNLOm6H9z7vVa3e1rJSdFVvpEOHP+S8Kx3j9d5XV/mXDMrUD4LvwuwrG0e4Eno/POaFkDPhvV+XNgxwrW9QrCj3qPAg9Hp2vrXRvwYuBnUV27gT+L2s8HfgrsJfy4nYnas9H1vdHt56/C3/TVPLvXTV3ritb/SHR6bG77rvffMVrXZcDO6G/5baBjjdTVBAwC6xa0rYW6Pgz8Itru/wHIrPb2pUMgiIjEXCMP3YiIyBIo6EVEYk5BLyIScwp6EZGYU9CLiMScgl5EJOYU9CIiMff/AUvDp5pfM4dmAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#import sys\n",
    "#print(sys.path)\n",
    "#print(sys.executable)\n",
    "\n",
    "import cnn_utils\n",
    "from matplotlib import pyplot as plt\n",
    "import keras\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Flatten\n",
    "from keras.layers import Conv2D, MaxPooling2D\n",
    "import keras.callbacks as cb\n",
    "from keras import regularizers\n",
    "\n",
    "class LossHistory(cb.Callback):\n",
    "    def on_train_begin(self, logs={}):\n",
    "        self.losses = []\n",
    "\n",
    "    def on_batch_end(self, batch, logs={}):\n",
    "        batch_loss = logs.get('loss')\n",
    "        self.losses.append(batch_loss)\n",
    "\n",
    "def init_model():\n",
    "    model = Sequential()\n",
    "    model.add(Conv2D(32, kernel_size=(3, 3),\n",
    "                    activation='relu',\n",
    "                    input_shape=(28, 28, 1)))\n",
    "    model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "    model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "    model.add(Dropout(0.25))\n",
    "    model.add(Flatten())\n",
    "    model.add(Dense(128, kernel_regularizer=regularizers.l2(0.85), activation='relu'))\n",
    "    model.add(Dropout(0.5))\n",
    "    model.add(Dense(10, activation='softmax'))\n",
    "\n",
    "\n",
    "    model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer=keras.optimizers.Adadelta(),\n",
    "              metrics=['accuracy'])\n",
    "    return model\n",
    "\n",
    "def plot_losses(losses):\n",
    "    plt.plot(losses)\n",
    "    plt.title('Loss per batch')\n",
    "    plt.show()\n",
    "    \n",
    "filePath = 'C:/Users/computer/Desktop/Assignment8/'   # the training set is stored in this directory\n",
    "mnistData= cnn_utils.loadData(filePath)\n",
    "mnistData_preprocessed = cnn_utils.preprocessData_CNN(mnistData)\n",
    "(X_train, y_train), (X_test, y_test) = mnistData_preprocessed\n",
    "\n",
    "(X_train_mini_batch, y_train_mini_batch) = cnn_utils.mini_batch(X_train, y_train, 10000, 0)\n",
    "(X_test_mini_batch, y_test_mini_batch) = cnn_utils.mini_batch(X_test, y_test, 5000, 0)\n",
    "print(X_train_mini_batch.shape)\n",
    "print(y_test_mini_batch.shape)\n",
    "\n",
    "model = init_model()\n",
    "history = LossHistory()\n",
    "model.fit(X_train_mini_batch, y_train_mini_batch, epochs=20, batch_size=256,\n",
    "                callbacks=[history],\n",
    "                validation_data=(X_test_mini_batch, y_test_mini_batch), verbose=2)\n",
    "\n",
    "score = model.evaluate(X_test_mini_batch, y_test_mini_batch, batch_size=16)\n",
    "print(score)\n",
    "plot_losses(history.losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As illustrated after 20 epochs we reached a cost of approximately 0.05557 and a test accuracy of 0.9312. The cost and test accuracy levelized to an unacceptable level here. Intuitively, L2 regularization attempts to decrease the weights towards zero; therefore, the weights naturally find a balance between fitting the training data and pushing towards zero. Since we had a high L2 hyper-parameter of 0.85, the weights of the first dense layer had high tendedancy to decrease and likely cause the model to underfit the training data (essentially smoothing out the curve too much). \n",
    "\n",
    "The last thing we wish to do is investigate the impact of the hyper-paramters of the network. One of the most fundamental hyper-parameters we have is the minibatch size; so we will focus on adjusting that one and see the effect on the results. We will decrease the minibach size down to 64 and see what results we achieve.\n",
    "\n",
    "We will use the following:\n",
    "\n",
    "1. CNN architecture: \n",
    "    a. Convolution layer (kernel 3X3), then MaxPooling (3X3) Relu\n",
    "    b. Convolution layer (kernel 2X2), then MaxPooling (2X2) Relu, Dropout (Keep Probability 0.25)\n",
    "    c. Dense layer (units 128), Relu, Dropout (Keep Probability 0.5)\n",
    "    d. Output layer (units 128), Softmax\n",
    "2. Dropout: \n",
    "    a. As specified in 1\n",
    "3. Optimization algorithm: \n",
    "    a. Adam, which is a more robust extension of Adagrad that adapts learning rates based on a moving window of gradient updates.\n",
    "4. Minibatches: \n",
    "    a. 64\n",
    "5. Epochs:\n",
    "    a. 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10000, 28, 28, 1)\n",
      "(5000, 10)\n",
      "Train on 10000 samples, validate on 5000 samples\n",
      "Epoch 1/20\n",
      " - 3s - loss: 0.5596 - acc: 0.8251 - val_loss: 0.1715 - val_acc: 0.9458\n",
      "Epoch 2/20\n",
      " - 2s - loss: 0.1822 - acc: 0.9453 - val_loss: 0.1044 - val_acc: 0.9662\n",
      "Epoch 3/20\n",
      " - 2s - loss: 0.1270 - acc: 0.9604 - val_loss: 0.0846 - val_acc: 0.9724\n",
      "Epoch 4/20\n",
      " - 2s - loss: 0.1077 - acc: 0.9675 - val_loss: 0.0763 - val_acc: 0.9760\n",
      "Epoch 5/20\n",
      " - 2s - loss: 0.0832 - acc: 0.9757 - val_loss: 0.0895 - val_acc: 0.9724\n",
      "Epoch 6/20\n",
      " - 2s - loss: 0.0746 - acc: 0.9780 - val_loss: 0.0636 - val_acc: 0.9788\n",
      "Epoch 7/20\n",
      " - 2s - loss: 0.0595 - acc: 0.9820 - val_loss: 0.0528 - val_acc: 0.9820\n",
      "Epoch 8/20\n",
      " - 2s - loss: 0.0520 - acc: 0.9850 - val_loss: 0.0546 - val_acc: 0.9828\n",
      "Epoch 9/20\n",
      " - 3s - loss: 0.0462 - acc: 0.9852 - val_loss: 0.0534 - val_acc: 0.9840\n",
      "Epoch 10/20\n",
      " - 2s - loss: 0.0436 - acc: 0.9865 - val_loss: 0.0598 - val_acc: 0.9816\n",
      "Epoch 11/20\n",
      " - 2s - loss: 0.0369 - acc: 0.9880 - val_loss: 0.0517 - val_acc: 0.9830\n",
      "Epoch 12/20\n",
      " - 2s - loss: 0.0328 - acc: 0.9897 - val_loss: 0.0518 - val_acc: 0.9830\n",
      "Epoch 13/20\n",
      " - 2s - loss: 0.0289 - acc: 0.9913 - val_loss: 0.0521 - val_acc: 0.9842\n",
      "Epoch 14/20\n",
      " - 3s - loss: 0.0282 - acc: 0.9903 - val_loss: 0.0510 - val_acc: 0.9846\n",
      "Epoch 15/20\n",
      " - 3s - loss: 0.0264 - acc: 0.9920 - val_loss: 0.0542 - val_acc: 0.9838\n",
      "Epoch 16/20\n",
      " - 2s - loss: 0.0224 - acc: 0.9931 - val_loss: 0.0568 - val_acc: 0.9832\n",
      "Epoch 17/20\n",
      " - 2s - loss: 0.0204 - acc: 0.9938 - val_loss: 0.0586 - val_acc: 0.9840\n",
      "Epoch 18/20\n",
      " - 2s - loss: 0.0190 - acc: 0.9945 - val_loss: 0.0537 - val_acc: 0.9842\n",
      "Epoch 19/20\n",
      " - 2s - loss: 0.0132 - acc: 0.9956 - val_loss: 0.0568 - val_acc: 0.9844\n",
      "Epoch 20/20\n",
      " - 2s - loss: 0.0184 - acc: 0.9935 - val_loss: 0.0519 - val_acc: 0.9844\n",
      "5000/5000 [==============================] - 1s 237us/step\n",
      "[0.05193657198720566, 0.9844]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xl8XHW9//HXJ0v3fYOWFsJSERDZSoGLgoIooIIoKOgVwYWLy1Wv3ovlB8paURFUFtlXq+wI1baUUlqgpVtauqR7uiZp2ixNkyZt9u/vjzkzmSSzJZlkcqbv5+ORR2bOnDnnezKT9/me7/me7zHnHCIikl4yUl0AERFJPoW7iEgaUriLiKQhhbuISBpSuIuIpCGFu4hIGlK4i3SBmd1uZtN6YD05ZubMLKu71yXpQeEuPcLMtpvZ51Jdjt7EzK4zswWpLoekJ4W7SIJUaxY/UbhLypnZD8ws38z2mtl0MxvnTTcz+5OZlZhZpZmtNrNPeK9dambrzGy/mRWZ2f9GWfZ1ZrbQzB70lrHBzC4Me32omT1lZsXecu42s8w27/2Tme0Fbo+yCf3M7CWvLCvM7JSw5U8xsy3ea+vM7Apv+gnAo8A5ZlZtZvu86f3N7D4z2+GVd4GZ9Q9b17fMbKeZlZnZLZ39m0v6U7hLSpnZBcA9wNeBscAO4EXv5c8D5wEfA4YB3wDKvdeeAv7LOTcY+ATwbozVnAVsBUYBtwGvm9kI77XngEbgOOA0b53fj/DeMcDUKMu/HHgFGAH8A3jDzLK917YAnwaGAncA08xsrHNuPXAjsMg5N8g5N8yb/4/AGcB/eMu7CWgOW9engOOBC4HfeDsJkXYU7pJq3wKeds6tcM7VATcTqM3mAA3AYODjgDnn1jvnir33NQAnmtkQ51yFc25FjHWUAH92zjU4514CNgJfNLPDgEuAnzvnapxzJcCfgKvD3rvLOfegc67ROXcwyvKXO+dedc41APcD/YCzAZxzrzjndjnnmr11bwYmR1qImWUA3wV+5pwrcs41Oec+9P4uQXc45w4651YBq4BTIi1LROEuqTaOQG0dAOdcNYHa+RHOuXeBh4CHgT1m9riZDfFm/RpwKbDDzN4zs3NirKPItR4hb4e33qOAbKDYzPZ5TSOPEailBxUksA2heZxzzUCht3zM7FozWxm2/E8QOIKIZBSBHcOWGOvaHfb4ADAogfLJIUjhLqm2i0DIAmBmA4GRQBGAc+4B59wZwEkEmmf+z5u+zDl3OYEgfgN4OcY6jjAzC3t+pLfeAqAOGOWcG+b9DHHOnRQ2byLDpk4IK38GMB7YZWZHAU8APwFGek0veUCwLG2XXQbUAscmsE6RmBTu0pOyzaxf2E8WgTbq683sVDPrC/wWWOKc225mZ5rZWV77dQ2B4Gsysz5m9i0zG+o1hVQBTTHWOwb4qZllm9lVwAnATK+J523gPjMbYmYZZnasmZ3fwe06w8y+6m3PzwnsMBYDAwkEeCmAmV1PoOYetAcYb2Z9IFTrfxq438zGmVmmmZ3j/V1EOkThLj1pJnAw7Od259xc4NfAa0AxgVprsM17CIGabwWBppRyAiccAb4NbDezKgInJv8zxnqXABMJ1IynAlc654InZq8F+gDrvPW8SuDEbke8SeBkb4VXrq967fvrgPuARQSC/GRgYdj73gXWArvNrMyb9r/AGmAZsBf4Pfo/lU4w3axD0pmZXQd83zn3qVSXRaQnqUYgIpKGFO4iImlIzTIiImlINXcRkTSUsoGQRo0a5XJyclK1ehERX1q+fHmZc250vPlSFu45OTnk5uamavUiIr5kZjviz6VmGRGRtKRwFxFJQwp3EZE0pHAXEUlDCncRkTSkcBcRSUMKdxGRNOS7cN+4ez/3vb2R8uq6+DOLiByifBfu+SXVPPhuPuU19akuiohIr+W7cM/MCNyhrLFJA56JiETju3DP8sK9qVnhLiISje/CPVRzb25OcUlERHov34a7au4iItH5LtwLKw4C8MLSghSXRESk9/JduG8trQbgg82lKS6JiEjv5btwv3ryBAB+/NnjUlwSEZHey3fh3i87E4D+3m8REWnPd+GeYYETqs26sbeISFS+DXdFu4hIdD4M98Bv1dxFRKLzXbhbqFkmxQUREenFfBjugd9ONXcRkah8F+6hNndlu4hIVD4M98BvtbmLiETnu3BXm7uISHw+DPfAb7W5i4hE57twV5u7iEh8Pgz3wG+1uYuIROfDcFebu4hIPL4L9yDV3EVEovNduAdr7iIiEp0Pwz3wu1ntMiIiUcUNdzObYGbzzGy9ma01s59FmMfM7AEzyzez1WZ2evcUV23uIiKJyEpgnkbgl865FWY2GFhuZnOcc+vC5rkEmOj9nAU84v1OOlNvGRGRuOLW3J1zxc65Fd7j/cB64Ig2s10OPO8CFgPDzGxs0ktLyxWquohJRCS6DrW5m1kOcBqwpM1LRwAFYc8Lab8DwMxuMLNcM8stLe38Da4zTDfrEBGJJeFwN7NBwGvAz51zVW1fjvCWdvnrnHvcOTfJOTdp9OjRHStpmAwzNcuIiMSQULibWTaBYP+7c+71CLMUAhPCno8HdnW9eJEFwr27li4i4n+J9JYx4ClgvXPu/iizTQeu9XrNnA1UOueKk1jONoXSCVURkVgS6S1zLvBtYI2ZrfSm/T/gSADn3KPATOBSIB84AFyf/KK2yDDU6C4iEkPccHfOLSBym3r4PA74cbIKFY/a3EVEYvPdFaqgNncRkXh8Ge6G2txFRGLxZ7ibbtYhIhKLL8M9I8N0haqISAz+DHe1uYuIxOTTcFebu4hILL4Md1DNXUQkFl+Ge+CGHUp3EZFofBruRnNzqkshItJ7+TTc1eYuIhKLL8Pd1FtGRCQmn4Y7OLW5i4hE5ctwzzDTFaoiIjH4NNzV5i4iEosvw11t7iIisfky3Gsbmti172CqiyEi0mslciemXqe4spbiytpUF0NEpNfyZc1dRERiU7iLiKQhhbuISBpSuIuIpCGFu4hIGlK4i4ikIYW7iEga8nW46ybZIiKR+TrcNQSBiEhkPg93pbuISCQKdxGRNOTrcFe2i4hE5stw/845RwGquYuIROPLcB8/fACgE6oiItH4MtzNAr+blO4iIhH5MtxfXFYAwPSVRSkuiYhI7+TLcC/YewCATXuqU1wSEZHeKW64m9nTZlZiZnlRXv+MmVWa2Urv5zfJL2ZrdY3NAPxt8Y7uXpWIiC8lcpu9Z4GHgOdjzPOBc+5LSSmRiIh0Wdyau3PufWBvD5RFRESSJFlt7ueY2Sozm2VmJ0WbycxuMLNcM8stLS1N0qpFRKStZIT7CuAo59wpwIPAG9FmdM497pyb5JybNHr06CSsWkREIulyuDvnqpxz1d7jmUC2mY3qcslERKTTuhzuZna4WeCyIjOb7C2zvKvLFRGRzovbW8bMXgA+A4wys0LgNiAbwDn3KHAl8EMzawQOAle7br6Lxgljh7C+uKo7VyEi4mtxw905d02c1x8i0FWyx9xw3tH8z0urenKVIiK+4ssrVDOCg8uIiEhEvgz3c44ZCcD5H1OPGxGRSHwZ7qMH9wXg1AnDUlwSEZHeyZfh7nXOQQP+iohE5stwB8gw6OZOOSIivuXbcDcz3WZPRCQK34Z7oOae6lKIiPROvg13w3QPVRGRKPwb7gZOp1RFRCLybbhnmKlZRkQkCt+Guxk0q11GRCQi34Z7hpkaZUREovBtuJuhrpAiIlH4N9xRV0gRkWh8G+4ZGaYrVEVEovBtuBuon7uISBS+DffACVWlu4hIJL4N98DYMqkuhYhI7+TjcNeokCIi0fg23DVwmIhIdD4Odw35KyISjW/DvbiyltdXFKW6GCIivZJvwx2gUWdURUQi8nW4i4hIZAp3EZE0pHAXEUlDCncRkTTk+3BvaGpOdRFERHod34f71BnrU10EEZFex/fhvmJnRaqLICLS6/g+3EVEpD2Fu4hIGlK4i4ikobjhbmZPm1mJmeVFed3M7AEzyzez1WZ2evKLGZ0GDxMRaS+RmvuzwMUxXr8EmOj93AA80vViJW7drqqeXJ2IiC/EDXfn3PvA3hizXA487wIWA8PMbGyyChiPmfXUqkREfCMZbe5HAAVhzwu9aT0iU+EuItJOMsI9UrpGbAg3sxvMLNfMcktLS5OwasjQKWERkXaSEY2FwISw5+OBXZFmdM497pyb5JybNHr06CSsWjV3EZFIkhHu04FrvV4zZwOVzrniJCw3IRkZCncRkbYS6Qr5ArAION7MCs3se2Z2o5nd6M0yE9gK5ANPAD/qttKG+cqp4wDIVLiLiLSTFW8G59w1cV53wI+TVqIEDRvQp6dXKSLiG74/HalrmERE2vNtuAfPozqlu4hIO74N9yBFu4hIe74Ndwt2r1e6i4i049twD1K2i4i059twd16sV9c18ouXV6a4NCIivYtvwz3c6yuKUl0EEZFeJS3CXUREWvNtuKsHpIhIdL4NdxERic634a6Ll0REovNtuIuISHS+DXfV20VEovNvuCvdRUSi8m24i4hIdL4Nd6eGGRGRqHwb7iIiEp1vw11t7iIi0fk23EVEJDrfhrsq7iIi0fk33JXuIiJR+TbcL/j4mFbPi/YdTFFJRER6H9+G+0UnHtbq+cbdVSkqiYhI7+PbcG8r2Eyzt6Y+tQUREekF0ibcmx3M21jC6XfN4f1NpakujohISqVRuDs+2lEBwIqdFSkujYhIaqVNuDvnwMx7nOLCiIikWFaqC5AsN05bEXqsbBeRQ13a1NzD6S5NInKo83W4X3XG+IjTmxXuInKI83W49++TGXG6sl1EDnW+DneLMl3ZLiKHOn+Hu0WO92CzTMn+WmobmpK2vt2VyV2eiEh38Xm4R54ebJaZPHUu33tuWdLWd/Y9c/nhtOVJW56ISHdJKNzN7GIz22hm+WY2JcLr15lZqZmt9H6+n/yiRihXlIaZ8N4yC/PLk7rOeRt19auI9H5x+7mbWSbwMHARUAgsM7Ppzrl1bWZ9yTn3k24oY1TZWZHDvay6nuJKjRIpIoeuRGruk4F859xW51w98CJwefcWKzGnTRgecfo/PyrinHveDT1/bXkhd/xrbU8VS0Qk5RIJ9yOAgrDnhd60tr5mZqvN7FUzmxBpQWZ2g5nlmlluaWnXmzcu/sThCc33y1dW8czC7V1en4iIXyQS7pHaPtr2NvwXkOOc+yTwDvBcpAU55x53zk1yzk0aPXp0x0rqUzvKa9ilG4mISA9LJNwLgfCa+HhgV/gMzrly51yd9/QJ4IzkFK/36OyQBuffO5//+N278WcUEUmiRMJ9GTDRzI42sz7A1cD08BnMbGzY08uA9ckrYu+gq15FxE/i9pZxzjWa2U+A2UAm8LRzbq2Z3QnkOuemAz81s8uARmAvcF03lllEROJIqJ+7c26mc+5jzrljnXNTvWm/8YId59zNzrmTnHOnOOc+65zb0J2FDnfPV09OeN6SqloenpffqSYWVdxFxE98P577iWOHJDzv5N/OBaCipp7/uehjDOyb+OYnYxjhkv21jBzYl8yMaKPiiIgkh6+HHwA4bsygDr/nyQXbuH36Wsqq69haWt0NpWqvrLqOyVPn8oe3euygRkQOYb4P94xoA8zEsbuqlvP/MI8L7nsv7rzNzY4Za4o7tZ6gvTX1AMzdUNKl5YiIJML34d7JbAegpj6xER5fzi3gZy+u7PyKaLlYoKfvElVceZAbns/lQH1jj65XRFLL9+He2Zp7R+7WVLK/Lv5McQSHJ+7pE7P3zt7I2+v2MHPN7h5es4ikUhqEe+fe1zbbN+3ZT8HeA10vUBShfZC63YhID/B9b5loN+yIp+JAQ+jxFX9dyEc79wGw/XdfBKC6rpF/flTEf551ZKeW/+h7W2hobG43PdXZvmhLORNG9Gf88AEpLomIdCffh3tna+7ri6tCj4PBHu6O6Wt5ZXkhR48c2Knl/25W614xqWpzb+uaJxaTnWlsnnppSsshIt3L980yna25xxPs3XIwSbfVC29zr6ptIGfKDJ78YGtSlh1ThH1JQ1Oqjx9EpLv5Pty7mwH3z9mUlOUElVTVAvCPpTu7vFwRkUgU7m3U1DXyg+dzKa6s7fB7Kw82sKcq9vucazmZ29GePjvLD/BKbkH8GcPpYliRQ5Lv29yTbeaaYuas2xN63pEGjE///l2qahvZdk/79uyMULOMozkU7tGX9UpuASeNG8qJ41qGV7jirwspr6nnyjPGd1tzlIikh7SouV904mFJW1ZXQrOqNnCh0LQl7Ztbps4M3HK2ubmlj32smvv/vbqaSx/4oNW0cu88gIYfbu+x97awaEtyb4Yu4mdpEe5PXDspacv631dWdXkZH2xqfwvB2WtbjgaC4d7ZHUmHsv0Q2RHcM2sD1zyxONXFEOk10iLcAY4Y1r9blvuD53M7/J54V79Wen3s1xdXsW5XVcx5I0lmd0rnHEu2lqe8i6aIJFfahPvko0eEHs/46adSWBJinox1zvHNJ5eEnn//uWUdXn6sGK6oqac+/OKpOAcHLy0r4BuPL+bm19dwwR/nU9eYnK6fcuhwzvV45SBnygwenpffo+v0m7QJ93AnjRua0vWvjVEbb/svsCtsR7DvQD0/nLY8bo+b8CODkv213DNzPU3NgX+w0+6aw4/+viL0emmccXG2ldUA8OKyAraW1VBYkX43865vbOZggoPESccdffNMbp++tsfWF9yR3Dt7Y4+t04/SJtxPnTAs1UVISKQKTnl1HTlTZnD+vfOZlbebS/7SciL1zZVFXPbQglbzf/2xxWzesx+AKa+t4bH3t7J4aznPL9oBwDvrA+37Jftr+WBzWTdtiX9c9tACTvjNW6kuRlp7zvvu9QS1ICYmbbpCXnvOUQzok0nRvu6vee4sP0B+6X4u+HjHe+nsjlAr//E/AjXtyoOBtvjg1bFAaKjh8MPeVQX7+O3M9Txz/eRQE0yzc7zbZqz48OUkqqOneN/K282ZOcMZOahvh9fVUzbs3p/qIkRUUVNPdlYGgzpwR7DeJhXnapTtiUmbmruZcdWkCfz8cx/r9nWdd+88vvtsy4nWl5bt5M2VRZ1e3uKte+PO09Tc+is9b2Mpl4bV8Nv+j32wuZTMsN44zc2ODbvbNxfF+0epqKnn648toriy/U5z34F6bpy2nO89F/2kc31jMzV18ceSX19cRVl1YkMr766sJWfKDHK3x/+7RfLdZ5fx5soiZq/dzRl3zenyeYZZa4oprOj4iKKn3TWH/7hnbpfWnWqpqEV3ZLju3uaJ97eyo7ymR9aVNuGeCje/voacKTP41Wtrunwzj3h+/WZeu2nriqta3aykOixEv/3UUsL3B/e+vZGL/9y633wiXltRyNJte3nyg22tplfVNoSOGoLBtjC/jJwpM1odMVzzxGJOum121OWXV9eRV1TJJX/5gEl3v8PTC7ZFnTfobC8Q/x7heoJEvLuhhJ+9uJI7/7WO8pp6fvHyKvYd6PhRTtAP/76Cyx9a2Kn3Bq+N8KtUxKxfs73yYANTZ67nm08siT9zEijcu+CFHhwb5oWl8YcdWL6jotXzW/65JvS47YnV9zaVMnvt7rjbEKwlPbVgG7XeIGoVNfV88va3+dM7m1vN++h7WwB49sPtrCms5MMtZaEy7dp3kJwpM1iytfWFRpc9tJAvPdhyTuHOf6+Lu51ByWoSmLG6mKkz1ndpGeWdaALrjAWby7q0I2rrO08v5f63O39iMjXNMj5Nd0/VwYb4MyXBIRHu//j+WakuQrcJ9m6J9HXPbRP24b7z9FL+62/L2R+h5phfsp+8osp20yu8UNnr/Y62Y3hg7ma+/NCCVjWUZV4Tyt8Wtz7x1pVzJF39Fw8/6qmLMPZ+b3OgvpH/fGoJ1z/b8e6z0by3qZQH3u18l8LmFORs+P6koan3f25BweFGeqpZKW3D/Sunjgs9/qRPetJ0RrArYzJ97v73Q7Xp8O9hcLiE9idd45+G7ZuVCXQtREv31/HswpZmm2j/I845bnszj3kbS1rtpLaX1dCcxDRaX1zFTa+2XNG8pbQ6acuOpNEre/6e7l1PR6SiFh3+uf979a4eX39nBYvdpHDvmj9ffRoD+2Smuhi+Ez4kQll1XauamVngxOyO8tYnD8uq6/j6Y4uoa4ge3DdOWw4E/jErauqZv7Ekbm+enCkzuP6ZpQB84rbZnDn1HW7/V/xmm8KKgzy3aAfXP7OsVZPPZ/44n61lLcHY9n+sdH9d3OsCwn3/uVxezi0MPX9mYfzzBet2VTFvY0lo2OeWsjimzljHxh7s2VNV29I8sLqw5YY1e2vq49aIa+oaeWT+FlbsaH+jm+4WvkPJK6riu88uCzUZRhO8h8K/VnXPzuCLD3zAL16Kfd7NeX/S5h462PBvH6wEvHjDOby6vOCQCPnGJB2ehrehTrr7HW66+PjQ80//fl7UmvfSbYn1XHln/R6ue3YZqwr2kZ0ZucY/bfEOThgbGA1z3sZStpfVtDpZHCprlHXUx/hbFO1rCdWS/a0D9syp7wAtt1qMJ6NN1Si4I8wrqmRBfhk3nn9su/e0HQwuqKy6nic+2MY/P9pF7q2fS2j94Rqamsk0I6MDtyb7ZthYPJc9tJDtv/sizc2O0++aw1dPO4L7v3Fq1Pf+4a0NPdq3fd+Bev69uphvnXVkq53yU94J+I927uOcY0dGff+OskCF5LH3t/DlU8ZFnS/ie8trMIwjR0a/NeXaXVWs3VUV+pu9vXY3Z+aMYPjAPqF5gjslNcskwcnjh3LH5Z84JIbHjdUdsSPa3qUp/MrOZLVLb/Jqp9HuCHXrG3l87ZEPQ88/88f5Eeerb2yKWMOM1fQSvhOMtv6cKTPY6R2dvLeplJwpM5h09zvkTJnB4rATwm1H9Qyu90sPLmh3m8W8okq+8nAiPWpal2nZ9r3MXFPMW3m7Q6G2v66RnCkzWg1NPfGWWfyXd3SUiEAvpfZdY4NNBq9/VMQHm9sPgBcU6VxNLFf8dSEf//WsDr0n3P+9uppb38gjr6gq4k49XmAu3BK4mC+vqKr18BwJOP/e+Zx377yE56+oqeeGvy1vNy5V8GupZhlJiS/8+f1Wzx/swsm2aNr22e+s2Wv3MPGWWXyY33IVrnMu1DYdSbRAbzv1fS/YXl8RaHYJ9sG/+vGW2u7uNmMIvbisoNXRQPhO5ksPLmBlQWJNGA/O3cy3nlzMv1fv4qpHF/Gjv68INGu1KeTMNcVA4KI2oFXYB+07UB96PVykczW/enV1q+aNbz+1NHRhXXAn2tjUTHmC1yOE+2jnPmpjNNvFU+E14dU2NkUM8njhHr6zfWvt7k6V4e4Ee3IF/1Y79rZuvgweFfdUByOFexR9s/Sn6S7JrrnMDxtiudkRc2yejvauiHQe4SsPL2T5jr0Rj2S+FdZD6JH3tnDFXxcmdBFXuPvmbGJhfjk/+cdHraa3DbCmZkfB3gNcHuOI4JtPLAm9nldUGeq1FOlo9qXcgnY7iIamZhZtKWfiLbO49Y01/PKVVZxx9zsRm8k6qqGpmf21rbsFVtc1cvLts/lgcykl+2tDF5gFj5ICYyi1X1b4/vzlZQWhbsBz1+8hZ8qMVvN2tvvmkwlcg1HX2BS1O29P9yxSgkVx25dPYuzQfqkuRlpKVs09KPzq0MbmZq57JnpXwWjh3vYf3nnTItXyVhbs42uPLIq4nM0lLSds7529kY927ot5ERfAy2G3ToyVOxf9qfVR1fRVu1odSQS9uyEQaLPWFLOuOND08rMXP+JLDy7gqkcD5Y7WNN8Y4chmQX5g5zlt8U7eXBk4IRkr3J9duI27/72OR+Zvib4xwE9f+IiTb38bgDWFlZxzz1yWbitnf20j98/ZxOSpc/nRtMDQHMHzG83OURfh5Gn4UdJNr60OXeD2xsr2J1C3ltZQVdsQOhrojJtfX8O9sze0m/722j38e3XgiKrtZ9nTPYsOuXCPdzLli58cC8CRIwaw6OYLe6JI0kUfht2BqSjOqJaRwiuSX7+Rxye94OluN726mntmBS6iinUxVKThGdpeJ3DrG2tCQ2P8MGx00DfDQi6/pJrMKOl+02urWz2fOmM9e2sSu+jmLq/Gevu/1vHkgm38/q0NPDi39YVuDU3N/G3RdhqbmpmV17Lj/PJDCyiurGWJNxRHMKznbijhgbmbyd0euGZj0ZZyJv+2/ZANdY3NnPeHecxuszPOirCdf5m7mUl3vcNpd82JuB3VdY1R7+rlnOP3b23ghaU7eXhe7J0XwP1zNjFvY4n33rizJ1Va95YJN354fyYfPYJ7vnpyzO5QD11zGj+9YCLHHz64B0snXbHvQEv4/DVObfE309sP4wCEalvh9ieh6SFRyzo5Tk5b0xbHv2r6Fy+v5LdXnJzQ8v75UeJjJj21YFuo90rQfXM2MahfS8w8s3Abv525geq6ltp3q55eXhaHN3ndP2dT6PGSKOMwrSzYx869B7jln60/32i3sgz2qHpzZRF9MjO45OSxodd+8dJK3l63h4VTLmBo/+xW75u5ZnfcI5Kgsuo6HvB2btt/90WFe3dZ8KsLQo/X3fkFGhodmZlG/+xMMiwwJnWfzAzMrFWwL775QgoqDnDVo4sYNiC7VZAAvHLjOYwa1JfPej06Pj1xVLthdrMzLeqJPEmuV5cXxny9Kyf1ulPB3p4bR391YWWXh874sAP3q70j7NqE4BXVFWFDKJx2Z0sN2rx0j3a9wdIoO8Hg0Bfh5yVeXlZAZpy2ieCYUBvvvjh0oV1wFNFzf/duu/mDI7gGffOJxUz7XssV8NF6GDU1O779VM+MKRN0yDXLAAzok8XQAdkM6ptFZoZhZvzioo/xxo/PbTfv4UP7ccr4YQzok8kdl53EpScf3ur1M3NGcPSogaHmnivPGN/q9Ye+eRqbp17afRsj0gmdHXStq4L3HHj8/a2haeFHSMGQ7uxYPeEXxt302upWF5nF8nMv5GsbmkI9hBLx4ZZy/jq/pUdZtPUd+/9mtjof0xMSqrmb2cXAX4BM4Enn3O/avN4XeB44AygHvuGc257conavn144MeprfbIyWHdT/VCiAAAKMklEQVTnxQBcfuoRobPv4YdswetxzIxnrz+TxibH505sGe/9pouPZ3C/bH79Rsth45B+WRFHBRw7tF/MW/WJSHLNytvNrW+sYe76kg6FO8Af394Uf6Y2yqvruv0eCBavW5CZZQKbgIuAQmAZcI1zbl3YPD8CPumcu9HMrgaucM59I9ZyJ02a5HJzk3PhTU97b1Mpy7bt5RtnTmDCiMBVa2XVdfxpziZu+/JJ9InRjbK+sZk9VbX8Ze5m7vnqyWRnZoR2Fjecdww7yw9w51dOYvLUuTxz/Znsr23k1PHDOnQRBcCPPnNs3Pbntn74mWND7Yn//u9Ptbp0P5KckQPYXt7xccxFDnWfP/EwHr92Uqfea2bLnXNx35xIuJ8D3O6c+4L3/GYA59w9YfPM9uZZZGZZwG5gtIuxcD+He7Llbt9LXlEl1517dNR5/rFkJyMGZnP2MSPJysxg176DrC+u4vjDB5MzciCLt5bT7BxF+2r51HGjyBk5gFvfyOPvS3by0g1nM2ZIPw4b0pesjAzmbyzh7XV7yCuqZFDfLHJ3VPDOL87nuDGDaG52ZGQYzjl+8Pzy0C372vrmWUfy2ytO5kB9I6X76zj/3vmh18J3EgCnTBgWupDmmFED2Zqkwc6uPeeo0GF+Tzhq5IB24+qIdMYRw/qzcMoF8WeMIJnhfiVwsXPu+97zbwNnOed+EjZPnjdPofd8izdPWZtl3QDcAHDkkUeesWNHz/1jSudsL6th3sYSrj/3aA7WN7F2VyVHjhjAiIF9yAo7W1V5oIG+2Rn0yw6clFq3K7Dj2bXvIOOG9adg7wGOHDGAjAyjvrGZZufom5VBs4NtZdUcN2YwBXsPMKBPJkP6B05cD+iTycC+WdQ1NlF5oIGq2kbmrt/DuceNYvzw/gwbEBi3Y9e+g9Q1NpOdaVz+0EIW/OoC+mVnsHTbXrKzMqioqeex97byq0uO50B9E2cfM5LszAwO1jfRv08mDU3NZJgxK6+YrAzj8fe38t8XTuTIEQOY5926cHt5Db+86HjqvLKXV9ezdlcln/34GAb3y2J3ZS3HjB5Ewd4DTFuygxPHDuELJx3Oih0VDB/YhzWFlTQ5R8WBer508jgamgMXB720rIDbvnwi8zeWcuEJY7h/ziaumjSBY0cPZHVhJWMG9+Wzx49he3kNF9z3HoP7ZvG1M8aTYcbwAdnU1Dexo7yGw4b04+xjRnDcmMFMW7yDE8cNYWj/bEr31/HSsgLWFFXy4DWnMXJQH376wkrKqus47chhXHbKOMYPH0BFTT0v5xaQu6OCZ68/k9GD+9I3K4PvPZfLjvIDvHjD2eQVVfKPpTs5/rDBXPDxMYwY2IfVhZVsLavhX6t2ccr4odTUN3Hj+cfywNzNfPX0Ixg3rD+z1hRz7OhBDOmfzbayGgb0yWT6ql2thjA4adwQBvXNYsm2vdz6xRP4/Vsb+OzxYxjYN4uBfTNDvYCGD8hm+IA+bC2r4WOHDWLTnmruu+oU/jJ3MzvbXBH66YmjmDhmMGuK9rFsewXXn5vD4H7ZvLh0Jwcbmmhqdhw2pF+nR1Yd3C8r4jAMHz98MAP7Br4Tbbuqfu6Ew3hn/R7++q3TuTSsh05HJDPcrwK+0CbcJzvn/jtsnrXePOHhPtk5F/WUumruIiIdl2i4J9JbphCYEPZ8PNC2o3hoHq9ZZiiQnI67IiLSYYmE+zJgopkdbWZ9gKuB6W3mmQ58x3t8JfBurPZ2ERHpXnG7QjrnGs3sJ8BsAl0hn3bOrTWzO4Fc59x04Cngb2aWT6DGfnV3FlpERGJLqJ+7c24mMLPNtN+EPa4Frkpu0UREpLMOyStURUTSncJdRCQNKdxFRNKQwl1EJA3FvYip21ZsVgp09hLVUUBZ3Ll6N79vg9/LD/7fBr+XH/y/Dako/1HOudHxZkpZuHeFmeUmcoVWb+b3bfB7+cH/2+D38oP/t6E3l1/NMiIiaUjhLiKShvwa7o+nugBJ4Pdt8Hv5wf/b4Pfyg/+3odeW35dt7iIiEptfa+4iIhKDwl1EJA35LtzN7GIz22hm+WY2JdXlicbMtpvZGjNbaWa53rQRZjbHzDZ7v4d7083MHvC2abWZnZ6iMj9tZiXenbWC0zpcZjP7jjf/ZjP7TqR19WD5bzezIu9zWGlml4a9drNX/o1m9oWw6Sn7jpnZBDObZ2brzWytmf3Mm+6LzyFG+X3zOZhZPzNbamarvG24w5t+tJkt8f6eL3lDoGNmfb3n+d7rOfG2rUc453zzQ2DI4S3AMUAfYBVwYqrLFaWs24FRbab9AZjiPZ4C/N57fCkwCzDgbGBJisp8HnA6kNfZMgMjgK3e7+He4+EpLP/twP9GmPdE7/vTFzja+15lpvo7BowFTvceDyZwc/oT/fI5xCi/bz4H7285yHucDSzx/rYvA1d70x8Ffug9/hHwqPf4auClWNvWU98lv9XcJwP5zrmtzrl64EXg8hSXqSMuB57zHj8HfCVs+vMuYDEwzMw6d4PFLnDOvU/7O2h1tMxfAOY45/Y65yqAOcDF3V/6qOWP5nLgRedcnXNuG5BP4PuV0u+Yc67YObfCe7wfWA8cgU8+hxjlj6bXfQ7e37Lae5rt/TjgAuBVb3rbzyD42bwKXGhmRvRt6xF+C/cjgIKw54XE/uKkkgPeNrPlFrgxOMBhzrliCPwTAGO86b15uzpa5t64LT/xmiyeDjZn4IPye4f3pxGoOfruc2hTfvDR52BmmWa2EighsGPcAuxzzgXviB1enlBZvdcrgZGkeBv8Fu4WYVpv7ct5rnPudOAS4Mdmdl6Mef20XUHRytzbtuUR4FjgVKAYuM+b3qvLb2aDgNeAnzvnqmLNGmFayrcjQvl99Tk455qcc6cSuGf0ZOCEGOXpldvgt3BP5GbdvYJzbpf3uwT4J4EvyJ5gc4v3u8SbvTdvV0fL3Ku2xTm3x/tHbQaeoOWwuNeW38yyCQTj351zr3uTffM5RCq/Hz8HAOfcPmA+gTb3YWYWvHtdeHlCZfVeH0qgeTCl2+C3cE/kZt0pZ2YDzWxw8DHweSCP1jcS/w7wpvd4OnCt1/PhbKAyeAjeC3S0zLOBz5vZcO/Q+/PetJRoc+7iCgKfAwTKf7XX0+FoYCKwlBR/x7y22qeA9c65+8Ne8sXnEK38fvoczGy0mQ3zHvcHPkfg3ME84EpvtrafQfCzuRJ41wXOqEbbtp7RU2duk/VDoHfAJgJtYLekujxRyngMgbPkq4C1wXISaIebC2z2fo9wLWfnH/a2aQ0wKUXlfoHAIXMDgVrH9zpTZuC7BE4e5QPXp7j8f/PKt5rAP9vYsPlv8cq/EbikN3zHgE8ROHRfDaz0fi71y+cQo/y++RyATwIfeWXNA37jTT+GQDjnA68Afb3p/bzn+d7rx8Tbtp740fADIiJpyG/NMiIikgCFu4hIGlK4i4ikIYW7iEgaUriLiKQhhbuISBpSuIuIpKH/D93qHdAxqvt/AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#import sys\n",
    "#print(sys.path)\n",
    "#print(sys.executable)\n",
    "\n",
    "import cnn_utils\n",
    "from matplotlib import pyplot as plt\n",
    "import keras\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Flatten\n",
    "from keras.layers import Conv2D, MaxPooling2D\n",
    "import keras.callbacks as cb\n",
    "from keras import regularizers\n",
    "\n",
    "class LossHistory(cb.Callback):\n",
    "    def on_train_begin(self, logs={}):\n",
    "        self.losses = []\n",
    "\n",
    "    def on_batch_end(self, batch, logs={}):\n",
    "        batch_loss = logs.get('loss')\n",
    "        self.losses.append(batch_loss)\n",
    "\n",
    "def init_model():\n",
    "    model = Sequential()\n",
    "    model.add(Conv2D(32, kernel_size=(3, 3),\n",
    "                    activation='relu',\n",
    "                    input_shape=(28, 28, 1)))\n",
    "    model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "    model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "    model.add(Dropout(0.25))\n",
    "    model.add(Flatten())\n",
    "    model.add(Dense(128, activation='relu'))\n",
    "    model.add(Dropout(0.5))\n",
    "    model.add(Dense(10, activation='softmax'))\n",
    "\n",
    "\n",
    "    model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer=keras.optimizers.Adadelta(),\n",
    "              metrics=['accuracy'])\n",
    "    return model\n",
    "\n",
    "def plot_losses(losses):\n",
    "    plt.plot(losses)\n",
    "    plt.title('Loss per batch')\n",
    "    plt.show()\n",
    "    \n",
    "filePath = 'C:/Users/computer/Desktop/Assignment8/'   # the training set is stored in this directory\n",
    "mnistData= cnn_utils.loadData(filePath)\n",
    "mnistData_preprocessed = cnn_utils.preprocessData_CNN(mnistData)\n",
    "(X_train, y_train), (X_test, y_test) = mnistData_preprocessed\n",
    "\n",
    "(X_train_mini_batch, y_train_mini_batch) = cnn_utils.mini_batch(X_train, y_train, 10000, 0)\n",
    "(X_test_mini_batch, y_test_mini_batch) = cnn_utils.mini_batch(X_test, y_test, 5000, 0)\n",
    "print(X_train_mini_batch.shape)\n",
    "print(y_test_mini_batch.shape)\n",
    "\n",
    "model = init_model()\n",
    "history = LossHistory()\n",
    "model.fit(X_train_mini_batch, y_train_mini_batch, epochs=20, batch_size=64,\n",
    "                callbacks=[history],\n",
    "                validation_data=(X_test_mini_batch, y_test_mini_batch), verbose=2)\n",
    "\n",
    "score = model.evaluate(X_test_mini_batch, y_test_mini_batch, batch_size=16)\n",
    "print(score)\n",
    "plot_losses(history.losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As illustrated, after 20 epochs we reached a cost of approximately 0.0519 and a test accuracy of 0.9844. The results are very good and reflect upon the power of CNNs. These results are comparable to the first network with minibataches of 256; however, the learning curve reveals a much more sporatic path towards the minima. Intiutively, this is because with a smaller minibatch size comes less precise gradients; therefore, the gradients vary more during each iteration and the cost changes rapidly. Though this can have the benefit of preventing the network from getting stuck on local minima, there is a trade-off in terms of the smoothness of the learning curve."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}