94-775 Neural Net Demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 94-775: Handwritten Digit Recognition with Neural Nets (single and two layer)\n",
    "\n",
    "This demo draws heavily from the handwritten digit example in Chapter 2 of Francois Chollet's \"Deep Learning with Python\" book. I've added a simpler single-layer example first before moving to the 2-layer example. -George Chen (georgechen [at symbol] cmu.edu)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We begin with some boilerplate code..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "plt.style.use('ggplot')  # if you want your plots to look like ggplot (like how R makes plots)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load in data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We load in the MNIST (handwritten digit) dataset from the `keras` package (`keras` is a popular neural net package in Python)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/georgehc/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n",
      "Using TensorFlow backend.\n",
      "/Users/georgehc/anaconda3/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6\n",
      "  return f(*args, **kwds)\n"
     ]
    }
   ],
   "source": [
    "from keras.datasets import mnist\n",
    "\n",
    "(train_images, train_labels), (test_images, test_labels) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 28, 28)\n"
     ]
    }
   ],
   "source": [
    "print(train_images.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10000, 28, 28)\n"
     ]
    }
   ],
   "source": [
    "print(test_images.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "flattened_train_images = train_images.reshape(len(train_images), -1)  # flattens out each training image\n",
    "flattened_train_images = flattened_train_images.astype(np.float32) / 255  # rescale to be between 0 and 1\n",
    "flattened_test_images = test_images.reshape(len(test_images), -1)  # flattens out each test image\n",
    "flattened_test_images = flattened_test_images.astype(np.float32) / 255  # rescale to be between 0 and 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(60000, 784)\n"
     ]
    }
   ],
   "source": [
    "print(flattened_train_images.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10000, 784)\n"
     ]
    }
   ],
   "source": [
    "print(flattened_test_images.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall from the lecture slides that we set up the neural net for this handwritten digit recognition problem to output 10 numbers, corresponding to probabilities of seeing digits 0, 1, ..., 9. We will want to convert the raw labels also to be in this format (i.e., the number 5 would be converted to a 1D array of 10 numbers, where it's all 0's except for index 5 which has value 1). This vector representation is called a categorical representation. We do this conversion next:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras.utils import to_categorical\n",
    "train_labels_categorical = to_categorical(train_labels)\n",
    "test_labels_categorical = to_categorical(test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see what the 0th training data point's label is, and what this same label is when represented in as the categorical representation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_labels[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 0., 0., 0., 1., 0., 0., 0., 0.])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_labels_categorical[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below is an example of how to plot out a randomly chosen training image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-0.5, 27.5, 27.5, -0.5)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAABuNJREFUeJzt3U+Izfsfx3HDzYTFbFi4SpOQpZTU\nsKOksKChGUt2lpRSSJOaLUoRsbCzsPEnlCRNlJRmJWV2o5RE1EyYczd38avbeR+/OTOHmdfjsX31\nPd/v5tl38ZlzpqvRaDQWAHEW/u4HAH4P8UMo8UMo8UMo8UMo8UMo8UMo8UMo8UOovzp5s66urk7e\nDiL96h/tevNDKPFDKPFDKPFDKPFDKPFDKPFDKPFDKPFDKPFDKPFDKPFDKPFDKPFDKPFDqI5+nx/+\nV09PT7m/f/++3JcsWVLuJ0+ebLoNDw+X1yb8IytvfgglfgglfgglfgglfgglfgjlqI/f5tixY+Xe\n3d1d7lNTU+Xe19fXdGv1M/KO+oB5S/wQSvwQSvwQSvwQSvwQSvwQqqvRwQNN/6I7z5YtW5puT548\nKa9dvHhxW/feunVr0+358+dtffafzL/oBkrih1Dih1Dih1Dih1Dih1Dih1C+z09bent7y/3mzZtN\nt3bP8UdGRsr9zZs3bX3+fOfND6HED6HED6HED6HED6HED6HED6Gc89OWoaGhcl+zZs20P3tiYqLc\njx8/Xu6fPn2a9r0TePNDKPFDKPFDKPFDKPFDKPFDKEd9lM6dO1fuAwMDs3bvwcHBcn/x4sWs3TuB\nNz+EEj+EEj+EEj+EEj+EEj+EEj+Ecs4frvoX2gsWLFhw5MiRcm/n367fvn273O/evTvtz6Y1b34I\nJX4IJX4IJX4IJX4IJX4IJX4I1dVoNBodu1kbZ8JMz6ZNm8r93r175b5ixYq27j86Otp027ZtW3nt\n169f27p3ql9N2psfQokfQokfQokfQokfQokfQokfQvk+/zxw8ODBptvVq1fLa5cuXdrWvVv9G+zq\n9wBaneMvXFi/m3p6esq9Hd+/fy/3+fA3CN78EEr8EEr8EEr8EEr8EEr8EEr8EMo5/xzQ6rf1z58/\n33Rr9xz/48eP5X7ixIlyf/nyZdOt1W8FXLx4sdz7+/vLvR3j4+Plvnv37nJ//fr1TD7OrPDmh1Di\nh1Dih1Dih1Dih1Dih1B+uvsPsHHjxnJ/8OBBuS9fvnza9251lLd3795yP3DgQLn39fU13Vod9fX2\n9pb77/Thw4dyX7lyZYee5L/8dDdQEj+EEj+EEj+EEj+EEj+EEj+E8pXeDuju7i73s2fPlns75/it\nTE5OlvuNGzfKfd26dTP4NHPH2NjY736EtnnzQyjxQyjxQyjxQyjxQyjxQyjxQyjn/DNg2bJl5X7t\n2rVyb/Uz0LPp77//ntXP//LlS9NtdHS0vHZkZKTcX716Ve47duxouh0+fLi8tpWhoaG2rv8TePND\nKPFDKPFDKPFDKPFDKPFDKPFDKOf8v2jRokVNt+vXr5fX7t+/f6Yf548xMTFR7oODg023+/fvt3Xv\n7du3l/vmzZvb+vzK27dvZ+2zO8WbH0KJH0KJH0KJH0KJH0KJH0KJH0I55/9FR48ebbrN53P8Z8+e\nlfupU6fK/enTp9O+d39/f7m3+k599T8Ffv78WV575syZcn/37l25zwXe/BBK/BBK/BBK/BBK/BBK\n/BDKUd+/Wv0b7FbHTn+qHz9+lHurI63Lly+X+7dv38r90KFDTbd9+/aV1+7Zs6fcq69Zt3L69Oly\nHx4envZnzxXe/BBK/BBK/BBK/BBK/BBK/BBK/BDKOf+/Lly4UO59fX0depKZtXPnznLfsGFDud+6\ndavcV61aVe7r168v93Y8fPiw3O/cudN0u3Tp0kw/zpzjzQ+hxA+hxA+hxA+hxA+hxA+hxA+huhqN\nRqNjN+vq6tSt/mPt2rXl/ujRo3JfvXr1TD5Ox7T6iep2vhPfrvHx8XIfGBgo95GRkXKfmpr6v59p\nPvjVpL35IZT4IZT4IZT4IZT4IZT4IZT4IVTMOf/Y2Fi5z9Vz/N/t8ePH5X737t2m2/Xr18trP3/+\nPK1nSuecHyiJH0KJH0KJH0KJH0KJH0LFHPW1+gnp2fz6Z6uvE+/atWvW7j05OVnuV65caevzWx2h\ntvpKMTPPUR9QEj+EEj+EEj+EEj+EEj+EEj+EijnnhxTO+YGS+CGU+CGU+CGU+CGU+CGU+CGU+CGU\n+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU\n+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CGU+CHU\nX528WaPR6OTtgII3P4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4QS\nP4QSP4QSP4QSP4QSP4QSP4QSP4QSP4T6B+wWJ5zK/YQrAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x181d47e390>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# this plots out one of the training images\n",
    "idx = np.random.randint(len(train_images))  # random training image index\n",
    "plt.imshow(train_images[idx], cmap='gray')\n",
    "plt.axis('off')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-0.5, 9.5, 783.5, -0.5)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAADEAAAD8CAYAAADe6kx2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAAxJJREFUeJztnLGOeVEQhwcrSqLwBCRKkYhoJFov\n4Cl4C+/CC2g1EoVGKRGhVojQEfy781fucTbZbye/r5pmEt/OnZmz914yr9frZX+c7G9/gJ9AEhQk\nQUESFCRB4eu3P4CZWSaTCfEnBwgXlZAEBUlQkAQFSVCQBAVJUJAEBUlQkAQFxP/YxWIxKT9DuLWf\nzf6/IJ7PZ3Q+ohKpf0cXPSEJCpKggJhOj8cjKR+xJ3RD2STBQRIUXEhoT/wU2hMmCQ6SoIAYsdvt\nNikfMWLL5XKIT6dTdD5CQnvCJMFBEhQQe0JHcXNyOUmCgiQoIEbseDxOykeM2Pcnpu9PUr8LQkJ7\nwiTBQRIUEHvCxVE8n8+H+H6/R+cjJLQnTBIcXEggRmyz2UzKR0wnHcXNSU9IgoIkKCD2hIujeKvV\nCvFqtYrOR1Rit9sl5SMqoWVnkuCAaOx2u52Uj2hsF0fxSqUS4uPxGJ2PkHBRCe0JkwQHxJ5wcRTv\ndDohXi6X0fkIidQR66InEJXQnjAnEogRWygUkvIRPaHpZJLggOgJ7QlzIoHYE5PJJCkf0ROpewJR\nicvlEuJSqRSdj6jEYDAI8XQ6jc5HVOJ6vSblIyqhPWGQyymXyyXlIy4nHcUNUol+vx/i2WwWnY+Q\nWK/XIW40GtH5CAmNWHMigdgTo9EoKR/REy6O4ovFIsTdbjc6H1GJ1OmEqMTtdkvKd1EJFyPWhQSi\nJ1KfnrqoBKKxXfyWzfs3ut6/6fVdEBIasSYJDi72BKKxe71eiOfzeXQ+QmKz2YS4Xq9H5yMup8Ph\nEOJPJBCV0J4wJxKIntjv90n5iJ5wccvmfdn9WQkXT4o0Yk0SHBCNrVs25kQCMWLfXw86n8/R+QgJ\n7QmTBAfEnqhWq0n5CInhcJiUj5hOtVotxJ/87j5CQiPWJMFBEhQkQUESFCRBQRIUJEFBEhQkQUES\nFCRBQRIUJEFBEhQkQUESFCRBQRIUJEFBEhQkQUESFCRBQRIUEO8Apr785qISkqAgCQqSoCAJCpKg\nIAkKkqAgCQqSoCAJCpKg8A83qoXLSguZaQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1085fa400>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# this is just to visualize what the flattened version of the image above looks like;\n",
    "# after flattening the image I replicate it horizontally (by 10 pixels) just for\n",
    "# visualization purposes since otherwise the image is so thin (horizontally) that\n",
    "# it's not easy to see anything\n",
    "plt.imshow(np.hstack([train_images[idx].flatten().reshape((784,1))]*10), cmap='gray')\n",
    "plt.axis('off')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1-layer neural net"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_1 (Dense)              (None, 10)                7850      \n",
      "=================================================================\n",
      "Total params: 7,850\n",
      "Trainable params: 7,850\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "from keras import models\n",
    "from keras import layers\n",
    "\n",
    "shallow_single_layer_model = models.Sequential()  # this is Keras's way of specifying a model that is a single sequence of layers\n",
    "shallow_single_layer_model.add(layers.Dense(10, activation='softmax', input_shape=(784,)))\n",
    "\n",
    "# the line below is not required for constructing the model and just gives a summary of the model\n",
    "print(shallow_single_layer_model.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# the lecture slides explains the loss (which is an error function) that the optimizer will try to minimize;\n",
    "# there are some optimizers that are popularly used for training neural nets such as stochastic gradient\n",
    "# descent (SGD), RMSProp, and Adam (for the purposes of this class, don't worry about what these are)\n",
    "shallow_single_layer_model.compile(optimizer='rmsprop',\n",
    "                                   loss='categorical_crossentropy',\n",
    "                                   metrics=['accuracy'])  # metrics says what accuracy metrics to display when training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The fitting is similar to fitting predictive models in scikit-learn where you specify the feature vectors and their corresponding labels. Here, the twist is that we also specify what fraction of the data to hold out as a validation set (note that this is *not* doing cross-validation!). Also, because training neural nets can be really computationally intensive, usually the optimizers will not look at all the training data at once. Instead it will sequentially go through the training data a number of times (referred to as the number of \"epochs\", which is set to 5 below via parameter `epochs`), looking at a batch of training examples at a time (the number of training examples in a batch is specified as the `batch_size` below)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 48000 samples, validate on 12000 samples\n",
      "Epoch 1/5\n",
      "48000/48000 [==============================] - 2s 36us/step - loss: 0.6790 - acc: 0.8307 - val_loss: 0.3582 - val_acc: 0.9044\n",
      "Epoch 2/5\n",
      "48000/48000 [==============================] - 1s 21us/step - loss: 0.3511 - acc: 0.9032 - val_loss: 0.3073 - val_acc: 0.9137\n",
      "Epoch 3/5\n",
      "48000/48000 [==============================] - 1s 22us/step - loss: 0.3157 - acc: 0.9119 - val_loss: 0.2902 - val_acc: 0.9187\n",
      "Epoch 4/5\n",
      "48000/48000 [==============================] - 1s 19us/step - loss: 0.2993 - acc: 0.9165 - val_loss: 0.2815 - val_acc: 0.9220\n",
      "Epoch 5/5\n",
      "48000/48000 [==============================] - 1s 19us/step - loss: 0.2897 - acc: 0.9187 - val_loss: 0.2795 - val_acc: 0.9237\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x182ae60c88>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "shallow_single_layer_model.fit(flattened_train_images,\n",
    "                               train_labels_categorical,\n",
    "                               validation_split=0.2,\n",
    "                               epochs=5,\n",
    "                               batch_size=128)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember: the validation accuracy (displayed at the end of each of the epoch training lines above right after the text \"val_acc\"; note that the first \"acc\" is the training set accuracy) is meant to be a guess of what the accuracy is on the actual test data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now predict on the test data and display the accuracy on the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 0s 30us/step\n",
      "Test accuracy: 0.9211\n"
     ]
    }
   ],
   "source": [
    "test_loss, test_acc = shallow_single_layer_model.evaluate(flattened_test_images,\n",
    "                                                          test_labels_categorical)\n",
    "print('Test accuracy:', test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2-layer neural net"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code here is almost the same as in the 1-layer case...except now we add 2 layers. Note that we only have to specify the input shape for the very first layer added (in subsequent layers, `keras` can automatically figure out what the input shape should be)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_2 (Dense)              (None, 512)               401920    \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 10)                5130      \n",
      "=================================================================\n",
      "Total params: 407,050\n",
      "Trainable params: 407,050\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "two_layer_model = models.Sequential()  # this is Keras's way of specifying a model that is a single sequence of layers\n",
    "two_layer_model.add(layers.Dense(512, activation='relu', input_shape=(784,)))\n",
    "two_layer_model.add(layers.Dense(10, activation='softmax'))\n",
    "\n",
    "print(two_layer_model.summary())\n",
    "\n",
    "\n",
    "two_layer_model.compile(optimizer='rmsprop',\n",
    "                        loss='categorical_crossentropy',\n",
    "                        metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 48000 samples, validate on 12000 samples\n",
      "Epoch 1/5\n",
      "48000/48000 [==============================] - 7s 136us/step - loss: 0.2835 - acc: 0.9189 - val_loss: 0.1472 - val_acc: 0.9585\n",
      "Epoch 2/5\n",
      "48000/48000 [==============================] - 5s 109us/step - loss: 0.1167 - acc: 0.9654 - val_loss: 0.1118 - val_acc: 0.9655\n",
      "Epoch 3/5\n",
      "48000/48000 [==============================] - 6s 122us/step - loss: 0.0766 - acc: 0.9773 - val_loss: 0.0976 - val_acc: 0.9706\n",
      "Epoch 4/5\n",
      "48000/48000 [==============================] - 6s 116us/step - loss: 0.0559 - acc: 0.9838 - val_loss: 0.0821 - val_acc: 0.9742\n",
      "Epoch 5/5\n",
      "48000/48000 [==============================] - 5s 112us/step - loss: 0.0416 - acc: 0.9875 - val_loss: 0.0826 - val_acc: 0.9758\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x182ab4a5f8>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "two_layer_model.fit(flattened_train_images,\n",
    "                    train_labels_categorical,\n",
    "                    validation_split=0.2,\n",
    "                    epochs=5,\n",
    "                    batch_size=128)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Important: note that the training accuracy (`acc`) keeps increasing but can be an over-estimate of the true test accuracy. In contrast, the validation accuracy (`val_acc`) tends to be a better guess of the true test accuracy, and does not keep growing as we keep training (which happens with the training accuracy  `acc`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we compute the test accuracy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 1s 72us/step\n",
      "Test accuracy: 0.9777\n"
     ]
    }
   ],
   "source": [
    "test_loss, test_acc = two_layer_model.evaluate(flattened_test_images, test_labels_categorical)\n",
    "print('Test accuracy:', test_acc)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}