Perceptrons.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#The Perceptron "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##<font color='grey'>Learning the NAND function</font> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook is a walkthrough of the simple example of perceptrons found on this Wikipedia page:\n",
    "\n",
    "https://en.wikipedia.org/wiki/Perceptron"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the Wikipedia article already does a great job of explaining the background of perceptrons (as well as the full setup of the problem), I'm going to skip that here, and launch straight into the code.\n",
    "\n",
    "__If you're not sure what's going on or you just want more information, visit the link above!__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The example even gives some python code. Below I'm going to explain what exactly is happening at each step (with a couple super minor changes I think will improve clairity). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "threshold = 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The threshold is a constant. It's the stimulus a perceptron must percieve in order to \"fire\". If a perceptron reaches the threshold and fires, it outputs a 1; otherwise, it outputs a 0. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "learning_rate = 0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the speed at which a perceptron makes changes to its weight (which I'll explain in a second) when its output is different than expected. Personally, I think the name \"learning_rate\" is a little misleading. Why not set the learning rate as high as possible?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you set the learning rate too high, you're going to overcorrect. Just like when you're driving your car, if you have a large, jerk reaction to a mistake, you'll suddenly find yourself in the other lane or in a ditch. What you want is just enough correction to get you back in the center of your lane. If the learning rate is too high, you'll oscillate back and forth around where you're trying to be. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "weights = [0,0,0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A perceptron is given a set of weights. In a way, weights are what define the perceptron and make it unique. Weights are also where the \"learning\" occurs, since the values of the weights are going to be updated over the course of the algorithm.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "training_set = [[1,0,0],[1,0,1],[1,1,0],[1,1,1]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the raw data that we are going to feed into our perceptrons. We'll feed it it row by row. Each time, the same perceptron will get the same column entry of the data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "desired_output_array = [1,1,1,0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I've extracted out the desired_output into it's own variable from the beginning, instead of keeping it wrapped in the training set list. \n",
    "\n",
    "Anyway, the desired output is what the inputs (training_set[1], training_set[2]) would give us if we fed it to a NAND gate, and what we hope our mini neural net will output for each of the rows. \n",
    "\n",
    "So, for example, we hope that when we feed our perceptrons the data [1, 0, 0] that it gives us a 1, but when we feed it [1, 1, 1] it gives 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def dot_product(values, weights):\n",
    "    return sum(value * weight for value, weight in zip(values, weights))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function just defines a standard mathmatical function:\n",
    "\n",
    "https://en.wikipedia.org/wiki/Dot_product\n",
    "\n",
    "You could also employ something like numpy.dot(), but one of the things that I really like about this piece of code is that it only uses the standard Python library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "perfect = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "perfect is going to act as a flag for our loop. We'll continue to update our weights until we get all of our outputs correct."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The rest of the code is wrapped in a loop, so I'm going to heavily comment between each line, print a lot of stuff, then rewrite the code a second time so we can see the final product."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------------------------------------\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0, 0, 0]\n",
      "\n",
      "Input: [1, 0, 0] should output: 1\n",
      "Actual result: 0,\n",
      "means an error of 1\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.1, 0.0, 0.0]\n",
      "\n",
      "Input: [1, 0, 1] should output: 1\n",
      "Actual result: 0,\n",
      "means an error of 1\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.2, 0.0, 0.1]\n",
      "\n",
      "Input: [1, 1, 0] should output: 1\n",
      "Actual result: 0,\n",
      "means an error of 1\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.30000000000000004, 0.1, 0.1]\n",
      "\n",
      "Input: [1, 1, 1] should output: 0\n",
      "Actual result: 0,\n",
      "means an error of 0\n",
      "------------------------------------------------------------\n",
      "------------------------------------------------------------\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.30000000000000004, 0.1, 0.1]\n",
      "\n",
      "Input: [1, 0, 0] should output: 1\n",
      "Actual result: 0,\n",
      "means an error of 1\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.4, 0.1, 0.1]\n",
      "\n",
      "Input: [1, 0, 1] should output: 1\n",
      "Actual result: 0,\n",
      "means an error of 1\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.5, 0.1, 0.2]\n",
      "\n",
      "Input: [1, 1, 0] should output: 1\n",
      "Actual result: 1,\n",
      "means an error of 0\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.5, 0.1, 0.2]\n",
      "\n",
      "Input: [1, 1, 1] should output: 0\n",
      "Actual result: 1,\n",
      "means an error of -1\n",
      "------------------------------------------------------------\n",
      "------------------------------------------------------------\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.4, 0.0, 0.1]\n",
      "\n",
      "Input: [1, 0, 0] should output: 1\n",
      "Actual result: 0,\n",
      "means an error of 1\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.5, 0.0, 0.1]\n",
      "\n",
      "Input: [1, 0, 1] should output: 1\n",
      "Actual result: 1,\n",
      "means an error of 0\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.5, 0.0, 0.1]\n",
      "\n",
      "Input: [1, 1, 0] should output: 1\n",
      "Actual result: 0,\n",
      "means an error of 1\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.6, 0.1, 0.1]\n",
      "\n",
      "Input: [1, 1, 1] should output: 0\n",
      "Actual result: 1,\n",
      "means an error of -1\n",
      "------------------------------------------------------------\n",
      "------------------------------------------------------------\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.5, 0.0, 0.0]\n",
      "\n",
      "Input: [1, 0, 0] should output: 1\n",
      "Actual result: 0,\n",
      "means an error of 1\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.6, 0.0, 0.0]\n",
      "\n",
      "Input: [1, 0, 1] should output: 1\n",
      "Actual result: 1,\n",
      "means an error of 0\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.6, 0.0, 0.0]\n",
      "\n",
      "Input: [1, 1, 0] should output: 1\n",
      "Actual result: 1,\n",
      "means an error of 0\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.6, 0.0, 0.0]\n",
      "\n",
      "Input: [1, 1, 1] should output: 0\n",
      "Actual result: 1,\n",
      "means an error of -1\n",
      "------------------------------------------------------------\n",
      "------------------------------------------------------------\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.5, -0.1, -0.1]\n",
      "\n",
      "Input: [1, 0, 0] should output: 1\n",
      "Actual result: 0,\n",
      "means an error of 1\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.6, -0.1, -0.1]\n",
      "\n",
      "Input: [1, 0, 1] should output: 1\n",
      "Actual result: 0,\n",
      "means an error of 1\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.7, -0.1, 0.0]\n",
      "\n",
      "Input: [1, 1, 0] should output: 1\n",
      "Actual result: 1,\n",
      "means an error of 0\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.7, -0.1, 0.0]\n",
      "\n",
      "Input: [1, 1, 1] should output: 0\n",
      "Actual result: 1,\n",
      "means an error of -1\n",
      "------------------------------------------------------------\n",
      "------------------------------------------------------------\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.6, -0.2, -0.1]\n",
      "\n",
      "Input: [1, 0, 0] should output: 1\n",
      "Actual result: 1,\n",
      "means an error of 0\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.6, -0.2, -0.1]\n",
      "\n",
      "Input: [1, 0, 1] should output: 1\n",
      "Actual result: 0,\n",
      "means an error of 1\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.7, -0.2, 0.0]\n",
      "\n",
      "Input: [1, 1, 0] should output: 1\n",
      "Actual result: 0,\n",
      "means an error of 1\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.7999999999999999, -0.1, 0.0]\n",
      "\n",
      "Input: [1, 1, 1] should output: 0\n",
      "Actual result: 1,\n",
      "means an error of -1\n",
      "------------------------------------------------------------\n",
      "------------------------------------------------------------\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.7, -0.2, -0.1]\n",
      "\n",
      "Input: [1, 0, 0] should output: 1\n",
      "Actual result: 1,\n",
      "means an error of 0\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.7, -0.2, -0.1]\n",
      "\n",
      "Input: [1, 0, 1] should output: 1\n",
      "Actual result: 1,\n",
      "means an error of 0\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.7, -0.2, -0.1]\n",
      "\n",
      "Input: [1, 1, 0] should output: 1\n",
      "Actual result: 0,\n",
      "means an error of 1\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.7999999999999999, -0.1, -0.1]\n",
      "\n",
      "Input: [1, 1, 1] should output: 0\n",
      "Actual result: 1,\n",
      "means an error of -1\n",
      "------------------------------------------------------------\n",
      "------------------------------------------------------------\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.7, -0.2, -0.2]\n",
      "\n",
      "Input: [1, 0, 0] should output: 1\n",
      "Actual result: 1,\n",
      "means an error of 0\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.7, -0.2, -0.2]\n",
      "\n",
      "Input: [1, 0, 1] should output: 1\n",
      "Actual result: 0,\n",
      "means an error of 1\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.7999999999999999, -0.2, -0.1]\n",
      "\n",
      "Input: [1, 1, 0] should output: 1\n",
      "Actual result: 1,\n",
      "means an error of 0\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.7999999999999999, -0.2, -0.1]\n",
      "\n",
      "Input: [1, 1, 1] should output: 0\n",
      "Actual result: 0,\n",
      "means an error of 0\n",
      "------------------------------------------------------------\n",
      "------------------------------------------------------------\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.7999999999999999, -0.2, -0.1]\n",
      "\n",
      "Input: [1, 0, 0] should output: 1\n",
      "Actual result: 1,\n",
      "means an error of 0\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.7999999999999999, -0.2, -0.1]\n",
      "\n",
      "Input: [1, 0, 1] should output: 1\n",
      "Actual result: 1,\n",
      "means an error of 0\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.7999999999999999, -0.2, -0.1]\n",
      "\n",
      "Input: [1, 1, 0] should output: 1\n",
      "Actual result: 1,\n",
      "means an error of 0\n",
      "------------------------------------------------------------\n",
      "Current weights\n",
      "[0.7999999999999999, -0.2, -0.1]\n",
      "\n",
      "Input: [1, 1, 1] should output: 0\n",
      "Actual result: 0,\n",
      "means an error of 0\n",
      "------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "#Continue until we get the outputs right\n",
    "while not perfect:\n",
    "    \n",
    "    #Start new block of output\n",
    "    print \"-\"*60\n",
    "    print \"-\"*60\n",
    "    \n",
    "    #This will keep track of how many of the 4 desired_outputs we get wrong\n",
    "    #Eventually, we won't get any incorrect, and we'll finish the loop\n",
    "    error_count = 0\n",
    "    \n",
    "    #Feed the perceptrons each of the 4 possible combination of input features\n",
    "    #Also keep track of what value you expect to get for each set of inputs\n",
    "    for input_vector, desired_output in zip(training_set, desired_output_array):\n",
    "                \n",
    "        #The weights are the most important variable to keep track of\n",
    "        #Watch how they change over iterations of the loop\n",
    "        print \"Current weights\"\n",
    "        print weights\n",
    "        print \"\"\n",
    "        print \"Input: {} should output: {}\".format(input_vector, desired_output)\n",
    "        \n",
    "        #The result will be a 1 if the perceptron \"fires\".\n",
    "        #It will fire if the sum of each of the inputs times the weights\n",
    "        #is greater than the threshold we set\n",
    "        result = dot_product(input_vector, weights) > threshold\n",
    "        \n",
    "        #This simple error calculation will always be 1, 0, or -1.\n",
    "        error = desired_output - result\n",
    "        print \"Actual result: {},\".format(int(result)) \n",
    "        print \"means an error of {}\".format(error)\n",
    "        \n",
    "        #If we didn't get the result we expected, increment the error counter\n",
    "        if error != 0:\n",
    "            error_count += 1\n",
    "            \n",
    "            #Since our output was incorrect, change each of the weights\n",
    "            #Again, this is where the learning occurs\n",
    "            for index, value in enumerate(input_vector):\n",
    "                weights[index] += learning_rate * error * value\n",
    "        print \"-\"*60\n",
    "    \n",
    "    #If all 4 of our results matched our desired outputs,\n",
    "    #we have the weights we want and can quit the loop\n",
    "    if error_count == 0:\n",
    "        perfect = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There we go! As we can see here, as well as the Wikipedia page, if we set the weights to weights[0] = .8, weights[1] = -.2, and weights[2] = -.1, we get the output we would expect for all four cases of input. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's all of the code collected into one block. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------------------------------------\n",
      "[0, 0, 0]\n",
      "[0.1, 0.0, 0.0]\n",
      "[0.2, 0.0, 0.1]\n",
      "[0.30000000000000004, 0.1, 0.1]\n",
      "------------------------------------------------------------\n",
      "[0.30000000000000004, 0.1, 0.1]\n",
      "[0.4, 0.1, 0.1]\n",
      "[0.5, 0.1, 0.2]\n",
      "[0.5, 0.1, 0.2]\n",
      "------------------------------------------------------------\n",
      "[0.4, 0.0, 0.1]\n",
      "[0.5, 0.0, 0.1]\n",
      "[0.5, 0.0, 0.1]\n",
      "[0.6, 0.1, 0.1]\n",
      "------------------------------------------------------------\n",
      "[0.5, 0.0, 0.0]\n",
      "[0.6, 0.0, 0.0]\n",
      "[0.6, 0.0, 0.0]\n",
      "[0.6, 0.0, 0.0]\n",
      "------------------------------------------------------------\n",
      "[0.5, -0.1, -0.1]\n",
      "[0.6, -0.1, -0.1]\n",
      "[0.7, -0.1, 0.0]\n",
      "[0.7, -0.1, 0.0]\n",
      "------------------------------------------------------------\n",
      "[0.6, -0.2, -0.1]\n",
      "[0.6, -0.2, -0.1]\n",
      "[0.7, -0.2, 0.0]\n",
      "[0.7999999999999999, -0.1, 0.0]\n",
      "------------------------------------------------------------\n",
      "[0.7, -0.2, -0.1]\n",
      "[0.7, -0.2, -0.1]\n",
      "[0.7, -0.2, -0.1]\n",
      "[0.7999999999999999, -0.1, -0.1]\n",
      "------------------------------------------------------------\n",
      "[0.7, -0.2, -0.2]\n",
      "[0.7, -0.2, -0.2]\n",
      "[0.7999999999999999, -0.2, -0.1]\n",
      "[0.7999999999999999, -0.2, -0.1]\n",
      "------------------------------------------------------------\n",
      "[0.7999999999999999, -0.2, -0.1]\n",
      "[0.7999999999999999, -0.2, -0.1]\n",
      "[0.7999999999999999, -0.2, -0.1]\n",
      "[0.7999999999999999, -0.2, -0.1]\n"
     ]
    }
   ],
   "source": [
    "threshold = 0.5\n",
    "learning_rate = 0.1\n",
    "weights = [0,0,0]\n",
    "training_set = [[1,0,0],[1,0,1],[1,1,0],[1,1,1]]\n",
    "desired_output_array = [1,1,1,0]\n",
    "perfect = False\n",
    "\n",
    "def dot_product(values, weights):\n",
    "    return sum(value * weight for value, weight in zip(values, weights))\n",
    "\n",
    "while not perfect:\n",
    "    print \"-\"*60\n",
    "    error_count = 0\n",
    "    for input_vector, desired_output in zip(training_set, desired_output_array):\n",
    "        print weights\n",
    "        result = dot_product(input_vector, weights) > threshold\n",
    "        error = desired_output - result\n",
    "        if error != 0:\n",
    "            error_count += 1\n",
    "            for index, value in enumerate(input_vector):\n",
    "                weights[index] += learning_rate * error * value\n",
    "    if error_count == 0:\n",
    "        perfect = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a pretty awesome algorithm. We've just taught a perceptron to act like a NAND gate, even though it has no knowledge of what a NAND gate is. \n",
    "\n",
    "Try running this code yourself! Use different input data. Tweak the learning rate and see how quickly you can get perfect classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}