Introduction to Neural Networks

introduction-to-neural-networks.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Introduction to Neural Networks",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/gist/JonasSchroeder/500d85590ea172cb56c0c0c11a8cce1e/introduction-to-neural-networks.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lm1ZGIIRD52-",
        "colab_type": "text"
      },
      "source": [
        "# Introduction to Neural Networks\n",
        "\n",
        "1. Neurons\n",
        "2. Layers\n",
        "3. Different types of activation functions\n",
        "4. Example\n",
        "5. Training neural networks\n",
        "\n",
        "# 1. Neurons\n",
        "\n",
        "A neural network is not a fixed program but a model and system that processes information (inputs). Characteristics of a neural network are:\n",
        "\n",
        "- Information processing occurs over simple elements called **neurons**\n",
        "- Neurons are connected and they exchange signals between them through connection links\n",
        "- **Connection links** can be stronger or weaker, which determines how information is processed\n",
        "- Each neuron has an internal state that is determined by all the incoming connections from other neurons\n",
        "- Each neuron has a different **activation function** that determines its output signal\n",
        "- A neural network can be described as a computational graph of mathematical operations.\n",
        "\n",
        "A neural network has an architecture which descibes the set of connections between neurons like *feed forward, recurrent, multi- or single-layered, ...*. A neural network is most commonly trained with gradient descent and backpropagation.\n",
        "\n",
        "**Gradient Descent:** \n",
        "Gradient Descent is an optimization algorithm that is used to minimize the loss function through updating the loss function by moving in the direction of steepest descent as defined by the negative of the gradient. Example: Walking down from the mountain top to find the lowest point by always taking a step in the direction with the steepest slope. \n",
        "The learning rate defines the size of the steps.\n",
        "\n",
        "**Backpropagation:**\n",
        "Adjust each weight in the network in proportion to how much it contributes to overall error.\n",
        "\n",
        "A **Neuron** is a mathematical function that takes one or more input values and ouputs one single numerical value. **Activation value** is the weighted sum of inputs xi and the weights wi. Activation value plus Bias b are the input of the **activation function** which determines the output of the neuron. The *Bias* allows the hyperplane to shift away from the center of the coordinate system.\n",
        "\n",
        "In a one-dimensional input space, the neuron defines a line. Activation function f(x)=x, neuron output y=wx+b\n",
        "\n",
        "# 2. Layers\n",
        "\n",
        "The **input layer** represents the dataset and the initial condition and is usually not counted as part of the other layers. Thus, a *1-layer net* consists of an input layer, a simple network with just one layer, and the output, which can consist of multiple neurons as well.\n",
        "\n",
        "Neurons of one layer can be connected to the neurons of other layers, but not to the other neurons of the same layer.\n",
        "\n",
        "Since neurons can only process one value, combinding them in layers allows to process a vector with multiple values at once.\n",
        "\n",
        "A **multi-layer neural network** consist of an input layer, an output layer, and *hidden layers* between them.\n",
        "\n",
        "# 3. Different types of activation fucntions\n",
        "\n",
        "\n",
        "\n",
        "# 4. Example\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OUg-EBtvD1vz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}