Stanford Online/ DeepLearning.AI. Supervised Machine Learaning: Regression and Classification, Cost Function J (w,b).

C1_W1_Lab03_Cost_function_Soln.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optional  Lab: Cost Function \n",
    "<figure>\n",
    "    <center> <img src=\"./images/C1_W1_L3_S2_Lecture_b.png\"  style=\"width:1000px;height:200px;\" ></center>\n",
    "</figure>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Goals\n",
    "In this lab you will:\n",
    "- you will implement and explore the `cost` function for linear regression with one variable. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tools\n",
    "In this lab we will make use of: \n",
    "- NumPy, a popular library for scientific computing\n",
    "- Matplotlib, a popular library for plotting data\n",
    "- local plotting routines in the 'lab_utils_uni.py' file in the local directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np                # Get numpy library as 'np' constructor\n",
    "\n",
    "# matplotlib backend 'widget' used for loading interactive plots\n",
    "%matplotlib widget                \n",
    "\n",
    "import matplotlib.pyplot as plt   # Get matplotlib, pyplot library as 'plt' constructor\n",
    "\n",
    "# 'lab_utils_uni.py' file is loaded to use some functions when you plot. \n",
    "from lab_utils_uni import plt_intuition, plt_stationary, plt_update_onclick, soup_bowl \n",
    "    \n",
    "plt.style.use('./deeplearning.mplstyle')   # Use or import 'available style sheets', using its 'name',\n",
    "                                           # on a common set of example plots: scatter plot, image,\n",
    "                                           # bar graph, patches, line plot and histogram."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem Statement\n",
    "\n",
    "You would like a model which can predict housing prices given the size of the house.  \n",
    "Let's use the same two data points as before the previous lab- a house with 1000 square feet sold for \\\\$300,000 and a house with 2000 square feet sold for \\\\$500,000.\n",
    "\n",
    "\n",
    "| Size (1000 sqft)     | Price (1000s of dollars) |\n",
    "| -------------------| ------------------------ |\n",
    "| 1                 | 300                      |\n",
    "| 2                  | 500                      |\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Shorter data set of training points\n",
    "x_train = np.array([1.0, 2.0])           # x_train rows/samples (size in 1000 square feet) 1D numpy array -> [1.000, 2.000]\n",
    "y_train = np.array([300.0, 500.0])       # y_train rows/samples (price in 1000s of dollars) 1D numpy array -> [300.000, 500.000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computing Cost\n",
    "The term 'cost' in this assignment might be a little confusing since the data is housing cost. Here, cost is a measure how well our model is predicting the target price of the house. The term 'price' is used for housing data.\n",
    "\n",
    "The equation for cost with one variable is:\n",
    "  $$J(w,b) = \\frac{1}{2m} \\sum\\limits_{i = 0}^{m-1} (f_{w,b}(x^{(i)}) - y^{(i)})^2 \\tag{1}$$ \n",
    " \n",
    "where \n",
    "  $$f_{w,b}(x^{(i)}) = wx^{(i)} + b \\tag{2}$$\n",
    "  \n",
    "- $f_{w,b}(x^{(i)})$ is our prediction for example $i$ using parameters $w,b$.  \n",
    "- $(f_{w,b}(x^{(i)}) -y^{(i)})^2$ is the squared difference between the target value and the prediction.   \n",
    "- These differences are summed over all the $m$ examples and divided by `2m` to produce the cost, $J(w,b)$.  \n",
    ">Note, in lecture summation ranges are typically from 1 to m, while code will be from 0 to m-1.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below calculates cost by looping over each example. In each loop:\n",
    "- `f_wb`, a prediction is calculated\n",
    "- the difference between the target and the prediction is calculated and squared.\n",
    "- this is added to the total cost."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_cost(x, y, w, b):      # Function inputs are (x_train, y_train, SLOPE, INTERCEPT)\n",
    "    \"\"\"\n",
    "    Computes the cost function for linear regression.\n",
    "    \n",
    "    Args:\n",
    "      x (ndarray (m,)): Data, m examples \n",
    "      y (ndarray (m,)): target values\n",
    "      w,b (scalar)    : model parameters. w (slope), b (intercept)  \n",
    "    \n",
    "    Returns\n",
    "        total_cost (float): The cost of using w,b as the parameters for linear regression model\n",
    "               to fit the data points in x and y (x_train, y_train)\n",
    "    \"\"\"\n",
    "    # number of training examples\n",
    "    m = x.shape[0]                 # Select the number of samples/elements at 'x' 1D numpy array, and assign to (m = 2)\n",
    "    \n",
    "    cost_sum = 0                   # Initialize the first Cost function value/counter to be added, as '0'.\n",
    "    for i in range(m):             # Because m= 2, iterate through i = 0,1 times\n",
    "        f_wb = w * x[i] + b        # Linear Regression Model Equation: y^ = f_wb(x[i]) = w*x[i] + b\n",
    "        cost = (f_wb - y[i]) ** 2  # Get each error^2 as actual_cost = (Predicted value - True value)^2 = (y^[i] - y[i])^2 \n",
    "        cost_sum = cost_sum + cost            # new_cost = prev_cost + actual_cost\n",
    "                                              #           (init as 0)                 \n",
    "    total_cost = (1 / (2 * m)) * cost_sum     # total_cost = average_cost = J(w,b) = (1/2m) * new_cost  \n",
    "\n",
    "    return total_cost        # total_cost = average_cost = J(w,b) = (1/2m) * SUM i=0 -> m-1 (y^[i] - y[i])^2  is returned"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cost Function Intuition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"left\" src=\"./images/C1_W1_Lab02_GoalOfRegression.PNG\"    style=\" width:380px; padding: 10px;  \" /> Your goal is to find a model $f_{w,b}(x) = wx + b$, with parameters $w,b$,  which will accurately predict house values given an input $x$. The cost is a measure of how accurate the model is on the training data.\n",
    "\n",
    "The cost equation (1) above shows that if $w$ and $b$ can be selected such that the predictions $f_{w,b}(x)$ match the target data $y$, the $(f_{w,b}(x^{(i)}) - y^{(i)})^2 $ term will be zero and the cost minimized. In this simple two point example, you can achieve this!\n",
    "\n",
    "In the previous lab, you determined that $b=100$ provided an optimal solution so let's set $b$ to 100 and focus on $w$.\n",
    "\n",
    "<br/>\n",
    "Below, use the slider control to select the value of $w$ that minimizes cost. It can take a few seconds for the plot to update."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e7b6df177cc54803951d3382e9912cb2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(IntSlider(value=150, description='w', max=400, step=10), Output()), _dom_classes=('widge…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt_intuition(x_train,y_train) # Use ‘plt_intuition()’ function to get a graph with the Linear Regression model on the \n",
    "                               # left, and the cost function graph on the right for a fixed value of b=100, so we\n",
    "                               # want to find the w parameter value, which give us the minimum value for the Cost J(w,b) = 0."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The plot contains a few points that are worth mentioning.\n",
    "- cost is minimized when $w = 200$, which matches results from the previous lab\n",
    "- Because the difference between the target and pediction is squared in the cost equation, the cost increases rapidly when $w$ is either too large or too small.\n",
    "- Using the `w` and `b` selected by minimizing cost results in a line which is a perfect fit to the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cost Function Visualization- 3D\n",
    "\n",
    "You can see how cost varies with respect to *both* `w` and `b` by plotting in 3D or using a contour plot.   \n",
    "It is worth noting that some of the plotting in this course can become quite involved. The plotting routines are provided and while it can be instructive to read through the code to become familiar with the methods, it is not needed to complete the course successfully. The routines are in lab_utils_uni.py in the local directory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Larger Data Set\n",
    "It is instructive to view a scenario with a few more data points. This data set includes data points that do not fall on the same line. What does that mean for the cost equation? Can we find $w$, and $b$ that will give us a cost of 0? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Larger Data set of training points\n",
    "# x_train rows/samples (size in 1000 square feet) 1D numpy array -> [1.000, 1.700,2.000, 2.500, 3.000, 3.200] \n",
    "x_train = np.array([1.0, 1.7, 2.0, 2.5, 3.0, 3.2])\n",
    "\n",
    "# y_train rows/samples (price in 1000s of dollars) 1D numpy array -> [250.000, 300.000, 480.000, 430.000, 630.000, 730.000] \n",
    "y_train = np.array([250, 300, 480, 430, 630, 730])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the contour plot, click on a point to select `w` and `b` to achieve the lowest cost. Use the contours to guide your selections. Note, it can take a few seconds to update the graph. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8862d70005554192bab2a7e7ab492c4a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.close('all')    # It closes all the figure windows \n",
    "\n",
    "fig, ax, dyn_items = plt_stationary(x_train, y_train)  # Call function 'plt_stationary()' with training set (x_train, y_train)\n",
    "                                                       # as inputs, and return a Figure object 'fig', the figure axes 'ax' \n",
    "                                                       # and an array including some objects for plotting \n",
    "                                                       # dyn_items = [scatter points obj, \n",
    "                                                       #              horizontal contour dotted lines obj, \n",
    "                                                       #              model vertical dotted lines obj] \n",
    "\n",
    "updater = plt_update_onclick(fig, ax, x_train, y_train, dyn_items) # Update the graph, when click on a point \n",
    "                                                                   # for selecting w (SLOPE) and b (INTERCEPT) parameters \n",
    "                                                                   # to achieve the lowest Cost J (w,b) value. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above, note the dashed lines in the left plot. These represent the portion of the cost contributed by each example in your training set. In this case, values of approximately $w=209$ and $b=2.4$ provide low cost. Note that, because our training examples are not on a line, the minimum cost is not zero."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convex Cost surface\n",
    "The fact that the cost function squares the loss ensures that the 'error surface' is convex like a soup bowl. It will always have a minimum that can be reached by following the gradient in all dimensions. In the previous plot, because the $w$ and $b$ dimensions scale differently, this is not easy to recognize. The following plot, where $w$ and $b$ are symmetric, was shown in lecture:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bac3763c28e94c69a499f4fdf01cc3ec",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "soup_bowl() # Plot the Cost J (w,b) function as 3D (Bowl Shape) using the function 'soup_bowl()'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Congratulations!\n",
    "You have learned the following:\n",
    " - The cost equation provides a measure of how well your predictions match your training data.\n",
    " - Minimizing the cost can provide optimal values of $w$, $b$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "toc-autonumbering": false
 },
 "nbformat": 4,
 "nbformat_minor": 5
}