Intro to gradient descent, and why feature scaling leads to better convergence

gistfile1.txt

ml gives computers ability to learn without being explicitly programmed.

# Linear Regression
linear regression is a simple model to find best for for data.
http://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Linear_regression.svg/400px-Linear_regression.svg.png
view above x and y axis as features, generalize to n-dimensional.

# Example
You have a data set with commute and sleep time, salary and happiness (1-10)
for the three features, commute, sleep and salary you want to predict happy

# Cost function and finding a good fit
How well does this line predict the data? concept of a cost function, once it converges we know we have a good fit. How find it? gradient descent.

# Contour plot
Here is a contour plot, like hills in a landscape:
http://upload.wikimedia.org/wikipedia/commons/f/fa/Cntr-map-1.jpg

# Gradient/Steepest descent
imagine we are walking up a hill by taking the steepest path. Thats gradient (ascent) descent, but in n-dimensions of course.

# Proportionality and feature scaling
Commute time on scale of 10m-2h, salary in 5-6 figure salary, obv not proportional.
A way of normalizing our values. Using  x' = (x - min) / (max - min).
x=15 with min=0, max=120 gives x' = (15 - 0) / (120 - 0) = 15/120 = 0.125

# Convergence speed
Feature scaling helps with convergence speed, without proportionality we get uneven descent.
good: http://upload.wikimedia.org/wikipedia/commons/thumb/7/79/Gradient_descent.png/350px-Gradient_descent.png
bad: http://komarix.org/ac/papers/thesis/thesis_html/img30.png
3d: http://upload.wikimedia.org/wikipedia/commons/6/68/Gradient_ascent_%28surface%29.png