Nonlinear least squares example. Notice the "normal equations" on line 10. Three things are different than the linear case: 1) The X matrix is now the derivative of the mean function with respect to beta 2) We're solving for the change in beta from the last iteration 3) We have to iterate See http://en.wikipedia.org/wiki/Non-linear_least_squares …

nonlinear_least_squares.R

N <- 10000
x <- rnorm(N)
y <- exp(-.75 + 1.2*x) + rnorm(N)

alpha <- -.5 
beta <- 1 
for ( i in 1:10) { # 10 iterations, for no particular reason
  X <- cbind(exp(alpha + beta*x), x*exp(alpha + beta*x))
  pred <- exp(alpha + beta*x) 
  diff.soln <- solve(t(X) %*% X) %*% t(X) %*% (y - pred) 
  
  alpha <- alpha + diff.soln[1]
  beta <- beta + diff.soln[2]
  cat("iteration: ", i, ", alpha: ", alpha, ", beta: ", beta, "\n")
}