Levenberg-Marquardt.md

What is it?

• an algorithm to solve non-linear least squares minimization problems
• finds a local minima, not necessarily the global minima
• alternatives: Gauss–Newton, gradient-descent

Why use it?

• LM algorithm combines the advantages of gradient-descent and Gauss-Newton
• allows for initial values to be further away from solution than Gauss-Newton
• gradient-descent dominates until a canyon is reached, after which Gauss-Newton takes over

Mathematical representation

• set of m data points, (x₁, y₁), (x₂, y₂), ..., (x_m, y_m)
• curve function y = f (x, b)
• n parameters, b = (b₁, b₂, ..., b_n)
• residuals (errors) r_i, for i from 1 to m, are given by r_i = y_i - f (x_i, b)
• find b such that the sum of the sum of the squared residuals is minimized, S = sum_i=1..m ( r_i² )

Sources

• http://mads.lanl.gov/presentations/Leif_LM_presentation_m.pdf
• https://en.wikipedia.org/wiki/Non-linear_least_squares
• https://en.wikipedia.org/wiki/Levenberg%E2%80%93Marquardt_algorithm