Finite difference approach to calculating the Hessian

hessian.py

#!/usr/bin/env python
"""
Some Hessian codes
"""
import numpy as np
from scipy.optimize import approx_fprime


def hessian ( x0, epsilon=1.e-5, linear_approx=False, *args ):
    """
    A numerical approximation to the Hessian matrix of cost function at
    location x0 (hopefully, the minimum)
    """
    # ``calculate_cost_function`` is the cost function implementation
    # The next line calculates an approximation to the first
    # derivative
    f1 = approx_fprime( x0, calculate_cost_function, *args) 

    # This is a linear approximation. Obviously much more efficient
    # if cost function is linear
    if linear_approx:
        f1 = np.matrix(f1)
        return f1.transpose() * f1    
    # Allocate space for the hessian
    n = x0.shape[0]
    hessian = np.zeros ( ( n, n ) )
    # The next loop fill in the matrix
    xx = x0
    for j in xrange( n ):
        xx0 = xx[j] # Store old value
        xx[j] = xx0 + epsilon # Perturb with finite difference
        # Recalculate the partial derivatives for this new point
        f2 = approx_fprime( x0, calculate_cost_function, *args) 
        hessian[:, j] = (f2 - f1)/epsilon # scale...
        xx[j] = xx0 # Restore initial value of x0        
    return hessian
   

Thank you so much for this code, is saved so much of my time! Just wanted to say thank you for uploading it!

I tried your version of the hessian and it gives me some bad results if I compare it to the hessian returned by the package numdifftools. I think that you should have a look and correct your file. But thanks for sharing some code!

I tried your version of the hessian and it gives me some bad results if I compare it to the hessian returned by the package numdifftools. I think that you should have a look and correct your file. But thanks for sharing some code!

Hi were you able to get the hessian matrix for a dataset using the numdifftools

Hi there. I second @glederrey. There are huge discrepancies between this result and that of numdifftools.