Adaptive soft threshold and smooth abs 2017-03-21 Mar

0-Adaptive-soft-threshold-smooth-abs.md

Adaptive soft threshold and smooth abs: scale by average |X|

The soft threshold and smooth absolute value functions

are widely used in optimization and signal processing. (Soft thresholding squeezes small values to 0; if "noise" is small and "signal" large, this improves the signal-to-noise ratio. Smooth abs, also called Huber loss, rounds off the sharp corner in |X|, which many optimizers don't like.)

But: the user must set a fixed threshold for his or her problem; the value 3.14 may be ok for one run, terrible for the next. For example, scipy.optimize.least_squares.html says "f_scale is of crucial importance" .

A simple way of making these functions adaptive for vector signals X is to scale the threshold t by the average |X|:

import numpy as np

def softthresh( X, t=1, adapt=True ):
    """ squeeze small values, |X| < t, to 0 """
	absx = np.abs(X)
    if adapt:
        t *= absx.mean()
	return np.sign(X) * np.fmax( absx - t, 0 )

def smoothabs( X, t=1, adapt=True ):
    """ smooth |X|: linear for |X| > t, spline < t """
    absx = np.abs(X)
    if adapt:
        t *= absx.mean()
    ...

Small to big ratio

X is a vector with some components X_i small, some large. Smoothabs makes the small ones smaller but leaves the big ones alone (after abs). If the X_i follow an exponential or Laplace distribution, the ratio of small to big, |X_i| < t to |X_i| > t, is 1 - e^-t small, e^-t big:

    t     < t    > t
    --------------
    0.7   50 %   50 %
    1     63 %   37 %
    2     86 %   14 %

Notes

If signals change in both time and space, one can of course smooth / denoise in time too; there's a huge literature, tl;dr .

This smoothabs uses a cubic spline between [0 0] and the linear part after [t t], which is a bit different from standard Huber loss .

rosenabs test function

The Rosenbrock function is widely used as a test case for optimization methods in 2 .. 1000 variables. It's a sum of squares; changing the squares to absolute values, rosenabs, is tough on optimizers that use gradients / jacobians and want them smooth. With smoothabs instead of abs ... TODO

See also

L1 regularization
Robust regression

scipy.optimize.least_squares.html
Robust nonlinear regression in scipy

Autodiff programs that can be told "use smoothabs for abs, and smoothsign for its derivative" ?

Comments are welcome, test cases most welcome.

cheers
-- denis

Last change: 21 March 2017

adasoft.py

#!/usr/bin/env python2
""" Adaptive soft threshold and smooth abs: scale by average |X| """
# see Soft-threshold-smooth-abs.md under https://gist.github.com/denis-bz

from __future__ import division
import numpy as np

__version__ = "2017-03-21 mar  denis"


def softthresh( X, t=1, adapt=True ):
    """ squeeze small values, |X| < t, to 0 """
    absx = np.abs(X)
    if adapt:
        t *= absx.mean()
    return np.sign(X) * np.fmax( absx - t, 0 )


def smoothabs( X, t=1, adapt=True ):
    """ smooth |X|: linear for |X| > t, spline < t
        X^2 (2 - |X|): 0 1 -> 0 1,  deriv 0 1,  deriv2 4 -5
    """
    absx = np.abs(X)
    if adapt:
        t *= absx.mean()
    absx /= t
    return t * np.where(
        absx < 1,   absx**2 * (2 - absx),   # spline for |X| < t
                    absx )                  # linear for |X| > t

def smoothsign( X, t=1, adapt=True ):
    """ sign(X) where |X| > t, spline in between """
    absx = np.abs(X)
    if adapt:
        t *= absx.mean()
    absx /= t
    return t * np.sign(X) * np.where(
        absx < 1, absx * (1 + absx - absx**2),
                  1 )