A3: logaddexp

A3-logaddexp.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`np.logaddexp`:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\log \\mathcal{L}(\\theta \\mid x, y) = \\log \\prod_{i=1}^{n}{\\Big[g_i P(y_i \\mid a, b, x, e) + (1-g_i) P(y_i \\mid a, b, x, \\sigma)\\Big]}=\\\\\n",
    "=\\sum_{i=1}^{n}{\\log{\\Big[g_i P(y_i \\mid a, b, x, e) + (1-g_i) P(y_i \\mid a, b, x, \\sigma)\\Big]}}\\\\\n",
    "=\\sum_{i=1}^{n}{\\log{\\Big[L1 + L2\\Big]}}\n",
    "$$\n",
    "$L1,L2$ can be very small, and can exceed the range of normal floating point numbers. If we want precise calculation here, we do the trick:\n",
    "\n",
    "$$\n",
    "\\sum_{i=1}^{n}{\\log{\\Big[L1 + L2\\Big]}}=\\sum_{i=1}^{n}{\\log{\\Big[\\exp(\\log(L1)) + exp(\\log(L2))\\Big]}}\\\\\\\\\n",
    "=np.logaddexp(logL1, logL2).sum()\n",
    "$$\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:scipy]",
   "language": "python",
   "name": "conda-env-scipy-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}

np.logaddexp calculates the expression effectively, without explicitly computing the intermediate exp values. If the power of exponential is high, exp value could exceed the integer range (with result inf), np.logaddexp solves this problem