focal_loss.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Background: Accuracy isn't a good metric\n",
    "\n",
    "Before we can make a decision, we have to make a prediction. By scoring our models with accuracy, we are combining both steps and making it impossible to evaluate our models' predictions.\n",
    "\n",
    "This doesn't matter so much when classes are distinct, when it is possible to draw relatively simple borders separating each class in Euclidean space. But when classes overlap, it becomes more clear that what we really want to evaluate are our models' probability estimates."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple illustration of the problem with accuracy\n",
    "\n",
    "Consider a data set with two classes, -1 and 1. For simplicity, suppose there is a subset S of our data set, every member of which has identical features. Suppose that 60% of S belongs to class -1 and 40% belongs to class 1.\n",
    "\n",
    "Estimate 1 judges, for each member of S, that it has a 60% chance of belonging to -1 and a 40% chance of belonging to class 1. If we judge by accuracy, estimate 1 scores 0.6.\n",
    "\n",
    "Estimate 2 judges, for each member of S, that it has a 99.999% chance of belonging to -1 and a 0.001% chance of belonging to class 1. Estimate 2 is horrible, but if we judge by accuracy, estimate 2 also scores 0.6.\n",
    "\n",
    "If it's not clear why this matters, imagine that class -1 is \"the tumor is benign\" and class 1 is \"the tumor is malignant.\" We very likely want to take further action if there is a 40% chance the tumor is malignant, but estimate 2 leads us astray.\n",
    "\n",
    "Similarly, imagine that class -1 is \"the object in front of the car is not a pedestrian.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Focal loss has the same problem as accuracy\n",
    "\n",
    "Namely, it's not a *proper scoring rule*: it's not optimized by the best probability estimates. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting for the following examples\n",
    "\n",
    "In the following examples, again assume there is a subset S of our data set with identical features. Suppose further that for the `n` classes in S, there appear with proportions `p = [p[0], ..., p[n-1]]`, and we will call estimated class probabilities in S: `q = [q[0], ..., q[n-1]]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def cross_entropy(p, q):\n",
    "    return sum(-p[i] * np.log(q[i]) for i in range(len(q)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def focal(gamma, p, q):\n",
    "    return sum(-p[i] * (1-q[i])**gamma * np.log(q[i]) for i in range(len(q)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def accuracy(p, q):\n",
    "    return p[np.argmax(q)]    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Perverse Example A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider this example with two classes and two estimates. Estimate 1 is optimal; estimate 2 is worse."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data_A = [0.8, 0.2]\n",
    "estimate_A_1 = [0.8, 0.2]\n",
    "estimate_A_2 = [0.6, 0.4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Estimate 1 has a lower cross entropy loss than estimate 2:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5004024235381879"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_entropy(data_A, estimate_A_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5919186453876236"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_entropy(data_A, estimate_A_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But focal loss rewards the worse estimate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.2131486464336196"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "focal(2, data_A, estimate_A_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.131358612536986"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "focal(2, data_A, estimate_A_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And with accuracy, both esimates score identically:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy(data_A, estimate_A_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy(data_A, estimate_A_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So focal loss rewards the worse estimate, and accuracy can't distinguish between them. Let's plot the two functions,\n",
    "so we can clearly see that cross entropy is minimized when q[0]=0.8 and focal loss is not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xt83FWd//HXmVsyud/T+/1OKW0p\nUKDcERErXlYFFZWfKIq63nbddXUv7q77U9ddXV0VRfCCF0BR5CqiUCggvVN6pfTeJk2TNGnuyUxm\n5uwfZxJKSdppm8l8J3k/H4/vY6aTb2Y+3yR9z5nzPd9zjLUWERHJHr5MFyAiIqdGwS0ikmUU3CIi\nWUbBLSKSZRTcIiJZRsEtIpJlFNwiIllGwS0ikmUU3CIiWSaQjietqKiwU6ZMScdTi4iMSOvXrz9i\nra1MZd+0BPeUKVNYt25dOp5aRGREMsbsT3VfdZWIiGQZBbeISJZRcIuIZBkFt4hIllFwi4hkGQW3\niEiWUXCLiGQZTwX3d57cyTOvNGa6DBERT/NUcL9t5ZuxT38t02WIiHiap4K7xHQQirRkugwREU/z\nVHD3mFx8sa5MlyEi4mmeCu6oyVFwi4ichLeC25dLIN6d6TJERDzNU8Hd6wsTiPdkugwREU/zVHDH\n/GGCCQW3iMiJeCq44/5cchLqKhERORFvBXcgj5BVi1tE5EQ8Fdw2ECbXRjJdhoiIp3kruEP55BLB\nWpvpUkREPMtTwU0wTB4ReqLxTFciIuJZngpuE8rHZyydXR2ZLkVExLM8Fdy+UD4Aka72DFciIuJd\n3gruHBfcPV2dGa5ERMS7PBXc/txki7tbLW4RkcEEUtnJGLMPaAfiQMxauyQdxQSTLe5eBbeIyKBS\nCu6kK6y1R9JWCRAMFwDQ262uEhGRwXiqq6Q/uCMaVSIiMphUg9sCTxhj1htjbh1oB2PMrcaYdcaY\ndY2Np7duZG5eIQDxHrW4RUQGk2pwL7PWLgbeBHzCGHPp8TtYa++w1i6x1i6prKw8rWJyki3uuFrc\nIiKDSim4rbW1ydsG4AHg/HQUk5vvWtw2olVwREQGc9LgNsbkG2MK++4D1wBb0lFMMNe1uG2vgltE\nZDCpjCqpBh4wxvTt/ytr7eNpqSaY526jCm4RkcGcNLittXuAc4ahFvD5iRDEqMUtIjIoTw0HBIiQ\ni9FK7yIig/JecPty8Su4RUQG5b3gNrn4tdK7iMigPBfcMX8ugbgWDBYRGYz3gtsXJphQi1tEZDDe\nC+5AmJyEWtwiIoPxXHDHA2FCWuldRGRQngvuRCCPHKuuEhGRwXguuG0gTJgIsXgi06WIiHiS54Kb\nUB55ROjqjWe6EhERT/JccJtQPmETpaunN9OliIh4kieDG6C7S3Nyi4gMxHPB7c9xMwT2dGnBYBGR\ngXgwuN2c3BG1uEVEBuS54A7kuq6S3m61uEVEBuK54O5f6V0LBouIDMhzwR3K7QtudZWIiAzEe8Hd\nt9K7gltEZECeC+6+ld4TEXWViIgMxHPBndPX4taCwSIiA/JccPuSF+AQVYtbRGQgngtuQu4CHKuV\n3kVEBuS94A6EATAKbhGRAXkvuH0+esjB9GoVHBGRgXgvuEmu9B5TcIuIDMSTwR315eLXSu8iIgPy\nbHAHFNwiIgNKObiNMX5jzIvGmEfSWRBAzB8mqJXeRUQGdCot7k8D29NVyLFccGvBYBGRgaQU3MaY\nCcCbgTvTW46T8IfJUXCLiAwo1Rb3/wB/Bwy69Lox5lZjzDpjzLrGxsYzKioRDJNre7DWntHziIiM\nRCcNbmPMcqDBWrv+RPtZa++w1i6x1i6prKw8o6ISwTzCJkIkNuj7hIjIqJVKi/ti4HpjzD7gXuBK\nY8wv0lpVMI8wUbqi8bS+jIhINjppcFtr/8FaO8FaOwW4EXjKWntTOosywTzyiNAZiaXzZUREspIn\nx3GbnHxyTC/dkWimSxER8ZxTCm5r7dPW2uXpKqaPL8dN7drTpQWDRUSO58kWdyAZ3JEuLV8mInI8\nTwd3tFvBLSJyPG8Gd1grvYuIDMaTwR3KTQZ3t/q4RUSO583gzksuGNyjdSdFRI7nyeDOzSsCtNK7\niMhAPBncwVx3cjIRUR+3iMjxPBncBJMrvavFLSLyOt4M7lBfi1t93CIix/NmcCdb3B2dGlUiInI8\nbwZ3IIcEhq6ONs3JLSJyHG8GtzHE/WH8sW4a2iOZrkZExFO8GdyADeYRpoed9RpZIiJyLM8Gt6+w\nimrTws4G9XOLiBzLs8Htr5jBdP9hdjaoxS0icizPBrcpn84EGth9uCXTpYiIeIpng5uy6QSI01G/\nVyNLRESO4d3gLp8OQEW0hiMdWsJMRKSPd4O7bBoAU8xhnaAUETmGd4O7oJpEMN8Ft4YEioj0825w\nG4Mpn8aMQINa3CIix/BucAOmbDoz/PVqcYuIHMPTwU35dKri9eyr15BAEZE+3g7usun4iRPurqWp\nQ3OWiIiA54O7b2RJva6gFBFJ8nZwJ8dyuyGBCm4REYBApgs4ofxKbKiQmbaBV+o1skREBFJocRtj\nco0xa4wxLxljthpj/nU4Cku+OKZ8GvNyGtlY0zpsLysi4mWpdJVEgCuttecAC4FrjTFL01vWMcqm\nM9V3mM01LbR06dJ3EZGTBrd1+jqYg8lt+GZ9Kp9OcaQOv43x/K6mYXtZERGvSunkpDHGb4zZCDQA\nf7LWrk5vWccom4axCWbnNvPszsZhe1kREa9KKbittXFr7UJgAnC+MWb+8fsYY241xqwzxqxrbBzC\ngC1zI0uuHdPFszuPaIpXERn1Tmk4oLW2BVgBXDvA1+6w1i6x1i6prKwcqvr6hwQuLTlKbUs3uxs7\nh+65RUSyUCqjSiqNMSXJ+2HgDcDL6S6sX1455BQzO+ha8eouEZHRLpUW91hghTFmE7AW18f9SHrL\nOoYxUDGTwradTK3I59mdR4btpUVEvOikF+BYazcBi4ahlsFNOA/W/5TL5pdw3/rDRGJxcgL+jJYk\nIpIp3r7kvc+kCyDWzZsqGunujbN+/9FMVyQikjHZEdwTLwDgHPsyAZ9Rd4mIjGrZEdxF46B4Erl1\nazl3cil/2lavYYEiMmplR3CD6y45uIa3LRzHroYONh7U4goiMjplT3BPvADa63jL5Bi5QR+/XleT\n6YpERDIiu4IbKGjYwHVnj+Xhlw7RHY1nuCgRkeGXPcFdNQ9CBXBwFTcsmUhHJMYfttRluioRkWGX\nPcHtD8CEJXBwNedPLWNKeR6/Xncw01WJiAy77AlucN0l9Vsx0Q7etWQiq/Y0s79Jc5eIyOiSfcFt\nE1CzlncsHo/PwP3rdZJSREaX7AruCecBBg6uYWxxmMtmVXLv2oP09OokpYiMHtkV3LlFUH0W7P8L\nALdeOp3G9gj3rVVft4iMHtkV3ADTr3DB3dPK0mllnD+ljNuf3k0kpla3iIwO2Rfcs98MiV7Y9WeM\nMXzqqpkcbuvhN7ogR0RGiewL7onnQ14FvPwYABfPKGfxpBJuf3o30Vgiw8WJiKRf9gW3zw+zroWd\nf4J4L8YYPn31LGpbuvndBrW6RWTky77gBphzHURaYd9zAFw6s4JzJpbwv0/t0ggTERnxsjO4p10B\ngTDscN0lxhj+/trZ1LZ0c/vTuzNcnIhIemVncIfyYPqVrp87OS/3RdMruP6ccdz+zG5dTSkiI1p2\nBje47pK2Gji8qf+hL715LkGf4csPbdVCCyIyYmVvcM+6FowPXn60/6Hqolw++4ZZrNjRyJ+3N2Sw\nOBGR9Mne4M6vcHOXbH+4v7sE4IMXTWFWdQFffmgr7T29GSxQRCQ9sje4Ac5+JzRsg0Mv9j8U9Pv4\n6jvOpq61my8/tC2DxYmIpEd2B/f8d0IgF178+WsePndyGZ+8cia/3VDDwy8dylBxIiLpkd3BHS6B\neW+DzfdDtOs1X/rUlTNYNKmELz6wmdqW7gwVKCIy9LI7uAEWvx8ibbD9odc8HPD7+PYNi0gkLJ+9\nbyOxuC6HF5GRIfuDe/LFUDYNNtz9ui9NKs/jK2+fz5q9zfzHY9szUJyIyNA7aXAbYyYaY1YYY7YZ\nY7YaYz49HIWlzBhY9H7Y/zw0vf6qybcvmsCHLp7KT57fx31rD2SgQBGRoZVKizsG/I21dh6wFPiE\nMWZeess6RQvfC8b/upOUfb543RwumVnBP/5+C2v3NQ9zcSIiQ+ukwW2trbPWbkjebwe2A+PTXdgp\nKRwDM6+BF38JvT2v+3LA7+O771nMhNI8Pvbz9ew7okviRSR7nVIftzFmCrAIWJ2OYs7I0o9BZwNs\nunfALxfnBbnzg0tIWMtNd63mcOvrA15EJBukHNzGmALgt8BnrLVtA3z9VmPMOmPMusbGxqGsMTVT\nL4OxC+H570Bi4Kldp1cW8LMPnU9LVy833bWa5s7oMBcpInLmUgpuY0wQF9q/tNb+bqB9rLV3WGuX\nWGuXVFZWDmWNqTEGln0GmnfDy48MutuCCSXc+cElHGzu4uafrKFNl8WLSJZJZVSJAe4Ctltrv5n+\nks7A3Ovd0MDnvvWa+UuOt3RaOd9/32K217Xx3h+tUstbRLJKKi3ui4H3A1caYzYmt+vSXNfp8fnh\nok+5uUv2rjzhrlfNreaO9y/hlfoObrzjBRra1ectItkhlVElz1lrjbV2gbV2YXJ7bDiKOy3nvAfy\nq1yr+ySumFPFT28+j5qj3bz7By9wsLnrpN8jIpJp2X/l5PGCuXDhJ2DPCtj/l5PuftGMCn5+y/k0\nd0Z5+/efZ+PBlmEoUkTk9I284AY4/1YoHAdP/NMJ+7r7nDu5jN99/CJyg35uvOMFHt9yeBiKFBE5\nPSMzuEN5cMUXoXYdbHswpW+ZUVXIAx+/mNljirjtl+v53opdJBJa/kxEvGdkBje4y+Ar58KT/wbx\n1Ib8VRbmcO9HlrJ8wTi+8ccdfOwX67WKjoh4zsgNbp8frv6yG9e9/qcpf1s45Oc7Ny7kn5bP48mX\nG3jrd59nx+H2dFUpInLKRm5wA8x6I0xeBk9/DbpTP+lojOGWZVP51YcvoK0nxlu++xx3v7BPK8eL\niCeM7OA2Bt74H9DdDE/9+yl/+wXTyvnDpy/hounl/PODW/nI3eto6oikoVARkdSN7OAGGLcQzv8o\nrL0LDq495W+vLMzhJzefxz8vn8fKV45wzbdW8tjmujQUKiKSmpEf3ABXfgkKx8Ijn0n5ROWxjDF8\naNlUHv7rZYwrCfPxX27gE7/coNa3iGTE6AjunEK47htQvwVWff+0n2b2mEIe+PhFfP6Ns3li22Gu\n/O9nuHfNAQ0bFJFhNTqCG2Ducpj9Zljx1QGXOEtVwO/jE1fM4LFPXcLsMYV84XebedcPX2B73etm\nuhURSYvRE9zgWt2BEPzu1tPqMjnWzOpC7rt1Kd945wL2NHbw5u88yz/+frNmGhSRtBtdwV08Hpb/\nj7uicuV/nfHTGWN415KJrPjby/nAhVO4Z81BLv/GCu58dg+R2MCLOYiInKnRFdwA89/hZhBc+Z9w\ncM2QPGVJXogvX38Wj3/6EhZOKuUrj27nyv96hgderFH/t4gMOZOOi0qWLFli161bN+TPO2R62uAH\nF4PxwUefhdyiIX3653Ye4WuPb2dLbRtzxhTymatncs28Mfh8ZkhfR0RGDmPMemvtklT2HX0tbnBB\n/Y4fQctB+P1tkEgM6dMvm1nBQ59Yxnfes4hoLMHHfrGB5f/7HH/celgtcBE5Y6MzuAEmLYVrvuLW\np3z+5IsunCqfz3D9OeN44rOX8s13n0NnNMZHf76eN337WR7cWEssPrRvFiIyeozOrpI+1sJvPwxb\nfgs3/RZmXJW2l4rFEzyyqY7vrdjFzoYOJpaF+fCyabxryQTyQoG0va6IZIdT6SoZ3cENEO2EO98A\nbbVw6wq32HAaJRKWJ7bV88OVu3nxQAvF4SDvu2AS779wMmOLw2l9bRHxLgX3qWreAz+6EsJlcMuf\nIL98WF52/f5m7li5hye21eMzhmvPGsMHL5rCeVNKMUYnMkVGEwX36TiwCn52vZuU6gMPQnD4Wr8H\nm7v4+ar93LvmAG09MWZXF/K+pZN426LxFOUGh60OEckcBffp2vp7+M3NMO96eOdPwTe85267ojEe\n3HiIX67ez5baNsJBP9edPZYbzpuoVrjICKfgPhN/+S488SVY8iF48zfdnN4ZsKmmhXvWHOChjYfo\njMaZVpHPX507gbctGs/4EvWFi4w0Cu4zYS38+V/g+W/D0k+4hRgy2NLtisZ4dFMdv1lXw5p9zRgD\nS6eW8/ZF43nj/DEUh9WVIjISKLjPlLXw+Bdg9Q/gkr+Bq/450xUBcKCpiwderOV3L9awv6mLkN/H\nFXMqWb5gHFfNrdKwQpEspuAeCtbCw5+GDT+DSz8PV3wpoy3vY1lr2VTTyoMbD/HwpkM0tkfIDfq4\nak41bzp7DJfPrqIgRyEukk0U3EMlkYCH/xpe/AVccBtc+1XPhHefeMKydl8zj26q4w9b6jjSESUU\n8HHpzEquOauaq+ZUUV6Qk+kyReQkhjS4jTE/BpYDDdba+ak86YgJbnDh/ccvwurbYdH74S3fBp8/\n01UNKJ6wrNvXzONbD/PHLYc51NqDz8C5k0u5em41V82tYnplgUaniHjQUAf3pUAHcPeoDG5w3SYr\n/r+bCnbOcjdBVSgv01WdkLWWrYfaeGJbPX/aVt+/Qs+ksjyunFPFZbMruXBaOblBb74JiYw2Q95V\nYoyZAjwyaoO7z6rb4fF/gPHnwnvuhYLKTFeUstqWbla83MCT2+t5YU8TPb0JcgI+LphWzqUzK7h0\nViUzq9QaF8kUBXc6bX/ETUxVUAXvux8qZ2W6olPW0xtn9d5mnt7RwMpXGtnd2AlAdVEOF0+v4KIZ\nFVw8o1xzp4gMo4wEtzHmVuBWgEmTJp27f//+lIrNSjXr4Z4bIBaBt/8Q5lyX6YrOSG1LN8++0shz\nu47wl91N/etmTinP48Lp5SydVs4FU8sZU5yb4UpFvCuesPgMp/2pVS3u4dByEO67Ceo2wmVfgMv+\nftgvkU+HRMKy/XAbL+xuYtWeJlbvaaY9EgNgcnkeF0wt47wpbptcnqeuFZGk25/ezcpXGrnr5iWn\ndU3FqQS3BvuerpKJ8KHH4ZHPwTNfg9r18PYfQH5Fpis7Iz6f4axxxZw1rpgPXzKNeMKy7VAbq/c2\nsXpvM09sq+fX62oAqCzM4dxJpZw7uZTFk0uZP76InIBOdsroc6ilm71P/ZhbSmrJM7eT7mhNZVTJ\nPcDlQAVQD/yLtfauE33PqGhx97EW1t7phgzmlbsRJ1MvyXRVaZNIWHY3drBmXzPr9h1l/f6jHGju\nAiDk93HW+CIWTypl4cQSFk4sYUJpWK1yGfE++YvV/P3Om6iuHkvotmdO63oPXYCTCXWb4P7/5+b2\nvuRvXNeJf3TMI9LQ1sOGA0d58UALGw4cZVNNK5GYW5qtPD/E2ROKWTChhAXji1kwoZiqIvWVy8jx\n7M5GHvnp1/l68Efw3l/DrDee1vMouDMl0gGPfR5e+hWMWeBOXFbPy3RVw643nmDH4XY2Hmxh48EW\nNte0srOhnb51kqsKczh7fDFnjS/mrHFFnDWuiPElaplL9onE4rzlW09xd9dtVI2diO8jT5321dXq\n486UnAJ4++1ulMnDn4E7LoMrvggXfnLUtL4Bgn4f88cXM398MTctnQxAZyTG1kNtbKltZUttK5tr\nW1mxo6E/zIvDQeaNLWLu2CLmji1k7tgiZlQV6AIh8bTvPLmTc1v+wJhgI1xx+7BNiaEWd7p0NMKj\nn4XtD0P12XD9t92FO9KvOxpn++E2th5qY9uhNrbVtbHjcBs9va6bxe8zTK3IZ/aYQmZXFzKrupBZ\n1QVMLs/H71PrXDInkbB89Q/b+emzO1ld+HnKqibCh/98RsGtFrcXFFTCDb9wwf3Y5+HOq+G8j7gW\neLgk09V5QjjkZ/GkUhZPKu1/LJ6w7Gvq5OW6drbXtbGjvp3NNa08uqmuf59QwMf0ygJmVRcws6qA\nGcltUlk+oUD2D8kUb4vGEnz+/pd4cOMhvjvjJcpq6uGK7w/rBHRqcQ+HnlZ48t9g7V1u5MnV/wIL\nbxoR476HS2ckxq6GDl6pb09uHexq6KC2pbt/H7/PMLksj2mV+UyrLGBaRT5TK/KZWplPZUGO+tDl\njG2va+NLD2xmw4EW/vOiBO/afCtm3GK4+ZEzDm6dnPSqupfgsb+Dg6tg7EK45isjeujgcOiMxNjd\n2MGexk52NXT039/b1Ek0ObIFoCAnwOTyPKZU5DO1PJ/J5XlMTt5WFSrU5cTaenr51p9e4e4X9lOU\nG+Brbyjnjc+/F/wh+MiTbgqMM6Tg9jJrYfNv4M//Cm01MPONcPWXR+Xok3SKJyy1R7vZ29TJviOd\n7E1u+5o6qTnaTTzx6t99btDHxNI8JpfnMaE0j4lleUwsDTOxLI8JpWEKc0fPiWV5rdqWbu5+YR/3\nrD5AeyTG+y6YxN9ePp6Se6+H5n1wyxND9n9XwZ0Nerth9Q/h2W9CpA3Ofqe7dL5iRqYrG/F64wlq\nj3azv7mL/U2dHGjq4kCz2w42d9EZjb9m/+JwkPElYcaVhJlQGu6/P64kl3ElYSoKcnSydATpjsZ5\n6uUGHn7pEH/aXo+1ljfNH8ttl09nfrmBX38A9j4D7/0NzLx6yF5XwZ1Nuprh+f+BNT+CWA8suBEu\n+RxUzMx0ZaOStZajXb0cbO6i5mg3NUfdbW1LN7XJ247k3C19Aj5DdVEuY4tzGVOcy5gid1td5LYx\nRblUFeVoaKOHHWzu4rldR3h2ZyMrXm6kuzdORUEO71g8ng9cOJkJpXnQcgB+dQM07oDr/xcWvW9I\na1BwZ6OOBrey/No73ayD866HZZ+DcQszXZkcw1pLW0+MQy3dbmvt4XBrN3UtPdS19nC4rYe61u7+\nIY3HKsoNUFWUS1VhDpWFOVQV5lBR4LbKwhzKC0JUFuRQmh8i6NeJ63TpjsbZUd/OxgNH2XiwhQ0H\nWvqnbaguyuGqudUsXzCWC6aWv/pJqmY93HOj+7/57p/B9CuGvC4FdzbraIRV33cBHmmDqZe6C3hm\nvEGjULKEtZa27hj17T0cbu2hvq2HhvYIDW091LdFaOyI0NgeoaG9Z8CAB9c9U14Qojw/RGleiPIC\nd1uWH6IkL0RpXpCSvBAleUFKwkGKw0ECCvt+iYSloT3CwaNd7tzGkU72NHayo76dfU2d2GOu4l04\nsYSl08q5ZGYFM45fTCTe67ozV/4nFI13l7RXzUlLzQrukaCnFdb9BNbcAW21UD4Tzv8InHMj5BZn\nujoZAtZaOqNxjrS7MG/qiHCkI8qRjgjNnVGaOqI0dUY42tlLU2eUo13R15xUPV5+yE9xOEhR35Yb\npCg3QGFugILcAIW5QfJzAhTmBMjPCZCf4yc/5O7nhfzkhfyEQ35Cfp/nRtlYa+nujdPeE6O1u5eW\nrl5auqLu55T8WTW099DQFun/1NMbf/VnFfAZJpXlMau6kDljC5kzppAFE0oYW5w7+LEe3gK/vw0O\nb4Kz3w1v+jrklaXtGBXcI0m8F7Y96FrhteshmAdnvwvOvRnGLfLcqvOSPn3dNC1dUY4mg6u1u5ej\nnVFau12gtXb30tbTS1t3L209Mdp7emlP3p4g81/DZyA36Ccc9JMT8JEb9JMT9BMK+Mjx+wgFfAT9\nhqDfR9Dvw+8zBPyGgM/g9xmMMfiN6f/TNIDFDaiyWBLWtYhjCUs8YYnGE8TiCaKxBJHk1tMbpzsa\npysapysaoyMSO2H9+SF/fzdUdZE7aTy+1J1Mnlqez4TScOqfSDoa4OmvwvqfuaBe/i2Y+5bUvvcM\nKLhHqtoNsO4u2PxbiHVD9XxY/AGY/07IL890deJhfS3Wjh4Xgh2RGJ2RV0OxLyS7e+P9oenuJ4jE\n3G00niAaixOJJYjFLb3xBL3xBPFkCMfiloR1W98ng77ANsYFuDEGn3EXS/mNIeD3EfAbgj4fOUEf\nOQEfOQH3hhFOfgrICwUoyOn71BCgONk1VBIOUZbsThqSE7/dLe4T7vPfdgMFltwCl38hra3sYym4\nR7ruFthyP2z4uVuBxxeAmdfAghtg1rUQ1LSpIilrr0+eV7oLou0wZzlc/a/DPjRXc5WMdOESOO/D\nbju8GV66FzbfDzseg1Chm53wrHe4M9+BnExXK+I91sLB1W4QwLYHIRGDeW+DZZ+FsQsyXd1JqcU9\nUiTi7qKALb91K9H3tLgQn3WN65+bcTXkFGa6SpHMaqtzVy6/dC80bIWcIjjnPXDBR6F8ekZLU1fJ\naBeLwp6nYftDrhXe1eTmVJiyzHWlzLwGyqZmukqR4dHRCC8/Att+D3ueAaybYnnxB90Vy6H8TFcI\nKLjlWIk4HFgFr/wBdjwOTTvd42XTYPpVMP1KmHKxhhjKyGEtHHkFXvmj2w78BWwCSqe6EVkLbvDk\n1BIKbhlc027Y9WfY9STsexZ6u8D43NDCqZfC5GUw8XzILcp0pSKp62xyXYV7n4HdK6Blv3u86iyY\nu9x1F1bP9/TwWQW3pCYWgYNrYO9K9wdfu96dpDE+t2bmpKUuxCde4K4a8/AfvYwi1rqL0g6sggMv\nwP4XXH81uD7rKcvcOZ2Z10DJxMzWegoU3HJ6op1Qsxb2Pe/+Q9Sudy1ygIIxrl9w/GI3f8rYRRo7\nLsOj+ygc2uiGvtauh5p10J5cESlU4BoXky+CqZe7T47+7Bwsp+GAcnpC+TDtcreBu2qzfotrldeu\nd9uOR1/dv3iia5mPmQ9jzoaqeVA6BXyaBU9OQyIOR/dBwzao3+qGuh7e/Gq3B7i/rynLYML5MGGJ\n+/vL0qA+E6PviCV1/qBrwYxb9Opj3S1uJZ+6ja4VVL/Fnfi0ycmSAmE3CU/lHKiYBZWz3W3plFG1\n0r2cQLTTnWtp3g1HdrppUo/scPdjPcmdDJTPcJ/wzv2g+xscu3DYrmL0OgW3nJpwCUy7zG19ol3Q\nuB3qt7nWUsM2N+zqpXte3cf4XXiXT3dn98umuX+XToaSyRDKG+4jkXSJx1xXRutBOLrfzWN9dB8c\n3etu2+uO2dm4fuiKWTD1MqhydBN2AAAHRElEQVSa67bKOZ4ZpudFCm45c6G8ZP/3ua99vKfVtaKa\ndiVvd0LzXncyKdr+2n3zKtx/4OKJUDzBnQwtGuduC8e4TVeBZl6kAzrqof0wdBx2F7S017mThW2H\noLUW2g+9+gmsT+E4d+3A9Cvdm3b5DPcmXjZdb9qnQcEt6ZNb7PohJxx3vsVa6DziWl8t+11LrOWg\na6E1bHfDFftOih4rXAYF1W5h1oIqyK+E/Ap3m1futnAZhEvdNgr7Pk9JLOqusO1ucScAu49Cd7O7\nYKtv6zwCnY1u62gY+PcSCCffZMe5xa/73nhLJrlPU8UTNH/OENNftgw/Y6Cg0m0Tz3v91611gdJa\n61p27clWXfth19rraHAnTLuaINox+OuECl3XTm6x23KK3Pj0nEI3GiGnwO0TynetvmA+BMNu6txg\n2G2BHAjkult/yG3DOSwykYB4BOJRF7SxHjeMM9bj1i2NdUNvD/R2ui6r3i73M4l2ui3S7rZoB/S0\nucU5elrdNlAI9/EF3Rth35tj6RT3pplf6W4Lq6FwrLsfLtVQ0WGWUnAbY64Fvg34gTuttV9La1Uy\nuhnzaqt5zPwT7xvtgq4jbu3OriZ32996PPpqSPW0QFsNNLS9GmTx6OnV5wu4YPOHXKve+JOP+d0Y\n+L5bTDLQjg0167oRbN9t3N1PxN0Y+kSvux/vdfXZ+CBFnIxJvjkVujeonEK3FY559Y0st+TVN7Zw\nGeQlf+Z5FW5fhbFnnTS4jTF+4HvAG4AaYK0x5iFr7bZ0FydyUqE8CE1yH8tPVSyabJ12uDeAaGey\nBdvtWqP9Ldse1+qNJVu+8V4XsPHeZNj2bX1BnHDhi3WhjOU14W18yc0kQz8Z9P6gewMwfnffH3Rv\nEIEQ+HOSrf9jPgH0fTIIhJOfGMLuU0Mo+clBwTtipdLiPh/YZa3dA2CMuRd4K6DgluwWCEGgTEPM\nJOukspbPeODgMf+uST72GsaYW40x64wx6xobG4eqPhEROc6QLQttrb3DWrvEWruksrJyqJ5WRESO\nk0pw1wLHztQyIfmYiIhkQCrBvRaYaYyZaowJATcCD6W3LBERGcxJT05aa2PGmE8Cf8QNB/yxtXZr\n2isTEZEBpTSO21r7GPBYmmsREZEUDNnJSRERGR4KbhGRLJOWFXCMMY3A/pPu+HoVwJEhLicbjMbj\nHo3HDKPzuEfjMcOpH/dka21KY6nTEtynyxizLtWle0aS0Xjco/GYYXQe92g8ZkjvcaurREQkyyi4\nRUSyjNeC+45MF5Aho/G4R+Mxw+g87tF4zJDG4/ZUH7eIiJyc11rcIiJyEhkJbmPMtcaYHcaYXcaY\nLwzw9RxjzH3Jr682xkwZ/iqHXgrH/TljzDZjzCZjzJPGmMmZqHMoneyYj9nvr4wx1hgzIkYfpHLc\nxph3J3/fW40xvxruGodaCn/fk4wxK4wxLyb/xq/LRJ1DyRjzY2NMgzFmyyBfN8aY7yR/JpuMMYuH\n5IWttcO64eY72Q1MA0LAS8C84/b5OPCD5P0bgfuGu84MHfcVQF7y/m3ZftypHHNyv0JgJbAKWJLp\nuofpdz0TeBEoTf67KtN1D8Mx3wHclrw/D9iX6bqH4LgvBRYDWwb5+nXAH3BLIC0FVg/F62aixd2/\noo61Ngr0rahzrLcCP0vevx+4ypisX4fppMdtrV1hre1bwXUVbgrdbJbK7xrg34GvAz3DWVwapXLc\nHwG+Z609CmCtbRjmGodaKsdsgaLk/WLg0DDWlxbW2pVA8wl2eStwt3VWASXGmLFn+rqZCO5UVtTp\n38daGwNagfJhqS59UlpJ6Bi34N6ps9lJjzn50XGitfbR4SwszVL5Xc8CZhljnjfGrEouyJ3NUjnm\nLwM3GWNqcJPW/fXwlJZRp/r/PiUpzQ4ow8sYcxOwBLgs07WkkzHGB3wTuDnDpWRCANddcjnuk9VK\nY8zZ1tqWjFaVXu8Bfmqt/W9jzIXAz40x8621iUwXlm0y0eJOZUWd/n2MMQHcx6qmYakufVJaScgY\nczXwJeB6a21kmGpLl5MdcyEwH3jaGLMP1wf40Ag4QZnK77oGeMha22ut3Qu8ggvybJXKMd8C/BrA\nWvsCkIubz2MkS8sKYpkI7lRW1HkI+GDy/juBp2yypz+LnfS4jTGLgB/iQjvb+zzhJMdsrW211lZY\na6dYa6fg+vWvt9auy0y5QyaVv/Hf41rbGGMqcF0ne4azyCGWyjEfAK4CMMbMxQX3SF9Z/CHgA8nR\nJUuBVmtt3Rk/a4bOxF6Ha2HsBr6UfOzfcP9pwf1CfwPsAtYA0zJ99niYjvvPQD2wMbk9lOma033M\nx+37NCNgVEmKv2uD6ybaBmwGbsx0zcNwzPOA53EjTjYC12S65iE45nuAOqAX9ynqFuBjwMeO+T1/\nL/kz2TxUf9+6clJEJMvoykkRkSyj4BYRyTIKbhGRLKPgFhHJMgpuEZEso+AWEckyCm4RkSyj4BYR\nyTL/B1ciGJIcU/V6AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "X = np.arange(0.001, 0.999, 0.01)\n",
    "y_ce = np.array([cross_entropy(data_A, [q, 1-q]) for q in X])\n",
    "y_fl = np.array([focal(3, data_A, [q, 1-q]) for q in X])\n",
    "plt.plot(X, y_ce)\n",
    "plt.plot(X, y_fl)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = np.array([1,2,3.0])\n",
    "y = [z**2 for z in x]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Perverse Example B"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Example A, if we classify based on which class has the highest estimated probability,\n",
    "both estimates above produce the same answer. So how does the model in the focal loss\n",
    "paper get a higher accuracy using focal loss?\n",
    "\n",
    "Consider this example with four classes and three different estimates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data_B = [0.4, 0.38, 0.11, 0.11]\n",
    "estimate_B_1 = [0.4, 0.38, 0.11, 0.11] # an optimal estimate\n",
    "estimate_B_2 = [0.38, 0.39, 0.12, 0.11] # still a good estimate\n",
    "estimate_B_3 = [0.37, 0.23, 0.2, 0.2] # by any reasonable metric, much worse than estimate 1 or 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cross entropy loss ranks the three estimates in a reasonable order:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.2197987036308489"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_entropy(data_B, estimate_B_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.2208740850837707"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_entropy(data_B, estimate_B_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.3102541186954466"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_entropy(data_B, estimate_B_3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Focal loss gets them exactly backwards:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3333470510847732"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "focal(5, data_B, estimate_B_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.32434150427747027"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "focal(5, data_B, estimate_B_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3066606608971082"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "focal(5, data_B, estimate_B_3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Accuracy cannot distinguish between the optimal estimate and estimate 3, the worst one. Moreover, it claims estimate 2 is worse than estimate 3. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy(data_B, estimate_B_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.38"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy(data_B, estimate_B_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy(data_B, estimate_B_3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This example illustrates how it is possible for focal loss to give higher accuracy by producing worse estimates. If the actual proportions of class 0 and class 1 are 0.4 and 0.38, it's much better to have the estimates\n",
    "0.38 and 0.39 than the estimates 0.38 and 0.22. *After* we have a good estimate we can decide what decision to make. If the dangers of incorrectly choosing class 0 are less than those of incorrectly choosing class 1, we can see that their probabilities are very similar and choose class 0 instead of class 1.\n",
    "\n",
    "But if we only have the poor estimate, we don't have good information available to make a reasonable decision. Even if the dangers of incorrectly choosing class 0 are worse, we may decide to go with class 0 since, according to our bogus estimate, class 1 appears much more rarely than class 0 anyway."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A better approach (maybe)\n",
    "\n",
    "There *are* proper scoring rules that are asymmetric in that they penalize mistakes in one class more heavily than mistakes in another class. For instance, here's one (that I'll only implement in the two-class case)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def loss_class_0(n, q):\n",
    "    return -(n-1)*(1-q) - n * np.log(q)\n",
    "    \n",
    "def loss_class_1(n, q):\n",
    "    return -np.log(q) + (n-1)*(1-q)\n",
    "\n",
    "def asymmetric_loss(n, p, q):\n",
    "    return p[0] * loss_class_0(n, q[0]) + p[1] * loss_class_1(n, q[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the mean loss for the case of a class distribution (0.1, 0.9). Note that both are minimized when q[0] is 0.1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XecVdW9/vHPmt6HafQuTZqCY8Ua\nLGgMaqLGGo2FaG403Zib/BJzExPTY8nVkKuxl0RNxN4VUYoDKiq9twEGptfT1u+PdUBEcIY5Z84+\n5Xm/OK/TNuzvnhmeWWfttdcy1lpERCTxpXldgIiIRIcCXUQkSSjQRUSShAJdRCRJKNBFRJKEAl1E\nJEl0GujGmHuMMduNMR/t8drvjTHLjDGLjTH/Nsb06tkyRUSkM11pod8LTNvrtZeB8dbaicAK4MdR\nrktERA5Qp4FurZ0N1O712kvW2kD46TxgYA/UJiIiByAjCv/GFcBjXdmwvLzcDh06NAq7FBFJHQsX\nLtxhra3obLuIAt0Y8xMgADz0OdvMAGYADB48mKqqqkh2KSKScowx67uyXbdHuRhjLgfOBC62nzMh\njLV2prW20lpbWVHR6S8YERHppm610I0x04AbgBOsta3RLUlERLqjK8MWHwHmAqONMZuMMVcCdwCF\nwMvGmPeNMXf1cJ0iItKJTlvo1toL9/Hy3T1Qi4iIREBXioqIJAkFuohIklCgi4gkCQW6iEhP8rXA\n8zdC7Zoe35UCXUSkJ338b5h/JzRt6/FdKdBFRHrSwnuhfDQMPqrHd6VAFxHpKds+hk3vwmGXgzE9\nvjsFuohIT1l4H6RnwSEXxGR3CnQRkZ7ga4XFj8LYsyCvNCa7VKCLiPSEJU9Be4PrbokRBbqISE9Y\neC+UjYAhU2K2SwW6iEi0bV8KG+fB5MticjJ0FwW6iEi0vfVHyMiFQy+K6W4V6CIi0bTtY/jwcTjq\nGsgvj+muFegiItH02s2QXQTHXB/zXSvQRUSiZVMVLH8WplwXs6GKe1Kgi4hEy6v/A3nlcOS1nuxe\ngS4iEg1r3oS1b8Jx34fsAk9KUKCLiEQq0AHP/RCKB0PlFZ6V0emaoiIi0om3b4Udy+Gif0Fmjmdl\nqIUuIhKJHatg9h9g3Jdh1KmelqJAFxHpLmvhme+4Vvm0W7yuRl0uIiLd9t6DsO4t+NKtUNjH62rU\nQhcR6Zadq+GFG2HIsTDpa15XAyjQRUQOXMAHT1wFaRlwzl2QFh9Rqi4XEZED9fqvYMsiOP8B6DXI\n62p2i49fKyIiiWL1a26YYuUVMHa619V8SqeBboy5xxiz3Rjz0R6vlRpjXjbGrAzfl/RsmSIicaB+\nAzxxNVSMgVNv9rqaz+hKC/1eYNper90IvGqtHQm8Gn4uIpK8fK3w6MUQ9MFXH4SsPK8r+oxOA91a\nOxuo3evls4D7wo/vA86Ocl0iIvHDWpj1Ldj6IXzlbigf6XVF+9TdPvQ+1trq8OOtwH4HYBpjZhhj\nqowxVTU1Nd3cnYiIh97+C3z0BEz9medXg36eiE+KWmstYD/n/ZnW2kprbWVFRUWkuxMRia2PnoRX\nbnKX9h/7Xa+r+VzdDfRtxph+AOH77dErSUQkTqybA//+Bgw+Gs6+M6YLPndHdwN9FnBZ+PFlwFPR\nKUdEJE5sXwqPXgQlQ+GChz2dRbGrujJs8RFgLjDaGLPJGHMlcAtwijFmJXBy+LmISHKo3wAPngsZ\nOXDx454sJ9cdnV4paq29cD9vTY1yLSIi3mvaCvefBR1NcPkzUDLE64q6TJf+i4js0rLThXnTNvja\nf6DfRK8rOiAKdBERgLY6ePAcqFvnulkGHeF1RQdMgS4i0loLD5ztToR+9SEYdpzXFXWLAl1EUtuu\nbpYdK9xolpGneF1RtynQRSR1NW+H+8+G2tVw4SMwIrHHeijQRSQ11W9wYd64BS56DIaf6HVFEVOg\ni0jqqVnh+sx9zfC1p2DwkV5XFBUKdBFJLZsXwUPngkmHy5+DvuO9rihqtGKRiKSOFS/BvV+ErHy4\n4oWkCnNQoItIqlh0PzxyAZSNgCtfgbKDvK4o6tTlIiLJzVp44zfw5m/hoC/A+fdDdqHXVfUIBbqI\nJC9/Ozz1X/DR43DoxfClWyE90+uqeowCXUSSU8sOtwboxnlupaFjvxf385lHSoEuIsln2xLXX968\nDc67F8ad43VFMaFAF5HksuxZeHIGZBXA5c/CwEqvK4oZjXIRkeRgLcz+vVtlqHwUzHg9pcIc1EIX\nkWTQ0QT/vgaWPQMTzofpt0FmrtdVxZwCXUQS246VrlW+czWc9ms46ptJf/JzfxToIpK4lsyC/3wT\nMrLdCkPDjve6Ik8p0EUk8QT98MpNMPcOGHAYnHcf9BrkdVWeU6CLSGJp3AKPXwEb5sLhV8NpN7sW\nuijQRSSBrHrFDUn0t8OX/w8mnud1RXFFgS4i8S8YgDd+DW/9EXqPdV0sFaO8riruKNBFJL7Vb4Qn\nrnKX8E/+Gkz7LWTleV1VXFKgi0j8WjILZn0LQiF1sXSBAl1E4o+vFV76CVTdA/0nwbn3QOlwr6uK\nexEFujHmu8BVgAU+BL5urW2PRmEikqKqF7sulh3L4Zjr4As/g4wsr6tKCN2ey8UYMwC4Hqi01o4H\n0oELolWYiKSYUAjeuR3+byp0NLrFm0/9lcL8AETa5ZIB5Bpj/EAesCXykkQk5TRscnOxrHsLxpwJ\n02+HvFKvq0o43Q50a+1mY8wfgA1AG/CStfalqFUmIsnPWvjwcXj2+2CDMP0OmHRJys7FEqlIulxK\ngLOAYUB/IN8Yc8k+tpthjKkyxlTV1NR0v1IRSS4tO+Ffl8OTV0HFaLjmLZh8qcI8ApHMh34ysNZa\nW2Ot9QNPAsfsvZG1dqa1ttJaW1lRURHB7kQkaax4Ee482i1GMfXncMULGsUSBZH0oW8AjjLG5OG6\nXKYCVVGpSkSSU3sDvPDf8P6D0HscXPIE9J3gdVVJI5I+9PnGmMeBRUAAeA+YGa3CRCTJrHoVZl0H\nTVvhuB/ACTdoUq0oi2iUi7X258DPo1SLiCSj9gZ48Sfw3gNQPhquetlNeStRpytFRaTnrHwZnv42\nNFXDlO/AiT+GzByvq0paCnQRib7WWnjxv+GDR6BiDJz/AAxUq7ynKdBFJLqWPAXP/gBad6qvPMYU\n6CISHY3V8PwPYenT0HeiG8HSb6LXVaUUBbqIRMZaWHQfvPQzCLTD1J/BMddDeqbXlaUcBbqIdN+O\nlfD0d2D9HBh6HHzpVig7yOuqUpYCXUQOXMAHb/8FZv8eMnPhS7e51YR02b6nFOgicmDWv+Na5TuW\nw7gvw7RboLCP11UJCnQR6arWWnj5Z+4CoeLBcNG/YNSpXlcle1Cgi8jns9aNJ3/pp9BWD1O+DSf8\nCLLyva5M9qJAF5H9277MzVW+fg4MPALO/DP0He91VbIfCnQR+SxfC7z5O5h7B2QVuJOeky6FtEhm\n3JaepkAXkU9Y6+Yof+FGaNgIh14Cp/wC8su9rky6QIEuIs7O1fD8j2DVy26u8q+/AEOO9roqOQAK\ndJFU52uFOX9248rTs+G038ARV+tKzwSkQBdJVdbCsmfcCkING2DCeXDqr6Cwr9eVSTcp0EVS0Y6V\nrntl9auue+XyZ2HosV5XJRFSoIukkvZGmP07mHcnZObDtN/C4VdBuqIgGei7KJIKQiFY/Ci8chM0\nb4dJl8DUn0NBhdeVSRQp0EWS3aaF8PwNsLkKBlTChY9oTc8kpUAXSVaN1fDq/8AHD0NBHzj7Lpj4\nVV0clMQU6CLJxt8O8/4Ks/8IIb9bnPn4H0B2odeVSQ9ToIskC2vdep4v/z+o3wBjzoRTfwmlw72u\nTGJEgS6SDLa858aTb3jHDUP82lMw/ESvq5IYU6CLJLLGLfDqL930tnllcOZf3MpBaeleVyYeUKCL\nJCJfC7xzO7x9K4QCMOV6OO77kFPsdWXiIQW6SCIJBV1r/LVfQVM1jD0LTv4FlA7zujKJAxEFujGm\nF/B/wHjAAldYa+dGozAR2cuaN9yqQVs/dOPIz7sXBh/ldVUSRyJtod8KvGCtPdcYkwXkRaEmEdnT\n9qVuLc+VL7m1PL9yt1ucWePJZS/dDnRjTDFwPHA5gLXWB/iiU5aI0LQVXv+1W5Q5qxBO+SUcMQMy\nc7yuTOJUJC30YUAN8A9jzCHAQuDb1tqWPTcyxswAZgAMHjw4gt2JpIiOJnfC853bIeiHI74BJ9wA\neaVeVyZxLpLPbBnAZOBOa+0koAW4ce+NrLUzrbWV1trKigpNBCSyX0E/LPg73DYJ3vwtjDoNvrUA\nTr9FYS5dEkkLfROwyVo7P/z8cfYR6CLSCWth6Sx45RdQuxqGTIELH4WBlV5XJgmm24Furd1qjNlo\njBltrV0OTAWWRK80kRSw7m13wnNzFVSMgQsfcy1zY7yuTBJQpKNcrgMeCo9wWQN8PfKSRFLA1o/c\nTIgrX4TC/jD9DjjkQi00IRGJ6KfHWvs+oM+FIl1Vt86NXFn8T8gpgpNvgiOvgcxcjwuTZKDmgEgs\nNG+H2X+AqnvcPCtTrodjvwu5JV5XJklEgS7Sk9ob3PDDuf8LgXaYfCmc8CMo6u91ZZKEFOgiPcHX\nCu/+Heb8GdrqYNw5cNJPoXyE15VJElOgi0RTwAfv3e+6V5qqYcQp8IWfQv9Dva5MUoACXSQaQkF3\novON30D9ehh8tJtzZegUryuTFKJAF4lEKARLn3IjV3asgL4T4eLHYcTJGksuMadAF+kOa2HFC/D6\nzW4624oxcN59cPB0zYIonlGgixwIa2H1ay7INy+EkmFwzkyYcK6WfRPPKdBFumrtbNe1smEuFA+C\n6beHr+7M9LoyEUCBLtK59e+4IF/3FhT2gzP+4BZizsj2ujKRT1Ggi+zPhnkuyNe+Cfm9YdotcNjX\ntcCExC0FusjeNsx3ww/XvA75FXDqzVB5BWRphUWJbwp0kV02zIc3b3EnPfPK3ZJvh18JWfleVybS\nJQp0kfXvwBu3uK4VBbkkMAW6pK51c9xSb2tnh7tWfhXuWlGQS2JSoEtqsRbWvAFv/g42vAMFfeC0\nX7uTneojlwSnQJfUYC2sfMkF+eYqt0rQ6b9309lqcQlJEgp0SW6hECx7Bt76A1R/AMWD4Yt/gkmX\naBy5JB0FuiSnYAA+/rcL8pplUDo8vG7nBbqyU5KWAl2SS6ADPngE5vwF6tZCxcFuGtuxZ2sBZkl6\n+gmX5OBrgYX3wjt3QNMW6D8JTn0QRn9Rsx9KylCgS2JrrYUFf4f5d0FbLQw9Ds7+Kww/SfORS8pR\noEtiatgM8/4Xqv4B/hYYdToc9z0YdITXlYl4RoEuiaVmObx9Gyx+DGzIzUM+5dvQZ5zXlYl4ToEu\niWHjAneic/mzkJELh10Ox1wHJUO8rkwkbijQJX6FQrDyRXj7VreoRE4vOP4GOPIbkF/udXUicSfi\nQDfGpANVwGZr7ZmRlyQpL9DhulTeuQN2LHerA027BSZdCtkFXlcnErei0UL/NrAUKIrCvyWprLUW\nFv4D5v8NmrdBnwnw5b/DuHN0MZBIF0QU6MaYgcAXgZuB70WlIkk9tWtg3p3w3oPgb4WDpsI5f4Ph\nJ2roocgBiLSF/hfgBqAwCrVIKrEWNs6HuXfAsmfBpMOE8+Do/4K+472uTiQhdTvQjTFnAtuttQuN\nMSd+znYzgBkAgwcP7u7uJFkE/bDkKTeGfPNCd6JzynfgiBlQ1M/r6kQSWiQt9CnAdGPMGUAOUGSM\nedBae8meG1lrZwIzASorK20E+5NE1lbnLs1f8Hdo3AylB8EZf4BDL9KCEiJR0u1At9b+GPgxQLiF\n/oO9wzxqPnoCtrznVpSRxLJ9mbssf/Fjrn982PFu+tqRp2qOFZEoS4xx6NuWwNz/heO+D7klXlcj\nnQmF3GIS8++CNa9DejZMPA+OvAb6TvC6OpGkFZVAt9a+AbwRjX9rn0ZNc/Nar3rVXeot8amtHt5/\nyHWr1K2Fwn5w0k+h8uu6EEgkBhKjhT5gsluNfcULCvR4tG0JLJj5SbfKoCNh6v+Dg6dr/LhIDCVG\noKeluz7X5c+5lWi0UIH3gn5Y+jS8ezesnwMZOe6X7eFXQ/9Dva5OJCUlTjKOngYfPAybFsCQY7yu\nJnU1bIKF98Gi+6F5K/QaDCf/wl2Wn1/mdXUicaep3c+Pn/yQ66eOZFSfnr1kJ3ECffhJkJYJy59X\noMdaKASrX4Oqe2DF8+6ioJGnwOG3wYiT3ScoEdmn++eu55nF1Xzj+IN6fF+JE+g5RTB0Cqx4EU79\npdfVpIambfD+g278eP0Gdx5jyrfd1LUlQz0uTiT+tfoC3D1nLSeOrmDCwOIe31/iBDq40S4v3Ojm\n/igd7nU1ySkUgjWvuRBf/jyEAm5Zt5NvgjFnQka2xwWKJI6H52+gtsXHdV8YEZP9JVign+YCfcVL\ncNQ1XleTXOo3uiGH7z0EDRsgrwyOuhYmXwblI72uTiThtPuD/G32Go45qIzDhpTGZJ+JFeilw6F8\nlOvHVaBHzt/uRg6996DrI8e6cxWn3KTWuEiE/lm1kZqmDm69IHajvhIr0MF1u8y7ExqrNZlTd1gL\n1e/D+w/D4n9Cez0UDYTjfwiTLtGSbiJR4AuEuOuN1Rw2pISjh8du9FfiBXrlFe6S8ld/Aefc5XU1\niaOxGj78F3zwCGxf4i7HH/NFmHwpDDtBI1VEoui+d9axpaGdm788ARPDOf0TL9BLh7k5s+f8GQ6/\nCgZWel1R/PK1uLnGP3gE1rwBNgQDKuHMP7tVgDQvjkjUrdzWxO9fWs7JB/fhxFEVMd134gU6uEm6\n3n8Ynv8RXPmyZu3bU9APq1+HD//pwtzf6i7+Oe77MPECKI/N2XaRVOQPhvjePz+gIDuD38S4dQ6J\nGujZhW4Y3X+udcF1yAVeV+StUBDWv+OmGV7yFLTVuoUjJn4VJp4Pg47SLz2RGLjjtVV8uLmBOy+e\nTEVh7AcVJGagg2ttLvg7vPxzN5wx1boPQiHYMBeW/MeFePM2yMyD0WfA+K+4KzgzsryuUiRlvL+x\nnjteX8U5kwZw+gRvBmwkbqCnpbkVb+45DR7+Klz67+Rf+Sboh3VzYOks153SvM1NijXyVBh3thsB\nlOxfA5E4tLG2lavvr6JvUQ43fWmcZ3UkbqADDDwMzr0b/nU5PHYJXPho8o2d7miG1a/Csufc9MHt\n9a4lPvIUNz3tqGmQXeB1lSIpq77Vx+X/WECHP8jDVx1JcZ53U0YndqADjD0Lpt8OT/0XPHElnHtv\n4k+vW7fOXQ274gVY9xYEfa5LafTpbqjhQVMhK8/rKkVSXrs/yNX3V7Gxto0HrjyCkT08m2JnEjz5\nwiZdAh1NblqAf0yDL89MrLlefK3upOaqV9xt50r3etkIOGKGO0cw+JjE/0UlkkTa/UG++dAi3l1X\nxx0XTeLIGF5AtD/JkxBHXQsFveGZ78Kdx8K038Dkr0GMhw11ScAHWxa51veaN2HjfNcKT8+Goce6\ni6dGnqohhiJxqqHNz9X3VfHu+lpuPmc8Z07s73VJQDIFOrjRHYOOgv9cA09fD4vugynfcfOSeDls\nr70RNr0LGxe4kSkbF0Cgzb3XdwIc+Q0YdqKb511dKSJxbXtTO5fd8y6rtjdx+4WT4ibMIdkCHaB4\nAFz6FLz3AMz5E/zzUigb6ebwHnNGz3fFtDe6S+urF8OW99ytZhlgwaRB73Fw2GUwZIq7aZUfkYSx\neFM91z64iNoWH3dfdjjHx/hK0M4Ya23MdlZZWWmrqqpitj+CAVj6FLx9m5uQCqD3WDejYL+J0Hei\nmxr2QBcy9rVC42a3HFvtGti52vV7b1/mpp7dJb+3W+C6/yS3cPKAw9xCHSKScB5dsIGfPfUxFYXZ\n3HnJZCYO7BWzfRtjFlprO53nJLkDfU+1a92CDcufc90fgfbwG8b1vRf2hfwKN647IxvSs1y/dqDD\n3drq3BWYrbVu6OCeMnLdCcyKUdBnHPQZ725F/eOzD19Euqyp3c8vnl7C4ws3cdzIcm69YBKl+bG9\naK+rgZ58XS77UzoMjv6muwUDsHMVbF0MO1a6xY4bq6F1xycBHvS7lvuucM8tcVPL5pa6aXuLBrru\nnZJhUNhPl9aLJKG3V+3ghscXU93QxnVfGMF3Th5Felr8NtJSJ9D3lJ4Bvce4m4jIXhra/Pz+xWU8\nOG8Dw8rz+dc1x3DYkPifXiQ1A11EZB9CIcvjizbx2+eXUdvq44opw/jhaaPJzUqM9QIU6CIiwIK1\ntfz6uaW8v7Gew4aUcN/0Ixg/oNjrsg5ItwPdGDMIuB/oA1hgprX21mgVJiISC0u2NPL7F5fx+vIa\nehdm86fzD+GcSQNiPpd5NETSQg8A37fWLjLGFAILjTEvW2uXRKk2EZEe8+GmBv76+ipe+HgrxbmZ\n3Hj6GC47emjCdK/sS7cD3VpbDVSHHzcZY5YCAwAFuojEJWstc1fv5K7Za5i9ooainAyu/8IIrjxu\nOMW53s2SGC1R6UM3xgwFJgHz9/HeDGAGwODBg6OxOxGRA9LuD/LU+5v5x9vrWLa1ibL8LG6YNppL\njxpCYU7iB/kuEQe6MaYAeAL4jrW2ce/3rbUzgZngLiyKdH8iIl21clsTDy/YwJOLNtPQ5mdM30J+\nd+5Eph/Sn5zMxO1a2Z+IAt0Yk4kL84estU9GpyQRke5raPPz7OJqnli0iYXr68hMN5w2ri8XHzmE\no4aXJuTJzq6KZJSLAe4Gllpr/xS9kkREDky7P8ibK2qY9cEWXl6yDV8gxIjeBdx4+hjOPWwg5QVJ\ntpLZfkTSQp8CXAp8aIwJz3zFf1trn4u8LBGRz9fuDzJn5Q6e/2grL328laaOAKX5WVxw+CC+Mnkg\nEwcWJ3VrfF8iGeUyB0itr5aIeKq+1ccby2t4eek23li2nRZfkKKcDKaN78uXDunPMQeVkZGeuvMq\n6UpREYlb1lqWbW3izRU1vL5sO1Xr6wiGLOUFWUw/dADTxvfl6OFlZGWkbojvSYEuInFlR3MHb6/a\nwVsrdzBn5Q62Nrqprsf0LeTaEw5i6sG9OWRgL9LieNZDryjQRcRT9a0+5q+tZe7qncxbs5NlW5sA\nKM7N5JiDyjhxdAUnjOpN3+IcjyuNfwp0EYmpLfVtVK2vo2pdLQvW1u4O8JzMNCqHlPLD0/pz7Ihy\nxg8ojuu5x+ORAl1EekxHIMiSLY28t6GeRRvqWLS+ji0NrgslLyudw4aU8MUJ/ThyeBmHDComOyP5\nLvaJJQW6iERFMGRZXdPM4k0NLN5UzwebGli6pRFfMARAv+IcJg8p4eohJVQOKeXgfoUpPSKlJyjQ\nReSA+QIhVm5v4uMtjXy8ucHdb2mkzR8EID8rnQkDi/n6lKFMGtyLQweVqA88BhToIrJf1lpqmjpY\ntrWJ5VubWFrdyJLqRlbXNOMPuqmZ8rLSGduviK8ePogJA4qZMLCYgyoK1P/tAQW6iABQ2+JjxbYm\nVm5vZuU2F+ArtjVR1+rfvU3vwmzG9i/ipDG9GduviHH9ixhalq8hhHFCgS6SQoIhy+a6NlbvaGZN\nTQurtjezenszq2qaqW3x7d6uIDuDUX0KOG1cX0b1KWRMv0LG9C2iND/Lw+qlMwp0kSSzq5tk7Y4W\n1u1sYe2OVtbuaA4/b8UXCO3etldeJiMqCjh1bB9G9C5gZJ9CRvQuoH9xTsrNg5IMFOgiCcgXCLGl\nvo31ta1sqG1lw84W1u90j9fvbN19chIgM90wpCyfYeX5nDS6N8Mr8hleUcDw8nzKUmQWwlShQBeJ\nQ4FgiK2N7Wyqa2NzXRsb61rZWNvGprpWNtW1Ud3QRmiP5WKyMtIYVJLLsPJ8jjmonKHleQwty2do\nWT79e+VoeGCKUKCLxJi1lsb2ANUNbVTXt7O5vo0tu2/u+dbGdoJ7JLYx0Kcwh4EluRwxrJRBpXkM\nKsllcGkeQ8ry6V2YrROTokAXiaZQyFLb6mNbYztbG9rZuus+/Li6oZ3q+jZafMFP/b2MNEOfohwG\nlORy5LBS+vfKZWBJLgNL8hhQkkv/Xjm6ilI6pUAX6YJQyFLX6mN7U4e7NbZ/6n5rYzvbGzvY3tS+\ne3z2LmkGKgqz6Vecy8jeBRw3spz+xbn065VDv2IX3OUF2Rq3LRFToEvKstbS2BagprmDHbtuTR3u\neZOPmuYOaprcbUdzB4HQZ9c4L8zJoE9RDn2LcjhyWD59inPoU5jtXit2t4qCbPVhS0wo0CVpWGtp\n6ghQ1+JjZ4uP2mYftS0+drR0UNvsXtvZ4mNncwc7m33sbOn4TGsaID3NUJafRXlBNr2LshnTt5CK\nwmx6F2ZTUZhDRWE2fYqy6V2YQ26WukEkfijQJS6FQpbGdj/1rX7qWn277+ta/dS3uqCub/VT2+Kj\nLvy8rtW3z4AGyM1Mp6wgi7IC13oe17+IsoJsyvKzqCjMprxg1y2LkrwsnWCUhKRAlx5jraXNH6Sh\nzU9jW4CGNv9nb60+Gtr81Le58G5oc8Hd0ObH7jubSTNu8YPSfBe+g0rzOHRQL0rysyjNy6I0P4vS\nAve4rCCLsvxstaQlJSREoFetq2VbYwdfGNNb/zFjxFpLRyBEc0eA5vYATe0Bmtr9NIbvm8KvNXe4\nx43tLrTdvduusc2/z37nXYyBopxMinMz6ZXn7geW5FKSl7X7eUleFiX5mfTKc+FdkpdJUU6mWtAi\n+5AQgf7Igo08sWgTeVnpnHxwH6aN78v4/sUMLMnVf+ywXQHc6gvS0hGgze/uWzqCtPgCtPrCjzsC\n7hbermnX844AzR1Bmjv8NLcHaO4I7Lf7Yk+5mekU5mRQlJtJYU4GvfKyGFKWT1FOBsW5mRSGA7s4\nN5Oi3Izdj3e9p5EdItFj7P4+1/aAyspKW1VVdcB/LxiyzF+zk6cXV/PCR9W7Z3/Lz0pnRJ9CBvbK\ndSMKinLcx/D8TIpzsyjMySA/O4P8rHRyMtPJzkjr8fkprLUEQpZA0OILhvAHQ/gCn9x3BEL4giE6\n/CE6AkE6wq+1+8OP/UHa/UHa/e619kCQNl+I9kCQdl+QVl+QNn+QNl+QVn+ANl+INp8L8M9pDH9G\nXlY6BdkZ7paTQX6W+1q5r1ntulpuAAAFvUlEQVQ6hTmZu98vzMnY/bwwJ4OinEwKctzjTI3eEOlx\nxpiF1trKTrdLhEDfkz8Y4sPNDSwPz8+8cntT+GKN9k/NX7EvxkB2RhpZ6WlkpqeRkW7ISEsjLQ3S\njSHNGHB/MMZgrcUCWLBAyFpC1hIMWoLWEgy5WyAYDvFQqEut2q5IM5CT6X4R5Wamk5vl7nMy08jN\nyiA3My38egZ54ffystPJz8ogN8vd52Wnk5eZ7n6pZbvt8rMzyMtM1ycbkQTS1UBPiC6XPWWmpzF5\ncAmTB5d86vVdl1PXh0dC1LX6PtWV0O4PutZv4JMW864gDoXDOfSpALcYPgn4NGNIMy7o09MMGWmG\ntPD9rucZ6WlkppnwL4s0sjLSyEp3z7Mydj1399kZ6eH7tN2fHrIzP3mcld7znyZEJLkkXKDvjzFm\nd9/skDKvqxERib2IOkCNMdOMMcuNMauMMTdGqygRETlw3Q50Y0w68FfgdGAscKExZmy0ChMRkQMT\nSQv9CGCVtXaNtdYHPAqcFZ2yRETkQEUS6AOAjXs83xR+TUREPNDjg4iNMTOMMVXGmKqampqe3p2I\nSMqKJNA3A4P2eD4w/NqnWGtnWmsrrbWVFRUVEexOREQ+TySB/i4w0hgzzBiTBVwAzIpOWSIicqC6\nPQ7dWhswxnwLeBFIB+6x1n4ctcpEROSAxPTSf2NMDbD+AP5KObCjh8qJdzr21JOqxw069s6OfYi1\nttM+65gG+oEyxlR1Zf6CZKRjT71jT9XjBh17tI5dU+WJiCQJBbqISJKI90Cf6XUBHtKxp55UPW7Q\nsUdFXPehi4hI18V7C11ERLooLgK9s2l4jTHZxpjHwu/PN8YMjX2V0deF4/6eMWaJMWaxMeZVY8wQ\nL+rsCV2detkY8xVjjDXGJM0IiK4cuzHm/PD3/mNjzMOxrrGndOFnfrAx5nVjzHvhn/szvKgz2owx\n9xhjthtjPtrP+8YYc1v467LYGDO5Wzuy1np6w12UtBoYDmQBHwBj99rmm8Bd4ccXAI95XXeMjvsk\nIC/8+NpkOO6uHnt4u0JgNjAPqPS67hh+30cC7wEl4ee9va47hsc+E7g2/HgssM7ruqN07McDk4GP\n9vP+GcDzuAXSjgLmd2c/8dBC78o0vGcB94UfPw5MNYm/Plunx22tfd1a2xp+Og83X04y6OrUy78E\nfgu0x7K4HtaVY78a+Ku1tg7AWrs9xjX2lK4cuwWKwo+LgS0xrK/HWGtnA7Wfs8lZwP3WmQf0Msb0\nO9D9xEOgd2Ua3t3bWGsDQAOQ6AvNHej0w1fifoMng06PPfyRc5C19tlYFhYDXfm+jwJGGWPeNsbM\nM8ZMi1l1Pasrx34TcIkxZhPwHHBdbErzXFSmI0+aNUWTmTHmEqASOMHrWmLBGJMG/Am43ONSvJKB\n63Y5EfepbLYxZoK1tt7TqmLjQuBea+0fjTFHAw8YY8Zba0NeF5YI4qGF3pVpeHdvY4zJwH0U2xmT\n6npOl6YfNsacDPwEmG6t7YhRbT2ts2MvBMYDbxhj1uH6FGclyYnRrnzfNwGzrLV+a+1aYAUu4BNd\nV479SuCfANbauUAObq6TZNelPOhMPAR6V6bhnQVcFn58LvCaDZ9JSGCdHrcxZhLwN1yYJ0s/KnRy\n7NbaBmttubV2qLV2KO78wXRrbZU35UZVV37e/4NrnWOMKcd1wayJZZE9pCvHvgGYCmCMORgX6Kmw\nMs4s4Gvh0S5HAQ3W2uoD/le8Pvu7xxneFbgz4D8Jv/Y/uP/E4L6p/wJWAQuA4V7XHKPjfgXYBrwf\nvs3yuuZYHfte275Bkoxy6eL33eC6nJYAHwIXeF1zDI99LPA2bgTM+8CpXtccpeN+BKgG/LhPYFcC\n1wDX7PE9/2v46/Jhd3/edaWoiEiSiIcuFxERiQIFuohIklCgi4gkCQW6iEiSUKCLiCQJBbqISJJQ\noIuIJAkFuohIkvj/S3zris8AG1YAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_better = [0.1, 0.9]\n",
    "X = np.arange(0.01, 0.99, 0.01)\n",
    "y_ce = np.array([cross_entropy(data_better, [q, 1-q]) for q in X])\n",
    "y_al = np.array([asymmetric_loss(10, data_better, [q, 1-q]) for q in X])\n",
    "plt.plot(X, y_ce)\n",
    "plt.plot(X, y_al)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now plot the asymmetric loss given for misclassification of each of class 0 and class 1 as a function of predicted probability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8XFd99/HPT/u+L5bkRV7jeI/t\nOAkhZCMhpCmBllK2EkoghZQWaJ+W9unTltLS0paltC9CG0hIgISlhCUQloTgxAmJ7djxvtuyZcvW\nvu+jmTnPH/fKUhwrGtvSjGbm+3695jXbnbm/a8lfH5977jnmnENEROJfSqwLEBGRqaFAFxFJEAp0\nEZEEoUAXEUkQCnQRkQShQBcRSRAKdBGRBKFAFxFJEAp0EZEEkRbNnZWVlbna2tpo7lJEJO5t3769\nzTlXPtl2UQ302tpatm3bFs1diojEPTOrj2Q7dbmIiCQIBbqISIJQoIuIJAgFuohIglCgi4gkCAW6\niEiCUKCLiCSIuAj0pw80c98zR2NdhojIjBYXgf7ckTa+svFYrMsQEZnR4iLQy/Iy6B0OMjQSinUp\nIiIzVlwEemleJgAd/YEYVyIiMnPFR6DnZgDQ3qdAFxGZSHwEut9Cb+sfjnElIiIzV1wEelmeWugi\nIpOJk0D3WujtfWqhi4hMJC4CPScjlaz0FNp1UlREZEKTBrqZZZnZVjPbZWb7zOwf/NcfMrPjZrbT\nv62ZriLNjNLcTNrUQhcRmVAkKxYNAzc55/rMLB143sx+7r/3F865709feWPK8jLUhy4i8hombaE7\nT5//NN2/uWmt6jxK8zJp1ygXEZEJRdSHbmapZrYTaAGecs5t8d/6jJntNrMvmlnmtFWJNxZdLXQR\nkYlFFOjOuZBzbg0wG9hgZiuAvwaWAlcCJcAnz/dZM7vHzLaZ2bbW1taLLrQ0L5P2vgDORf0/ByIi\nceGCRrk457qAjcBtzrlGvztmGPg6sGGCz9zvnFvvnFtfXl5+0YWW5WUQCIXpGQpe9HeIiCSySEa5\nlJtZkf84G7gFOGhmVf5rBrwV2DudhZaevbhI/egiIucTySiXKuBhM0vF+wfge865n5rZr82sHDBg\nJ/DhaayT0lz/4qL+AAsuvqEvIpKwJg1059xu4IrzvH7TtFQ0AbXQRUReW1xcKQpjl/+3aaSLiMh5\nxU2gF+dogi4RkdcSN4GekZZCYXa6Li4SEZlA3AQ66PJ/EZHXEleBXpqnCbpERCYSV4FelpehKXRF\nRCYQV4FempupYYsiIhOIr0DPy6BzYIRgKBzrUkREZpw4C3RvLHrHgLpdRETOFVeBXparsegiIhOJ\nq0AvPbtYtAJdRORccRbofgtdFxeJiLxKXAV6Wa7mcxERmUhcBXpBdhppKaahiyIi5xFXgW5mlOry\nfxGR84qrQAf/4iL1oYuIvEr8BXpehvrQRUTOI+4CvSxPLXQRkfOJZJHoLDPbama7zGyfmf2D//p8\nM9tiZkfN7LtmljH95WoKXRGRiUTSQh8GbnLOrQbWALeZ2dXAvwJfdM4tAjqBu6evzDGleZkMBEL0\nDwejsTsRkbgxaaA7T5//NN2/OeAm4Pv+6w8Db52WCs8xpzgHgPr2gWjsTkQkbkTUh25mqWa2E2gB\nngKOAV3OudFmcgNQM8Fn7zGzbWa2rbW19ZILnl+WC8Dxtv5L/i4RkUQSUaA750LOuTXAbGADsDTS\nHTjn7nfOrXfOrS8vL7/IMseMBXrfJFuKiCSXCxrl4pzrAjYC1wBFZpbmvzUbOD3FtZ1XdkYq1YVZ\n1KmFLiLyCpGMcik3syL/cTZwC3AAL9jf7m92F/Dj6SryXPPLc9XlIiJyjkha6FXARjPbDbwEPOWc\n+ynwSeDPzOwoUAo8MH1lvtL8slzqWvtxzkVrlyIiM17aZBs453YDV5zn9Tq8/vSom1+WR/fgCJ0D\nI5TkRmX4u4jIjBd3V4oCLNCJURGRV4nLQB8d6VLXqn50EZFRcRnos4uzSUsxnRgVERknLgM9LTWF\nuaU5CnQRkXHiMtDB60dXoIuIjInbQJ/vB3o4rKGLIiIQ14Gex3AwTGPPUKxLERGZEeI40P2hixrp\nIiICxHGgLyjXWHQRkfHiNtAr8jPJyUjVJF0iIr64DXQzO3tiVERE4jjQAQW6iMg4cR3oC8pyOdUx\nwHAwFOtSRERiLq4DfX55LmEHJ7W+qIhIfAf6sqpCAPae6Y5xJSIisRfXgb6oIo+cjFR2nVKgi4jE\ndaCnphgrawrZcaor1qWIiMRcJGuKzjGzjWa238z2mdnH/Nc/ZWanzWynf7t9+st9tTVzijhwpkcn\nRkUk6U26BB0QBP7cOfeymeUD283sKf+9LzrnPjd95U1u9ZwiAqEwBxt7WT2nKJaliIjE1KQtdOdc\no3PuZf9xL3AAqJnuwiI1GuK7GtTtIiLJ7YL60M2sFm/B6C3+Sx81s91m9qCZFU9xbRGpLsyiLC+T\nnepHF5EkF3Ggm1ke8BjwcedcD/AVYCGwBmgEPj/B5+4xs21mtq21tXUKSn7V97NmTiG7FOgikuQi\nCnQzS8cL80eccz8AcM41O+dCzrkw8FVgw/k+65y73zm33jm3vry8fKrqfoXVs4s41tpPz9DItHy/\niEg8iGSUiwEPAAecc18Y93rVuM3eBuyd+vIiM9qPvqdB49FFJHlFMsrlWuAPgD1mttN/7f8C7zKz\nNYADTgB/NC0VRmD1bC/Qd57q4tpFZbEqQ0QkpiYNdOfc84Cd562fTX05F6cwJ50FZbk6MSoiSS2u\nrxQdb/WcInae6sI5LRotIskpcQJ9diGtvcM0adFoEUlSiRPo/onRl+vV7SIiySlhAn1FTSF5mWk8\nf7Qt1qWIiMREwgR6emoKr1tYyqbDrepHF5GklDCBDnDdknJOdw1qnVERSUoJFejXL/auRN10eOqn\nGBARmekSKtDnluYwrzSH546oH11Ekk9CBTrAGxaX82JdO4FgONaliIhEVcIF+nWLyxgIhNhe3xnr\nUkREoirhAv2ahaWkpRjPHVE/uogkl4QL9PysdNbOLWaTAl1EkkzCBTrAG5aUsfd0D+19w7EuRUQk\nahIy0K/zhy/qqlERSSYJGegragopy8vkF3ubYl2KiEjUJGSgp6YYt6+cxa8PttA3HIx1OSIiUZGQ\ngQ5wx6pqhoNhnj7QHOtSRESiImEDff28YmYVZPGTXY2xLkVEJCoiWSR6jpltNLP9ZrbPzD7mv15i\nZk+Z2RH/vnj6y41cSopx+8oqNh1upXtwJNbliIhMu0ha6EHgz51zy4CrgT82s2XAXwFPO+cWA0/7\nz2eUO1ZXEQiFeWq/ul1EJPFNGujOuUbn3Mv+417gAFAD3Ak87G/2MPDW6SryYl0xp4iaomx+uvtM\nrEsREZl2F9SHbma1wBXAFqDSOTfaQd0EVE5pZVPAzLhjVRXPH2mjsz8Q63JERKZVxIFuZnnAY8DH\nnXM9499z3hJB510myMzuMbNtZrattTX6l+PfsaqaYNjxy30aky4iiS2iQDezdLwwf8Q59wP/5WYz\nq/LfrwJazvdZ59z9zrn1zrn15eXlU1HzBVlRU8D8slwee7kh6vsWEYmmSEa5GPAAcMA594Vxbz0O\n3OU/vgv48dSXd+nMjHdtmMNLJzo51NQb63JERKZNJC30a4E/AG4ys53+7Xbgs8AtZnYEeKP/fEZ6\n+7o5ZKSm8OiW+liXIiIybdIm28A59zxgE7x989SWMz1KcjO4feUsfvDyaT755qXkZEx62CIicSdh\nrxQ913uunkfvcJCf7NIQRhFJTEkT6OvnFbOkMo9HtpyMdSkiItMiaQLdzHjPVfPY3dDN7oauWJcj\nIjLlkibQAd62tobs9FQe2axWuogknqQK9IKsdN62toYf7jhNS89QrMsREZlSSRXoAH/0hgUEw2Ee\neP54rEsREZlSSRfo80pz+a1V1Xxrcz3dA5pWV0QSR9IFOsBHrl9IfyDEN148EetSRESmTFIG+rLq\nAm5aWsHXXzjBYCAU63JERKZEUgY6wL03LKSjP8B3XtKIFxFJDEkb6OtrS9hQW8L/PFvH0Iha6SIS\n/5I20AE+ccsSmnqGePiFE7EuRUTkkiV1oF+zsJQbLyvnyxuP0jWgFY1EJL4ldaADfPLNS+kdDnLf\nM8diXYqIyCVJ+kBfOquA37liNg+9cILTXYOxLkdE5KIlfaAD/NmtSwD4/JOHYlyJiMjFU6ADNUXZ\n/OHravnhjtPsPKWZGEUkPkWypuiDZtZiZnvHvfYpMzt9zpJ0ce2jNy2iPC+T//ejPYTCLtbliIhc\nsEha6A8Bt53n9S8659b4t59NbVnRl5+Vzt/esYy9p3v41matPSoi8WfSQHfObQI6olBLzN2xqorr\nFpfxuV8e0vS6IhJ3LqUP/aNmttvvkimesopiyMz4h7csZzgY5jM/OxDrckRELsjFBvpXgIXAGqAR\n+PxEG5rZPWa2zcy2tba2XuTuomdBeR4fvmEhP955ho0HW2JdjohIxC4q0J1zzc65kHMuDHwV2PAa\n297vnFvvnFtfXl5+sXVG1b03LGRJZR6ffGy3riAVkbhxUYFuZlXjnr4N2DvRtvEoKz2VL7xjDR39\nAf7ux/tiXY6ISEQiGbb4beBF4DIzazCzu4F/M7M9ZrYbuBH4xDTXGXUragr505sX8/iuMzyxuzHW\n5YiITCptsg2cc+86z8sPTEMtM869Nyzk6QPN/L8f7eHK2mIqCrJiXZKIyIR0pehrSEtN4fPvWMPg\nSIiPfWenLjgSkRlNgT6JRRV5/OOdK3ixrp0v/epwrMsREZlQfAT6zkfhsQ+Bi00L+ffWz+Ht62bz\nXxuPsunwzB96KSLJKT4CfaAD9nwPNn8lZiX8450rWFKRz8e/u5Ombl1FKiIzT3wE+jV/DJfdDk/9\nHTRsj0kJ2RmpfPk9axkaCfFH39ymdUhFZMaJj0A3g7feB/lV8L/v91rsMbCoIo//+P017D7dzV9+\nfzcuRl1AIiLnEx+BDpBdDL/3EPQ2wo/uhdBITMq4dfks/uJNl/H4rjN8eePRmNQgInI+8RPoALPX\nwW3/Aod/Do+8HQZjsxjFR65fyNuuqOFzTx7m53t00ZGIzAzxFegAGz4Ed94HJ34DX3sjtEd/cWcz\n419+ZyVr5xbxse/uZEtde9RrEBE5V/wFOsAV74H3/RgG2uBrN8Oe70d9SGNWeioP3HUlc4qz+eA3\ntnGwqSeq+xcROVd8BjpA7bXwoV9D8Xx47G549Peh61RUSyjOzeAbd19FbkYa73tgK6c6BqK6fxGR\n8eI30AFKFsAHfwVv+mc48RzcdzVs+ncI9EethJqibB7+wAaGRkK894EtGqMuIjET34EOkJLqjVO/\n90WYfz38+p/gS2tg61chGJ25zC+blc9DH9hAe1+Ad391s5avE5GYiP9AH1VcC+96FO5+CsoWw8/+\nD3xpFfzmP2Fo+vu3184t5qE/vJKmniHe/bUttPYOT/s+RUTGS5xAHzVnA7z/CfiDH3rB/tTfwheX\nwy//ZtpHxKyvLeHr77+S052DvPurm2lWS11EosiiebXj+vXr3bZt26K2PwBOvwwv/Ccc+AmEg7Dw\nZlj/AVjyJkhNn5ZdvnisnQ8+/BKleZk88sGrmFOSMy37EZHkYGbbnXPrJ90u4QN9VE8jvPwN2P4Q\n9J6BnDJY9fuw5t0wa8WU727nqS7uenArWekpPPLBq1hUkT/l+xCR5KBAn0goCMeehh3fgkM/h/AI\nVCyDlW+HFW+H4nlTtquDTT2892tbCYXDfO2uK1k3r3jKvltEkseUBbqZPQjcAbQ451b4r5UA3wVq\ngRPAO5xznZPtbEYE+nj97bDvB96FSac2e69Vr4Xlb4Vld3onWi/RibZ+3v/1rTR2D/Gld67hthVV\nk39IRGScqQz0NwB9wDfGBfq/AR3Ouc+a2V8Bxc65T062sxkX6ON11sO+H8L+H8GZHd5rs1bC0ju8\nW+Vyb9bHi9DeN8wHv7GNnae6+NvfWsYHXj9/CgsXkUQ3pV0uZlYL/HRcoB8CbnDONZpZFfCMc+6y\nyb5nRgf6eB3HvZOoB5+AU1sAB4Vz4bLbYMltUPt6SMu8oK8cGgnxse/s4Jf7mnn3VXP51G8vJyMt\n8QYZicjUm+5A73LOFfmPDegcff5a4ibQx+tt9mZ3PPQLqHsGgoOQngsLrodFb4TFt0DR3Ii+KhR2\nfP7JQ9z3zDE21JZw33vXUpZ3Yf8wiEjyiVqg+887nXPnPeNnZvcA9wDMnTt3XX19fUQHMCONDMLx\nTXDkSTj8JHSf9F4vXQyLbvaGRNZeCxm5r/k1P955mr/8/m7K8jL5ynvXsmr2pP8WikgSU5fLdHMO\n2g7D0V/B0aeh/jcQHIKUdJh7tdeCn38DVF8BqWmv+viehm4+/K3ttPYO83e/vYz3XDUXu8g+ehFJ\nbNMd6P8OtI87KVrinPvLyb4noQL9XCODcPJFOLYR6jZC0x7v9cwCmHctzH8DzL8OKpZDitd33tkf\n4BPf28kzh1q5c001//y2leRmvjr8RSS5TeUol28DNwBlQDPw98CPgO8Bc4F6vGGLky70mdCBfq7+\nNq975vizUPcsdB73Xs8u8bplaq+D2tcTLlvKfc/W8YWnDjOvNJcvvXONumBE5BV0YdFM090Ax5/z\npvk98Rx0+f3v2SUw73WcyFvNp3YV8WJ/FZ940zLuuW4BKSnqghERBfrM13USTjzv9b2f+M3ZFvyQ\nZbM1uIjGwiu48dY7qbj8WkjPjnGxIhJLCvR403MG6l/A1b9I96FnKeo9AkDI0kipXoPNvdo72Trn\nasgrj3GxIhJNCvQ4d6bpDI/+7/fJbd7KjTl1XBY6ioX8OdaL5/vhvgHmXAXlS72FPkQkISnQE4Bz\njke3nuSzPzuICw3z6SuD3Fl6itSGrXBys7dINngjaWrWeQE/ewPMXgfZmghMJFEo0BNIU/cQf//4\nXn65r5mls/L5p7euYP28Yuiog4aXvOkJTm2Flv3gwt6HypbA7Cth9nrvvvzy846HF5GZT4GegH6x\nt4lP/2QfZ7qH+N21s/nr25e+cuqA4V5vQY+GrdCw3bsfaPfeS8/xLnKqWeeFfM16KKyJzYGIyAVR\noCeogUCQ//r1Ub72XB1Zaan86c2Luet1teef6Ms5b/RMw3avJd/wknfBU3jEez+/ygv40ZCvWgNZ\nBdE9IBGZlAI9wR1t6eMzT+xn46FWaktz+OvbL+fWZZWTTx8wMgTNe6FhG5zeBqe3e103AJjXVVOz\nDmrWenPDz1pxwTNLisjUUqAniWcOtfBPTxzgaEsfV9YW81dvXsq6eSUX9iUDHXDmZa+75vR279bf\n6r2Xku6FevUVXsDXrIWyy9QfLxJFCvQkEgyF+e62U/zHr47Q2jvMrcsq+fNbL+OyWRe5jqlz3pWt\nZ/yAP7MDzuyE4R7v/fQcmLXKb8Vf4d1KFp6do0ZEppYCPQkNBII8+Pxx/vvZOvoDQe5YVc3H37iY\nheV5l/7l4TB0HPNa8Wd2eGHfuNubHx4gIx+qVkP1Gi/gq9ZAyQKFvMgUUKAnsa6BAPdvquOhF04w\nNBLiLaur+ehNi1hUcZEt9omEgt4Uwmde9lrwZ3Z4J11HL4DKLPBCvmq1Ql7kEijQhba+Ye7fVMc3\nX6xnKBji9hVV3HvjQpZXF07fTkMj0HIAGnd6Id+4E5r2joV8Rj5UrfLCfbRFX7pIV7qKvAYFupzV\n0R/ggefrePiFevqGg1y3uIyPXL+QaxaWRmdRjbMhv8sL+MZdXks+OOS9n57jLcg92pqvWu1NZ5Ca\nPv21icQBBbq8SvfgCI9sqefB50/Q1jfMipoCPvj6Bdy+sir6C1aPdteMtuSbdnt98iP93vupmVC5\nzAv3WX6LvnKZZp6UpKRAlwkNjYT4wcuneeD5Oo619lNZkMn7rqnlnVfOoTSWi1aHw9B+1A93vyXf\nuBuGurz3LdUbJ1+12uu2mbXKa9lna0EQSWwKdJlUOOx49kgrDzx3nOePtpGRmsIdq6t43zW1rJ5d\nODPWOHXOmzu+cddYK75xF/Q1jW1TNM8P+NV+180q7yrYmVC/yBRQoMsFOdLcyzc31/PY9gb6AyGW\nVxfwnqvm8ZY11eTNxHVO+1q8cG/yW/FNe7xhlaNyyvyQX+m35FdB6UKdfJW4FJVAN7MTQC8QAoKT\n7VCBPvP1Do3wo51neGRzPQebesnNSOWOVdW848o5rJ1bNDNa7RMZ7vVG1DTtHmvNtxwYm7smPQcq\nl/sh7wd9xTLIyIlt3SKTiGagr3fOtUWyvQI9fjjn2HGqi29vOckTexoZCIRYXJHH766bzVvX1DCr\nMCvWJUYmGIC2Q2Ot+NHbcLf3vqV4wybPhvxKqFwJ+ZWxrVtkHAW6TJm+4SBP7D7D97Y1sL2+kxSD\naxeV8bYrarh1+ayZ2SXzWkb75Zv2+K35vd7j7pNj2+RW+AG/wmvJV67wgl9z2EgMRCvQjwOdgAP+\nxzl3/3m2uQe4B2Du3Lnr6uvrL3p/Ensn2vr5wcsNPPbyaU53DZKVnsIbL6/kt1dXc/2ScrLS47iP\neqADmvd5s1GOtuTHd9mkZUHF5WOt+FkrvC6crGm8UEuE6AV6jXPutJlVAE8Bf+Kc2zTR9mqhJw7n\nHNvrO/nxzjM8saeRjv4A+Zlp3LKskttXVvH6xWXxHe6jggFvvHzTnlcG/WDH2DZFc8da8bNWePdF\n8zTFgUyZqI9yMbNPAX3Ouc9NtI0CPTGNhMK8eKydn+4+wy/2NtEzFCQvM40bl1bw5hWzuH5JObnx\n1i3zWpyD3iY/5Pd4XTbNe70x9KNLAGbkexdCnQ35lV7rPnMKJkqTpDPtgW5muUCKc67Xf/wU8Gnn\n3C8m+owCPfEFgmFerGvn53saeXJ/Mx39ATLSUnj9ojJuWVbJzUsrqCiIkxOqFyowAK0HxgK+aa/X\nhTN6AhaDkvleN03lSn/Ejd+an8mjhyTmohHoC4Af+k/TgEedc595rc8o0JNLMBRmW30nT+5r5sn9\nTTR0elPtrppdyM1LK7lxaTkrqgtJSUngMBs9ATu+b755n79KlP93LyN/LNwrl3ut+orLIXOKZ8eU\nuKULi2RGcc5xqLmXpw+08PSBZnac6sI5KMvL4A1Lyrl+STmvX1QW26kHomm4D1oPjgX86O1sax4o\nnj8W8JXLvVvxfPXNJyEFusxo7X3DPHekjY2HWnj2cCtdAyOYwYrqQq5dVMZ1i8tYN684MU6sRso5\n6D411mUzGvIdx8b65tNzvdb7aP985XLv4qicC1x2UOKKAl3iRijs2Hu6m02HW9l0pJUdJ7sIhh2Z\naSmsry3mmgWlXLOwjFWzC0lPTcLW6cigN3xytNtm9H6wc2ybgpqxVvxo0Jcu0hTECUKBLnGrbzjI\nS8c7eO5IGy/WtXOg0VvLNCcjlXXzirlqfgkb5peyanZhcrXgx3MOehuheT+07PNa9S37ofXQ2Lj5\n1AxvQe/KZX5L3g/8/Fk6CRtnFOiSMDr6A2yua2dzXTtb6jo41NwLQEZaCqtnF7K+toT184pZN6+Y\nopyMGFcbY6Pj5lv2v7JvvvfM2DbZxWPhXrnMe6whlTOaAl0SVkd/gG0nOnjpRAdbT3Sy73Q3wbD3\ne7ywPJe1c4u5Ym4xa+YUsaQyj7Rk7KY510DHK0O+Zb/XjRPoG9umuNYP+mVev3zlCm8NWE13EHMK\ndEkag4EQuxu62Fbfycv1new41UVHfwCA7PRUVs4uZM2cIlbWFLJ6dhFzSrJn9qyR0RIOQ1f9q1vz\n40/CpmZC+WVj/fMVfvdNXqW6baJIgS5JyzlHffsAO051sutUN7sauth3uodAyAupopx0VlQXsqKm\nkJU1hSyvLmBuSU5ij4e/ECODXl/8aNC37Pf66scvKpJd4gf85WMhr7Hz00aBLjJOIBjmcHMvuxu6\n2d3QxZ7T3Rxu7mUk5P3+52WmcXlVPsuqCrjcvy2pzCc7I0lPup5Pf7vfVbN/4m6bormv7LapWAZl\nizXa5hIp0EUmMRwMcbipj31nutnf2MO+Mz0caOxhIBACvB6F+aW5XDYr37tV5rNkVj7zSnLULz8q\nHPamHR4dbdPsB37bEXDenyMp6d5asKPj50cDv3COum0ipEAXuQjhsONU5wAHGns40NjLoaZeDjX3\ncqK9n9G/KhmpKSwoz2VxZT6LyvNYVOHdastyyExTix6A4LAX6mdb9P5996mxbTLy/S6by8d13yyH\n3NLY1T1DKdBFptBgIMTRlj4ONfdyuLmXoy19HGnp5VTH4NltUgzmlOSwoCyXBeV5LCjPZX5pLvPL\nc6nMz1IfPcBQN7QcHNeaP+A9Hn+RVG7FuC4bP+TLL0vqYZUKdJEoGAyEqGvr42hLH8da+jjW1k9d\naz91rX0MB8Nnt8tKT6G2NJd5pTnUluYytzSHeSXe86rCrOTuwnEO+ppf2ZJv2e8Ff3DsH8yzwypH\nW/UVy7yrYdMS/9oDBbpIDIXDjqaeIY639VPX1k99Wz8n2vs53tbPqY7BsyNuANJSjOqibOaUZDOn\nOIc5JTnMLs5mdrF3X56XmZyt+3AIOk/4rXj/RGzrwXP659OgdPG4/nm/VV9Um1CTmCnQRWao0bCv\nbx/gZEc/JzsGONkxyMmOARo6Bmj3x9CPykhNoaooi+rCbGqKs6kuyqa6MIvqomyqCrOoKsqOv3Vd\nL8XZ/vkD41rz+71pikel50D50nHdNn6LPk6nPVCgi8SpgUCQhs5BTncO0tA5QEPnIGe6hzjdOcDp\nrkFaeoc5969tfmYalYVZVBVmUVmQxayCLCoLMqksyKKiIIuK/EzK8zMTe3Kz4d6x8fOjk5m1HvS6\nc0ZlFb065Csun/GzVSrQRRLUSChMc88Qjd3+rWuQxu4hmrqHaOrx7lt6hwif81fbDEpyMij3w708\nP5PyPO++LM+/5WdQmptJcU564vTr97d7K0k17/fuW/zH4+eez5v1yoCvuNw/ETszLpRSoIsksVDY\n0d43THPPMM09Q7T0DtPS69239g579z1DtPUFXtGfP8oMinMyKMn1bqXj7ov9x0U5GZTkZFCUk05J\nbgY5GanxM6XC6GyVoydiWw91mv6TAAAHO0lEQVSe/0Rs0dxxIb/M68YpWwLp0V1GMdJAv6SONzO7\nDfgSkAp8zTn32Uv5PhGZGqkp5nW1FGSxksIJt3PO0TMUpLV3mPa+Ydr6ArT1DdPeH6C9b5j2vgAd\n/QGOtPTR0R+gcyDwqu6eUempRmG2F/BF2ekU5aRTmJ1BYXa6f0ujwH9ckJ1OQVY6+Vnea7nR/sfA\nDAqqvduiN469Hg7589uc06I/+vTYtMSWAiULoWLpWMhXLIPShTG/IvZS1hRNBQ4DtwANwEvAu5xz\n+yf6jFroIvEtFHb0DI7QMeAFfWd/gK4B73n34AhdAyN0jXvcPejd+oaDr/m9KeZNv5Dvh3x+Vhp5\nmWnkZaX7r6eRm5FGbmYqeZlp5GSmkZeZSk6Gt11Ohvc4JzOVnPTUqe8uCo1A+7Gx/vnRoO+oG5vI\nLCXdm+bgbJeNf19cCymXdsFZNFroG4Cjzrk6f4ffAe4EJgx0EYlvqSlGsd/tsrA88s8FQ2F6hoL0\nDI7QM+SFfM9gkN4h73nvUJDeoeDZx/3DQdr6ApxoH6BvOEjfUJDBkVDE+8tITSE7I5WcjFSyM1LJ\nTvceZ6V7j7MzUslKSyUrPYWs9FQy0/3Had42WekpZKalkpmWQmZ6ChmpKWSmzyKjpJrMylu952kp\nZBAgs/sY6W0HSW07hLXsh4aXYO9jY8WkZUP5ErjtX2HeNRfwp33hLiXQa4Bx1/HSAFx1aeWISCJK\nS0052x9/sYKhMP2BEP3DXuD3B0IMDAfpG/bCvn84xEAgyEAg5N+CDAZCDIyEGPJf6xv2upcGR0IM\njYQYGgkzOBIiEHz1eYQLU4DZlaSnXkVGagoFKUMsSTnDYmtgESdZ2HKKzLYwK+dd4m4mMe2DV83s\nHuAegLlz50737kQkQaWlplCYnUJh9tT3U4fDjuFgmKGREMPBMMNBL+wD4x+HQv5z7xYYvYXCjATD\njITCDIfCBEOOkVCYkdB8OkOOF0JhNoUc91YtnPK6z3UpgX4amDPu+Wz/tVdwzt0P3A9eH/ol7E9E\nZFqkpJjXNRPn0yVfypmDl4DFZjbfzDKAdwKPT01ZIiJyoS66he6cC5rZR4Ff4g1bfNA5t2/KKhMR\nkQtySX3ozrmfAT+bolpEROQSJMi1vSIiokAXEUkQCnQRkQShQBcRSRAKdBGRBBHV6XPNrBWov4CP\nlAFt01TOTKdjTz7JetygY5/s2Oc55yadPSeqgX6hzGxbJDOMJSIde/Ide7IeN+jYp+rY1eUiIpIg\nFOgiIglipgf6/bEuIIZ07MknWY8bdOxTYkb3oYuISORmegtdREQiNCMC3cxuM7NDZnbUzP7qPO9n\nmtl3/fe3mFlt9KucehEc95+Z2X4z221mT5vZNK93Ej2THfu47X7XzJyZJcwIiEiO3cze4f/s95nZ\no9GucbpE8Ds/18w2mtkO//f+9ljUOdXM7EEzazGzvRO8b2b2n/6fy24zW3tRO3LOxfSGN/XuMWAB\nkAHsApads829wH/7j98JfDfWdUfpuG8EcvzHH0mE44702P3t8oFNwGZgfazrjuLPfTGwAyj2n1fE\nuu4oHvv9wEf8x8uAE7Gue4qO/Q3AWmDvBO/fDvwcMOBqYMvF7GcmtNDPLjbtnAsAo4tNj3cn8LD/\n+PvAzWZmUaxxOkx63M65jc65Af/pZrxVoRJBJD9zgH8E/hUYimZx0yySY/8Q8GXnXCeAc64lyjVO\nl0iO3QEF/uNC4EwU65s2zrlNQMdrbHIn8A3n2QwUmVnVhe5nJgT6+RabrploG+dcEOgGSqNS3fSJ\n5LjHuxvvX/BEMOmx+//lnOOceyKahUVBJD/3JcASM/uNmW02s9uiVt30iuTYPwW818wa8NZa+JPo\nlBZzF5oH5zXti0TLpTOz9wLrgetjXUs0mFkK8AXg/TEuJVbS8LpdbsD7X9kmM1vpnOuKaVXR8S7g\nIefc583sGuCbZrbCOReOdWHxYCa00CNZbPrsNmaWhvdfsfaoVDd9Ilpk28zeCPwN8Bbn3HCUaptu\nkx17PrACeMbMTuD1KT6eICdGI/m5NwCPO+dGnHPHgcN4AR/vIjn2u4HvATjnXgSy8OY6SXQR5cFk\nZkKgR7LY9OPAXf7jtwO/dv6ZhDg26XGb2RXA/+CFeaL0o8Ikx+6c63bOlTnnap1ztXjnD97inNsW\nm3KnVCS/7z/Ca51jZmV4XTB10SxymkRy7CeBmwHM7HK8QG+NapWx8TjwPn+0y9VAt3Ou8YK/JdZn\nf8ed4T2Mdwb8b/zXPo33lxi8H+r/AkeBrcCCWNccpeP+FdAM7PRvj8e65mgd+znbPkOCjHKJ8Odu\neF1O+4E9wDtjXXMUj30Z8Bu8ETA7gVtjXfMUHfe3gUZgBO9/YHcDHwY+PO5n/mX/z2XPxf6+60pR\nEZEEMRO6XEREZAoo0EVEEoQCXUQkQSjQRUQShAJdRCRBKNBFRBKEAl1EJEEo0EVEEsT/BxP6iq7m\nfh0hAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X = np.arange(0.01, 0.99, 0.01)\n",
    "y_class_0 = np.array([loss_class_0(10, q) for q in X])\n",
    "y_class_1 = np.array([loss_class_1(10, q) for q in X])\n",
    "plt.plot(X, y_class_0)\n",
    "plt.plot(X, y_class_1)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The blue curve (loss for a misprediction of a class 0 instance) is much greater than the orange curve (loss for a misprediction of a class 1 instance) when the predicted probability is less than 0.2. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The whole concept of a \"class imbalance\"\n",
    "\n",
    "Much of what is written about \"class imbalance\" is not coherent due to people not thinking probabilistically. If a class really is rare, then the fact that your model predicts a probability of 0.05 that an example is a member of that class may be a perfectly reasonable estimate. \n",
    "\n",
    "Consider, again, a dataset S whose members have identical features. Suppose 90% are of class 0 and 10% are of class 1. Your model manages to make the perfect estimate, and always gives predicted probabilities of 0.9 and 0.1. But if you only measure by accuracy, it appears to you that it's always predicting class 0, and always gets instances of class 1 incorrect. This leads you to think that your model is not adequately learning instances of class 1 and that you need to do something about the \"class imbalance.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other thoughts on the paper\n",
    "\n",
    "- The particular problem solved is suspicious to me. It's not clear why focal loss would apply to this dense object detection problem moreso than a more straightforward classification problem with a \"class imbalance\". So why not include results on a straightforward classification problem also? After reading this paper I did try to apply an approach similar to focal loss on the Fashion MNIST classification. I couldn't get any better accuracy than with regular old cross entropy loss. (But I didn't spend massive effort trying and, as I said, it wasn't quite the same approach as this paper.)\n",
    "\n",
    "- The authors say, \"this scaling factor can automatically down-weight the contribution of easy examples during training and rapidly focus the model on hard examples.\" I understand why that might be one's intuition, but they provide no evidence that this is what's really happening. How do we know that's what's happening and not something like my Perverse Example B?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Some Conjectures\n",
    "\n",
    "Finally, I have a mathematical conjecture about the focal loss. This may not be of much interest to ML or AI people, but I can't help myself. \n",
    "\n",
    "It's known that cross entropy loss is uniquely optimized on a set like S in the examples above when the probability estimates are equal to the actual proportions.\n",
    "\n",
    "Suppose there are `n` classes in S. For each `γ > 0`, let `q(γ) = [q(γ)[0], q(γ)[1], ..., q(γ)[n-1]]` be the estimates that minimize the focal loss on S. \n",
    "\n",
    "I conjecture that for each index i, `q(γ)[i]` monotonically approaches `1/n` as `γ` approaches infinity. Moreover, it's strictly monotone as long as `q(0)[i]` is not already `1/n`.\n",
    "\n",
    "I'm reasonably confident this conjecture is true, but I haven't made any real attempt to prove it. Maybe it's easy. Or maybe there's a simple counterexample."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}