bhtucker/prediction_variance.ipynb

## prediction_variance.ipynb
{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finding Noisy Data Sources\n",
    "\n",
    "* Suppose you have a multiclass prediction problem with an intermediate category\n",
    "* That is, suppose there are 500 'tags' that comprise 17 'classes'\n",
    "* Some 'tags' might actually correspond to phenomena outside the observations themselves, while others are directly useful\n",
    "* The input dataset is high dimensional (>50k features)\n",
    "\n",
    "An underperforming tag likely:\n",
    "* has an error rate higher than the average for its class\n",
    "* highly varying predictions\n",
    "* low gap between its correct prediction and incorrect predictions (low prediction margin)\n",
    "\n",
    "\n",
    "Another tool to think with is the bias-variance decomposition of error:\n",
    "* expected prediction error ~ bias^2 + variance + irreducible error\n",
    "\n",
    "If we collect class prediction probabilities from a variety of models that we in general accept as appropriate to the problem and landing at different parts of the bias/variance tradeoff, we might be able to find high error groups of observations.\n",
    "\n",
    "In particular, irreducible error will be subsumed into empirical measures of the other terms:\n",
    "* high bias tags may be mislabeled in their final class\n",
    "* high variance tags may not contain signal in their feature space\n",
    "\n",
    "Below are simulated test set prediction probabilities for:\n",
    "* a 'good' tag, whose predictions consistently favor the true class\n",
    "* a 'bad' tag, whose predictions favor the true class, but with high variance\n",
    "* a 'weak' or 'wrong' tag, whose predictions favor an incorrect class ahead of the true one, but with low variance\n",
    "\n",
    "By measuring the magnitude of each tag's prediction covariance and prediction bias, we might be able to determine certain tags contain little signal and omit them from our training. If our application will be able to refuse to classify inputs it's uncertain of, this approach might help us better model the true class of high-signal inputs and be appropriately befuddled when low-signal inputs appear in production use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "n_classes = 6\n",
    "good_mean = np.array([0.01, 0.01, 0.02, 0.3, 0.05, 0.08])\n",
    "good_covariance = (\n",
    "    np.ones(shape=(n_classes, n_classes)) + \n",
    "    (np.eye(n_classes) * np.array([1,1,1,5,1,1]))\n",
    ") * .0002\n",
    "\n",
    "good_preds = np.random.multivariate_normal(good_mean, good_covariance, size=500)\n",
    "good_preds += np.abs(good_preds.min())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.0004,  0.0002,  0.0002,  0.0002,  0.0002,  0.0002],\n",
       "       [ 0.0002,  0.0004,  0.0002,  0.0002,  0.0002,  0.0002],\n",
       "       [ 0.0002,  0.0002,  0.0004,  0.0002,  0.0002,  0.0002],\n",
       "       [ 0.0002,  0.0002,  0.0002,  0.0012,  0.0002,  0.0002],\n",
       "       [ 0.0002,  0.0002,  0.0002,  0.0002,  0.0004,  0.0002],\n",
       "       [ 0.0002,  0.0002,  0.0002,  0.0002,  0.0002,  0.0004]])"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "good_covariance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x112cde750>"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeAAAAFVCAYAAAA30zxTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAF79JREFUeJzt3X9s3Pd93/HnkZRkS0dRss0u2SKrBdZ9sqHQplloHZOW\n3AXu0C4CXCuBf+THqtSAmwRbEGCtnAYOsGZB5wUe0HZwkWnpbDfVPDhuugpp3axdLJkM4G1dFnc/\n8pHtOBI7FAmtHyeSsmiRd/vjjjGjkLzzifd9H++eD0AQ7773vc+bR/Je9/l+P9/Pp1Sr1ZAkScUa\niC5AkqR+ZABLkhTAAJYkKYABLElSAANYkqQABrAkSQGG1tqYUhoAHgP2APPAAznnV5Zt/wTwi8B0\n464Hc86nOlSrJEk9Y80ABu4CNuecb0sp/RTwaOO+JX8f+GDO+RudKlCSpF7U7BD0GPAsQM75BWDf\nVdtvAX41pfR8SumhDtQnSVJPahbA24GLy24vNg5LL/kPwIPAPwDGU0r/aJ3rkySpJzU7BH0RGF52\neyDnXF12+zdyzhcBUkpfAfYCX1ntyRYWFmtDQ4Pt1ipJ0kZUWunOZgE8CRwEnk4p3Qq8uLQhpTQC\nvJhS+jvAJeq94C+s9WTnz196KwVLkrThjY4Or3h/aa3FGFJKJd4cBQ1wmPp533LO+WhK6T7gE9RH\nSP9pzvmfr1XE9PSMKz9IkvrK6Ojwij3gNQN4vRnAkqR+s1oAOxGHJEkBDGBJkgIYwJIkBTCAJUkK\nYABLkhTAAJYkKYABLElSAANYkqQABrAkSQEMYEmSAhjAkiQFMIAlhahWq1Sr1eYPlHqUASwpxMTE\nCSYnT0aXIYVpth6wJK272dkZnnnmKQD27t1HuVwOrkgqnj1gSQFWXJ1N6iv2gCUVrlwuc+jQvZRK\nJXu/6lulWq1WWGPT0zPFNSapqy0NwBoY8ECcetvo6PCKh3zsAUsKYfCq3/kXIElSAANYkqQABrAk\nSQEMYEmSAhjAkkI4FaX6nQEsKYRTUarfeRmSpMI5FaVkAEsKUaLAOYCkrmQASypcuVxmz569TkWp\nvmYASyrc7OwM3/zmnze+fr8hrL5kAEsKUHIqSvU9A1hS4VwNSXI1JElBXA1J/cLVkCR1FYNX/c6/\nAEmSAhjAkiQFMIAlSQpgAEuSFMBBWJLaNjl5komJE23tW6lcAGBkZEdb+4+PH2BsbH9b+0rdwB6w\npBCVSoVKpRJdhhTG64AlhXjkkc8AcOTIw8GVSJ212nXA9oAlSQpgAEuSFMAAliQpgAEsSVIAA1iS\npAAGsCRJAQxgSZICGMCSJAUwgCVJCmAAS5IUwACWJCmAASxJUgADWJKkAAawJEkBDGBJkgIMrbUx\npTQAPAbsAeaBB3LOr6zwuH8LnM05f7IjVUqS1GOa9YDvAjbnnG8DHgIevfoBKaUHgZ8AautfniRJ\nvalZAI8BzwLknF8A9i3fmFK6DfhJ4PNAqRMFSpLUi9Y8BA1sBy4uu72YUhrIOVdTSm8HPg38PHBP\nK43t3LmVoaHB9iqV1FM2baq/F4yODgdXIsVoFsAXgeV/HQM552rj6/cCNwF/BLwN2JpS+r855ydX\ne7Lz5y9dS62SesiVK4sATE/PBFciddZqHzKbBfAkcBB4OqV0K/Di0oac828BvwWQUvrHwDvXCl9J\nkvSmZgH8ZeDOlNJk4/bhlNJ9QDnnfPSqxzoIS5KkFq0ZwDnnGvCRq+4+tcLjnljPoiRJ6nVOxCFJ\nUgADWJKkAAawJEkBDGBJkgIYwJIkBTCAJUkKYABLkhTAAJYkKYABLElSAANYkqQABrAkSQEMYEmS\nAhjAkiQFMIAlSQpQqtWKW8Z3enrGNYOlLnPs2JNMTZ0uvN0zZ+pt3nzz7sLb3rVrN/ff/6HC21V/\nGh0dLq10/5rrAUvqfVNTp3n5pZfZdv3OQtutLQ4C8Fd/ebbQdudeP19oe9JqDGBJbLt+Jz/x4z8T\nXUYh/tdLX40uQQI8ByxJUggDWJKkAAawJEkBDGBJkgIYwJIkBTCAJUkKYABLkhTAAJYkKYABLElS\nAANYkqQATkUp9blK5QJzr5/vmyka514/T6UyGF2GZA9YkqQI9oClPjcysoNLM4t9tRjDyMiO6DIk\ne8CSJEUwgCVJCmAAS5IUwACWJCmAASxJUgADWJKkAAawJEkBDGBJkgIYwJIkBTCAJUkKYABLkhTA\nAJYkKYCLMUgKWY7wjSuvA7B50/WFtjv3+nngxkLblFZiAEt9bteu3SHtnjlzGoC3v6PoMLwx7HuW\nlivVarXCGpuenimuMUld7ZFHPgPAkSMPB1ciddbo6HBppfs9ByxJUgADWJKkAAawJEkBDGBJkgIY\nwJIkBTCAJUkKYABLkhTAAJYkKcCaM2GllAaAx4A9wDzwQM75lWXbDwFHgBrweznn3+xgrZIk9Yxm\nPeC7gM0559uAh4BHlzaklAaBXwfeDbwL+GhK6YZOFSpJUi9pFsBjwLMAOecXgH1LG3LOi8A7c84z\nwCgwCLzRoTolSeopzQJ4O3Bx2e3FxmFpAHLO1ZTS3cA3gK8Bl9a/REmSek+z1ZAuAsPLbg/knKvL\nH5Bz/v2U0peBx4EPNf5f0c6dWxkaGmyvUkk9ZdOm+nvB6Ohwk0dKvalZAE8CB4GnU0q3Ai8ubUgp\nbQeOA3fmnN9IKc0Bi2s92fnzdpAl1V25Un+7mJ6eCa5E6qzVPmQ2C+AvA3emlCYbtw+nlO4Dyjnn\noymlLwInU0pXgG8CX1yvgiVJ6mVrBnDOuQZ85Kq7Ty3bfhQ42oG6JEnqaU7EIUlSAANYkqQABrAk\nSQEMYEmSAhjAkiQFMIAlSQpgAEuSFMAAliQpgAEsSVIAA1iSpAAGsCRJAQxgSZICGMCSJAUo1Wq1\nwhqbnp4prjFJHTc5eZKJiRNt7XvmzGkAbr55d1v7j48fYGxsf1v7SkUaHR0urXR/s/WAJakjRkZG\nokuQQtkDliSpg1brAXsOWJKkAAawJEkBDGBJkgIYwJIkBTCAJUkKYABLkhTAAJYkKYABLClEtVql\nWq1GlyGFMYAlhZiYOMHk5MnoMqQwTkUpqXCzszM888xTAOzdu49yuRxckVQ8e8CSAqw4M5/UV+wB\nSypcuVzm0KF7KZVK9n7Vt1yMQVKIpQFYAwMeiFNvczlCSV3F4FW/8y9AkqQABrAkSQEMYEmSAhjA\nkiQFMIAlSQpgAEuSFMAAliQpgAEsSVIAA1iSpAAGsCRJAQxgSZICGMCSJAUwgCVJCmAAS5IUwACW\nJCmAASxJUgADWJKkAAawJEkBDGBJkgIYwJIkBTCAJUkKYABLkhTAAJYkKcDQWhtTSgPAY8AeYB54\nIOf8yrLt9wEfBxaAvwA+mnOuda5cSZJ6Q7Me8F3A5pzzbcBDwKNLG1JK1wOfAe7IOY8DI8B7OlWo\nJEm9pFkAjwHPAuScXwD2Ldt2GXhXzvly4/YQ8Pq6VyhJUg9qFsDbgYvLbi82DkuTc67lnKcBUkr/\nBNiWc/7TzpQpSVJvWfMcMPXwHV52eyDnXF260QjjfwX8TeBQs8Z27tzK0NBgO3VKktRTmgXwJHAQ\neDqldCvw4lXbP0/9UPTPtzL46vz5S20VKUnSRjU6Orzi/aVabfXcTCmVeHMUNMBh4BagDPz3xr+T\ny3b5jZzzH6z2fNPTM46QliT1ldHR4dJK968ZwOvNAJYk9ZvVAtiJOCRJCmAAS5IUwACWJCmAASxJ\nUgADWJLUE6rVKtVqtfkDu4QBLEnqCc8//xzPP/9cdBktazYRhyRJXW92doannvoiALfc8pOUy+Xg\nipqzByxJ2vDm5uaYn7/M/Pxl5uZmo8tpiT1gSdKGt21bmS1btnz/643AAJYkbXjlcpl77vkApVJp\nQxx+BqeilCT1iKUR0AMD3XV2dbWpKO0BS5J6QrcFbzMbq1pJknqEPWBJUteYnDzJxMSJtvatVC4A\nMDKy4y3vOz5+gLGx/W212y57wJKknlCpVKhUKtFltMxBWJKknvDII58B4MiRh4Mr+UGuByxJUhcx\ngCVJCmAAS5IUwACWJCmAASxJXWyjrXGr1hnAktTFJiZOMDl5MroMdYATcUhSl5qdneGZZ54CYO/e\nfRtmkQG1xh6wJHWtFS8fVY+wByxJXapcLnPo0L0baok9tc4AlqQuNj5+ILoEdYgBLEldbKMtsafW\nGcCS1GFRK/xAzCo/ao0frSSpi220FX7UOnvAktRhY2P72+6FdusKP7p29oAlSQpgAEuSFMAAliQp\ngAEsSVIAB2FJktbVsWNPMjV1uvB2z5ypt7k0cK0ou3bt5v77P/SW9zOAJUnramrqNC99+9tsuWG0\n0HYXNm0B4MyFmcLanD833fa+BrAkad1tuWGU3T/7vugyOu70Hz/d9r6eA5YkKYABLElSAANYkqQA\nBrAkSQEMYEmSAjgKWpK0riqVC8yfO3tNI4Q3ivlz36NSWmxrX3vAkiQFsAesllSrVQAGBvzMJmlt\nIyM7qNQG++Y64JGR4bb29d1ULZmYOMHk5MnoMiSpZ9gDVlOzszM888xTAOzdu49yuRxckSRtfAaw\nWlCiVouuoTiTkyeZmDjR1r6VygWgfgiuHePjBxgb29/WvpI2FgNYTZXLZfbs2UupVLL320SlUgHa\nD2BJ/cMAVlOzszN84xv/DYB77nl/z4fw2Nj+tnuhS8ugHTny8HqWJKkHGcBqQen7o6AlqRXz56YL\nvw544fU5AIau31ZYm/PnpmFHe6OgeyKAo87Z9c/5uhpQii5CCtVvi8xD+wvN79q1uwPVNHfm4jkA\nbn7724prdMdw299vTwTwtfCcXStKbNq0KboIKdTU1Gm+8+opRm/cWmi7122uH32au/iXhbY7ffZS\n2/u2E9rrYaOdAuqJAPacXWeVy2Xe+957HYSlvjd641buOfjO6DIK8R+Pfyu6hJ7XUgCnlAaAx4A9\nwDzwQM75lasesxX4z8CHc855vQvVtbuWQ/Xnz9cP7Xz968+3tX//HK6XpNa0OhPWXcDmnPNtwEPA\no8s3ppT2ASeBH6N+wlA9ZmZmhpmZmegyJKlntHoIegx4FiDn/EIjcJfbTD2kf3cda9M681C9JHWP\nVgN4O3Bx2e3FlNJAzrkKkHP+OkBKac0n2blzK0NDg+3U2TGbNtXrGR1tbxh5v/B1ao2vU+9a+tn2\nk02bBjfU7/JG+/trNYAvAsu/o++H71tx/nz7o+o65cqV+jqO09MeXl2Lr1NrfJ1619LPtp9cubK4\noX6Xu/Xvb7UPBK2eA54Efg4gpXQr8OL6lCVJUn9qtQf8ZeDOlNJk4/bhlNJ9QDnnfLQzpUlS96hU\nLnDu3KW+uTzne2cvcUPtQnQZPa2lAM4514CPXHX3qRUe99PrUZQkSb2uJybikKROGxnZwVBptq8m\n4ti23RkCO6nVc8CSJGkdGcCSJAUwgCVJCmAAS5IUwACWJCmAo6AlqUXTZ4u/Dnju0hUAtm0tdk3u\n6bOX2La90Cb7jgEsSS3YtWt3SLtnL5wG4Efe9o5C2922Pe577hcGsHrWsWNPMjV1utA2z5ypt7e0\nelSRdu3azf33f6jwdvtF1GvrSmS9ywBWz5qaOs2rp77FjYPF/ZpvrtbXKLn4ysuFtQlwdnGh0PYk\nXTsDWD3txsEhDu64IbqMjjt+4Vx0CZLeIgNYktQ1JidPMjFxoq19v/OdV4H2TgGNjx9gbGx/W+22\nywCWJPWEoaEhSqXoKlrXNQEcMWAG4gbNOGBGkn7Y2Nj+tnqis7Mz/PIv/1MAPvaxT1Aul9e7tHXX\nNQE8NXWa/NLLDF5X7Oob1cVBAF6eeq2wNhcvu8amJK2n2dk55ufnAZibmzWA36rB63ZQ/tE7osvo\nuNnvPNf2vv12pAA8WiCpuXK5zJYt1wGwbVv3hy90WQCruamp05x6JTM0srnQdquDiwB8+7VXC213\nofJGoe1J2pjK5TL33vuB73+9ERjAG9DQyGZ23P43ossoxIXn/1/b+1YqFzi3sNAXl+icXVigVvHU\nhvrb7bffEV3CW2IAS5J6wsDAxlpfyABWzxoZ2UHptdf6ZiKO7SPFDmCUdG021scFSZJ6hAEsSVKA\nrjkEXalcYPHyhWu6RGejWLx8gUqlvZe+UrnAQmX+mgYnbSQLlXkqmxxcJKn32AOWJClA1/SAR0Z2\nMH1xoW8m4hhpc8DMyMgOzl4531eXIbX7WkF9mb4iL0O61FiOcGvBozHPLi6wvdAWJV2rrglgab3t\n2rW78DYvNGYMe9vNxba9nZjvV1L7DGD1rIjpK5em6jxy5OHC25a0sXgOWJKkAAawJEkBDGBJkgJ0\n1TngiOuAqwuXARgYuq6wNuvrAd/U9v4LlTcKvw64Ol9fDWlgy2Ch7S5U3riWl0rqCpOTJ5mYONHW\nvte6FOj4+IG2FrhX53VNAEeN4Fz65b55V5Hv8je1/f2Gv043Fdz+TY7uVX8bHh6OLkEdUqrVaoU1\nNj09U1xjLXLUamt8nVrj66T1dvLk1yiVShtuqT29aXR0uLTS/V3TA5Yk/aDZ2Rm+9KWnANi7d9+G\nWWherTGAJalrlbhy5Up0EeoQA1iSulat8U+9yACWpK5VYmjIt+le5U9WkrpUuVzmfe+7n1Kp5Pnf\nHmQAS1IXGx8/EF2COsQAlqQuNlDw0pYqjj9ZSZICGMCSJAUwgCVJCmAAS5IUwACWJCmAASxJUgAD\nWJKkAF4HLF3FxdMlFcEAltbRyMhIdAmSNggDWLrK2Nh+e6GSOs5zwJIkBTCAJUkK4CHoPuLgos6r\nVquAE+hLam7NAE4pDQCPAXuAeeCBnPMry7YfBB4GFoDfyTn/uw7WqkDDw8PRJWwIExMnKJVK3H77\nHdGlSOpypVqtturGlNLdwHtyzh9OKf0U8Mmc812NbZuA/wPsAy4Bk43Hfm+155uenlm9sWuwHj27\nm2/e/Zb37ZdeHcBzz/0ZAHfc8e7gSrrX7OwMn/rUPwPgs5991AXUJQEwOjpcWun+ZsfJxoBnAXLO\nL1AP2yV/G3g551zJOV8BJoANl0YjIyNeOtLE7OwMx449wbFjTzA7OxtdThdb8W9MklbU7BzwduDi\nstuLKaWBnHO1sa2ybNsMEJJkXjbSWdPT32VhYQGA1177rj27VZTLZQ4dupdSqeRrJKmpZgF8EVh+\n8m8pfKEevsu3DQPn13qy1brh6m4f/vCndwOvAvzarz38Y8ePHz8dXFLXevzxoyWAu+8+2JHTLZJ6\nR7MAngQOAk+nlG4FXly27VvAj6eUdgJz1A8/f64jVSpUI3Ad1tuC48ePG7ySWtJsEFaJN0dBAxwG\nbgHKOeejKaX3AJ+m/ub8hZzzb3e4XkmSesKaASxJkjrDw4qSJAUwgCVJCmAAS5IUwACWJClA3y/G\n0Jhi81/mnH86upZu1Zh29HeA3cAW4F/knI/HVtV9UkqDwFHgbwE14Jdyzv87tqrulVL6EeDPgXfn\nnE9F19OtUkr/gzcnPfp2zvkXI+vpVimlT1K/bHYT8G9yzk8El9RUXwdwSulXgA8Azq+4tvcD0znn\nDzau+/6fgAH8w94DVHPO4ymlA8BngbuCa+pKjQ91n6c+h4BWkVK6DsAOwtpSSncA78o535ZS2gb8\nSnBJLen3Q9AvA3fjJL7NPE39em+o/84sBNbStXLO/wl4sHHzR2kyM1yf+xzw28BfRRfS5f4usDWl\n9CcppT9rHLHTD/sZ4C9SSn9AvXPwh8H1tKSvAzjn/PsYJk3lnOdyzrMppWHqYfyp6Jq6Vc55MaX0\nOPCbwLHgcrpSSukXqB9R+WrjLj8Ar24O+FzO+R8CvwT8XmOZWP2gUeqTRL2XxusUW05r/EGqJSml\nXcB/AZ7MOT8VXU83yzn/AvXzwEdTStcHl9ONDgN3ppS+Bvw94ImU0l8LrqlbnaIRJjnnl4CzwNtD\nK+pOrwFfzTkvNMYTXE4p3RRdVDN9fQ5YrWm8OX4V+GjO+WvR9XSrlNIHgXfknH8deB2oNv5pmZzz\ngaWvGyH8YM75u4EldbPD1KcC/lhK6a9TX4XOw/Y/bAL4OPCvG6/TNuofVrqaAVznfJxr+1XqS01+\nOqW0dC74Z3POlwNr6kZfAh5PKZ2gPhLz4znn+eCatLF9Afj3KaWTjduHl61Ip4ac81dSSvtTSv+V\n+pHdj+acu/593bmgJUkK4DlgSZICGMCSJAUwgCVJCmAAS5IUwACWJCmAASxJUgADWJKkAP8f1lwi\nvDmPodEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x112cdeed0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.boxplot(good_preds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.28453946924024442"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# np.linalg.norm(np.cov(good_preds), ord='fro')\n",
    "sample_cov = np.cov(good_preds)\n",
    "np.mean(np.apply_along_axis(np.linalg.norm, 1, sample_cov))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.012807878787833685"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.linalg.norm(np.cov(good_preds), ord='fro') / float(len(good_preds))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "bad_mean = np.array([0.1, 0.1, 0.2, 0.3, 0.15, 0.08])\n",
    "bad_covariance = (\n",
    "    np.ones(shape=(n_classes, n_classes)) + \n",
    "    (np.eye(n_classes) * 5)\n",
    ") * .001\n",
    "\n",
    "bad_preds = np.random.multivariate_normal(bad_mean, bad_covariance, size=500)\n",
    "bad_preds += np.abs(bad_preds.min())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x114783810>"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeAAAAFVCAYAAAA30zxTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAG99JREFUeJzt3X+Q3Hd93/Hn3g/JllY6yfYFu0EY3JIP7TAmilVsdCfJ\nNDUdWjQxyBn/CLgITIjJNEz5IzJ0zD9uS1zGTAOpM6CQAkP8o7JQWpXiukmw5Fs3Zur+cNzWH8kS\nSIIhyVm6W92e7OPu9vrH7pntId2eVrv7+e7u8zHj8X33u9/7vG93ta/9fD+f/XxzCwsLSJKk9upL\nXYAkSb3IAJYkKQEDWJKkBAxgSZISMIAlSUrAAJYkKYGB5XaGEPqAh4HrgRngnhjjseq+NwCP1dz9\nF4E9McavtKhWSZK6xrIBDNwKrIoxbg0h3Ag8VL2NGONfAe8GCCG8C3gA2NvCWiVJ6hr1TkGPAE8C\nxBifA7YsvUMIIQd8Ebg3xuiqHpIkrUC9AF4PnK3Znq+elq61E3gxxni0qZVJktTF6p2CPgusq9nu\nizGWl9zn14B/vZLG5ubmFwYG+i+iPEmSOl7ufDfWC+AClR7uvhDCTcAL57nPlhjjf11JBRMT51Zy\nN0mSusbw8Lrz3l4vgA8At4QQCtXt3SGEO4F8jHFvCGEYKDavTEmSekOunVdDGh+fcpKWJKmnDA+v\nO+8paBfikJREuVymXF46pUTqHQawpCTGxg5RKBxOXYaUTL0xYElqulJpiv37Kwvpbd68hXw+n7gi\nqf3sAUtK4LxDYlJPsQcsqe3y+Twf+MDtr/8s9SJ7wJKSaOc3MKQssgcsqe1KpSkOHPh3ANxwwzvt\nBasn2QOWlIBjwJI9YEltl8/n2bXrDnK5nL1f9SxXwpKUxOIiHH19nohTd7vQSlj2gCUlYfCq1/kv\nQFISLkWpXmcAS0rCpSjV6zwFLantXIpSMoAlJZHz9LN6ngEsqe3y+TzveMcvvf6z1IscA5bUdqXS\nFM8//z2ef/57lEql1OVISRjAktpuenqamZkZZmZmmJ42gNWbPAUtqe3Wrs2zevXq13+WepEBLKnt\n8vk8t9/+QZeiVE9zKUpJSbgUpXqFS1FKyhSDV73OAJbUsELhMGNjhxo6tlicBGBoaENDx4+O7mBk\nZHtDx0pZ4EdQSUkUi0WKxWLqMqRkHAOWlMSDDz4AwJ499yeuRGqtC40B2wOWJCkBA1iSpAQMYEmS\nEjCAJUlKwACWJCkBA1iSpAQMYEmSEjCAJUlKwACWJCkBA1iSpAQMYKmJyuXy65fZk6TlLHs1pBBC\nH/AwcD0wA9wTYzxWs//vAg8BOeBHwN0xxp+0rlwp28bGDpHL5di27ebUpUjKuHo94FuBVTHGrcB9\nVMIWgBBCDvgK8OEY4zbgT4G3tKpQKetKpSn273+MJ554lFKplLocSRlXL4BHgCcBYozPAVtq9v0C\ncBr4VAjhaWBDjDG2okipM5z3gieSdF71Ang9cLZme756WhrgKmAr8CXg7wO/HEJ4d/NLlDpDPp9n\n1647uO22O8nn86nLkZRxy44BUwnfdTXbfTHGxRkmp4GXF3u9IYQnqfSQv3uhX7Zx4xoGBvovoVwp\n297//vcB0Nfn/MZ6Bgcr7wXDw+vq3FPqTvUCuADsBPaFEG4CXqjZdxzIhxD+ZnVi1jbgD5b7ZRMT\n5y6lVkldZHZ2HoDx8anElUitdaEPmfUC+ABwSwihUN3eHUK4E8jHGPeGED4KPFKdkFWIMX6naRVL\nktTFlg3gGOMCcO+Sm4/U7P8ucGML6pIkqas5UCVJUgIGsCRJCRjAkiQlYABLkpSAASxJUgIGsCRJ\nCRjAkiQlYABLkpSAASxJUgIGsCRJCRjAkiQlYABLkpSAASw1Ublcplwu17+jpJ5nAEtNNDZ2iELh\ncOoyJHWAetcDlrRCpdIU+/c/BsDmzVvI5/OJK1I3WDyj0tdnf6nb+IxKTZNLXYC6kGdVupc9YKlJ\n8vk8v/Irt5HL5ez9qik8q9Ld7AFLTXT8+MscO3Y0dRnqGp5V6Wb2gKUm+cu//DHPPvsMADt3vp83\nvOHqxBWp0+XzeXbtusOzKl3KAJaaJJezt6LmGx3dkboEtYgBLDXJ2rVrX5+punatvRVJy3MMWGqS\n6enp1xfimJ4upS5HXcJZ0N3LHrDUJGvX5lm9evXrP0uXylnQ3c0Alpokn89z++0fdMKMmsh5Bd3M\nAJaWKBQOMzZ2qKFjJybOAPDnf15o6PjR0R2MjGxv6Fh1H2dBdzcDWGqiqakpADZuvCJxJeoWW7du\nS12CWsQAlpYYGdnecC/0wQcfAGDPnvubWZJ62LPPPkMul2PbtptTl6ImM4AlKaNKpSn27XsEcBJW\nNzKAJSmzcszNzaUuQi1iAEtSZi3gTOjuZQBLUmblGBwcTF2EWsQAlqSMyufz3HabX0PqVgawJGWY\nF2PoXgawJGXY4gU+1H0MYElqsUtZXa1YnARgaGhDQ8e7ulp2+dFKkjKsWCxSLBZTl6EWWLYHHELo\nAx4GrgdmgHtijMdq9v9T4KPAePWmj8cYj7SoVknqSK6upvOpdwr6VmBVjHFrCOFG4KHqbYt+CfhQ\njPF/tKpASZK6Ub0AHgGeBIgxPhdC2LJk/w3AZ0IIVwPfjjH+TgtqlNRCjzzyDU6dOtH2dk+erLS5\n2MNrp02bruWuu+5ue7tSrXoBvB44W7M9H0LoizGWq9uPAv8GmAIOhBD+UYzx2y2oU1KLnDp1gpeP\nvszayze2td2F+X4AfvzD021td/rViba2J11IvQA+C6yr2a4NX4DfjTGeBQghfBvYDFwwgDduXMPA\nQH+jtUqZNzhYeX0PD6+rc8/sGBzsZ+3lG3n7W9+TupS2ePHoUwwO9nfMc9SJrymtTL0ALgA7gX0h\nhJuAFxZ3hBCGgBdCCH8HOAf8PeCry/2yiYlzl1atlHGzs/MAjI9PJa5k5RZr7iWzs/Md8xx14mtK\n/78LfXiqF8AHgFtCCIXq9u4Qwp1APsa4N4RwH/BdKjOk/yTG+GSzCpYkqZstG8AxxgXg3iU3H6nZ\n/yiVcWBJknQRXIhDkqQEDGBJkhIwgCVJSsAAliQpAQNYkqQEDGBJUlcol8uUy+X6d8wIA1iS1BXG\nxg5RKBxOXcaK1VuIQ13Ei4JL6lal0hT79z8GwObNW8jn84krqs8esFbEi4JLyrZc6gIuWs/3gBfH\nC/r6uv+ziBcFl9St8vk8u3bdQS6X64jeLxjAjI0dIpfLsW3bzalLkSRdgtHRHalLuCg9HcCdOGYg\nSTq/TjuT2VnVNl3njRlIkrpDT/eAO3HMQJLUHXo6gKHzxgwkSd2h5wO408YMpGYrFieZfnWCF48+\nlbqUtph+dYJisT91GZIBLEnKjlQLBqVYLMgAlnrc0NAGzk3N8/a3vid1KW3x4tGnGl7RTdm2uFhQ\npzy/BrAkKTN6acEgB0AlSUrAAJYkKYGuOAXdS4P2kqTu0PM9YK/yI0lKoSt6wL00aC9J6g493wOW\nJCkFA1iSpAQMYEmSEjCAJUlKwACWJCkBA1iSpAQMYEmSEjCAJUlKwACWJCkBA1iSpAQMYEmSEjCA\nJUlKYNmLMYQQ+oCHgeuBGeCeGOOx89zvK8DpGOOnW1KlJEldpt7VkG4FVsUYt4YQbgQeqt72uhDC\nx4G3A0+3pEJJLTf96gQvHn2qrW3+ZPZVAFYNXt7WdqdfnQCubGub0vnUC+AR4EmAGONzIYQttTtD\nCFuBdwJfBt7WkgoltdSmTdcmaffkyRMAXPPGdofhlcn+ZqlWvQBeD5yt2Z4PIfTFGMshhGuAzwLv\nB25fSWMbN65hYKC/sUpbZHCwUs/w8LrElWRbJz5Oe/fu5fjx421t89SpSqh84Qv/sq3tAlx33XV8\n7GMfu+jjPvnJ32xBNfV9+tOVEavPfe5zSdrvFJ34by+VTnus6gXwWaD2L+mLMZarP98GXAX8J+Bq\nYE0I4f/GGL9xoV82MXHuUmptidnZeQDGx6cSV5Jtnfg4xXiU7x95iSv7673Mm2ewXPnncfql2LY2\nAU7PzzE7O99Rz08nvqZS8HFauaw+Vhf6QFDvnakA7AT2hRBuAl5Y3BFj/BLwJYAQwj8G3rZc+Eop\nXNk/wM4NV6Quo+UOTp5JXYKki1QvgA8At4QQCtXt3SGEO4F8jHHvkvsuNL06SZK61LIBHGNcAO5d\ncvOR89zv680sSpKkbudCHJIkJWAAS5KUgAEsSVICBrAkSQkYwJIkJWAAS5KUgAEsSVICBrAkSQkY\nwJIkJWAAS5KUgAEsSVICBrAkSQkYwJIkJWAAS5KUgAEsSVICy14PWOpkxeIkZ+bmODh5JnUpLXd6\nbo6F4mTqMrraI498g1OnTrS93ZMnK20++OADbW9706Zrueuuu9vebq8wgCVpBU6dOsEPvn+E4SvX\ntLXdy1aVAZg++8O2tjt++lxb2+tFBrC61tDQBnKvvMLODVekLqXlDk6eYf3QhtRldL3hK9dw+863\npS6jLR4/+FLqErqeY8CSJCVgAEuSlIABLElSAgawJEkJGMCSJCVgAEuSlIABLElSAgawJEkJGMCS\nJCVgAEuSlIABLElSAgawJEkJGMCSJCVgAEuSlICXI+wwXhRckrqDAdxhTp06wZFjkYGhVW1tt9w/\nD8DxV77f1nbnij9pa3uS1C6ZCeBe69ldSq9uYGgVG7b9fJMryqbJZ36UugRJaonMBPCpUyeIR1+m\n/7INbW23PN8PwMunXmlbm/OvTbatLUlSNi0bwCGEPuBh4HpgBrgnxnisZv8uYA+wAPxRjPGLl1JM\n/2UbyL/55kv5FR2h9IOnU5fQM07Pz3Fw8kzb2jtXLgOwpq+98xtPz8+xvq0tSrpU9XrAtwKrYoxb\nQwg3Ag9VbyOE0A98DrgBmAb+TwjhmzHG9r3bScvYtOnatrc5WR3SuPpN7W17PWn+XkmNqxfAI8CT\nADHG50IIWxZ3xBjnQwhvizGWQwhvAPoBZ8woM1LMnF6cS7Bnz/1tb1utVSxOcubMOR4/+FLqUtri\nr0+f44oFh8taqV4ArwfO1mzPhxD6YoxlgGr4fgD4PeA/AueW+2UbN65hYKD/vPsGB89/e7caHOxn\neHhdQ8f1mkYfqxQWn59OqTelTnus+vt7b9mE/v6+jnl+oPNeU/UC+CxQ+5e8Hr6LYozfCiEcAL4G\n3F39/3lNTFw4n2dn5+uU0l1mZ+cZH59q6Lhe0+hjlcLi89Mp9abUaY9VPr+eXPkst+98W+pS2uLx\ngy+xNr++Y54fyO5r6kIfCOp9pCsA/xAghHAT8MLijhDC+hDCoRDCqhjjApVx4N5LB0mSGlCvB3wA\nuCWEUKhu7w4h3AnkY4x7QwjfBA6HEGaB/wV8s4W1SpLUNZYN4GrP9t4lNx+p2b8X2NuCuiRJ6mqZ\nWYhDktQdXNlwZQxgSVJTnTp1gqPHj7P6iuG2tjs3uBqAk5Ptm4Q1c2a84WMNYElS062+Yphr3/ur\nqctouRPf2dfwsb33xTZJkjLAAJYkKQEDWJKkBAxgSZISMIAlSUrAAJYkKQEDWJKkBPwesKSGFQqH\nGRs71NCxl7pq0ejoDkZGtjd0rJQFBrCkJIaGhlKXICWVmQAuFieZf22S0g+eTl1Ky82/Nkmx2NhD\nXyxOMlecYfKZHzW5qmyaK85QHJxMXYYuYGRku71QqUGOAUuSlEBmesBDQxsYPztH/s03py6l5Uo/\neJqhoQ0NHTs0tIHTsxNs2PbzTa4qmyaf+VHDj5UkZZk9YEmSEjCAJUlKwACWJCkBA1iSpAQyMwlL\nUm8pl8sA9PV1Tj9g/PQ5Hj/4UlvbnD43C8DaNYNtbXf89DnWrm9rkz3HAJaUxNjYIXK5HNu23Zy6\nlBXZtOnaJO2enqysGPZzV7+xre2uXZ/ub+4VBrCktiuVpti//zEANm/eQj6fT1xRfXfddXeSdheX\n6tyz5/4k7at1Oufcj6QukktdgJScPWBJbZfP59m16w5yuVxH9H6lVjCAJSUxOrojdQlSUgawpCQ6\nafaz1AoGsCSpqYrFSWbOnObEd/alLqXlZs78NcXcfEPH+hFUkqQE7AFLkppqaGgDxYV+rn3vr6Yu\npeVOfGcfQ0PrGjrWHrAkSQkYwJIkJWAAS5KUQKbGgOdfm6T0g6fb2mZ57jUA+gYua1ub869NAle1\nrT1JUvZkJoBTLfp98mRlofM3bWpnIF7lIueS1OMyE8AudC5J6iXLBnAIoQ94GLgemAHuiTEeq9l/\nJ/BJYA74C+ATMcaF1pUrqVt04vWApWaq98q/FVgVY9wK3Ac8tLgjhHA58ABwc4xxFBgC3teqQiV1\nl7GxQxQKh1OXISVTL4BHgCcBYozPAVtq9r0GvCvG+Fp1ewB4tekVSuo6i9cDfuKJRymVSqnLkZKo\nNwa8Hjhbsz0fQuiLMZarp5rHAUII/wRYG2P8kxbVqRpzxZ8w+cyP2tpmeaay1mnf6v62tjtX/IkT\nxruS1wOW6gXwWaB2ja2+GGN5caM6RvyvgL8F7KrX2MaNaxgYaO8beD2Dg5V6hocbW0qs3UJ46+s1\nt9Px48cBuO6a69rb8DVw3XXXdczz02mvp1SGh9dx9913k8vleMtbrkldTqZ14msqxXtUSoOD/Q09\nP/UCuADsBPaFEG4CXliy/8tUTkW/fyWTryYmzl10ga02O1vp2Y2PTyWuZGVuvfWOJO0uzhb/1Kc+\nk6T9Tnl+Ou31lFKpNAP4WNXTia+pxZp7xezs/LLPz4XCuV4AHwBuCSEUqtu7qzOf88B/Az4CHAb+\nLIQA8Lsxxj++uNIl9ZpSaYpvfetxAG644Z3k8/nEFUntt2wAV3u19y65+UjNz711nkFSkzgGLGVm\nIQ5JvSOfz7Nr1x3kcjl7v+pZBrCkJEZHd6QuQUrKAJaUhCtgqdf5L0CSpAQMYEmSEjCAJUlKwDFg\nSVLTzZwZ58R39rW1zblXpwEYuHxt29qcOTMOGxpbpcwAliQ11aZN1yZp9+TZMwC86Zqr29fohnUN\n/70GsCSpqe666+4k7S4umbtnz/1J2r9YjgFLkpSAPWBpiULhMGNjhxo69uTJE8BPP4lfrNHRHYyM\nbG/oWEmdxQCWmmhoaCh1CZI6hAEsLTEyst1eqKSWcwxYkqQEDGCpicrlMuVyOXUZkjqAASw10djY\nIQqFw6nLkNQBHAOWmqRUmmL//scA2Lx5i9e5lbQse8BS0+RSFyCpg9gDlpokn8+za9cd5HI5e7+S\n6jKApSYaHd2RugRJHcIAlpqor89RHUkr47uFJEkJGMCSJCVgAEuSlIABLElSAgawJEkJGMCSJCVg\nAEuSlIABLElSAgaw1ERejlDSShnAUhN5OUJJK+VSlFKTeDlCXUihcJixsUMNHXvy5AkAHnzwgYaO\nHx3dwcjI9oaOVWsZwFLTeDlCNd/Q0FDqEtQiBrDUJF6OUBcyMrLdXqh+hgEsNZGXI5S0Ugaw1ERe\njlDSSq0ogEMIfcDDwPXADHBPjPHYkvusAf4L8JEYY2x2oZIkdZOVfly/FVgVY9wK3Ac8VLszhLAF\nOAy8BVhoaoWSJHWhlQbwCPAkQIzxOWDLkv2rqIS0PV9JklZgpQG8Hjhbsz1fPS0NQIzx2RjjD5ta\nmSRJXWylk7DOAutqtvtijBe93t7GjWsYGOi/2MNaanCwUs/w8Lo69+xtPk6Ssq7T3qdWGsAFYCew\nL4RwE/BCI41NTJxr5LCWmp2dB2B8fCpxJdnm4yQp67L6PnWhDwQrDeADwC0hhEJ1e3cI4U4gH2Pc\n24T6JEnqKSsK4BjjAnDvkpuPnOd+725GUZIkdTtXDZAkKQEDWJKkBAxgSZISMIAlSUrAAJYkKYGu\nuBpSoXCYsbFDDR178uQJAB588IGLPnZ0dIfX+JQkNaQrAvhSDA0NpS5BktSDuiKAR0a22xOV1JXK\n5cqqv15ruvv4jEpSho2NHaJQOJy6DLVAV/SAJakblUpT7N//GACbN28hn88nrkjNZA9YkjIrl7oA\ntZA9YEnKqHw+z65dd5DL5ez9diEDWJIybHR0R+oS1CI9H8DOMJSUZb43da+ef2adYShJSqGne8DO\nMJQkpdLjPWBnGEqS0ujpHnCvzTBMtWY2uG62JC3V0wEMzjBcKdfMltJwomj36vkA7qUXtWtmS51n\nbOwQuVyObdtuTl2KmqznA1iSssqJot2td7p/ktRxnCjazewBS1JG9dpE0V5jAEtShjlRtHsZwJKU\nYb00UbTX+MxKkpSAASxJUgIGsCRJCRjAkiQlYABLkpSAASxJUgIGsCRJCRjAkiQlkFtYWGhbY+Pj\nU+1rTJLUcZpx3fI3venaiz62ldcsHx5ed95FvV0JS5LUFTrtuuX2gCVJaqGGesAhhD7gYeB6YAa4\nJ8Z4rGb/TuB+YA74wxjjHzStYkmSuli9SVi3AqtijFuB+4CHFneEEAaBLwC3ADuAXw8h/FyrCm2V\ncrlMuVxOXYYkqcfUC+AR4EmAGONzwJaafX8beDnGWIwxzgJjQGtGsFtobOwQhcLh1GVIknpMvQBe\nD5yt2Z6vnpZe3Fes2TcFdNQIeKk0xf79j/HEE49SKpVSlyNJ6iH1ZkGfBdbVbPfFGBfP1xaX7FsH\nTCz3yy40EJ3KRz5y1wYgAvzWb/16OHjw4GTikiRJPaJeABeAncC+EMJNwAs1+14C3hpC2AhMUzn9\n/PmWVNki1cB9Q+o6JEm9Z9mvIYUQcvx0FjTAbuAGIB9j3BtCeB/wWSqnsr8aY/z9FtcrSVJXaOv3\ngCVJUoVrQUuSlIABLElSAgawJEkJGMCSJCXQ81dDCiHcCPxOjPHdqWvJquqyo38IXAusBv55jPFg\n2qqyJ4TQD+wFfgFYAH4jxvi/01aVXdWla58HfjnGeCR1PVkVQvjv/HTRo+Mxxo+mrCerQgifpvK1\n2UHg92KMX09cUl09HcAhhN8GPgi4DNbyfg0YjzF+qPq97/8JGMA/631AOcY4GkLYAfwLKuupa4nq\nh7ovU1lDQBcQQrgMwA7C8kIINwPvijFuDSGsBX47cUkr0uunoF8GPgBkaoWuDNpH5fveUHnNzCWs\nJbNijP8e+Hh1883UWRmux30e+H3gx6kLybh3AGtCCP85hPCn1TN2+lnvAf4ihPDHVDoH/yFxPSvS\n0wEcY/wWhkldMcbpGGMphLCOShj/s9Q1ZVWMcT6E8DXgi8AjicvJpBDCh6mcUXmqepMfgC9sGvh8\njPEfAL8B/FHNevz6qWEqi0TdRvVxSlvOyvhEakVCCJuAPwO+EWN8LHU9WRZj/DCVceC9IYTLE5eT\nRbuBW0II3wV+Efh6CMElYc/vCNUwiTEeBU4D1yStKJteAZ6KMc5V5xO8FkK4KnVR9fT0GLBWpvrm\n+BTwiRjjd1PXk1UhhA8Bb4wxfg54FShX/1ONGOOOxZ+rIfzxGONfJSwpy3ZTWQr4N0MIf4PKVeg8\nbf+zxoBPAl+oPk5rqXxYyTQDuML1OJf3GSqXmvxsCGFxLPi9McbXEtaURU8AXwshHKIyE/OTMcaZ\nxDWps30V+LchhMWLlu+uuSKdqmKM3w4hbA8hfI/Kmd1PxBgz/77uWtCSJCXgGLAkSQkYwJIkJWAA\nS5KUgAEsSVICBrAkSQkYwJIkJWAAS5KUwP8D2ZSCYFNrQ4IAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x114783f90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.boxplot(bad_preds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.17324849127681441"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_cov = np.cov(bad_preds)\n",
    "np.mean(np.apply_along_axis(np.linalg.norm, 1, sample_cov))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.008204058148402249"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.linalg.norm(np.cov(bad_preds), ord='fro') / float(len(bad_preds))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def covariance_norm(preds, target_class=3):\n",
    "    sample_cov = np.cov(preds.T)\n",
    "    return np.linalg.norm(sample_cov, ord='fro')\n",
    "\n",
    "def target_mean_advantage(preds, target_class=3):\n",
    "    sample_mean = np.mean(preds, axis=0)\n",
    "    target_mean = sample_mean[target_class]\n",
    "    off_target_mean = np.delete(sample_mean, target_class)\n",
    "    return target_mean - off_target_mean.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfIAAAFVCAYAAAAUiG2GAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmUVdWdt/HnFlBVSFURlMIRR2A7MBhFZjUmbSdGjbZZ\nruBrTItjbDVZ2InJG7vb5E0npuOrnXS6TdJq1E7I0jaKeU23mihRmkFQQRGHjTigokSIyGgx3veP\nulXcW9M9VdxbeKqez1os797n7MOun4f63jPcczPZbBZJkpROFXt6ApIkqesMckmSUswglyQpxQxy\nSZJSzCCXJCnFDHJJklKsb0cLQwgVwC3AaGALcEmM8dW85Z8HvgFkgRkxxn8pNkaSJJVOsSPys4HK\nGOMk4JvATU0LQgh9gBuATwETgb8JIeyTG1PV1hhJklRaxYJ8MvAwQIxxATC2aUGMcQdwZIxxA1AP\n9AG25sY81NYYSZJUWsWCvA5Yn9fekTt1DkCMcWcI4RxgMfBHYFOxMZIkqXQ6vEZOYyDX5rUrYow7\n81eIMd4fQpgJ3Al8KcmYlrLZbDaTySSetCRJKVey0CsW5HOBM4F7QwgTgCVNC0IIdcCDwKkxxq0h\nhE3Ajo7GtCeTybB69YYu/gi9R319rXVKyFolY52Ss1bJWKdk6utri6+UULEgnwmcGkKYm2tPCyGc\nB9TEGG8NIfwKmB1C2AY8B/wqt17BmJLNVpIkFch8RL79LOs7uOJ8p5uctUrGOiVnrZKxTsnU19eW\n7NS6N6FJkpRiBrkkSSlmkEuSlGLFbnaT2vXhhx8S40u88cZrbNiwgerq/gwZsi9HHXU0Q4bsu6en\nJ0m9gkGuTlux4g3uuut2nnhiFlu3bm21PJPJMHr0sVx44SWMHTsOgJtv/idmzvxN8zrTpl3KRRdd\n1m1zlqSeyiBXYtlsljvuuJW77rqdnTvbf8ZPNpvluecWM336lZx55tlcc803Wq3jA4AkqTQMciWy\nc+dOvvvdf+DRRx/p1LgHH3yA9evX8bGPDSrTzCSpdzPIlci///stbYb4yJGj+PSnP8thhw2jqqqK\nVaveYd68OTz66O/Ztq3xtPsTT/yRfv0qu3vKktQrGOQqaunS5/n1r/+joK+yspKvfe1/c9ppZxT0\nH3nkUXziE5/iggum8e1vX8eyZS8DNIe6JKm0/PiZivr5z/+V/CcAVlRUcMMNN7UK8XxDhx7MT37y\nM4488qjumKIk9VoGuTq0bNnLPPvsooK+qVO/yLhxE4qO3WuvAVx//feorq4u1/QkqdczyNWhRx75\n74J2VVUVF1yQ/HtwDjpoaIdH7pKk3WOQq0MLFjxZ0J406URqamo6tY0zzjirlFOSJOUxyNWujRs3\nsmLF6wV9I0eO6vR2hg0bQXV1/1JNS5KUxyBXu956a0WrvsMPH9bp7VRUVHDEEZ0fJ0kqziBXu9as\nWd2qr66urkvbqqsbuLvTkSS1wSBXuz788MOCdiaToX//vbq0rc5eV5ckJWOQq119+hQ+LyibzbJl\ny5YubautL1eRJO0+g1ztaus0+ubNm7q0rU2bNu7udCRJbTDI1a599tmnVd8776zs0rb+9KdVuzsd\nSVIbDHK165BDDmt1TfyVV5Z1ejsffvghb7/9VqmmJUnKY5CrXX369OHoo48p6FuwYF6nt7Nw4fyC\nZ7VLkkrHIFeHPvnJUwvaK1a8wZIlz3ZqG7/97cxSTkmSlMcgV4f+8i9PY8CAAQV9//zPP2T79u2J\nxj/++GM89dSTxVeUJHWJQa4OVVdXM3XqFwv6li9/hW9/+7qiYb548TPccMP/Kef0JKnXM8hV1Be/\neCGHHnpYQd8TT8zi0ku/xNNPL2x1/Xvt2vf5+c//jWuuuYrNmzd351QlqdfpW3wV9ToNDVTfPaPx\n5dTz6VtdzXe+832uuupyNmxY37za8uWvMH36ley9994ceujhVFVVs2bNe7z66nJ27tzZvN6gQXvz\nwQdrCwI/k8l0388jST2YR+Qq1NDAwC+cQ+2106m9djoDv3AONDRw+OHDuPnmn1BTU9tqyPvvv8+i\nRU8zf/4cXnllWUGI77PPYP7lX37W6qi9X79+Zf9RJKk3MMhVoPruGVTOn9Pcrpw/p/no/Mgjj+aX\nv/xPTj75k0W3k8lkOPHET3Dbbf/BwQcf0mp5ZWVV6SYtSb2Yp9bVKYMHD+Yf//GfeP311/jjHx/l\nqacWsGbNGtau/TOVlVXU19dz7LHHceqpn2HkyNEAbV4n79+/urunLkk9kkGuAg1Tz6dq5n3NR+Vb\nJ06hYer5rdY77LDDOeywy7joosuKbrOt56zvs8/g3Z+sJMkgVwvV1ay75/6Cm92o3r2j59Wr32vV\nN3jwkN3apiSpkUGu1qqrabjwYgDuu+8e5s6dw8EHH8zQoQcTwlHNp8yTev755wraffv2bfVxNklS\n1xjkKuqpp55sfjrb4MH1zJz534nH7tixgwcf/G1B37BhI7xrXZJKxLvW1aFhw0YUtNesWc28eXPa\nWbu1GTPuYsWK1wv6PvWpU9tZW5LUWQa5OjRq1BgGDdq7oO8HP/hu0S9O2b59O7/4xb9z660/Lej/\n2McGccYZZ5d8npLUW3lqXR2qqKjgoosu46abftDct3bt+1x11WWMGzeBKVNO4tBDD2fAgBq2bdvK\nmjVreOmlF3j00d+zatU7rbb1ta99k5qamu7+MSSpxzLIVdRZZ53D4sXPMGvWH5r7stksCxbMZ8GC\n+Ym2UVFRwdVXT0/0MBlJUnIGuYrKZDJ8+9vf44ADDuTuu3+V+CtMmwwZsi/XXnsd48dPLNMMJan3\nMsiVSCaT4fLLr+S0087gvvvu4bHH/sD69etaPUM9f/1hw0Zw+umf44wzzqKqykeySlI5ZNr7RdzN\nsqtXb9jTc/jIq6+v5aNSp2w2y5tvrmD58lf44IO1bN68iYqKCgYMqOHAAw9k2LARrW6S604fpVp9\nlFmn5KxVMtYpmfr62pJ9BaRH5OqSTCbDIYccyiGHHLqnpyJJvZofP5MkKcU6PCIPIVQAtwCjgS3A\nJTHGV/OWnwd8FdgOPA/8TYwxG0JYBKzLrfZajPHickxekqTertip9bOByhjjpBDCeOCmXB8hhP7A\nd4GRMcaGEMKvgTNCCH8AiDGeUsZ5S5Ikip9anww8DBBjXACMzVvWAEyMMTbk2n2BD4ExwF4hhEdC\nCI/l3gBIkqQyKBbkdcD6vPaO3Ol2YozZGONqgBDC1cCAGOOjwCbgxhjjp4EvAzOaxkiSpNIqdmp9\nPVCb166IMe5sauQC+ofAMODzue5lwHKAGOMrIYQ/A/sDKzv6i+rraztarBzrlJy1SsY6JWetkrFO\n3atYkM8FzgTuDSFMAJa0WP5zGk+x/1WMsekD6dNovDnuyhDCATQe1b9bbCJ+7rA4P5+ZnLVKxjol\nZ62SsU7JlPLNTrEgnwmcGkKYm2tPy92pXgM8DVwEzAZmhRAAfgTcDtwRQpjdNCb/KF6SJJVOh0Ge\nO8q+okX3srzXfdoZesHuTEqSJCXjTWiSJKWYQS5JUooZ5JIkpZhBLklSihnkkiSlmEEuSVKKGeSS\nJKWYQS5JUooZ5JIkpZhBLklSihnkkiSlmEEuSVKKGeSSJKWYQS5JUooZ5JIkpZhBLklSihnkkiSl\nmEEuSVKKGeSSJKWYQS5JUooZ5JIkpZhBLklSihnkkiSlmEEuSVKKGeSSJKWYQS5JUooZ5JIkpZhB\nLklSihnkkiSlmEEuSVKKGeSSJKWYQS5JUooZ5JIkpZhBLklSihnkkiSlmEEuSVKKGeSSJKWYQS5J\nUooZ5JIkpVjfPT0BSeoprrrqMp57bnFz+4wzzuIb3/i7PTgj9QYekUtS2WT29ATUCxjkkiSlmEEu\nSVKKdXiNPIRQAdwCjAa2AJfEGF/NW34e8FVgO/A88Dc0nktqd4wkSSqdYkfkZwOVMcZJwDeBm5oW\nhBD6A98FPhFjnAIMBM7Ijalqa4wkSSqtYkE+GXgYIMa4ABibt6wBmBhjbMi1++b6JgMPtTNGkiSV\nULEgrwPW57V35E63E2PMxhhXA4QQrgYGxBj/0NEYSZJUWsU+R74eqM1rV8QYdzY1cgH9Q2AY8Pkk\nY9pTX19bbBVhnTrDWiVjnZIrVqvKysJfqf379+uV9e2NP/OeVCzI5wJnAveGECYAS1os/zmNp9P/\nKsaYTTimTatXb0g86d6qvr7WOiVkrZKxTsklqdXWrdsL2h9+uK3VmJdffpE33nidNWtW07//XgwZ\nMoQRI45k3333K/mc9wT3qWRK+WanWJDPBE4NIczNtafl7lSvAZ4GLgJmA7NCCAA/amtMyWYrSSmU\nzWaZOfM33HPPDN55Z2Wb6xx11DF87nNnc/rpZ5HJ+CAZJddhkOeOsq9o0b0s73Wfdoa2HCNJvdK6\ndR9w3XXXFjy6tS0vvfQCL730AjNn3sd1113P4YcP66YZKu28CU2SymTjxg1Mn35V0RDPt2zZy1x9\n9eUsW/ZyGWemnsQvTZGkMnn88ccK2vvvfwCnn/45Ro4cTW1tLe+99yfmz5/LI4/8N1u2bGleb/36\n9VxzzVX88pf3MmjQoO6etlLGI3JJ6gbnnjuVGTN+w1//9cUcf/wJjBhxJFOmnMzXv/4t7rrrbkI4\nqmD9devW8aMf/XAPzVZpYpBLUpl96UsX8ZWv/C39+vVrc/mBBx7Ej398C0ccMbygf9asR3n55Ze6\nY4pKMYNckspo1KgxXHpp8ft/Bwyo4brrrqeiovDX8gMP/KZcU1MPYZBLUplkMhm+/OWrEq8/fHhg\n0qQTC/oee+z3bNu2rdRTUw9ikEtSmQwdejCjRx/bqTGf+cxnC9oNDQ3E6Ol1tc8gl6QyOfbY4zo9\n5phjRrXqW7LkuVJMRz2UQS5JZTJ8eOj0mMGD66mtrSvoa+9pcBIY5JJUNoMG7d2lcQMHDixor1+/\nrhTTUQ9lkEtSmdTU1JRk3MaNG0sxHfVQBrkklcnOnUW/wbmdcdmCdnV1dSmmox7KIJekMsl/7Gpn\nbNpUeAReW+v3e6t9Brkklcnate93ekw2m201rq5uYDtrSwa5JJXNypVvd2nM5s2bC/qOOMKvNFX7\nDHJJKpOlS5d0esyzzz5T0M5kMowaNaZUU1IPZJBLUpm88MLznT69/l//9f8K2oMH13PggQeVclrq\nYQxySSqT7du3c9ddv0i8/pw5T7B06fMFfWef/flST0s9jEEuSWV0//3/yZw5TxRdb+XKt7nxxu8X\n9FVVVRnkKsogl6QyymazXH/9t3jood+1u87zzz/H1VdfzvvvF56Gv+KKq71jXUX13dMTkKTUa2ig\n+u4ZVKxaVdBdU1PLxo0b2Lp1K9///nd44IH7+Oxnz+Tggw+hqqqKd95ZyeOPP8bs2Y+TzRY+BGby\n5BP5/Oe/0J0/hVLKIJek3dHQwMAvnEPl/Dn0Pegg2GsvyGTIZDJcffV07rjjNlategeAF19cyosv\nLi26yYkTJ/Od79xQ7pmrh/DUuiTthuq7Z1A5f86ujmwWslmy2SyDBg3i5pt/wqGHHpZoWxUVFZx7\n7lRuuOEmqqqqyjRj9TQekUtSCWUAMpnGP8DQoQdz222/5M47b+N3v/stH3ywttWYyspKxo+fxLRp\nlzJ8+IjunbBSL9Pyuswekl29esOensNHXn19LdYpGWuVjHVKrt1a5Z1aB9g6cQrr7rkf2viik+3b\nt7NkybO8++47rF37PnV1AxkyZF/GjPk4/fv3L/eP0C3cp5Kpr6/NlGpbHpFL0u6ormbdPfdTffcM\nABqmnt9miAP07duX444b252zUy9gkEvS7qqupuHCi/f0LNRLebObJEkpZpBLkpRiBrkkSSlmkEuS\nlGIGuSRJKWaQS5KUYga5JEkpZpBLkpRiBrkkSSlmkEuSlGIGuSRJKWaQS5KUYga5JEkpZpBLkpRi\nBrkkSSlmkEuSlGIGuSRJKda3o4UhhArgFmA0sAW4JMb4aot19gL+AFwUY4y5vkXAutwqr8UYLy71\nxCVJUpEgB84GKmOMk0II44Gbcn0AhBDGAj8DDgCyub5qgBjjKWWZsSRJalbs1Ppk4GGAGOMCYGyL\n5ZU0BnvM6xsD7BVCeCSE8FjuDYAkSSqDYkfkdcD6vPaOEEJFjHEnQIxxHkAIIX/MJuDGGOPtIYTh\nwEMhhBFNY9pTX1/b6cn3RtYpOWuVjHVKzlolY526V7EgXw/k/x+pKBbIwDJgOUCM8ZUQwp+B/YGV\nHQ1avXpDkc2qvr7WOiVkrZKxTslZq2SsUzKlfLNT7NT6XOCzACGECcCSBNucRuO1dEIIB9B4VP/u\nbsxRkiS1o9gR+Uzg1BDC3Fx7WgjhPKAmxnhrO2NuB+4IIcxuGpPgKF6SJHVBh0EeY8wCV7ToXtbG\neqfkvd4OXFCS2UmSpA75QBhJklLMIJckKcUMckmSUswglyQpxQxySZJSzCCXJCnFDHJJklLMIJck\nKcUMckmSUswglyQpxQxySZJSzCCXJCnFDHJJklLMIJckKcUMckmSUswglyQpxQxySZJSzCCXJCnF\nDHJJklLMIJckKcUMckmSUswglyQpxQxySZJSzCCXJCnFDHJJklLMIJckKcUMckmSUswglyQpxQxy\nSZJSzCCXJCnFDHJJklLMIJckKcUMckmSUswglyQpxQxySZJSzCCXJCnFDHJJklLMIJckKcUMckmS\nUswglyQpxQxySZJSzCCXJCnFDHJJklKsb0cLQwgVwC3AaGALcEmM8dUW6+wF/AG4KMYYk4yRJEml\nUeyI/GygMsY4CfgmcFP+whDCWGA2cBiQTTJGkiSVTrEgnww8DBBjXACMbbG8ksbgjp0YI0mSSqRY\nkNcB6/PaO3KnzgGIMc6LMb7dmTGSJKl0OrxGTmMg1+a1K2KMO8swhvr62mKrCOvUGdYqGeuUnLVK\nxjp1r2JBPhc4E7g3hDABWJJgm10Zw+rVG5Ks1qvV19dap4SsVTLWKTlrlYx1SqaUb3aKBflM4NQQ\nwtxce1oI4TygJsZ4a9IxJZinJElqQyabzRZfq/yyvoMrzne6yVmrZKxTctYqGeuUTH19baZU2/Im\nNEmSUswglyQpxQxySZJSzCCXJCnFDHJJklLMIJckKcUMckmSUswglyQpxQxySZJSzCCXJCnFDHJJ\nklLMIJckKcUMckmSUswglyQpxQxySZJSzCCXJCnFDHJJklLMIJckKcUMckmSUswglyQpxQxySZJS\nzCCXJCnFDHJJklLMIJckKcUMckmSUswglyQpxQxySZJSzCCXJCnFDHJJklLMIJckKcUMckmSUswg\nlyQpxQxySZJSzCCXJCnFDHJJklLMIJckKcUMckmSUswglyQpxQxySZJSzCCXJCnFDHJJklLMIJck\nKcUMckmSUqxvRwtDCBXALcBoYAtwSYzx1bzlZwJ/D2wHfhFjvC3XvwhYl1vttRjjxWWYuyRJvV6H\nQQ6cDVTGGCeFEMYDN+X6CCH0A24GxgKbgbkhhN8CGwBijKeUbdbqlK9//as8+eS85vbhhx/BXXfd\nnWjs//zP43zrW18v6Lv88iv54hcvTDT+G9+Yzrx5c5rb5547la985W8TjZUkFVfs1Ppk4GGAGOMC\nGkO7yVHA8hjjuhjjNmAOcDIwBtgrhPBICOGx3BsA7UFTppxc0H799df485/XJBr71FMLWvUtWvR0\norFbtmzhmWeeam5nMhlOOsn3d5JUSsWCvA5Yn9fekTvd3rRsXd6yDcBAYBNwY4zx08CXgRl5Y7QH\nTJ58EplMprmdzWZZuPDJRGPbCvLnn3+O7du3Fx27aNFTbNmypbldVzeQMWM+nujvlSQlU+zU+nqg\nNq9dEWPcmXu9rsWyWmAtsAxYDhBjfCWE8Gdgf2BlR39RfX1tR4uV05U61dfXcswxx7B06dLmviVL\nnuFLXzqvw3ErV67k7bffatXf0NDAypWvMnbs2DZG7bJoUeGbgE9+8hSGDKnrxMx3j/tUMtYpOWuV\njHXqXsWCfC5wJnBvCGECsCRv2cvA8BDCIBqPwk8CbgSm0Xhz3JUhhANoPHJ/t9hEVq/e0PnZ9zL1\n9bVdrtP48ZMLgnzOnLlFt/XII7PaXTZr1mwOOSR0OH7WrD8WtMeNm9xt/593p1a9iXVKzlolY52S\nKeWbnWKnvGcCDSGEuTTe6DY9hHBeCOHS3HXxa4BHgHnA7THGd4HbgboQwmzgbmBa3lG89pCW18k/\n+GAty5a93OGYtk6rN8m/9t2W5ctf4b33/tTcrq6u5oQTJiSYqSSpMzo8Io8xZoErWnQvy1v+O+B3\nLcZsBy4o1QRVGsOGDWe//fZn1apdJ0cWLnySESOObHP9nTt38swzC5vbI0YcyZtvrqCh4UMAXnxx\nKVu2bKGqqqrN8fPnzylon3DChHbXlSR1nTeh9SKTJ59U0O7ohrdly15m/fpd9zmOGzeBo48+prm9\nbds2lix5tt3x8+b9T0H7xBNPbmdNSdLuMMh7kSlTCoN86dIlNDQ0tLluy9PqJ5wwnpEjRxf0tfcx\ntHXrPuCFF3Zdj+/Tpw+TJ5/YlSlLkoowyHuRY489jgEDaprb27ZtY/HitsM4/2i9qqqKUaPGJA7y\n+fPnks1mm9ujRo2hrm7g7kxdktQOg7wX6du3L+PHTyzoW7Cg9en1hoYGli7d9QGFkSNH069fP445\nZhQVFbt2mRhfYvPmza3G5z/JDeDEEz+xmzOXJLXHIO9lWp5ef+qp1kG+ePEzBQ98Of74cQDU1dUx\nfPiuj5zt2LGDZ59dVDB2+/btLFw4v7nd+DS3T5Ri6pKkNhjkvczEiVPo06dPc/vNN1fwpz+tKlin\nZbifcMK45tdjx44rWNbyY2hLljzLpk2bmttHHDGM/fbbf7fnLUlqm0Hey9TU1DBmzHEFfS3vXs+/\n0a2mppYQjmpuH3/8CQXrLlpUGOQtT6u3/Py6JKm0DPJeqOXp9fwgX7NmNW+88Xpz++MfP77gOe2j\nRx9Lv36Vze1XX13O+vW7Hrmf/7EzT6tLUvkZ5L1QyyB/5pmnmu8yf/rphQXLWh6BN93B3iSbzbJ4\n8TMArFz5Nm+99Wbzsn333a/gmrokqfQM8l5o//0P4PDDj2hub9iwnhdffAEo/EhZJpNpdU0cWl8n\nf/rpxtPrc+fOLuj3tLoklZ9B3ku1fspb453mTUfXAPvsM5hDDjm01dj8m99g13Xy1h87M8glqdwM\n8l6q9XXy+axatargWewtT6s3CeEoamt3fR3pW2+9yYoVb/Dcc4ub+wYOHMixxx7X1nBJUgkZ5L3U\n0UePZJ99Bje3X3zxBWbPLvza0faCPJPJcNxxxze3s9kst9zy44LPnk+cOKXg4TGSpPLwN20vNmnS\nlObXO3fuZMaMO5vb7V0fb9L0kJgmfuxMkvYMg7wXaxm277//fvPrgw4aSn39kHbHdhTyVVVVTJgw\nsd3lkqTSMch7soYGqu+8neo7b4c2vuVs7NhxVFdXtzm0o6AGGDr0YIYM2bfdsVVVbW9XklRaBnlP\n1dDAwC+cQ+2106m9djoDv3BOqzCvrKxk7NjxbQ5veeq8Le2FvV+SIkndxyDvoarvnkHl/F3XrSvn\nz6H67hmt1mt59zpARUUFxx03tujf0VaQV1RUtLlNSVJ59N3TE9CeNWnSiVRUVBR8f/jw4YHa2tqi\nY8eOHVfw+FZo/O7xgQM/VvJ5SpLaZpD3UA1Tz6dq5n3NR+VbJ06hYer5rdYbNGgQTzyxoFV/EoMG\n7c3s2QuLryhJKhuDvKeqrmbdPfc3n05vmHo+tHNjmyQpvQzynqy6moYLL97Ts5AklZE3u0mSlGIG\nuSRJKWaQS5KUYga5JEkpZpBLkpRiBrkkSSlmkEuSlGIGuSRJKWaQS5KUYga5JEkpZpBLkpRiBrkk\nSSlmkEuSlGIGuSRJKWaQS5KUYga5JEkpZpBLkpRiBrkkSSlmkEuSlGIGuSRJKWaQS5KUYn07WhhC\nqABuAUYDW4BLYoyv5i0/E/h7YDvwixjjbcXGSJKk0il2RH42UBljnAR8E7ipaUEIoR9wM3AqcDJw\nWQhhSG5MVVtjJElSaRUL8snAwwAxxgXA2LxlRwHLY4zrYozbgDnASbkxD7UzRpIklVCxIK8D1ue1\nd+ROnTctW5e3bAMwsMgYSZJUQh1eI6cxkGvz2hUxxp251+taLKsFPigypj2Z+vraIqsIwDolZ62S\nsU7JWatkrFP3KnakPBf4LEAIYQKwJG/Zy8DwEMKgEEIljafV5xUZI0mSSiiTzWbbXRhCyLDrDnSA\nacDxQE2M8dYQwhnAP9D4huD2GONP2xoTY1xWrh9AkqTerMMglyRJH23ehCZJUooZ5JIkpZhBLklS\nihnkkiSlWLHPke+WEEJ/4FdAPY0PjPnrGOOaFutcClxG4/Pa/zHG+F/tjQsh/BVwI/BWbvj1McbZ\n5fwZyqmUz7IPIQwD7gR2AkuBK2OMPeZOxhLX6uPAg8ArueE/jTH+Z/f9NOXTlTrlLRsP/CDGeEqu\n3WP3qRLXqcfuT9Dlf3v9gF8AhwBVNP5uf9B9KnGdOrVPlfuI/ArguRjjScB/AH+XvzCEsB9wNTAJ\n+DRwQ+4z6e2NOx64NsZ4Su5PakM8p5TPsr8Z+FauZhngrG77KbpHKWt1PHBz3n7UY37p0rU6EUK4\nFriVxl8mTXryPlXKOvXk/Qm6VqvzgdW5feczwL/mhrhPJatTp/apcgd587Pac//9ixbLxwFzY4zb\nYozrgeU0vptpb9zxwEUhhNkhhP8bQuhT1tmXXymfZX9c3hubh2hd67QrZa2OB04PITwRQrgthFDT\nTT9Dd+hKnaDx3945NP5ybdKT96mS1omeuz9B12p1L43PGIHGnNmWe+0+laxOnfodVbIgDyFcHEJ4\nPv8Pjc9eb3ruetOz2PPVUvx57fnjfg9clXv3UgN8uVTz30NK9Sz7PhT+YtlI61qnXSlrtQD4Wozx\nZOA14Pqyzbr7daVOxBjvp/GUX76evE+Vsk4L6bn7E3ShVjHGTTHGjSGEWuA37Dqr6j7VqFidOvU7\nqmRBHmOVHW7IAAABm0lEQVS8PcY4Kv8Phc9jb3oWe76Wz2Vv63nt+ePuiDG+kXv9W+DjpZr/HlKq\nZ9nvoPGaU8t1e5JS1uqBGOPiXN8DpH8/ytfZOq3tYFs9eZ8qZZ1m9uD9CbpYqxDCUGAWcFeM8e7c\ncvepRsXq1Kl9qtyn1pufuw6cBrS8pr0QODGEUBVCGEjj6YelHYx7NoRwYO71XwBPl2vi3aSUz7Jf\nHEI4Ofe6rVqnXSlr9VAI4YTc60+R/v0oX2frNL+DbfXkfaqUdXq4B+9P0IVahRD2pfEM6rUxxjvz\n1nefSlanTu1TZX1Ea+7u87uA/Wm8i+9/xRjfCyFMp/F6wYMhhEtovGu9AvhejHFmB+M+BXwPaKAx\n8L+aO8JKpVI+yz6EMJzGm3AqgReBS3vK3aBQ8lqNAf6NxutR7wKXxRg3duOPUzZdqVPe2EOBX+du\n1qEn71MlrlOP3Z+gy//2fgycC8S8TZ0GDMV9KkmdAp3Yp3zWuiRJKeYDYSRJSjGDXJKkFDPIJUlK\nMYNckqQUM8glSUoxg1ySpBQzyCVJSrH/D9kdjvfGlASNAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x116a40ed0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "pred_groups = [good_preds, bad_preds, weak_preds]\n",
    "\n",
    "x = map(covariance_norm, pred_groups)\n",
    "y = map(target_mean_advantage, pred_groups)\n",
    "plt.scatter(x, y, color='red')\n",
    "for label, x_, y_ in zip(['g', 'b', 'w'], x, y):\n",
    "    plt.annotate(label, xy= (x_, y_), size=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x11683f590>"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAe0AAAFVCAYAAADCLbfjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEwJJREFUeJzt3WGMZfV53/HfLrvrATNYVjWtbKmqKsV9mirFqotkBqyk\npKVps0Yi8QuWtm5Zg5qYKEqbFy1Km7xoKrUKIkoslTQi2G5lJ7tCNWqtyBDHtkS8XqFGToNfNA8G\nv6qF1KmbshvMkl2Yvpg7y2W8O/fskGH2z3w+ksWce+4Z/jwc87333MvhwPr6egCAq9/BvV4AADCN\naAPAIEQbAAYh2gAwCNEGgEGINgAM4tB2O6vqYJKHk9yY5JUk93X383P7P5LkXyZZT/LZ7v7EomMA\ngJ1Z9E77ziRHuvuWJA8keWhzR1Vdk+TfJfnbSVaT3F9Vf252zDsudQwAsHOLon1rkieSpLufTnLT\n5o7ufjXJX+3us0lWklyT5E9nx3zhUscAADu3KNo3JDkzt/3q7PJ3kqS7X6uqH0/yB0m+kuSlRccA\nADuz7Wfa2Yjv8tz2we5+bf4J3f25qno8yaeT/OMpx2y1vr6+fuDAgcmLBoDB7Sh6i6J9KskdSR6r\nqpuTPLO5o6puSPL5JLd3959W1UtJXt3umMuu/MCBrK2d3cn695WVlWVzmsispjGn6cxqGnOaZmVl\nefGTLmFRtB9PcntVnZptH6+qu5Nc392PVNVnkjxVVeeT/GGSz8ye94ZjdrQyAOANDlwl/5Wvda/M\nFvMKdjqzmsacpjOracxpmpWV5R1dHvcFMQAYhGgDwCBEGwAGIdoAMAjRBoBBiDYADEK0AWAQog0A\ngxBtABiEaAPAIEQbAAYh2gAwCNEGgEGINgAMQrQBYBCiDQCDEG0AGIRoA8AgRBsABiHaADAI0QaA\nQYg2AAxCtAFgEKINAIMQbQAYhGgDwCAO7fUCWOzcueTEicNZXk6OHk2WlvZ6RQDsBdG+yp07l9x1\n17U5fXrjb9Xq6rU5efJl4QbYh1wev8qdOHH4YrCT5PTpQzlx4vAergiAvSLaADAI0b7KHTt2Pqur\nFy5ur65eyLFj5/dwRQDsFZ9pX+WWlpKTJ1+efRFtKUeP+jwbYL8S7QEsLSX33HM+KytLWVvb69UA\nsFdcHgeAQYg2AAxCtAFgEKINAIMQbQAYhGgDwCBEGwAGIdoAMAjRBoBBiDYADGLb25hW1cEkDye5\nMckrSe7r7ufn9t+d5GeSXEjyjST3d/d6VX09yYuzp32ru+/djcUDwH6y6N7jdyY50t23VNUHkzw0\neyxVdW2SX0zyA919rqp+M8mHq+qLSdLdt+3iugFg31l0efzWJE8kSXc/neSmuX3nkqx297nZ9qEk\nLyd5f5LrqurJqvrSLPYAwJu0KNo3JDkzt/3q7JJ5unu9u9eSpKp+Osk7u/t3k7yU5MHu/pEkP5nk\ns5vHAAA7t+jy+Jkky3PbB7v7tc2NWYx/Kcn3JfnI7OFnkzyXJN39zar6TpL3JPn2dn+ilZXl7XYz\nY07TmdU05jSdWU1jTrtnUbRPJbkjyWNVdXOSZ7bs//VsXCb/se5enz12PBtfXPupqnpvNt6tv7Bo\nIWtrZ69k3fvSysqyOU1kVtOY03RmNY05TbPTFzaLov14ktur6tRs+/jsG+PXJ/n9JB9L8lSSL1dV\nkvxKkkeTfKqqnto8Zv7dOQCwM9tGe/bu+eNbHn527udrLnPoR9/MogCA7+ULYgAwCNEGgEGINgAM\nQrQBYBCiDQCDEG0AGIRoA8AgRBsABiHaADAI0QaAQYg2AAxCtAFgEKINAIMQbQAYhGgDwCBEGwAG\nIdoAMAjRBoBBiDYADEK0AWAQog0AgxBtABiEaAPAIEQbAAYh2gAwCNEGgEGINgAMQrQBYBCiDQCD\nEG0AGIRoA8AgRBsABiHaADAI0QaAQYg2AAxCtAFgEKINAIMQbQAYhGgDwCBEGwAGIdoAMAjRBoBB\niDYADOLQdjur6mCSh5PcmOSVJPd19/Nz++9O8jNJLiT5RpL7kxzY7hgAYGcWvdO+M8mR7r4lyQNJ\nHtrcUVXXJvnFJH+ruz+U5F1JPjw75h2XOgYA2LlF0b41yRNJ0t1PJ7lpbt+5JKvdfW62fWj22K1J\nvnCZYwCAHVoU7RuSnJnbfnV2yTzdvd7da0lSVT+d5J3d/cXtjgEAdm7bz7SzEd/lue2D3f3a5sYs\nxr+U5PuSfGTKMZezsrK86CnEnK6EWU1jTtOZ1TTmtHsWRftUkjuSPFZVNyd5Zsv+X8/GJfEf6+71\nicdc0tra2cmL3q9WVpbNaSKzmsacpjOracxpmp2+sFkU7ceT3F5Vp2bbx2ffGL8+ye8n+ViSp5J8\nuaqS5FcudcyOVgYAvMG20Z69e/74loefnfv5msscuvUYAOBN8gUxABiEaAPAIEQbAAYh2gAwCNEG\ngEGINgAMQrQBYBCiDQCDEG0AGIRoA8AgRBsABiHaADAI0QaAQYg2AAxCtAFgEKINAIMQbQAYhGgD\nwCBEGwAGIdoAMIhDe70A4K137lxy4sThLC8nR48mS0t7vSJgCtGGfebcueSuu67N6dMb//dfXb02\nJ0++LNwwAJfHYZ85ceLwxWAnyenTh3LixOE9XBEwlWgDwCBEG/aZY8fOZ3X1wsXt1dULOXbs/B6u\nCJjKZ9qwzywtJSdPvjz7ItpSjh71eTaMQrRhH1paSu6553xWVpaytrbXqwGmcnkcAAYh2gAwCNEG\ngEGINgAMQrQBYBCiDQCDEG0AGIRoA8AgRBsABiHaADAI0QaAQYg2AAxCtAFgEKINAIMQbQAYhGgD\nwCBEGwAGcWi7nVV1MMnDSW5M8kqS+7r7+S3PuS7JF5N8rLt79tjXk7w4e8q3uvveP+uFA8B+s220\nk9yZ5Eh331JVH0zy0OyxJElV3ZTkPyZ5b5L12WNLSdLdt+3KigFgn1p0efzWJE8kSXc/neSmLfuP\nZCPiPffY+5NcV1VPVtWXZrEHAN6kRe+0b0hyZm771ao62N2vJUl3fy1Jqmr+mJeSPNjdj1bV+5J8\noar+yuYxl7OysnzFi9+PzGk6s5rGnKYzq2nMafcsivaZJPPTP7govkmeTfJcknT3N6vqO0nek+Tb\n2x20tnZ2wa9lZWXZnCYyq2nMaTqzmsacptnpC5tFl8dPJfnRJKmqm5M8M+F3Hs/GZ9+pqvdm4936\nCztaHQBw0aJ32o8nub2qTs22j1fV3Umu7+5HLnPMo0k+VVVPbR4z4d05ALDAttHu7vUkH9/y8LOX\neN5tcz9fSPLRP5PVAQAXubkKAAxCtAFgEKINAIMQbQAYhGgDwCBEGwAGIdoAMAjRBoBBiDYADEK0\nAWAQog0AgxBtABiEaAPAIEQbAAYh2gAwCNEGgEGINgAMQrQBYBCiDQCDEG0AGIRoA8AgRBsABiHa\nADAI0QaAQYg2AAxCtAFgEKINAIMQbQAYhGgDwCBEGwAGIdoAMAjRBoBBiDYADEK0AWAQog0AgxBt\nABiEaAPAIEQbAAYh2gAwCNEGgEGINgAMQrQBYBCiDQCDOLTdzqo6mOThJDcmeSXJfd39/JbnXJfk\ni0k+1t095RgA4Moteqd9Z5Ij3X1LkgeSPDS/s6puSvJUkr+cZH3KMQDAziyK9q1JnkiS7n46yU1b\n9h/JRqT7Co4BAHZgUbRvSHJmbvvV2eXvJEl3f627/9eVHAMA7My2n2lnI77Lc9sHu/u1XTgmKyvL\ni55CzOlKmNU05jSdWU1jTrtnUbRPJbkjyWNVdXOSZyb8zp0ck7W1s1Oetq+trCyb00RmNY05TWdW\n05jTNDt9YbMo2o8nub2qTs22j1fV3Umu7+5Hph6zo5UBAG9wYH19ffGzdt+6V2aLeQU7nVlNY07T\nmdU05jTNysrygZ0c5wtiADAI0QaAQYg2AAxCtAFgEKINAIMQbQAYhGgDwCBEGwAGIdoAMAjRBoBB\niDYADEK0AWAQog0AgxBtABiEaAPAIEQbAAYh2gAwCNEGgEGINgAMQrQBYBCiDQCDEG0AGIRoA8Ag\nRBsABiHaADAI0QaAQYg2AAxCtAFgEKINAIMQbQAYhGgDwCBEGwAGIdoAMAjRBoBBiDYADEK0AWAQ\nog0AgxBtABiEaAPAIEQbAAYh2gAwCNEGgEGINgAM4tB2O6vqYJKHk9yY5JUk93X383P770jy80ku\nJPlkd//G7PGvJ3lx9rRvdfe9u7B2ANhXto12kjuTHOnuW6rqg0kemj2Wqjqc5JeT3JTku0lOVdV/\nTXI2Sbr7tl1bNQDsQ4suj9+a5Ikk6e6nsxHoTd+f5LnufrG7zyf5apIfSvL+JNdV1ZNV9aVZ7AGA\nN2lRtG9IcmZu+9XZJfPNfS/O7Tub5F1JXkryYHf/SJKfTPLZuWMAgB1adHn8TJLlue2D3f3a7OcX\nt+xbTvLHSZ5N8lySdPc3q+o7Sd6T5Nvb/YlWVpa3282MOU1nVtOY03RmNY057Z5F0T6V5I4kj1XV\nzUmemdv3R0neV1Xvzsa76x9M8mCS49n44tpPVdV7s/GO/IVFC1lbO3vlq99nVlaWzWkis5rGnKYz\nq2nMaZqdvrBZFO3Hk9xeVadm28er6u4k13f3I1X1s0mezMZl9ke7+4WqejTJp6rqqc1j5t6dAwA7\ndGB9fX2v15Ak616ZLeYV7HRmNY05TWdW05jTNCsrywd2cpwviAHAIEQbAAYh2gAwCNEGgEGINgAM\nQrQBYBCiDQCDEG0AGIRoA8AgRBsABiHaADAI0QaAQYg2AAxCtAFgEKINAIM4tNcLAGB8584lJ04c\nzvJycvRosrS01yt6exJtAN6Uc+eSu+66NqdPbyRldfXanDz5snDvApfHAXhTTpw4fDHYSXL69KGc\nOHF4D1f09iXaADAI0QbgTTl27HxWVy9c3F5dvZBjx87v4YrevnymDcCbsrSUnDz58uyLaEs5etTn\n2btFtAF405aWknvuOZ+VlaWsre31at6+XB4HgEGINgAMQrQBYBCiDQCDEG0AGIRoA8AgRBsABiHa\nADAI0QaAQYg2AAxCtAFgEKINAIMQbQAYhGgDwCBEGwAGIdoAMAjRBoBBiDYADEK0AWAQog0AgxBt\nABjEoe12VtXBJA8nuTHJK0nu6+7n5/bfkeTnk1xI8snu/o1FxwAAO7PonfadSY509y1JHkjy0OaO\nqjqc5JeT3J7kh5L806r687Nj3nGpYwCAnVsU7VuTPJEk3f10kpvm9n1/kue6+8XuPp/kq0l+cHbM\nFy5zDACwQ4uifUOSM3Pbr84uf2/ue3Fu39kk71pwDACwQ9t+pp2N+C7PbR/s7tdmP7+4Zd9ykv+3\n4JjLObCysrzgKSSJOU1nVtOY03RmNY057Z5F74BPJfnRJKmqm5M8M7fvj5K8r6reXVVHsnFp/GsL\njgEAdujA+vr6ZXdW1YG8/k3wJDme5G8mub67H6mqDyf5hWzE/9Hu/rVLHdPdz+7WXwAA7BfbRhsA\nuHr4ghgADEK0AWAQog0AgxBtABjEon9Pe1dU1bVJPpNkJRs3Zfkn3f1/tjznV7Nxd7WzSdaT3Nnd\nZ7b+rrejndzzfU8WuscmzOmfJ7k3ydrsoZ/Yz/8mQ1V9MMm/7+7btjzufJqzzZycTzOz21h/Mslf\nSvKOJP+2uz8/t985NTNhVld0Xu1JtJN8PMkfdve/qaq7kvzrJP9sy3M+kOTvdvf/fctXt/cu3vN9\n9g+Qh2aPzd/z/aYk301yqqr+W3f/7z1b7d657JxmPpDko939B3uyuqtIVf2LJP8oyZ9sedz5NOdy\nc5pxPr3uHyZZ6+6PVtW7k/yPJJ9PnFOXcNlZzVzRebVXl8cv3tN89se/M79z9g7qfUkeqaqvVtXx\nt3h9e20n93zfj7abU7JxT4Gfq6rfq6oH3urFXWWeS/LjSQ5sedz59EaXm1PifJr3WDbu0ZFsdOTC\n3D7n1BttN6vkCs+rXY92Vd1bVd+Y/1827lG+eal7857l865L8olsvEL5e0nur6q/vttrvYrs5J7v\n+9Gi+9z/VpKfSPLDST5UVUffysVdTbr7c/nef1gkzqc32GZOifPpou5+qbv/pKqWsxGlfzW32zk1\nZ8Gskis8r3b98nh3P5rk0fnHquq/5PX7k2/es3zed5N8orvPzZ7/5STvT/KN3V3tVeNK7/n+x2/V\nwq4yi+5z/6ub34Ooqt9O8jeS/PZbuL4ROJ+mcz7Nqaq/mORzSf5Dd5+Y2+Wc2mKbWSVXeF7t1Wfa\nm/cn/+9J/n6Sp7bsryS/VVUfSHJNkg8l+fRbucA9dirJHUke2+6e70leysZlpwff+iVeFS47p6p6\nV5JnquqvZeNF4A9ny4tHkjifJnE+vVFV/YUkv5Pk/u7+ypbdzqk5281qJ+fVXkX715L8p6r6vWx8\n6/cfJBe/Rfdcd3++qv5zktNJzif5dHf/zz1a6154PMntVXVqtn28qu7O6/d8/9kkT+b1e76/sFcL\n3WOL5vRAkq9k4xz73e5+4nK/aB9ZTxLn00KXmpPz6XU/l41L3r9QVZuf1z6S5J3Oqe+xaFZXdF65\n9zgADMLNVQBgEKINAIMQbQAYhGgDwCBEGwAGIdoAMAjRBoBB/H+Dj4Bb1j83ngAAAABJRU5ErkJg\ngg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1166f80d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(\n",
    "    np.arange(3), map(target_mean_advantage, pred_groups), color='blue'\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.scatter(\n",
    "    np.arange(3), [off_class_covariance(good_preds), off_class_covariance(bad_preds), off_class_covariance(weak_preds)]\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from matplotlib.pylab import plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0857288813754\n",
      "0.109356994978\n"
     ]
    }
   ],
   "source": [
    "for (a, b) in [Out[78], Out[80]]:\n",
    "    print a/b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "weak_mean = np.array([0.05, 0.05, 0.45, 0.15, 0.02, 0.04])\n",
    "weak_covariance = (\n",
    "    np.ones(shape=(n_classes, n_classes)) + \n",
    "    (np.eye(n_classes) * np.array([1, 1, 2, 4, .8, .9]))\n",
    ") * .0001\n",
    "\n",
    "weak_preds = np.random.multivariate_normal(weak_mean, weak_covariance, size=500)\n",
    "weak_preds += np.abs(weak_preds.min())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1159a0310>"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeAAAAFVCAYAAAA30zxTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAF45JREFUeJzt3X+Q3PV93/Hnnk4S6O50EuhiSC2Ip00+bieD60JjrBNg\n6pBOWjQmiRmP1ZgiQ0tsN2acqQOOB/9DOw714E7iBMcBp5hpsDs4IQ11S90ktkDHDJ3+JO3UbxBg\n6ZIx9qEfe/qBDt3d9o/dwxch3Z7udve9t/t8zGh03+93v/d53+q0r/18vp/9fCu1Wg1JktRZA9kF\nSJLUjwxgSZISGMCSJCUwgCVJSmAAS5KUwACWJCnB4FIHSykDwAPAFcAMcHtEvLjo+N8F7gcqwF8C\nt0TE6+0rV5Kk3tCsB3wTsCEidgB3Uw9bAEopFeB3gVsj4hrgT4G3tatQSZJ6SbMAHgeeBIiIZ4Gr\nFh37CeAQ8CullG8DWyIi2lGkJEm9plkAbwamF23PNYalAbYBO4AvAD8NvLeUcn3rS5QkqfcseQ2Y\neviOLNoeiIj5xteHgP0Lvd5SypPUe8jfOtc3m52dqw0OrltFuZIkrTmVs+1sFsATwC7gsVLK1cBz\ni469BAyXUv56Y2LWNcBDS32zI0dOLr9cSZJ6wNjYyFn3V5a6GUNjotXCLGiAPcCVwHBEPNgYcv51\n6uk+ERGfWKqIqalj3vlBktRXxsZGztoDXjKAW80AliT1m3MFsAtxSJKUwACWJCmBASxJUgIDWJKk\nBAawJEkJDGBJkhIYwJIkJTCAJUlKYABLkpTAAJYkKYEBLElSAgNYkqQEBrAkSQkMYEmSEhjAkiQl\nMIAlSUpgAEuSlMAAliQpgQEsSVICA1iSpAQGsCRJCQxgSZISGMCSJCUwgCVJSmAAS5KUwACWJCmB\nASxJUgIDWJKkBAawJEkJDGBJkhIYwJIkJTCAJUlKYABLkpTAAJYkKYEBLElSAgNYaqHZ2VlmZ2ez\ny5C0BhjAUgs9/PBDfOUrX84uQ9IaYABLLfLKK9/jmWeeYmJiL9///ivZ5Ujqcgaw1CKVSiW7BElr\nyGB2AVK3mZh4in379p73eYuv/T700BcZHDz//147d17H+Pi1532epLXHHrAkSQkqtVqtY41NTR3r\nXGNSgk996lcA+OxnP59ciaRuMTY2ctbrUw5BSy20efNodgmS1oglA7iUMgA8AFwBzAC3R8SLi45/\nArgNmGrsuiMinm9TrVLXGxjwqo6k5WnWA74J2BARO0op7wLub+xb8HeAD0XE/2xXgZIk9aJmb9fH\ngScBIuJZ4Kozjl8J/Fop5elSyt1tqE+SpJ7ULIA3A9OLtucaw9ILvgrcAfw9YGcp5R+2uD5JknpS\nsyHoaWBk0fZARMwv2v6NiJgGKKV8A3gn8I1zfbOtWzcxOLhupbVKXW/9+vrv99jYSJNHSup3zQJ4\nAtgFPFZKuRp4buFAKWUUeK6U8reAk9R7wUsugnvkyMnVVSt1udOn5wCYmjqWXImkbnGuN+TNAvhx\n4IZSykRje08p5YPAcEQ82Lju+y3qM6T/JCKebFXBkiT1siUDOCJqwEfO2P38ouNfpX4dWJIknQc/\ntChJUgIDWJKkBAawJEkJDGBJkhIYwJIkJTCAJUlK4P2A1bMeffQRJicPdLTNgwfr7V122eUdbRdg\n+/bL2b37lo63K2lp3g9YfWdy8gAvP/8dLl7XuV/zDfP1lVqnX9zfsTYBDs3NdrQ9SatnAKunXbxu\nkF1bLsouo+2eOHo4uwRJ58lrwJIkJTCAJUlKYABLkpTAa8DqWdXqUQ7PzvbF9dFDs7PUqkezy5B0\nHuwBS5KUwB6wetbo6BYqr77aN7OgN49uyS5D0nmwByxJUgIDWJKkBAawJEkJvAasnnZorrOzoE82\nlqLcNNDZ97aH5mbZ3NEWJa2WAayetX1752+IcLRxM4ZLOnwzhs3k/LySVs67IUktdN999wJw1133\nJFciqVuc625IXgOWJCmBASxJUgIDWJKkBAawJEkJDGBJkhIYwJIkJTCAJUlKYABLkpTAAJYkKYEB\nLElSAgNYkqQEBrAkSQm8GYN0homJp9i3b++Kzj3YuBvSZSu8G9LOndcxPn7tis6V1J3OdTMGb0co\ntdDo6Gh2CZLWCHvAUgvNz88DMDDg1R1Jdd6OUOqAp5/+Nk8//e3sMiStAQ5BSy1y/Pgxvva1fwvA\nlVf+FMPDw8kVSepm9oClFjlx4gQzM6eYmTnFiRPHs8uR1OXsAUstMjQ0zMaNG9/4WpKWYgBLLTI8\nPMwHPvCLVCoVh58lNeUsaKmFnAUt6Ux+DljqAINX0nItGcCllAHgAeAKYAa4PSJePMvjfhc4FBGf\nakuVkiT1mGZv128CNkTEDuBu4P4zH1BKuQP4ScDhZUmSlqlZAI8DTwJExLPAVYsPllJ2AD8FfAk4\n6xi3JEl6s2YBvBmYXrQ91xiWppRyKfAZ4J9h+EqSdF6aTcKaBkYWbQ9ExHzj6/cD24D/CFwCbCql\n/L+IeORc32zr1k0MDq5bTb2SJPWEZgE8AewCHiulXA08t3AgIr4AfAGglPKPgbcvFb4AR46cXF21\nkiStMWNjI2fd3yyAHwduKKVMNLb3lFI+CAxHxINnPNZJWJIkLZMLcUiS1EbejlCSpC5iAEuSlMAA\nliQpgQEsSVICA1iSpAQGsCRJCQxgSZISGMCSJCUwgCVJSmAAS5KUwACWJCmBASxJUgIDWJKkBAaw\nJEkJDGBJkhIYwJIkJTCAJUlKYABLkpTAAJYkKYEBLElSAgNYkqQEBrAkSQkMYEmSEhjAkiQlMIAl\nSUpgAEuSlMAAliQpgQEsSVICA1iSpAQGsCRJCQxgSZISGMCSJCUwgCVJSmAAS5KUwACWJCmBASxJ\nUgIDWJKkBAawJEkJDGBJkhIYwJIkJTCAJUlKYABLkpTAAJYkKYEBLElSgsGlDpZSBoAHgCuAGeD2\niHhx0fFfAO4CasDvR8RvtrFWSZJ6RrMe8E3AhojYAdwN3L9woJSyDvgs8F7g3cBHSykXtatQSZJ6\nSbMAHgeeBIiIZ4GrFg5ExBzw9og4BowB64DX21SnJEk9ZckhaGAzML1oe66UMhAR8wARMV9K+Xng\nt4D/AJxc6ptt3bqJwcF1q6lXkqSe0CyAp4GRRdtvhO+CiPjDUsrjwMPALY2/z+rIkSXzWZKknjM2\nNnLW/c2GoCeAfwBQSrkaeG7hQCllcyllbyllQ0TUgBPAXGvKlSSpt1Vqtdo5D5ZSKvxwFjTAHuBK\nYDgiHiyl/BPgNuA08L+BX26E8VlNTR07d2OSJPWgsbGRytn2LxnArWYAS5L6zbkC2IU4JElKYABL\nkpTAAJYkKYEBLElSAgNYkqQEBrAkSQkMYEmSEhjAkiQlMIAlSUpgAEuSlMAAliQpgQEsSVICA1iS\npAQGsCRJCQxgSZISGMCSJCUwgCVJSmAAS5KUwACWJCmBASxJUgIDWJKkBAawJEkJDGBJkhIYwJIk\nJTCAJUlKYABLkpTAAJYkKYEBLElSAgNYkqQEBrAkSQkMYEmSEhjAkiQlGMwuQNLaNTHxFPv27V3R\nudXqUQBGR7es6PydO69jfPzaFZ0rdQN7wJJSVKtVqtVqdhlSmkqtVutYY1NTxzrXmKSudt999wJw\n1133JFcitdfY2EjlbPvtAUuSlMAAliQpgQEsSVICA1iSpAQGsCRJCQxgSZISGMCSJCUwgCVJSrDk\nUpSllAHgAeAKYAa4PSJeXHT8g8CdwCzw58BHI8LFNiRJaqJZD/gmYENE7ADuBu5fOFBKuRC4F3hP\nROwERoEb21WoJEm9pFkAjwNPAkTEs8BVi46dAt4dEaca24PAay2vUJKkHtQsgDcD04u25xrD0kRE\nLSKmAEopvwwMRcSftKdMSZJ6S7PbEU4DI4u2ByJifmGjEcb/CvgbwC80a2zr1k0MDq5bSZ2Sesz6\n9fXXgrGxkSaPlHpTswCeAHYBj5VSrgaeO+P4l6gPRf/cciZfHTlyckVFSuo9p0/PATA1dSy5Eqm9\nzvUms1kAPw7cUEqZaGzvacx8Hgb+G/Bh4Cngz0opAL8REX/UkoolSephSwZwo1f7kTN2P7/oa8eT\nJUlaARfikCQpgQEsSVICA1iSpASVWq1zK0dOTR1zmUqpyzz66CNMTh7oeLsHD9bbvOyyyzve9vbt\nl7N79y0db1f9aWxspHK2/c1mQUvqcZOTB9j/wn6GLtza0XZrc/U5nN/7i0MdbffEa0c62p50Lgaw\nJIYu3MpP/vjPZJfREf/nhW9mlyABXgOWJCmFASxJUgIDWJKkBAawJEkJDGBJkhI4C1rqc9XqUU68\ndqRvZgefeO0I1arL2CufPWBJkhLYA5b63OjoFk4em+urzwGPjm7JLkOyByxJUgYDWJKkBAawJEkJ\nDGBJkhIYwJIkJXAWtKSUzwG/fvo1ADasv7Cj7dZvR3hxR9uUzsYAlvrc9u2Xp7R78OABAC59a6fD\n8OK0n1larFKr1TrW2NTUsc41Jqmr3XffvQDcddc9yZVI7TU2NlI5236vAUuSlMAAliQpgQEsSVIC\nA1iSpAQGsCRJCQxgSZISGMCSJCUwgCVJSmAAS5KUwACWJCmBASxJUgIDWJKkBAawJEkJDGBJkhL0\nfQDPz88zPz+fXYYkqc/0fQDv27eXiYmnssuQJPWZwewCMh0/fow/+IOvAfDOd17F8PBwckXda2GU\nYGCg79+zSVJL9PmraSW7gDXDkQJJaq2+7gEPDw/zvve9n0qlYu93CY4USFLr9XUAA7z00n4Arr/+\np5Mr6WaOFOjsJiaeYt++vSs69+DBAwDcd9+9Kzp/587rGB+/dkXnSt2grwP4lVe+xzPPPA3Arl0/\nx1vecklyRe21mhdLqFCpwG//9r9e0dm+WOpMQ0ND2SVIqZYVwKWUAeAB4ApgBrg9Il484zGbgP8C\nfDgiotWFtkOlYs9uuV5//fXsEtSFxsevXfEbq4ce+h0qlQq33XZHi6tSv1prk0WX2wO+CdgQETtK\nKe8C7m/sA6CUchXwO8CPArWWV9nEanp2GzduBODhhx8873PXWq9uNS+WC8OEd911TytLUp+qjz7V\nJ/XdeOP7en70SZ2xb99eKpUK11zznuxSlmW5bxPGgScBIuJZ4Kozjm+gHshroue72Px8jfn5jr9n\nkPqao09qtYXJol//+lc5fvx4djnLstwe8GZgetH2XCllICLmASLiGYBSSovLWx57dtLaMjQ0xODg\nYONrZ9WrFdbem7rlBvA0MLJo+43wPR9bt25icHDd+Z7WVuvX1+sZGxtp8sj+5vOkVrrgAt4I4G3b\nhhkZ8fdKqzM2NsKtt95KpVLhbW+7NLucZVluAE8Au4DHSilXA8+tpLEjR06u5LS2On16DoCpqWPJ\nlXQ3nye10vHjx6jV6pd+Xn31OKdOJReknvCOd7wL6L7XqXN1XJYbwI8DN5RSJhrbe0opHwSGI+L8\nZy9J6nOVN3rAUqusldnPC5b1PyAiasBHztj9/Fked30ripLU24aHh7n55t2uQqc3Wc2nWqrVowCM\njm4573MzPtXiW1BJKXbuvC67BPWYarUKrCyAMxjAklKsteFCdUY/farF/wGSJCUwgCVJSmAAS5KU\nwACWJCmBk7DWmEcffYTJyQMdb3e1925dje3bL2f37ls63q4ktVPXBHC/BctKQ2Vy8gDPvxgMjm5o\nQ1XnNr+uvhLWS6++3NF2Z6veBlFSb+qaAJ6cPEC8sJ91F3T281vzc/U1jvdPvtqxNudOHV3V+YOj\nG9hyzV9rUTXd7ejTf5ldgiS1RdcEMMC6C7Yw/GPvyS6j7Y5/99vZJUjqoKzVnWDt3be8nzgJS5K6\nWLVafWOFJ/WWruoBq7lq9Siz1Zm+GZqdrc5QXb+6IXspWz+t7gTO6VmurgngavUoc6eO9sXw7Nyp\no1SrXfPUS1JLTU4e4IWXXmLjRWMdbXd2/UYADh7t3O0IZw5PrfhcU2CNGR3dwqHTR/pqEtZaWVhd\n0g9tvGiMy3/25uwy2u7Af3psxed2TQCPjm5hanq2byZhGSqS1N+chCVJUoKu6QFLknpDtXqUmcOH\nVjU8u1bMHP4B1crcis7tqgDOmIQ1P3sKgIHBCzrWZn0hjm0rPn+2+nrHZ0HPz9R/wQY2rutou7PV\n11fzVElS1+qaAN6+/fKUdhemrV+2vZOv8ttW/POmP0/bOtz+tryfWdLKjI5uoVpb1zeTsEZHR1Z0\nbtcEcNZi+2vtM3Y+T5LUG5yEJUlSgq7pAUuSesfM4amOT8Kafe0EAIMXDnWszZnDU7BljQ9BS1I3\n67flFWHlSyymzVWZPgzAZZde0rlGt4ys+Oc1gCVpGSYnD/Ddl59n7OJNHW33gg3zAJyY/ouOtjt1\n6OSKz3WuyvIYwJK0TGMXb+IDu96eXUZH/LsnvpNdQs9zEpYkSQnsAUvSMlSrRzl8+GTf9Ax/cOgk\nF9W8FWg72QOWJCmBPWBJWobR0S0MVo731TXgoc3eta2d7AFLkpTAAJYkKYFD0FqW+fn57BKkdFOH\nOj8J68TJ0wAMbVrf0XanDp1kaHNHm+w7PRHAExNPsW/f3hWdu5pVZnbuvI7x8WtX1G6G1TxPL7/8\nErDy1XjW2nMlnSlrdadDR+uvUT9yyVs72u7QZu9E1m49EcCrMTo6ml1C15udnWV2dvaNrwcH+/7X\nRn3I1Z3Uaj3xSjo+fq29q2VY6fN0/PhxPvnJjwNw552fZHh4uNWlSVLf6YkAVrvVGn8kSa1iAGsZ\nKg47S+qIfprT46uqmhoeHubmm3dTqVQcfpbUtdbanB4DWMuyc+d12SVI6gP9NKfHANayDAy4Zosk\ntZKvqpIkJTCAJUlKYABLkpRgyWvApZQB4AHgCmAGuD0iXlx0fBdwDzAL/F5EPNTGWiVJ6hnNJmHd\nBGyIiB2llHcB9zf2UUpZD3weuAo4CUyUUv44In7QzoIlaa1ZzWdbv/vdlwHXYe9FzYagx4EnASLi\nWephu+BvAvsjohoRp4F9gP/KktRCGzZsYOPGDdllqA2a9YA3A9OLtudKKQMRMd84Vl107Biwtj4F\nLUkdsPJ12I/x6U//c2o1+NjHPuFCOD2mWQBPAyOLthfCF+rhu/jYCHBkqW82NjZSOe8KJalPffjD\nu7cAAfDxj//T8sQTTxxNLkkt1CyAJ4BdwGOllKuB5xYd+w7w46WUrcAJ6sPPn2tLlZLUhxqB+5bs\nOtQelVrt3He5KaVU+OEsaIA9wJXAcEQ8WEq5EfgM9WvJX46IL7a5XkmSesKSASxJktrDhTgkSUpg\nAEuSlMAAliQpgQEsSVKCvr8fcGOJzV+PiOuza+lWjWVHfw+4HNgI/IuIeCK3qu5TSlkHPAj8BFAD\nfiki/m9uVd2rlPIjwH8H3hsRz2fX061KKf+DHy569FJE3JZZT7cqpXyK+sdm1wO/FRFfSS6pqb4O\n4FLKrwK/CBzPrqXL/SNgKiI+1Pjc9/8CDOA3uxGYj4idpZTrgH9JY+10/VWNN3Vfor6GgM6hlHIB\ngB2EpZVS3gO8u3HfgiHgV5NLWpZ+H4LeD/w84ApdS3uM+ue9of47M5tYS9eKiH8P3NHY/DGarAzX\n5z4HfBH4XnYhXe4dwKZSyn8upfxpY8ROb/YzwJ+XUv6Ieufgj5PrWZa+DuCI+EMMk6Yi4kREHC+l\njFAP409n19StImKulPIw8JvAo8nldKVSyq3UR1S+2djlG+BzOwF8LiL+PvBLwO83bhOrv2qM+iJR\n76fxPOWWszz+Q2pZSinbgT8DHomIr2XX080i4lbq14EfLKVcmFxON9oD3FBK+Rbwt4GvlFJcbvHs\nnqcRJhHxAnAIuDS1ou70KvDNiJhtzCc4VUrZll1UM319DVjL03hx/Cbw0Yj4VnY93aqU8iHgrRHx\nWeA1YL7xR4tExHULXzdC+I6I+H5iSd1sD/WlgD9WSvlR6nehc9j+zfYBdwKfbzxPQ9TfrHQ1A7jO\n9TiX9mvUbzX5mVLKwrXgn42IU4k1daOvAw+XUvZSn4l5Z0TMJNekte3LwL8ppTzV2N6z6I50aoiI\nb5RSri2l/FfqI7sfjYiuf113LWhJkhJ4DViSpAQGsCRJCQxgSZISGMCSJCUwgCVJSmAAS5KUwACW\nJCnB/wfEE//ZcihkjQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x115851e50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.boxplot(weak_preds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(500, 6)"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "good_preds.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6, 6)"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.corrcoef(good_preds.T).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.26401237916372872"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p_m = good_preds.mean(axis=0)\n",
    "p_m[3] - np.mean(np.hstack([p_m[:2], p_m[4:]]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.06638894,  0.0667836 ,  0.46704638,  0.1669964 ,  0.03704085,\n",
       "        0.05785215])"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(weak_preds, axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.10998001324079959"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p_m = weak_preds.mean(axis=0)\n",
    "p_m[3] - np.mean(np.hstack([p_m[:2], p_m[4:]]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.19255883728945977"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p_m = bad_preds.mean(axis=0)\n",
    "p_m[3] - np.mean(np.hstack([p_m[:2], p_m[4:]]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}