CalvinTChi/AUC.ipynb

## AUC.ipynb
{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AUC\n",
    "### Author: _Calvin Chi_\n",
    "AUC, short for area under the curve, is a measure of how good a classification algorithm performs in terms of sensitivity in light of the false positive rate. The curve under which the area is determined is the ROC curve, which stands for receiver operating characteristic. The receiver operating characteristic is a plot of the true positive rate against the false positive rate for various rejection thresholds. The name ROC bears more historic weight than it does its actual meaning, and more information on the history can be found [here](https://en.wikipedia.org/wiki/Receiver_operating_characteristic).\n",
    "\n",
    "To define the true positive rate (TPR) and false positive rate (FPR), first examine the following table:\n",
    "\n",
    "|Condition A|Not condition A\n",
    "-|-|-\n",
    "Test results A|True Positive (TP)|False Positive (FP)\n",
    "Test results not A|False Negative (FN)|True Negative (TN)\n",
    "\n",
    "The TPR and FPR are then defined as follows: \n",
    "$$TPR(true\\:positive\\:rate) = \\frac{TP}{TP + FN}$$\n",
    "$$$$\n",
    "$$FPR(false\\:positive\\:rate) = \\frac{FP}{FP + TN}$$\n",
    "\n",
    "The output of classification algorithms in machine learning are often probabilities that a sample belongs to a certain class. Classification is performed based off a rejection threshold. Knowing the true labels, one can then determine the TPR and FPR for the model under the threshold. For example, in a binary classification setting, the user could decide any probability values above 50% to be class 1 and any probability values below 50% to be class 0. On a ROC curve, this 50% threshold would then correspond to one point.\n",
    "\n",
    "The area under the ROC curve gives a measure of how good the model is at detecting true positives at the expense of false positives. The objective is to build a model that can achieve a high TPR while maintaining a low FPR, the values of which are determined by the chosen rejection threshold. Hence, the larger the area under the ROC curve, the better the model is. Models that perform similarly to random guessing have AUCs close to 0.5 and possess a roughly linear ROC curve with slope 1 and intercept of 0, since the TPR is about the same as the FPR at every rejection threshold. \n",
    "\n",
    "To see how this works through an example, let us demonstrate calculations for a toy example: \n",
    "\n",
    "Predicted probability|True label\n",
    "-|-\n",
    "0.75|1\n",
    "0.83|1\n",
    "0.4|0\n",
    "0.53|0\n",
    "0.98|1\n",
    "0.1|0\n",
    "0.89|1\n",
    "0.35|0\n",
    "\n",
    "With a classification threshold of 0.5, the TPR and FPR would be calculated as follows: \n",
    "\n",
    "$$TPR = \\frac{4}{4} = 1$$\n",
    "$$FPR = \\frac{1}{4} = 0.25$$\n",
    "\n",
    "Let us now graph the ROC curve by generating predictions for 20 samples, where the true labels are 1 for the first 10 and 0 for the last 10 samples. In this tutorial, a good model should output probabilities close to 1 for samples of class 1 and probabilities close to 0 for samples of class 0. In this particular simulated dataset, samples of class 1 are assigned probabilities drawn from a $N(0.85, \\sigma = 0.3)$ distribution and samples of class 0 are assigned probabilities drawn from a $N(0.15, \\sigma = 0.3)$ distribution. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAARAAAAEZCAYAAABfBXrdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGUlJREFUeJzt3X+QHOV95/H3ZxFriQtCYMo2EQHZFrJABdIiTEjOh8Ym\nhsW5Qy7gMJBwBp3r5LrASbm6snASwibW4SiJY9kRBIgl+ccFhAs5BhMnxsFMiDHYIGYRCCkLBsn8\nOgwYjG3gVtJ874/uFcN4ZzXTOz0zO/N5VW0xPfNsz5ed3Y/6efrppxURmJll0dfuAsxs6nKAmFlm\nDhAzy8wBYmaZOUDMLDMHiJll5gAxs8wcIF1G0k5Jr0p6RdIzkjZKOqiqzW9KuiNt85KkWyQdW9Xm\nYElrJe1K2z0q6a8kHTbBe/8PSQ9J+rmkH0m6SdKCvP5f6yWpIOk7kl6W9Hi76+kmDpDuE8BvR8RM\nYBEwAHxy7EVJvwF8C/h74AjgncBW4G5Jc9I2BwLfAY4FTk/39RvAC8DJ472ppM8DlwGXAocC84Cv\nA7/d6P+ApAMa/Z79+AWwHvhfTd6vRYS/uugLeAL4QMX2GuAbFdt3AX89zvd9E/hi+vhjwLPAjDrf\ncy6wB1g8QZs7gWUV2x8F/rViuwz8d2AE+CFwDfAXVfv4OrAyfXwEcDPw47T9ZXXUeRrweLs/o276\n8hFIF5N0JHAm8Gi6PQP4TZI/vGpfBT6YPj4N+KeIeK3OtzoNeDIitjRYYvV1FEuB9wLHATcC5429\nIGkWcDpwoyQB3wBKJEFyGrBC0gexlnKAdKevS3oF+BHwHDCUPn8YyWf+7Djf8yxwePr4rTXa1NJo\n+1quioifRsT/i4h/BULS+9LXzgW+FxHPkXSjDo+I/x0ReyNiJ/AF4Pwm1GANcIB0p6WRjFssAebz\nRjC8RNJVOGKc7zmCZIwD4MUabWpptH0tT1Vt3wRckD6+EPi79PFRwGxJP0m/XiIZ53lbE2qwBjhA\nupMA0n/FvwR8Jt1+FbgH+M/jfM95wD+nj/8ZOCPt8tTjDuBISSdO0OYXQOXZoHeM06a6S3MjcK6k\no4BfBzanzz9JMpZxWPp1aEQcEhH/qc56rUkcIN1vLfBBScen25cDH5V0qaRfkXSopNXAKcCfpm2+\nQvJHulnSe5R4q6RPShqsfoOIeIxk0PNGSUskHSjpLZI+IukTabNh4GxJMyTNBf7r/gqPiGGSo5sv\nkIzJvJK+9APgZ5I+IWm6pAMkLZB00nj7Set/C9AP9KW1Hbi/97f9c4B0nzf9Kx4RL5Achfxxun03\ncAZwDsm4xRPAQuDfR8QP0zajwG8BO4BvAz8F7iUZ6/j+uG8asQJYB1xN0lV6DPgwyWAnwGeB3cD/\nBTYC/2eiuivcQDJI+nf7GkaUgf9Icpr6CZIzMX8LzKyxj1OB14DbgF8DXiU5lW2TpPT0lplZw3wE\nYmaZOUDMLDMHiJll5gAxs8ymtbuAeknyaK9Zm0SExnt+Sh2BNOsCoCuvvLLtFyG5pu6pqVPralZN\nE5lSAWJmncUBYmaZ9WSAFAqFdpfwS1xTfTqxJujMulpR05SZiSoppkqtZt1EEtENg6hm1lkcIGaW\nmQPEzDJzgJhZZg4QM8vMAWJmmTlAzCwzB4iZZZZrgEhaL+k5SVsnaPP59L6rw5IW5VlPtyuXy2zZ\nsoUtW7ZQLpfbvh9LtPpzaeXnl/cRyEaSBXzHJelM4N0RcQywHLg253q6Vqm0jcWLV3Lqqbs49dRd\nLF68klJpW9v2Y4lWfy4t//zyvqQYOBrYWuO1a4GPVGxvB95eo23Y+Pbu3RuLFl0WsDcg0q/kub17\n97Z8P5Zo9eeS1+eX/u2N+/fd7gWFZpPcf2TM0+lzz7WnnKmpVCoxMlLgzQeUfWzfvoTPfa7E3LmL\n69rPY4+V2L598vuxRLN+nvXup1a7kZEllEolFi9u/ufX7gBpyNDQ0L7HhUKhI6+A7CS7d8PXvgaz\nZtXX/uWXk++Z7H4s0ayfZ737qdWuUcVikWKxWF/jWocmzfqisS7MDtyFadjevXtj9mx3YTpNL3Rh\nWhEgc4CHarz2IeAf0senAPdOsJ/MP4Bud/XVEe94x8Nx7LGXxUEH3RwHHXRzLFx4aTzwwMMN7+uB\nBx6ORYsmvx9LNOvnWe9+8vj8JgqQXNcDkXQDUCC5JeJzwJUk9yeNiLg+bbMOGCS5+fIlEfFAjX1F\nnrVOVddcA3/+5/Cd78CcOWVKpRIAAwMD9PVlO8lWLjdnP5Zo1s+z3v00+/ObaD0QLyg0hVWGx7ve\n1e5qrFt5QaEu5PCwTjClzsJ0o3oON6vbXHttn8PDOoK7MG1UKm1j2bLr0jkcMG9ekQ0bljMwsKBm\nm0MPLbJ373LuvnuBw8NawmMgHahcLrN48UqGh9fyRk+yzKJFK9myZS19fX012xx77EoefnitBzet\nJTwG0oFqzR4dHl7CAQeUkOCAA0oMD/9ym127luzr0pi1kwOkDSKgWITXXvvl1w46CO6/P2lz//3J\ntlmncoC0UATccguceCJ85SsDzJlTBCovty4zb96/MDAwACQDpvPmTdzGrJ18FqYFIuDWW2HsUp6h\nITjrrD6Gh5ezbNlKRkaWAHDMMUU2bPj4vrGNvr4+NmyYuI1ZO3kQNUdjRxx/8ifJdhIcoIrhqCyn\ncR0e1ko+C9Ni9QSH2VQxUYC4C5PB6Ogoa9asAWDVqlX09/cDDg7rPT4CadBVV13NH/3RPxFxEQDS\nV1i9epDjjvs9B4d1JXdhmmR0dJTp088h4hYqJ3bBUo4/fjOf+lS/g8O6jieSNcmaNWvSI483T+yC\n3+Xcc9ewdKnDw3qLA6RJHBzWi9yFaUCtLoy0lNdf37xvMNWsm7gL0yT9/f2sXj2ItBS4CbgJ6SxW\nrx50eFhP8hFIBqOjo/zhH67h2mvhxRdXOTysq/ksTA6eeQZOOin5r1k3cxemycrlMlu3bmH3bt87\n1nqbA6RBY/cePfvsXbzwgu8da73NXZgG1LOKmFm3cRemSWqtIjZ271GzXuMAMbPMHCAN8AphZm/m\nAGnA2AphixatZMaMzcBmFi5cwYYNyz3+YT3Jg6gZlMtlbr+9xEUXwXPPeYUw625eUKjJ+vr6OOGE\nxRx4IDg7rJfl/usvaVDSDkkjklaN8/pMSbdKGpb0kKSL867JzJoj1wCR1AesA84AFgAXSJpf1ez3\ngG0RsQh4P/AZST4yMpsC8j4CORl4NCJ2RcRuYBOwtKpNAAenjw8GXoyIPTnXZWZNkHeAzAaerNh+\nKn2u0jrgOEnPAA8CK3KuycyapBO6CmcApYj4gKR3A9+WdEJE/Ly64dDYnZmAQqFAoVBoWZFmvaJY\nLFIsFutqm+tpXEmnAEMRMZhuXw5ERKypaHMb8OmIuDvdvgNYFRH3V+2rY07jgi/nt97Rzmth7gPm\nSjpaUj9wPnBrVZtdwG8BSHo7MA94POe6zKwJcu3CRMReSZcCt5OE1fqI2C5pefJyXA+sBr4oaWv6\nbZ+IiJ/kWZeZNYdnombkLoz1Cl/Ob2a5cIBk4CUNzRIOkAZ5SUOzN3gMpAFe0tB6kcdAmsRLGpq9\nmQPEzDJzgDTASxqavZnHQBpUKm3j9NOv4+WXl9DfD8ccU2Tjxo8zMLCg3aWZ5cK3tmyiZ5+F444r\ns2lTicMPT45KPHhq3cwB0kQrV4IEn/1suysxaw0HSJM8+ywsWADbtsERR7S1FLOWcYA0iY8+rBc5\nQJrARx/WqxwgTeCjD+tVDpBJ8tGH9TIHyCT56MN6mQNkEnz0Yb3OATIJPvqwXucAychHH2a+uXZD\nyuXyvkvzv/zlAT760T6Hh1kNDpAKpdI2li27jpGRAhEwOvolvvnN5SS39TWzau7CpLzamNn4vCJZ\nHbzamFnjHCBmlpkDJOXVxswa5zGQCmODqI88soRyGRYs8GpjZp4H0oByucwVV5R45BHYvNmrjZl5\nHkgD+vr6OProxTz/PDg7zCaW+5+IpEFJOySNSFpVo01BUknSw5LuzLsmM2uOXI9AJPUB64DTgGeA\n+yTdEhE7KtocAlwNnB4RT0s6PM+a9qdcLrNrV4nnn4dy2V0Ys4nk/ddxMvBoROyKiN3AJmBpVZsL\ngc0R8TRARLyQc001jd339i//che33eb73prtT94BMht4smL7qfS5SvOAwyTdKek+SRflXNO4yuUy\ny5Zdx/DwWkZHz2bPnrMZHl7LsmXXUS6X978Dsx7UCcfn04ATgTOBQeAKSXNbXYRnopo1Lu+zME8D\nR1VsH5k+V+kp4IWIeB14XdJdwELgseqdDQ0N7XtcKBQoFApNLtfMisUixWKxrra5zgORdADwbySD\nqM8CPwAuiIjtFW3mA39NcvTxFuD7wEci4pGqffliOrM2aNs8kIjYK+lS4HaSv8r1EbFd0vLk5bg+\nInZI+hawFdgLXF8dHq3Q19fHhg3LWbZs5Ztmom7Y8HGHh1kNnolaxTNRzd7MM1Eb4JmoZvXzn0iV\nZCLZFp5/fotP35rthwOkgieSmTXGYyApn4UxG5+XNKyDJ5KZNc4BYmaZNRwgkvok/U4exbSTlzQ0\na1zNAJE0U9InJa2TdLoSlwGPA+e1rsTWGJtItmjRSvr7NzNt2mYWLlzBhg3LPf5hVkPNQVRJtwAv\nAfeQTEV/GyBgRUQMt6zCN+rxRDKzNsg6kexdEXF8uoMvkFzLclR60VvX8kQys/pNFCC7xx6k17Q8\n1e3hAbBnzx7uvXcTTzwBe/acz7RpnqxrVstE/8YulPSKpJ9J+hlwQsX2K60qsJVuvPFWZs48j40b\n+ykW+5k58zxuvPHWdpdl1rE8kSy1Z88eZs48j9deu5nKiWQzZpzLK6981Uci1rMyTSSTNF3SyvQs\nzH+T1NV/QZs2beK1186neiLZa699hE2bNrWrLLOONlEX5kvAScBDwIeAz7SkIjObMiY6jftQxVmY\nacAPIuLEVhZXVY+7MGZtkPVamMqzMHuaXlWHmTZtGuvXX8yMGecCNwE3MX36Oaxff7HDw6yGiY5A\nysDPxzaBGcCr6eOIiJktqfCNeloykWzPnj28852bWL4cLr/cp3HNMt1cW1IpIjrmIpBWBUgEzJoF\nO3fCoYfm/nZmHS9rF2ZqnN9tspdfTv47a1Z76zCbCiY6Pn+bpP9Z68WI+Ksc6mm7nTthzhzQuHlr\nZpUmCpADgF8hGfPoGWMBYmb7N1GAPBsRf9qySjqEA8SsfhONgfTUkccYB4hZ/SYKkNNaVkUHcYCY\n1a9mgETET1pZSKdwgJjVz0vmVIhwgJg1wgFSwXNAzBqTe4BIGpS0Q9KIpFUTtHuvpN2Szs67plo8\nB8SsMbkGiKQ+YB1wBrAAuEDS/Brt/gz4Vp717I+7L2aNyfsI5GTg0YjYFRG7gU3A0nHaXQbcDPw4\n53om5AAxa0zeATIbeLJi+6n0uX0k/Srw4Yj4G9o898QBYtaYThhEXQtUjo20LUQcIGaNyXuxi6eB\noyq2j0yfq3QSsEmSgMOBMyXtjohfWg59aGho3+NCoUChUGhqsQ4QMygWixSLxbra5roqu6QDgH8j\nmdX6LPAD4IKI2F6j/UbgGxHxtXFey3U9EK8DYja+rHemm7T0hlSXAreTdJfWR8R2ScuTl+P66m/J\ns56JeA6IWeN8X5hUqQQXXwwPPpjbW5hNSVlXJOspHv8wa5wDJOUAMWucAyTlADFrnAMk5QAxa5wD\nJOUAMWucAwSvA2KWlQMEzwExy8oBgtcBMcvKAYK7L2ZZOUBwgJhl5QDBAWKWlQMEB4hZVg4QHCBm\nWfV8gHgOiFl2PR8gngNill3PB4jngJhl5wDZ6e6LWVYOkJ0OELOsHCA7HSBmWTlAdjpAzLJygOx0\ngJhl1dMB4jkgZpPT0wHiOSBmk9PTAeI5IGaT4wCZ0+4qzKYuB8icdldhNnU5QOa0uwqzqcsBMqfd\nVZhNXQ6QOe2uwmzqyj1AJA1K2iFpRNKqcV6/UNKD6dd3JR2fd03gOSBmzZBrgEjqA9YBZwALgAsk\nza9q9jhwakQsBFYDf5tnTWM8B8Rs8vI+AjkZeDQidkXEbmATsLSyQUTcGxE/TTfvBWbnXBPgOSBm\nzZB3gMwGnqzYfoqJA+JjwD/mWlHK3RezyZvW7gLGSHo/cAnwvlpthoaG9j0uFAoUCoXM7+cAMRtf\nsVikWCzW1VYRkVshkk4BhiJiMN2+HIiIWFPV7gRgMzAYET+ssa9oZq0rViQB8vu/37RdmnUlSUTE\nuJ39vLsw9wFzJR0tqR84H7i1qrijSMLjolrhkQcfgZhNXq5dmIjYK+lS4HaSsFofEdslLU9ejuuB\nK4DDgGskCdgdESfnWRc4QMyaIdcuTDM1swsTkZy+3bkTDj20Kbs061rt7MJ0JM8BMWuOngwQzwEx\na46eDhAzmxwHiJll5gAxs8wcIGaWmQPEzDLruQDxOiBmzdNzAeI5IGbN03MB4jkgZs3TUwFSLpe5\n444tzJq1hXK53O5yzKa8ngmQUmkbixev5A/+YBff+94uFi9eSam0rd1lmU1pPXExXblcZvHilQwP\nr+WNzCyzaNFKtmxZS19fz+SoWcN6/mK6UqnEyEiBN//v9jEysoRSqdSmqsymvp4IEDPLR08EyMDA\nAPPmFYHKgdMy8+b9CwMDA+0pyqwL9MQYCCSDqMuWXcfWrUs48ECYP7/Ixo0fZ2BgQROrNOs+E42B\n9EyAQDKYumRJibPPhhUrBjx4alaHiQKkY27r0Ap9fX3MmrWYuXPB2WE2ef4zMrPMHCBmlpkDxMwy\nc4CYWWYOEDPLzAFiZpk5QMwsMweImWXmADGzzHIPEEmDknZIGpG0qkabz0t6VNKwpEV51TI6OsrI\nyKfYtOlTjI6O5vU2Zj0j1wCR1AesA84AFgAXSJpf1eZM4N0RcQywHLg2j1quuupqpk8/h5GR93DD\nDe9h+vRzuOqqq/N4K7OekevFdJJOAa6MiDPT7cuBiIg1FW2uBe6MiJvS7e1AISKeq9pX5ovpRkdH\nmT79HCJuoXJFMmkpr7++mf7+/kz7NesF7VyRbDbwZMX2U+lzE7V5epw2k7JmzRoiLqJ6RbKI32XN\nmjW1vs3M9mNKXY07NDS073GhUKBQKLStFrNuVSwWKRaLdbVtRRdmKCIG0+16ujA7gCXuwph1hnZ2\nYe4D5ko6WlI/cD5wa1WbW4H/AvsC5+Xq8Jis/v5+Vq8eRFoK3ATchHQWq1cPOjzMJiH3FckkDQKf\nIwmr9RHxZ5KWkxyJXJ+2WQcMAr8ALomIB8bZz6RXJBsdHd035rFq1SqHh1kdvKShmWXW8/eFMbN8\nOEDMLDMHiJll5gAxs8wcIGaWmQPEzDJzgJhZZg4QM8vMAWJmmTlAzCyzngyQei9VbiXXVJ9OrAk6\ns65W1OQA6RCuqT6dWBN0Zl0OEDPraA4QM8tsSl3O3+4azHrVlF8PxMw6j7swZpaZA8TMMuvaAOmk\nW2o2UpekCyU9mH59V9Lx7a6pot17Je2WdHYn1CSpIKkk6WFJd7a7JkkzJd2a/j49JOniFtS0XtJz\nkrZO0Ca/3/OI6LovkmB8DDgaOBAYBuZXtTkT+If08a8D93ZIXacAh6SPB/Ouq56aKtrdAdwGnN3u\nmoBDgG3A7HT78A6o6ZPAp8fqAV4EpuVc1/uARcDWGq/n+nverUcgJwOPRsSuiNgNbAKWVrVZCnwZ\nICK+Dxwi6e3trisi7o2In6ab99Lku/RlqSl1GXAz8OOc66m3pguBzRHxNEBEvNABNQVwcPr4YODF\niNiTZ1ER8V3gpQma5Pp73q0B0hG31MxYV6WPAf+Ya0V11CTpV4EPR8TfAOOezmt1TcA84DBJd0q6\nT9JFHVDTOuA4Sc8ADwIrcq6pHrn+nk+pW1v2EknvBy4hOURtt7VAZZ+/FSGyP9OAE4EPAP8OuEfS\nPRHxWBtrOgMoRcQHJL0b+LakEyLi522sKVfdGiBPA0dVbB+ZPlfd5tf206YddSHpBOB6YDAiJjo8\nbVVNJwGbJImkb3+mpN0RUX2XwVbW9BTwQkS8Drwu6S5gIck4RbtqugT4NEBE/FDSE8B84P6caqpH\nvr/neQ7wtOsLOIA3Brz6SQa8jq1q8yHeGFw6hdYMotZT11HAo8ApnfKzqmq/kfwHUev5Oc0Hvp22\nPQh4CDiuzTVdDVyZPn47SdfhsBZ8hnOAh2q8luvveVcegUTEXkmXArfzxi01t1feUjMivinpQ5Ie\nI72lZifUBVwBHAZck/6LvzsiTm5zTW/6lrxqaaSmiNgh6VvAVmAvcH1EPNLOmoDVwBcrTql+IiJ+\nkldNAJJuAArAWyX9CLiSJOBa8nvuqexmllm3noUxsxZwgJhZZg4QM8vMAWJmmTlAzCwzB4iZZeYA\nsUmTtFfSA+ml9Q9IOkrSEkkvp9vbJP1x2rby+Uck/UW767fsunIimbXcLyLixMonJL0TuCsizpJ0\nEDAsaWzq+9jz04GSpK9FxD2tLtomz0cg1gwTXlwXEa8CW4C5Vc+/TjIlPO+roC0nDhBrhhkVXZjN\nFc8LQNJbSRaz2Vb1/KEkoXJXK4u15nEXxprh1eouTOo/SNoClElW6tou6W3p8yXgGGBtRLRikSLL\ngQPE8nRXRJxV63lJc4B7JX01Imqu6Wmdy10Ya4ZMCwxFxE6S9TMub2o11jIOEGuGyVzSfR1Jl+ao\n/ba0juPL+c0sMx+BmFlmDhAzy8wBYmaZOUDMLDMHiJll5gAxs8wcIGaWmQPEzDL7/22g6lZZhTDo\nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x105c74a20>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "np.random.seed(2)\n",
    "\n",
    "num = 20\n",
    "trueLabels = num*[1] + num*[0]\n",
    "t = np.array([i if (i <= 1 and i>= 0) else 0.85 for i in np.random.normal(0.85, 0.3, num)])\n",
    "f = np.array([i if (i <= 1 and i>= 0) else 0.15 for i in np.random.normal(0.15, 0.3, num)])\n",
    "predict = np.concatenate([t, f])\n",
    "\n",
    "def TPR_FPR(predict, trueLabels, cutoff): \n",
    "    tp = predict[np.where(predict[:num] >= cutoff)]\n",
    "    fp = predict[np.where(predict[num:] >= cutoff)]\n",
    "    pos = [x for x in trueLabels if x == 1]\n",
    "    neg = [x for x in trueLabels if x == 0]\n",
    "    tpr = tp.size / len(pos)\n",
    "    fpr = fp.size / len(neg)\n",
    "    result = [tpr, fpr]\n",
    "    return result\n",
    "\n",
    "TPRs = []\n",
    "FPRs = []\n",
    "for i in np.arange(0, 1.05, 0.05): \n",
    "    TPRs.append(TPR_FPR(predict, trueLabels, i)[0])\n",
    "    FPRs.append(TPR_FPR(predict, trueLabels, i)[1])\n",
    "plt.plot(FPRs, TPRs, \"o-\", color=\"blue\")\n",
    "plt.xlim(-0.1, 1.1)\n",
    "plt.ylim(-0.1, 1.1)\n",
    "plt.xlabel(\"FPR\")\n",
    "plt.ylabel(\"TPR\")\n",
    "plt.title(\"ROC Curve 1\")\n",
    "plt.gca().set_aspect('equal', adjustable='box')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visually, the area under this ROC curve is close to 1, indicating that the model is capable of being sensitive to true positives while maintaining a low false positive rate. \n",
    "\n",
    "What does the ROC curve for a poor model look like? Let us generate sample predictions where the probability predictions for classes 1 and 2 are closer to each other than the previous example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAARAAAAEZCAYAAABfBXrdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt4VNW5+PHvO4SUTCVyq9wJCkaBFiaGomKFWKqiPYiC\nF2hBNG1Nj4WS8+vp0VataPFCT1W0WsUjQa1KUGMFq7VaNeBdE4cqAY0XCJcAIoLcSzLz/v6YyThM\nMiEZ5j7v53n2k9kza/Z+szPzZq21115bVBVjjImEI9EBGGNSlyUQY0zELIEYYyJmCcQYEzFLIMaY\niFkCMcZEzBKIMSZilkAyhIisE5F9IrJLROpFZJGIOEPKjBaRl/xldojIUhEZElKms4jMF5E6f7mP\nReR2EenWyr5/KSIfiMgeEVkvIktEZFisflcTP5ZAMocCP1TVXMAFFAC/aXpRRE4F/gH8FegNHAu8\nD7wuIgP9ZToCLwNDgLP82zoV+AIY1dJOReQuYBYwE+gK5ANPAz9s7y8gIh3a+x4TY6pqSwYswFrg\n+0Hr84BngtZXAH9q4X3PAQ/6H/8U2AzktHGfg4FGoLCVMq8AxUHrM4BXg9a9wJVALfAp8Gfgf0O2\n8TRQ6n/cG3gS+Nxfflaij306L1YDyUAi0g84B/jYv54DjMb3xQv1OHCm//E44HlV3d/GXY0DNqhq\ndTtDDL2+YiLwXWAosBi4uOkFEekCnAUsFhEBngHc+BLJOGC2iJyJiQlLIJnlaRHZBawHtgJz/M93\nw/dZ2NzCezYDPfyPu4cpE057y4dzs6p+par/VtVXARWR7/lfuxB4Q1W34mtG9VDVm1TVo6rrgAeA\nKVGIwbTAEkhmmai+fouxwIl8nRh24Gsq9G7hPb3x9XEAbA9TJpz2lg9nY8j6EmCq//GPgEf9jwcA\nfUXkS/+yA18/zzFRiMG0wBJIZhEA/3/xh4Db/Ov7gDeBi1p4z8XAP/2P/wmc7W/ytMVLQD8ROamV\nMnuB4LNBvVooE9qkWQxcKCIDgJOBCv/zG4DPVLWbf+mqqker6oQ2xmvayRJI5poPnCki3/GvXw3M\nEJGZInKUiHQVkbnAKcCN/jJ/wfclrRCRE8Snu4j8RkTGh+5AVT/B1+m5WETGikhHEfmGiFwiIv/j\nL7YSmCQiOSIyGPjJ4QJX1ZX4ajcP4OuT2eV/6R1gt4j8j4h0EpEOIjJMREZGcoDM4VkCyRyH/BdX\n1S/w1UJ+519/HTgbmIyv32ItMAI4TVU/9Zc5CPwA+BB4EfgKeAtfX8fbLe5UdTZwN3APvqbSJ8D5\n+Do7Ae4AGoAtwCLgkdbiDvIYvk7SRwMFVb3Af+A7Tb0W35mY/wNyw2zDHCHxn/oyxph2sxqIMSZi\nlkCMMRGzBGKMiZglEGNMxLISHUBbiYj19hqTIKoqLT2fUjWQaF0AdP311yf8IiSLKX1iSta4ohVT\na1IqgRhjkoslEGNMxDIygRQVFSU6hGYsprZJxpggOeOKR0wpMxJVRDRVYjUmnYgImg6dqMaY5GIJ\nxBgTMUsgxpiIWQIxxkTMEogxJmKWQIwxEbMEYoyJmCUQY0zEYppARGShiGwVkfdbKXOX//6qK0XE\nFct40p3X66W6uprq6mq8Xm/Ct2N84v13aWxs5JFHHuGRRx6hsbEx4v21RaxrIIvwTdTbIhE5Bxik\nqscDJcB9MY4nbbndNRQWljJmTB1jxtRRWFiK212TsO0Yn3j/XSoWP8m5ub3R6T9Bp/+Ec3N7U7G4\npRsORkmsLykG8oD3w7x2H3BJ0PoaoGeYsmpa5vF41OWapeBRUP/ie87j8cR9O8Yn3n+XhoYGPTOn\nh3q+LqQe0DNzemhDQ0PEv4f/u9fi9zvREwr1xXefkSab/M9tTUw4qcntdlNbW8ShFUoHa9aM5c47\n3QweXNim7XzyiZs1a458O8YnWsezrdt55ZVypu/fFVIKpu3fRXl5OdOmTTuSX6dFiU4g7TJnzpzA\n46KioqS8AjKZNDTAU09Bly5tK79zp+89R7od4xOt49nW7WzcCAWRhxtQWVlJZWVl2wqHq5pEa6F9\nTZgPsSZMu3k8Hu3b15owySYRTZiJ2UfHtQkTjwQyEPggzGvnAs/6H58CvNXKdiI+AOnunntUe/Va\npUOGzFKn80l1Op/UESNm6nvvrWr3tt57b5W6XEe+HeMTrePZpu288oq6j+qsE7OP1ofI1ofI1jM7\nddcnH3viiH6H1hJITOcDEZHHgCJ8tz7cClwPZPsDut9f5m5gPL6bLF+uqu+F2ZbGMtZU9ec/wx/+\nAC+/DAMHenG73QAUFBTgcER2ks3rjc52jE+0jmer26mshIsugieeoPF736O8vByAKVOmkJV1ZD0V\nrc0HYhMKpbDg5HHccYmOxiRMUPIgBv2CNqFQGrLkYYCYJ4/DSamzMOmoLdXb0DL33eew5JFmGhsb\nD9vsaPZZWbEiockDiH0narQW0rAT9euOsQp1OivU5ZrVrGMstEzfvrO0V69V+umnCQraRN3Sxx7T\nC3JydAnoEtALcnJ06WOPHVJm1Xvv6SyXSyucTq1wOnXW4MG66uijVV95JebxkcizMNFa0i2BtOXU\nXLgyQ4bYadV00dDQoBfk5DQ79XpBTk7g1KvH49FZLlezMrMGD47L56C1BGJ9IAkSbvToypVj6dDB\njQh06OBm5crmZerqxgaqsia1lZeXM2X//majRy/Zv5/yjh1BBHeHDhStXNmszNj6+oR/DiyBJICq\nr+9r//7mrzmdUFXlK1NV5Vs3aWjtWl8v+HXXhS/zl78k/QfBEkgcqcLSpXDSSfCXvxQwcGAlEHxZ\ntpf8/OUUFPgGJBcUFJCf33oZk0KaksbIkXDyyfDZZ0y57z7KO3UK+QvDkpwcpkyZAvg+B5X5+c3K\nLM/PT/znIFzbJtkWUrgPxOtVffppVZfLtzz9tO+5towutJGhKe7TT1VvvVW1sFD1W99SLSlR/ec/\nVYOGljd1opaDloOe36lT2E7UJ51OfdLp1JkjRuiq996Ly69AokaiRlMqDiRrqnHccINvfc4cOO88\nkKAhOZGcxrWRoUnus898p1afeALWr4dJk3ynW8eOhTCjQiM6jRunz4GNRI2ztiQOk2YiSBqpwhJI\nlB08eJB58+YBcNVVV5GdnQ1Y4kgnbfpvn8ZJI5glkCi6+eZ7uPba51GdDoDIX5g7dzxDh/7CEkea\nqHG7WVBcTFFtLQCV+fmUlJUxrKAgY5JGMEsgUXLw4EE6dZqM6lK+PoHlBSbyne9U8PvfZ1viSHFe\nr5fSwkLmB4278AKlvXszv3dvHBs2ZETSCGYX00XJvHnz/DWPZpPGceGF85g40ZJHqnO73RTV1jYf\ntLVtG+6f/hTq6+G++2DcuIxIHodjCSRKLHGkuexsGDXKkkYIa8K0Q7gmjMhEDhyoCHSmmtQVtgnj\ncjG/ujojT6FbEyZKsrOzmTt3PCITgSXAEkTOY+7c8ZY80oTD4aCkrIxSl4uKTp2ocDiYPWIEJWVl\nGZk8DsdqIBE4ePAg11wzj/vug+3br7LkkYa8Xi/uxx6Dm26ioKYmo5OHnYWJgfp63yUN9fWJjsTE\nTHU1XHGF72cGsyZMlHm9Xt5/v5qGBrt3rMlslkDaqekepZMm1fHFF3bvWJPZLIG0g9frpbh4AStX\nzmf//knAJFaunE9x8QKriZiMZAmkHcLNIlZbazOEmcxkCcQYEzFLIO1gM4QZcyhLIO3gcDgoKyvB\n5SolJ6cCqGDEiNmUlZVk9DgBk7lsHEgEvF4vL7zgZvp02LrVZghLWzYOBGh9HIhdGRQBh8PB8OGF\ndOwIljtMJov5x19ExovIhyJSKyJXtfB6rogsE5GVIvKBiFwW65iMORyv10v1mjVU79tnp+hbEdME\nIiIO4G7gbGAYMFVETgwp9gugRlVdwBnAbSJiNSOTMDVuN6WFhdT97GfU1dZSWlhIjZ2mb1GsayCj\ngI9VtU5VG4ByYGJIGQU6+x93BraramOM4zKmRV6vlwXFxcxfuZJJBw4wyetl/sqVLCgutppIC2Kd\nQPoCG4LWN/qfC3Y3MFRE6oF/AbNjHJMxYYWdkay21gYLtiAZmgpnA25V/b6IDAJeFJHhqrontOCc\nOXMCj4uKiigqKopbkMZkisrKSiorK9tUNqancUXkFGCOqo73r1+N7y5X84LK/A24RVVf96+/BFyl\nqlUh20qa07hgl/OnK5uRrLlEXs7/LjBYRPJEJBuYAiwLKVMH/ABARHoC+cBnMY7LmBYFZiQ79lgq\nHA4qnE6bkawVMR9IJiLjgTvxJauFqnqriJTgq4ncLyK9gQeB3v633KKqi1vYjtVATHyo4i0owH3p\npTB2bMbfTtRmJIsBSyBprOn2gtXVNt0+NhLVmLZT9d1a8IYbLHm0QebWy46ATWmYxpYt8yWOCRMS\nHUlKsATSTjalYRprqn3MmWO1jzayPpB28Hq9FBaWsnLlfIJvLOVylVJdPT+jO9rSgvV9tMhmZY8S\nm9IwjVntIyKWQIwB6/uIkCWQdrApDdOU1T4iZqdx26FpSsOzzipl586xZGfD8cdXUlb2c+v/SGVW\n+4iYdaK20+bNMHSol/JyNz16kPGjFFOeKpx0kq/z9LzzEh1NUrKBZFE0bx5cdpmDs88uTHQoJhqs\n9nFELIG0w+bN8PDDUGPDPlKa1+v1nTVTpeD663HceKP1fUTIEkg7zJsHM2ZA796HL2uSU43bzYLi\nYopqa6GxkYc6dKCkXz+GJTqwFGV9IG20eTMMG+arfVgCSU0210dkbCBZFFjtI/XZdIXRZ02YNrC+\nD2NaZjWQNrDaR3ooKCigsk+fkGGAsDw/3wYCRshqIIdhtY/04VixgpJt2ygdPJix/pmgKo8/np/b\ndIURs07Uwygt9Z3hu+OOuO/aRFNlJVx0ETzxBN4xYwJ9HjYQ8PBsSsMI2ZmXNBGUPLBbgbSbjURt\nh8AgI+DhhwuYMcNhySOFBP/9CgoKcKxYYckjhiyBBHG7ayguXkBtbRGqcPDgQzz3XAnYMKOUcMgg\nMeChPn0o2baNYU8/bckjRqwJ42ezjaW2sIPEBg9m/kcf2d/vCNhAsjaw2cZSW9hBYvX19veLIUsg\nJn3YDPlxZwnEz2YbS2E1NRTMm0dlY6MNEoszSyB+TbONuVylZGdXkJVVwYgRsykrK7H2c7KqqYFL\nLoFx43B897uULF9OqctFhdNp97SNE+tEDeH1ernuOjerV0NFhQ0ySko1NXDjjbB8OfzqV3DllfDN\nbwItnMa1v98Rs3Eg7eBwOMjLK2TbNrDPXpIJTRxlZYHE0cThcFBYaLPFxUvMvyIiMl5EPhSRWhG5\nKkyZIhFxi8gqEXkl1jGZFBPUVGHkSPj0U/j1r5slDxN/MU0gIuIA7gbOxjcaa6qInBhS5mjgHuA/\nVPXbwEWxjOlwvF4vdXXVbNtm972NF6/XS3V1NdXVIcfcEkfSi3UNZBTwsarWqWoDUA5MDCnzI6BC\nVTcBqOoXMY4prKb73v7xj3X87W9239t4qHG7KS0spG7MGOrGjKG0sJCaJ5+0xJEqVDVmCzAZuD9o\nfRpwV0iZO/DVUl4B3gWmh9mWxpLH41GXa5aCR31z/auC7zmPxxPTfWcqj8ejs1wu9Xx9wNUDOisr\nSz233qq6Z0+iQzSq6v/utfgdT4ZuwizgJOAcYDxwnYgMjncQNhI1/txvvEHRmjXNR49mZ+P+wQ+s\nxpECYn0WZhMwIGi9n/+5YBuBL1T1AHBARFYAI4BPQjc2Z86cwOOioiKK7AKp1LJ3L7z5pu/y+spK\nqK6GhoZER2VCVFZWUllZ2bbC4aom0ViADvgSQR6QDawEhoSUORF40V/WCXwADG1hW7GspVkTJhb2\n7FF98UXVa65RPe001W9+0/fzmmtUX3xRPbt2tdyEcbnsmCcRWmnCxLQGoqoeEZkJvICvdrpQVdeI\nSIk/qPtV9UMR+QfwPuDB12eyOpZxtaRpJGpxcSmrV4/F64Vhw+y+t+0SWsNYuRJcLt+l9HPmwKmn\nHtIscQAlZWWUFhcz1n8Jvk0xmFpsJGoIG4naDnv3whtvHJowCgp8CaOoqFnCCMdGjyY3G4naDjYS\ntRWtJYwbbmhzwghlo0dTlyWQEL6BZG62bQOvNzn+G0bzP3S7thWjhGHShzVhgjRNabh6dRFeL3z7\n25WUlZVQUJC4KQ1Dp+mrzM+npKyMYRFcon7YbUWpSWLSS2tNmJiehYnmQgaehQk70CqCsxRht3Xc\ncer5zW9UR49WdToPOUtiA7mMagLPwqSSww0kS0QbPew0fatW4Z48mcJvfavt29q2jaJVq5pva906\n3Fu3UmhNEhMBSyCpyOGAoUMhL6/t76mrg+eea/58p06++TSsE9NEIlzVJNyC7x/Xj9v7viNdsCZM\nbJowNmjLHAatNGHCdqKKSC7wC6AvsAzfaNGZwK+Af6lq6FW1MRXfTtSvB5ItWvTzxHeinnUWY3fu\nhOxs30CrRYuOqBP1kEFbEW7LZI6Ibm0pIkuBHcCbwDjgGECA2aq6MkaxhpWxA8k2b8Y7dCju8nLo\n0SO+p3GNIfIE8oGqfsf/uAOwGRigvove4i6e98a9/36oqvL9TDi7u7dJsEhHogYuk1TfNS0bE5U8\n4qmxsZG33ipn7VpobJxCVlYC+5k3b4aHH/bNzGVMEmqt/jpCRHaJyG4R2Q0MD1rfFa8A42nx4mXk\n5l7MokXZVFZmk5t7MYsXL0tcQPPmwYwZ2N29TbKykah+jY2N5OZezP79TxJ8b9ycnAvZtevx+NdE\nNm+GYcN8tQ9LICaBIro3roh0EpFSEblbRK4QkbQeM1JeXs7+/VMIHUi2f/8llJeXxz8gq32YFNBa\nUngIXz/Iq8C5+GZVnx2PoDKe9X2YFNHWszBZwDuqelI8gwuJJ3OaMP/1X76fdubFJIFonIVpFGn5\nYrx0kZWVxcKFl/GTn1zI/v2XANCpUzkLF14e3+SxeTM89JDVPkxKaK0G4gX2NK0COcA+/2NV1dy4\nRPh1PHEZB9LY2Mixx5ZTUgJXX52A07hW+zBJJtKBZG5VTZoxzvFKIKrQpQusWwddu8Z8d4eyMy8m\nCUV0FgZIjfO7UbZzp+9nly4J2Pkf/mBnXkxKaa1+foyI/L9wL6rq7TGIJ+HWrYOBA32jx+MhcG3K\nF19Q8OCDOFbHfUJ6YyLWWgLpAByFr88jYzQlkHg4ZIrBgwd5qGtXSrZsYZjVQEyKaC2BbFbVG+MW\nSZKIVwLxer0sKC5m/sqVgXbk+du2UVpczPzqartK1qSE1j6lGVXzaBKvBBJ2usLaWrsXr0kZrSWQ\ncXGLIonEswljTKoLm0BU9ct4BpIs4pVACgoKqMzPxxv0nBdYnp9Pgc0QZlKEXY0bJN5jQGrcbhZM\nm8bY1avB6bQpBk1SimggWbKJRwLZscNX+9i5M46ncSsqcN9+O9x1l00xaJJSpAPJorXz8SLyoYjU\nishVrZT7rog0iMikWMcUTrzHgAA43G4Kx42jsLDQkodJOTH9xIqIA7gbOBvfdABTReTEMOVuBf4R\ny3gOJyEdqNXVdk8Wk7Ji/S9vFPCxqtapagNQDrR0O4hZwJPA5zGOp1VxTyCqlkBMSot1AukLbAha\n3+h/LkBE+gDnq+q9JHjsSdwTyMaNvvZS376HL2tMEkqGRvd8ILhvJGFJJO4JpKrKV/tI87lWTPqK\n9WQXm4ABQev9/M8FGwmUi2/Goh7AOSLSoKrNpkOfM2dO4HFRURFFRUVRDTbuCaS6GkaOjOMOjTm8\nyspKKisr21Q2pqdx/Tek+gjfqNbNwDvAVFVdE6b8IuAZVX2qhddieho3IfOAnHMO/PznMDGudwk1\npl0indLwiPlvSDUTeAFfc2mhqq4RkRLfyxp677eEDUqJ+zwg1oFq0kDM5+tT1eeBE0KeWxCmbHGs\n4wkn7mNArAPVpIFk6ERNCtaBakz7WQLxsw5UY9rPEohfQhKI9X+YFGcJxC+uCcQ6UE2asATiF9cE\nYh2oJk1YAsFXIYhrArEOVJMmLIGQgDEg1oFq0oQlEBIwBsT6P0yasASCdaAaEylLIFgHqjGRsgSC\ndaAaEylLIMQ5gVjzxaQRSyAkIIHYGRiTJjI+gcR1DIh1oJo0k/EJJK5jQKwD1aSZjE8gcR0DYh2o\nJs1YAllnHajGRMoSyDrrQDUmUpZA1lkHqjGRsgSyLk4JxDpQTRqyBLIuTgnEOlBNGsroBBLXMSDW\nfDFpKKMTSFzHgFgHqklDGZ1A4jYGxDpQTZqyBDIwDjuyDlSTpiyBDIzDjqwD1aQpSyAD47Aja76Y\nNGUJZGAcdmQdqCZNWQIZGOOdWAeqSWMxTyAiMl5EPhSRWhG5qoXXfyQi//Ivr4nId2IdE4QfA9LY\n2MgjjzzCI488QmNj4xHtw+v1Uv3cc1Q3NuLt3fuItmVMMoppAhERB3A3cDYwDJgqIieGFPsMGKOq\nI4C5wP/FMqYmLY0BWbZ4MRfn5pI9fTrZ06dzcW4uyxYvjmj7NW43pYWF1E2aRN1XX1E6ciQ1bncU\nIjcmiahqzBbgFODvQetXA1e1Ur4LsCHMaxpN772nOnz41+sNDQ16QU6OenyVE1VQD+gFOTna0NDQ\nrm17PB6d5XI129Ysl0s9Hk9Ufw9jYs3/3WvxO5sV4/zUF9gQtL4RGNVK+Z8Cf49pRH6hzZfy8nKm\n7N9/SJXMAVyyfz/lF13EtBEj2rxtd309RatWNdvW2Npa3G43hdYfYtJErBNIm4nIGcDlwPfClZkz\nZ07gcVFREUVFRRHvL67zgBiTQiorK6msrGxb4XBVk2gs+Jowzwett9iEAYYDHwODWtlWVKtlv/yl\n6u23f71uTRhjWkYrTZhYn4V5FxgsInkikg1MAZYFFxCRAUAFMF1VP41xPAGhNZCsrCwuW7iQC3Ny\nWAIsASZ36sRlCxeSldW+iprD4aCkrIxSl4sKp5MKp5PZI0ZQUlaGw5HRZ85NmhFfgonhDkTGA3fi\n6wZYqKq3ikgJvqx2v4j8HzAJqAMEaFDVZv0kIqLRjHXECHjwQSgoOPT5xsZGyq+4AtauZcqLL7Y7\neQTzer24/WdeCgoKLHmYlCQiqGqL12HEPIFESzQTiKrv9O26ddC1awsF7r/fd/3K/fdHZX/GpLLW\nEkhG/kuM6zwgxqSxjEwgcb0XjDFpLKMTiDHmyCTNOJB4sgTSsoEDB1JXV5foMEyC5OXlsW7duna9\nxxKICairqyNVOtVN9EkEbXprwhhjImYJxBgTsYxLIHG9F4wxaS7jEoiNATEmejIugdgYkPSyYcMG\ncnNzU7Lz94YbbmD69OltKnvGGWdQVlYW44jaL6MSiNfr5aWXqunSpRqv15vocFKG1+ulurqa6urI\nj9uRbmPgwIE4nU5yc3Pp3Lkzubm5bNmyhf79+7Nr166IziA89NBDnH766e1+XzRFEncyyZgE4nbX\nUFhYym9/W8cbb9RRWFiK212T6LCSXtNxGzOmjjFjIjtu0diGiPDss8+ya9cudu/eza5du+jVq9dh\n39dazURVU/4LnHDhrvNPtoUjmA/E4/GoyzVLwaNfT9Hhe67F+TkWLFD92c8i3l+qCj3G7T5uLYjG\nNlRVBw4cqC+99FKz59etW6ciEthWUVGRXnPNNXraaaep0+nUTz/9VBctWqTHHXecdu7cWY877jh9\n7LHHdM2aNdqpUyfNysrSo446Srt27drifouKivTaa6/V0aNH61FHHaXnnXeebt++XX/84x9rbm6u\njho1Suvq6gLlX3/9df3ud7+rXbp00VGjRukbb7wReG3t2rU6duxYzc3N1bPOOktnzpyp06dPD7z+\n5ptv6ujRo7VLly7qcrm0srLykDgWLlzY5uMViXDfMVqZDyThiaGty5EkkKqqKnU6K4I+wL7F6XxS\nq6qqmr/BEoiqRnDcWhCNbai2nkAcDschCSQvL0/XrFmjHo9Hv/rqK83NzdWPP/5YVVW3bNmiq1ev\nVlXVBx98UE8//fRW91tUVKTHH3+8rl27Vnft2qVDhw7VE044QV9++WX1eDx66aWXanFxsaqqfvnl\nl9q1a1d99NFH1ePx6OLFi7Vr16765Zdfqqrqqaeeqv/93/+tBw8e1BUrVmjnzp0DCWTjxo3avXt3\nff7551VV9Z///Kd2795dv/jii0AcyZhAMqYJY6Jn3z7ffbJEDr+MHOkrHw3nn38+3bp1o1u3bkya\nNClsucsuu4wTTzwRh8NBVlYWHTp04IMPPuDAgQP07NmTIUOGtGu/l19+OQMHDqRz586cc845DBo0\niDPOOAOHw8FFF10UmPPl2WefJT8/nx/96Ec4HA6mTJnCiSeeyDPPPMOGDRuoqqrixhtvpGPHjpx+\n+ulMmDAhsI9HH32UH/7wh5x99tkAjBs3jpEjR/Lcc89FcKTiJyMSSEFBAfn5lUBw552X/PzlFITO\nKGQCwh03l2s5Hk9BSJ2i5cXjKcDlar6NSI790qVL+fLLL/nyyy956qmnwpbr379/4LHT6WTJkiXc\ne++99O7dmwkTJvDRRx+1a789e/YMPM7JyWm2vmfPHgDq6+vJy8s75L15eXls2rSJ+vp6unbtSk5O\nziGvNamrq+Pxxx8PJMiuXbvy+uuvs2XLlnbFGm8ZkUAcDgdlZSW4XKU4HBV84xsVjBgxm7KyEpsl\nrBXBx83prMDpbP9xi8Y2mmgrHaLBQjtGzzzzTF544QW2bNnCCSecwBVXXNFiuSPVp0+fZhejrV+/\nnr59+9K7d2927NjB/v37D3mtSf/+/bn00ksDCXLHjh3s3r2bX//611GNMdoy5ttTUDCM6ur5jB49\nkFtuGch7791JQcGwRIeV9JqO24oVA1mxIrLjFo1ttKa1xPL555+zbNky9u3bR8eOHTnqqKMCiatn\nz55s3LiRhoaGqMRx7rnn8vHHH1NeXo7H42HJkiWsWbOGCRMmMGDAAEaOHMn1119PQ0MDr732Gs88\n80zgvdOmTeOZZ57hhRdewOv1cuDAAZYvX059fX1UYouVjLoa1+Fw0KVLIYMHg1U82s7hcBzxvWyO\ndBut1RZNaK0xAAAOL0lEQVSCXwst5/V6uf3225kxYwYigsvl4t577wXg+9//PsOGDaNXr1506NCB\nzz//vF37DdWtWzf+9re/8ctf/pL//M//ZPDgwTz77LN09c+b+dhjj3HppZfSvXt3Tj31VGbMmMFO\n/9Dofv36sXTpUn79618zdepUsrKyGDVqVCDWZD3dnFFzonq9XsaOdTNpEsye3fIkxwcPHmTehRdC\nfT1XvfEG2dnZR7TPVOKf+zLRYZgECff3tzlR+fpetT9/Ywx9fzOG0sLCZveqvefmm5ncqRMnPPMM\nJ1RXM7lTJ+65+eYERWxM8suIGojX66W0sJD5K1cGMqYXKHW5mF9djcPh4ODBg0zu1ImlQee2vcBE\nESoOHMiImojVQDKb1UDCcLvdFNXWNr9X7apVuCdPhiuuYN7o0UwPGRjjAKapMm/evPgGbEyKyIgE\nEpbDAUOH+kY79emT6GiMSTnWhLEmTIA1YTKbNWHCCL5X7eM4eDzrG83uVZudnc34uXOZKBK4N+55\nIoyfOzcjkocxkciIGkgTr9fLX3qMhUmTmH7/7PCncf19HldddVVGJQ+rgWS2SGogGTeQbEjHLtB7\ncNhh1NnZ2Vx33XVxjsyY1BTzJoyIjBeRD0WkVkSuClPmLhH5WERWiogrVrHs27ePv+6s4o4nbmFf\ntC4RNYbEzm527LHH8vLLLx+2XF1dHQ6HI6qz8cU0gYiIA7gbOBsYBkwVkRNDypwDDFLV44ES4L5Y\nxHL5JTOY/M3OFBzcwgUfvcnkb3bm8ktmxGJXaScZpjR87bXXOO200+jSpQs9evTg9NNPp7q6OqJY\nYiVZh5sHi3aMsa6BjAI+VtU6VW0AyoGJIWUmAg8DqOrbwNEi0pMo2rdvH1sef4Rn8XIxcDHwLF62\nPP6I1UQOo2kEb92YMdSNaXkEb6y3sXv3biZMmMDs2bPZsWMHmzZt4vrrr+cb3/hGe38dE23hZhqK\nxgJMBu4PWp8G3BVS5hlgdND6P4GTWthWWydWamby5Mm6pIXpKspBJ0+eHPF2003oMfZ4PDrL5VJP\n0DHzgO+5dkxpeKTbqKqqCjvlYJOFCxfqkCFDtFu3bjp+/PhDphlctWqVnnnmmdqtWzft1auX3nLL\nLaqq+u9//1tnz56tffr00b59+2ppaakePHhQVVUrKyu1X79+etttt+kxxxyjffr00UWLFgW2uX37\ndp0wYYLm5ubqySefrNddd13Y2c2apl1ctGiR9u/fX7t166b33Xefvvvuuzp8+HDt2rWrzpw5M1De\n6/Xq73//e83Ly9OePXvqjBkz9Kuvvgq8/vDDD2teXp726NFDb7rppkNma/N6vXrLLbfooEGDtEeP\nHnrJJZfojh07AnEEz94WKtx3jHSZkWzOnDmBpbKyMtHhpL2wI3hrawOzcMVjG/n5+XTo0IHLLruM\n559/PnAFa5OlS5dy66238vTTT7Nt2zZOP/10pk6dCsCePXs488wzOffcc9m8eTOffPIJ48aNA2Du\n3Lm88847vP/++/zrX//inXfeYe7cuYHtbtmyhd27d1NfX88DDzzAL37xC7766isArrzySpxOJ1u3\nbmXhwoVtuuXCO++8wyeffMKSJUsoLS3l5ptv5uWXX2bVqlU8/vjjvPrqqwAsWrSIhx9+mOXLl/PZ\nZ5+xe/duZs6cCcDq1au58sorefTRR6mvr2f79u1s2rQpsI+77rqLZcuW8eqrrwYmMbryyivbdJyb\nVFZWHvJda1W4zBKNBTgFeD5o/WrgqpAy9wGXBK1/CPRsYVstZse22Lt3r47H0ey/4Hgcunfv3oi3\nm25Cj3FVVZVWOJ3Nam5PglYdfjIyVX+5ihaef9LpbNecqB9++KFefvnl2r9/f+3YsaOed955+vnn\nn6uq6jnnnKNlZWWBsh6PR51Op65fv14XL16sJ510UovbHDRoUGAOUlXVf/zjH3rssceqqq8G4nQ6\nD/lvfcwxx+jbb7+tHo9HO3bsqLW1tYHXfvvb37ZaA3E4HLp58+bAc927d9fHH388sD558mS98847\nVVV13Lhxeu+99wZe++ijjzQ7O1s9Ho/eeOONOnXq1MBre/fu1ezs7EANZMiQIfryyy8HXq+vr9eO\nHTuqx+NJyRrIu8BgEckTkWxgCrAspMwy4FIAETkF2KmqW6MZhNPppNfF0/ghjsAgsXNx0OviaTid\nzmjuKq0UFBRQmZ8fMhkhLHe5KPB42pRCCjweKl2u5tvIz2/XlIYnnHACZWVlrF+/nlWrVlFfX09p\naSngO7swe/bswHSA3bt3R0TYtGkTGzZsYNCgQS1us76+ngEDBgTW8/LyDpnAp3v37oec7nc6nezZ\ns4dt27bh8Xjo16/fIe89nGOOOSbwuD1TI+bl5dHY2MjWrVupr69vNmVj9+7dA+t1dXVccMEFgWMx\ndOhQOnbsyNatUf1KBcQ0gaiqB5gJvADUAOWqukZESkTkCn+Z54C1IvIJsABoX32rjRYteYiKvbt5\nYvJknpg8maf27mbRkodisau0ETyCt8LppMLpbDaCNx7bCJWfn89ll13GqlWrAN90gAsWLDhkOsA9\ne/Zwyimn0L9/fz799NMWt9O3b1/q6uoC63V1dfRpwzVR3/rWt+jQoQMbNmwIPBc8PeGR6tOnT7O4\nsrKy6NmzJ7179z5kv/v27WP79u2B9QEDBvD3v//9kGOxd+9eevfuHbX4DhGuapJsC0fQhDFtE+4Y\nezweraqq0qqqqnbdyyVa2/jwww/1tttu040bN6qq6vr16/W0007TkpISVVX961//qt/+9re1pqZG\nVVV37typTzzxhKqq7t69W/v06aN33nmn/vvf/9bdu3fr22+/raqq1157rZ522mm6bds23bZtm37v\ne9/T3/3ud6rqa8L079//kDiCOyunTJmiU6dO1X379mlNTY3269fvsJ2owb93v379dPny5YH1adOm\n6U033aSqqg888IDm5+fr2rVrdffu3XrhhRfqpZdeqqqqNTU12rlzZ3399df14MGD+qtf/Uo7duwY\niOuOO+7QoqKiQCfy559/rkuXLg0bR7Bwf3/SpRPVJEbTdISFhYUR1xqOZBudO3fm7bff5uSTT6Zz\n586MHj2a4cOH88c//hHw3e7h6quvZsqUKXTp0oXhw4fz/PPPA3DUUUfx4osvsmzZMnr16kV+fn6g\nA/7aa69l5MiRDB8+nBEjRjBy5EiuueaasHEEj6H405/+xO7du+nduzfFxcUUFxe3+juEjr9obb24\nuJjp06czZswYBg0ahNPp5K677gJg6NCh3HPPPUydOpU+ffrQvXv3Q5pSs2fPZuLEiZx11lkcffTR\njB49mnfeeSfsfo9URl0LY1pn18JkNrsa1xgTV5ZAjDERswRijImYJRBjTMQsgRhjImYJxBgTsYya\nkcy0Li8vLyXmtDCx0Zbh+KFsHIgxplU2DiREMk4FYDG1TTLGBMkZVzxisgSSJCymtknGmCA547IE\nYoxJapZAjDERS6lO1ETHYEymCteJmjIJxBiTfKwJY4yJmCUQY0zE0jaBJNMtNdsTl4j8SET+5V9e\nE5HvJDqmoHLfFZEGEZmUDDGJSJGIuEVklYi8kuiYRCRXRJb5P08fiMhlcYhpoYhsFZH3WykTu895\nuLkOU3nBlxg/AfKAjsBK4MSQMucAz/ofnwy8lSRxnQIc7X88PtZxtSWmoHIvAX8DJiU6JuBofBN1\n9/Wv90iCmH4D3NIUD7AdyIpxXN8DXMD7YV6P6ec8XWsgSXFLzUjiUtW3VPUr/+pbQN9Ex+Q3C3gS\n+DzG8bQ1ph8BFaq6CUBVv0iCmBTo7H/cGdiuqo2xDEpVXwN2tFIkpp/zdE0gfYENQesbaf5FDC2z\nqYUyiYgr2E+Bv8c0ojbEJCJ9gPNV9V4gHlfbteU45QPdROQVEXlXRKYnQUx3A0NFpB74FzA7xjG1\nRUw/53Y1bpISkTOAy/FVURNtPhDc5k+GS3azgJOA7wPfBN4UkTdV9ZMExnQ24FbV74vIIOBFERmu\nqnsSGFNMpWsC2QQMCFrv538utEz/w5RJRFyIyHDgfmC8qrZWPY1XTCOBcvFd698DOEdEGlQ19C6D\n8YxpI/CFqh4ADojICmAEvn6KRMV0OXALgKp+KiJrgROBqhjF1Bax/ZzHsoMnUQvQga87vLLxdXgN\nCSlzLl93Lp1CfDpR2xLXAOBj4JRkOVYh5RcR+07UthynE4EX/WWdwAfA0ATHdA9wvf9xT3xNh25x\n+BsOBD4I81pMP+dpWQNRVY+INN1S0wEsVP8tNX0v6/2q+pyInOu/peZefP89Eh4XcB3QDfiz/z9+\ng6qOSnBMh7wlVrG0JyZV/VBE/gG8D3iA+1V1dSJjAuYCDwadUv0fVf0yVjEBiMhjQBHQXUTWA9fj\nS3Bx+ZzbUHZjTMTS9SyMMSYOLIEYYyJmCcQYEzFLIMaYiFkCMcZEzBKIMSZilkDMERMRj4i857+0\n/j0RGSAiY0Vkp3+9RkR+5y8b/PxqEfnfRMdvIpeWA8lM3O1V1ZOCnxCRY4EVqnqeiDiBlSLSNPS9\n6flOgFtEnlLVN+MdtDlyVgMx0dDqxXWqug+oBgaHPH8A35DwWF8FbWLEEoiJhpygJkxF0PMCICLd\n8U1mUxPyfFd8SWVFPIM10WNNGBMN+0KbMH6ni0g14MU3U9caETnG/7wbOB6Yr6rxmKTIxIAlEBNL\nK1T1vHDPi8hA4C0ReVxVw87paZKXNWFMNEQ0wZCqrsM3f8bVUY3GxI0lEBMNR3JJ9wJ8TZoBhy1p\nko5dzm+MiZjVQIwxEbMEYoyJmCUQY0zELIEYYyJmCcQYEzFLIMaYiFkCMcZEzBKIMSZi/x8B7/lV\nACPxMwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x106608320>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "np.random.seed(2)\n",
    "trueLabels = num*[1] + num*[0]\n",
    "t = np.array([i if (i <= 1 and i>= 0) else 0.6 for i in np.random.normal(0.55, 0.3, num)])\n",
    "f = np.array([i if (i <= 1 and i>= 0) else 0.4 for i in np.random.normal(0.45, 0.3, num)])\n",
    "predict = np.concatenate([t, f])\n",
    "\n",
    "TPRs2 = []\n",
    "FPRs2 = []\n",
    "for i in np.arange(0, 1.05, 0.05): \n",
    "    TPRs2.append(TPR_FPR(predict, trueLabels, i)[0])\n",
    "    FPRs2.append(TPR_FPR(predict, trueLabels, i)[1])\n",
    "    \n",
    "plt.plot(FPRs, TPRs, \"o-\", color=\"blue\", label=\"First model\")\n",
    "plt.plot(FPRs2, TPRs2, \"o-\", color=\"red\", label=\"Second model\")\n",
    "plt.xlim(-0.1, 1.1)\n",
    "plt.ylim(-0.1, 1.1)\n",
    "plt.xlabel(\"FPR\")\n",
    "plt.ylabel(\"TPR\")\n",
    "plt.title(\"ROC Curve\")\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.gca().set_aspect('equal', adjustable='box')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It now becomes apparent that a good model should have a AUC close to 1 while a poor model should have a AUC close to 0.5, corresponding to the AUC of random classification. \n",
    "\n",
    "This concludes the AUC tutorial."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}