Naive Bayes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Naive Bayes\n",
    "### Author: _Calvin Chi_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Naive Bayes is a generative classificaiton algorithm. In a generative classification algorithm, we are trying to model $p(x|y)$, where $x$ is the feature and $y$ is the label. In contrast, a distriminative algoritm tries to model $p(y|x)$. Modeling $p(x|y)$ is useful because it can ultimately be used to calculate $p(y|x)$ using Bayes' Rule:\n",
    "\n",
    "$$p(y|x) = \\frac{p(x|y)p(y)}{p(x)}$$\n",
    "\n",
    "If there are two classes, we would use the above equation twice to calculate the probability of the sample being class $1$ and class $2$ respectively. Since $p(x)$ will be the same in both cases, the denominator is often omitted and we just compare the numerators to see which class is most probable. \n",
    "\n",
    "How do we compute $p(x|y)$? Assume that our sample has $n$ features. In Naive Bayes, it is assumed that each $x_{i}$ is conditionally independent of the rest of the $x_{j}'s$ given what $y$ is. Notice how this is different from $x_{i}$ being unconditionally independent from the rest of $x_{j}'s$. In the first case, the independence relationship would be written as:\n",
    "\n",
    "$$p(x_{i}|x_{j}, y) = p(x_{i}|y)$$\n",
    "\n",
    "Whereas the independence relationship in the latter case would be written as:\n",
    "\n",
    "$$p(x_{i}|x_{j}) = p(x_{i})$$ \n",
    "\n",
    "The independence assumption is convenient because now the joint probability of a sample with all its features given it belongs to a class can now be written as:\n",
    "\n",
    "$$p(x_{1},...,x_{n}|y) = \\prod_{i=1}^{n}p(x_{i}|y)$$\n",
    "\n",
    "What is $p(x_{i}|y)$? Assume we have two classes. Also assume that the data comes from the same distribution, but with different distribution paramters, then $p(x_{i}|y)$ is the probability of $x_{i}$ given the density with parameters dictated by $y$. These parameters would be calculated in the training stage, where we are given samples belonging to each class, and our goal would be to calculate the maximum likelihood parameters for each of the classes.\n",
    "\n",
    "Let us walk through an example. Assume that each of our $x_{i}'s$ are normally distributed with unknown parameters $\\mu$ and $\\sigma$. Given a training set $x^{(1)},...,x^{(m)}$ with known labels, where $x \\in \\mathbb{R}^{1xn}$, we would calculate the $\\mu$ and $\\sigma$ for each $x_{i}$ for each class, and use it to construct to $p(x_{i}|y)$. \n",
    "\n",
    "Let us run a simulation by generating $m$ samples where $x \\in \\mathbb{R}^{1x2}$, and each $x_{i}$ are conditionally independent of the other $x_{j}'s$. Then we will generate another simulation of samples, but this time violating our assumption by introducing a covariance term between each of the $x_{i}'s$. We will be applying Naive Bayes in both cases to evaluate its performance when the assumption is valid and when it is invalid. A normal distribution for each $x_{i}$ is assumed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEACAYAAACnJV25AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXt8FOX1/z/P7mZzJ4AQEoggoKiAQkDxAn6NF0gr1Gqr\nrVItiqjlW7+gggZRICCpCkVopa+f6U1pi2hNFbnJRSFIWqwSERQBQZTILQrhkuvuZvf5/TGZyczs\nzO7M7uw15+1rX2ZnZ555ZkM+c+ac85zDOOcgCIIgkgdbrCdAEARBWAsJO0EQRJJBwk4QBJFkkLAT\nBEEkGSTsBEEQSQYJO0EQRJJhibAzxh5jjH3OGNvNGFvOGHNaMS5BEARhnrCFnTHWE8D/ARjGOb8c\ngAPAXeGOSxAEQYSGw6Jx7AAyGWM+ABkAjlk0LkEQBGGSsC12zvkxAIsA1AA4CuAM5/y9cMclCIIg\nQsMKV0xnAD8G0AdATwBZjLHx4Y5LEARBhIYVrpibARzinNcBAGPsLQDXAnhNvhNjjIrSEARBhADn\nnJnZ34qsmBoAVzPG0hhjDMBNAPbqTC5hX3PmzIn5HDrq/BN57jT/2L8Sff6hYIWP/SMAFQB2AtgF\ngAH4Y7jjEgRBEKFhSVYM53wugLlWjEUQBEGEB608NUhRUVGspxAWiTz/RJ47QPOPNYk+/1Bgofpw\nTJ+IMR6tcxEEQSQLjDHwGARPCYIgiDiChJ0gCCLJIGEnCIJIMkjYCYIgkgwSdoIgiCSDhJ0gCCLJ\nIGEnCIJIMkjYCYIgkgwSdoIgiCSDhJ0gCCLJIGEnCIJIMkjYCYIgkgwSdoIgiCSDhJ0gCCLJIGEn\nCIJIMkjYCYIgkgwSdoIgiCSDhJ0gCCLJIGEnCIJIMkjYCYIgkgwSdoIgiCTDEmFnjOUwxt5kjO1l\njO1hjF1lxbgEQRCEeayy2H8HYB3n/FIAQwDstWhcgjDEhg3AmDHCa8OGWM+GIGIL45yHNwBjnQDs\n5Jz3D7IfD/dcBKHFhg3A7bcDzc3C+/R04O23geLi2M6LIKyAMQbOOTNzjBUWe18AJxljrzDGPmGM\n/ZExlm7BuARhiEWL2kUdEH5etCh28yGIWOOwaIxhAH7NOd/BGFsCYAaAOeodS0tLpZ+LiopQVFRk\nwekJgiCSh8rKSlRWVoY1hhWumB4AtnPO+7W9HwWghHP+I9V+5IohIgK5YohkJiauGM55LYBvGWMD\n2jbdBOCLcMclCKMUFwtCPnq08CJRJzo6YVvsAMAYGwLgzwBSABwCcD/n/KxqH7LYiZixYUO7333a\nNBJ+InEIxWK3RNgNnYiEnQgBKwSZXDVEIkPCTiQVVgnymDHApk3KbaNHAxs3WjNPgogksUp3JIiI\nEIs0RlroRCQDJOxE0jNtmmDti6SnC9vUiE8ImzYJr1tuAcrKojdPgrAKcsUQcYuVvnEjvnotl43N\nBqxbR/54InaQj51IOqKZzaIl7AD544nYQsJOEGGwYYPgfvH5lNtJ2IlYQsFTggiD4mJg3jzB/SKi\n548HKNBKxC9ksRNJj1l3jpH9KTeeiBbkiiEIFZESYMqNJ6IFuWKIqJIIrggq6dtOIvy+CGsgYSdC\nQp3zffvtsRGLWImV0dz4WKD1ncTL74uIEpzzqLyEUxHJwujRnAPK1+jR0Z3D+vWcp6e3nz89Xdhm\ndh+z5xw9WnjNn9/+czhjWone9cbD74sIjTbtNKW3VjTaIIiYoOdmkfvPxZK+VuTCq/31VVXxFzAl\n1xMBkCuGCJF4dkWoKS4WgpobN4YnwokqmidPJtbviwgfEnYiJOKhuQWJlT/TpgFOp3Lbnj3C/2P9\n+yKiB6U7EglNNEsOhJI6GYsGH8OGATt3KrdRKmbiQnnsRFITD12QzMwhVouYKMc+uSBhJ5KWRFzp\nGSvLOVLfVTzcWDsitECJSCjM5KBrBS7Hjw8/FztSefAbNgC7dlk3nhkiEf+IRR48LagKA7P5kaG+\nQHnshAyz+eVaedgA5zYb54WFoeWRq+fgdApjWZGXrjVfmy1+8t3NEigPXp7bb9X1Wb3+IJFBCHns\nJOxETDC7YEb9h65+hfKHr3ezsEJICgv9x+zfP/Tx5ERCSIOh9/uKlADTgqp2QhF2csUQCYHoXuja\nVftzq3PKI5Gj3qlT6MeKbolhw4Bbb/V3iUTabaGXWpqouf3JDgk7ERNCyUEvLgZee015XDhcf72y\n9rqVdOtmbJsR5P7tnTsBt7v9s+Zm4KmnIu//jva6BfW/D5tNWGhFvnaDmDXx9V4QbhKfAFil83lk\nn1eIhCNUl8L69YKrw2YL3QWgdiEwxrnDYX48vWuYPz+8+cnH79pV32UEaH9updsi0O8p0rV4wv09\nJwOIpY8dwGMA/kHCTkSLcHzNWj5cMXBqdDw9UVNvt9kEkQrl+gLFFcRzavnzrRL2QNdodTE0rXNF\n8toShZgJO4ACAJsAFJGwE4EIN/BnVeDQiuCc3hhGxw5mCRux1NevD/3pQH1+rfloCWthYfQCppF+\nGkkEYinsbwIYCuB6EnZCD7OP7VrCY5WgGB0rkPjqCZERMQqUajl/fnBLXS8rxejTgdb5nU7l96G+\nYURabPWeojp62mNMhB3AWABL234uArBaZz8+Z84c6bVly5bIfhtE1DBqRZuxkiP1WG7GhRBM/I24\nSvTEKFCqpZaY6o0Z6pNHoPMHEnBx3UAkhN2I26cjiPqWLVsUWhkrYf8NgBoAhwAcB9AA4G8a+0X8\nCyGsw+gfkxkr2owIae0rD24GOl5v7kaEurBQELTCQn03RGEh59nZnGdlCbnp4jFawqj3/WmNbeTF\nmHJBVrSFXTx3pKzojibiRohp8FQ4P7likgErxVptIQcTVnFfI8KnNa9A/uZgqyflrghRRLWEVb1N\ndKNoja0lVFrn0rKM9T5Tf7+hiKxRV4ze2NEQYBJ5ARJ2whLCtazlYqkWBrX7Q/zjLSxUCotaaOSv\nrCz9gKOeT1jtq1XPNVDJAiNiq+UL1hPGYNayeKxe8FTLXx9q2miw4GmsxJVKCrQTc2EPeCIS9ojx\n5udv8jc/f9Oy8cL1hRu1kNU5ylpiqefn1fojN+JeUFvageYqzsHIU4SWdW4mawYQngaystrdHVqW\nvXzOyYwVWUvJAgl7B6TJ3cQ7P9+Zd36+M2/2NFsyZrjZKyJ6f5xGg47ivlrir/VHblbYCwuV16Dl\nipFnmOi5UPTEVi9rxkjWizxwKPf7BxP1QPGFUC3vWFjtJOztkLB3QBZvX8wzyzJ5ZlkmX7J9iWXj\nWvHHrHeDMCLAcrE06r82esPQEwotwVX7lcXAaXq68MrOVgpuoJiCfMxArhb5TcDM9x8oqyTUbJNY\nuUTIFdMOCXsHQ7TWUQqOUlhqtVuFlniYFXY9X72RFZF6rh61lR3opqD1lGE02Kgn4GaeXPRKE6uv\nVes8Dod2NpHR/PBYWs4UPBUgYe9gLN6+mDufdUrC7nzWaanVHinUgmazCb5lLfHRyzk3Ijjr1wvp\niOr9+vc3l1ceyC8uf2VnGz9WHmvQCh5rvdRPD2aeTtQvo4uMyCUSe0IRdqrumMC4vC5clnsZhucP\nx/D84bgs9zK0eFtiPa2gqCsFrlsHXHON/367drVXLCwrEyr+bdxorKqgWBHxq6+E9zYbUFgIrF8P\nHDyoHGPDBqC6Wnuc9HShCuSHHwY/Z3299vZAlSyLi4Vr+uQTYNUq4fswUppYXS7XDOnpQJ8+xvYN\npQqnHtQRKYqYvROE+gJZ7EQAtKz4QJZiMB+sUUtTz/IV3R9Gl/drzVvLPx8oCGrEeg+Wnhno5XDo\nxyNstvbFVlYGXo3+vgh9QK4YIpEJtkDJTP62UWHX2k8MiBpdKCV/aVWI1ErtDFaewOkUhFbrmGCp\nounpnE+Y4L9dXUMm0Dih1M4JBLl0QoeEnUgawrXw5s/3F5IJE4wFco3UatGz1rUCnHoWf7AceSOZ\nP/InC/W1iQFVMcVSPiejuflW/U5I2EOHhJ1ISPTcFHrCZIRggi1mrKgt1nBEXWuOgVwmwc4biWCm\nmaCrelwjQWA9S55cMaETirA7YuPZJ5KdDRvaA33TpukHPDdsEHp4iu3e6uqAceOE4N7XXwM+n7C9\nrAy44orw2rGJYwFC4HH2bOW2/v2FvqQ7d2ofb7cDXq9yW9eu7YHIrVuV/w8UZGRMeW75zwDgdAqt\n4MaMCfz9mcVo0NVMkFQMVIvjVlX5t84TA+byfxOAcH3i+0i22utwmL0ThPoCWeymCLVMQLDjrC4/\noIVR62z9+uCLdKyyTLWKd+lZ3WaCpXo1aMTx8vO1P9ObD2PCePLPA6U5GrF8jbhexLiCVvA02O81\nlKcIsuCNA3LFJAbBxDXUMgHBjrOq/ECw+Wv9oV9+zQnFMaHkYRsVdnl2iShUWvnseucQXUNG3CdG\nxw33pc4IMhrA1ArKat1U9OrvBPpOA8Upgv2uyOduHBL2BMCIuIZaJiDYcVaUHzAyf60/WseFmxXH\nmBXEUMvRiscZzW6Ri4vWMepMlVB98uEIu1H0nojS0sydI5h1HYr1TcJunFCEnRYoRZny6nJ4vB54\nvB6U7yj3+7zZ04y5W+ei0dOIRk8jSreWoqU1+KKjYMeFOq7Z+QP+i1pSUj2wj1oiHVNW1r5wSI7D\nIfi5nc72beLCIrXPVg+1D1m+qEdNfr4wvvxc11/f/v6nP1V+np4u+ODVvnFbhP+KQlkUJPq96+r8\nP2tpEXz8RnnqKf/v9Kmn2hcbAcoFZ0Z+V1YufCL8IWGPIkbEtby6HE2eJul9k6dJV0BFNmwAho76\nDmf/9CZwcIzmcZPXTkaDu8HUuKHMXwyaXnKJIMg33uSF8xd3wXXBKumYRS9yzfHXrBFWhYorMMVV\nqZ98Ejj4amQ1Y7du/tsGDwbmzWsXZp9PCKgOGyYEa8vK2kXcZgOeflp7nCFD2uc7YYIQZFWTlSV8\nH1qfyXGo0hnS04XzAuZWbQYLknLVr0AurPLvtKxMWAGsRr4q+PbbhW0bNxpfGaxefWz0xk0Yg7Ji\nooieaE+9eqq0TSwTICdQmYD2jIQ+APqA1YxC/8nTkTPoQ+m4Zk8z3vj8DXDOUZhXCBuzBR03lPmr\nsyPS04Ff/mYVqhrWAG3ZJA1fjITvrA+AvsIVF/sv+dfKsCkrU2a2VFUJIlhVpZzDtGnAjh3A++8r\nhfr664UMFrUFvnOnf2aMzyfsO22a//jPPSfMST0fOddcI4jesGH6WTeAkGFz6FC78DY3A6WlwnzF\nzCGtrJNQsdmEG5N4Derfofw7k6POMFq0yHipB/nvcuPG8K+B8IeEPYoYEe3+nftjxsgZuGPQHYbG\nVFtm3JOGvnuXYuPv2reVV5fDbrPDbrNjwpAJihuJlfPXcoO8/9pQXDZJOObIJ4Pw3YqXwb3aoq4l\nDlpCM2+ekPqoFtHmZuDFF4WnBUCwrkUrVG59A8LPZWXt+xpFnbZ3/fXCz089BXz6qb8lLCK6eJ57\nTpneqeb0af8xWluV79U1YwD/dEH1DUh0b6nP6/MJ35N4rPp3qCXq2dn6dXECYSQtkrAIs075UF+g\n4GlQQslaCRaEanI38Yz5GX6lfSOR9hhoLk3uJu64cLPpAKHeQqNgwdBgnZzEV6B0xWDBWzOZPWJT\nD/lirP79/Uv/Gg3yGim7q7VqNVhVR6MLu0JJVaSAaWgghOApCXscIc9aue/t+wwJb7CMhBeqXpBE\nXSztu6BqgekbiJEbQaC5LN6+mNv6v2dKODnXF2Uj+e+ioAXaV57eqJfh4nCYm5vefPXqyqvFN1j5\nXrGOjN45Qv0d6X2uVa4glJoxJOyhQcKewKibZrBSxnOey+HLdy03JKh6f2TjXhvHbaU2biu18YJF\nBXx4+XA+7rVxptIezTxJaM1FurZ7xnA4GhUCFWhBjDieluAG65eqZdUGu5msX69dU13eQk+OGWEX\nr1O9XexxKs8Pnz9fuz69/EYTaMHVhAmBf59Glv+HW81R77xWL0rqCM04SNgTGHXTDJSCp8xL4Wnz\n00JeUKTVYamuqU6xLWN+Bl++e3nQuYWT/664tnvGcNZ/Ex8w4hvDf4jqTkiiIATrkKTl1pBXbtQT\nNbnFHKh5tJZ1zZhgTcu7FoljGL0RaLk7zL7M1tYxUlZY3E/9hGFGWPX2D0WgO8rq1VCEnQnHRR7G\nGI/WuRKRF/79At7c8yZ83Ifdtbvh5e1FSTIcGfjNTb8xHfRc8uESlLxXArdXiJg57U4M6TEEnxz/\nRDF+uiMddSV1SHOk+Y3R7GlGzxd74kzLGQBA57TOOD7tuOa+wa5Nzp2D7kTJyBLDY+hlxmzYAIwf\n75+vXVgoBAU3bVJuHz26PRMj0JhG6twA2lkuo0cLx6nHUNfFCYS8Bs3hw9r56MGOP3XK2L5a83I6\nhbRTdXaSPPipDsimp4cWDNXKpjIyzpgxgX+/yQJjDJxzEysPQBZ7PCD3X2tZ7qK1vXy3v1smkO/7\n+arn+fDy4dKr8OVC7pjn4Pa5dm4rtXFWyrit1BawpZ56Pva5dn7f2/cZuq5oPSbr+W71LH1xbkat\nPa3rELcFCkZqHWe2vnuo1nvXruF9f/LrCLZfoGNCPb+RcTqKzx6xcMUAKACwGcAeAJ8BmKKzX6Sv\nPyFR+69FMS54sYCzUqZwy6TPT1e4Zcxm0YguFeezTp4yN8XvxqE1hvzmUPhyIbfPtfO0+WnSvno3\nlmg+JusF/NR1y+XuCaOiYGRstRBr1U0PVDAr2Et0jWg1pQ7kijF6Y01UYSdXTGSFPQ/A0LafswDs\nB3CJxn6Rvv6EZPH2xTz12VSe+myqwmpWW9sFiwq481mnws9txvet9rerX0YaYavPF+jGEm1rSi1i\nwc5vtLa4loWtZaV37WqsMJZW1UmxJZ3o41YfJxboMmLtOxztom7miUQdK9CKLWgVFFOna4YirOEI\nNAVPIyTsfgMCKwHcpLE9ohefiGgFNwNVZQwUBA1mteu5eFgp4+e/eD4fXj6cP1/1vKm5LqhaoHtj\niaawy/+4xdS8YPnaWiIliqtcrLSCs3rCLhcWIzcOeTaMPCNGr+lGsFRIeVqm2e8/ksFTq/bpqIQi\n7JauPGWMXQBgKID/WjluslJeXY6zLWel9/Wuer8SA+J+8qX8De4GTFw1MWh5goo9FQCAOwbdIa0a\nPV5/HMcajgEAemb1RH52vqFAplY5gdmVs6VaMaVbS9E9szucNifuGHSH5tL7SBR5Ugfe1ME0EfH8\n8sDo008LZQJOngT27NFe6i8W+RJXYKanA48/Lqxala/QrKsT5iEf0+lsDyzabMK2DRvag4JagdSq\nKqBvX+0iacXFQkBz0SKguto/oKpuAmIGdRkHM/sFOk5v5bBY/8bs+QljWCbsjLEsABUApnLOG7T2\nKS0tlX4uKipCUVGRVaePW+TiquZ082lwcMU2eaEuEVGUxYwZzjka3Y0Bl/c3e5rx4JoHAQDjLh6H\nkpElKBlZgiUfLsEzm58BADw58knDmTbqcgK1jbU4Vn9Met/obsSkVZOQ6kjFuIvHobg4Da+/6cJP\nH/0PAOCN341EcbHTb9xwCVbsqmtXYPjw9puK1pL2RYsCZ6oMGdJe/EvMcLniCv9sHHVXJqdTqFYp\ndoLauVM4/9tvA7/+tfY5m5uB777zv5mI8xcFUCsjhPP2sgzRurEGQ6tEwezZ4XfDSmYqKytRWVkZ\n3iBmTXytF4QbxHoIoq63TySfVuKSYMHN21bcpuvn1gpKSsHPeU5+/SvXB1y4pOV/N+r6MUKwGMCb\nn7/JJ7w9Iez678EIFtAz4lcPNEagBhRGgolabhujmTFi6QS9fPtgfVLjwb1hNDBL6IMQXDFWle39\nK4AvOOe/C7pnB0KvdnnFngos370c679aL22zMzsGdR+EFm+LZG0/uOZBydUhL5nr9rmx9fBWTFo9\nSbOmul553VBKAovzFZ88REpGlmDHQzuw46Ed2Hb/NjR4GuD2uqXzTVo9CX/b9bew678HQ13XW45R\nKzXQGEOGCP/XKpmrPs5oXfbDh43tpy7QJae4WFlyGPC/3uJic6V09TBaGlmLadMiX6+e8Cfsr5wx\nNhLALwDcyBjbyRj7hDH2g/Cnltjoiaso2pNWTYKPt5fOEysvlowskW4ITe4mTF4zGYC/jxsQ3DZa\nwqzlky/fUS65U4bnD8fw/OG4LPeyoKV7tW4ycir2VGDy2smK89W76lHvrhfcTAfHAH/bgLN/ehOP\nLn3X2JdnAnVd7/nz9Wt8q4VYbBi9aJHg8y0s9G+8cfnlgvtEXntcFDf1uefN828e8fjj/tvEhUdG\nqK7WF9Snnxbq1UeyprnoI1dfv1GxN3IDIqyHVp5GCK1VnwtuXgAOjmc2PwO31428rDzkZuairllw\n1D58xcOYMmKKYqUnA0NdSR3Kq8ux4rMV2FWr7HqgtRJUvYoVAOYWzcXT/6OKWCFwDEC8DtEnX3Zj\nmcInL65KbfG04NLul8LGbMqVswfHAK+/DbRmABA6Ka1+JyWmvlUxeCoGTNWrJnfsUPrJ5b5ukUCr\nG7VWraq3AUpfvxFCXdUZLlq+/MJCYN8+cytFzazmJZSEsvKUHpIihJZ1XO+ul6x4j8+Denc9Nt27\nCadbTuN0y2lMvWqqn7XNwTHm72NQMrIEfTv39TuPmEkjR3ST/HLIL5HmSEOaIw1Zziy/Y4NZ48E6\nJolPFuLThnhOu62t3vp/pkmiDgAeV4pumzotwnEB6CG6J7p1UwYvxRrnWo03Qhlf7v5QbxMt/a5d\ntcew2YSOS3ICtfiLxPcUiMOHjbcfFLHKLUQYg4Q9Qsh90OIry5nl5+Oe+M5EhR/e5XVhUPdBsLP2\nZhQfH/sYf6r+k8InDwg1XvKy8jTdKUbb8AXqXxrIJ683vvyGlp3aKbQvD/ougFDGCVf0tNwI4Y5b\nXCxk62gxb57QcckIWt9TWZl1Qq/Vm9SMK4mIDeSKsZBgbg11MSwf92Hvyb2S4IpulZd3vKxw4wCA\njdlgZ3Z4fB5p28KbF2L6yOma59ZzBYmuFCPFvbSKd13c7WKcazmHenc9th/Zrjs+AKxa68KPb/cC\nHsFqT0vnWPk2M2SxWVHgKVBxKb3PAP/tYm46oO1KCaf41S23+D8VFBYKnZaMnEPre1KnSobrwgnm\nSoqVm6ijEIorhlrjWYQ6b1yr+qGYSy4iiq+IaBGLVru8yqOP+5CflQ8AOFp/FACw+ZvNmD5yuua5\ng7WxM9J/VZyveNMYO2As8hfl46xLWFQ1NG+o4slC/eRw6Lz/B3bXBvB/PwYAGPbzD1FcPDvAt2gt\nWq36xDzv4mJBsF98Ufjs8cfbhUne+k70B8sX1IwZoz+uGYqLhawb9cKoXbsEX/8llwhujz592nuS\nGiHUfqSB5qk+Xus7IuIHEnaLEN0aADRXj2qhJ74lI0uQak/Fk5ueVJTXPdV0Coy137grv6mU0hjV\n51bfRNTpisGEXy7m4k1j5qiZigVUfTv3xVs/f0vz2po9zSitLAXvfxboL7iQ/gPgTMsUdE7rHPiL\nQeQX2GzYoFw9WlbWvmgmmqsgn3vO32oXF/GI2wIFWdXfk1awNxLQStH4hlwxFhBKzXItt4182wv/\nfgFL/7sUR+uPSqtTGZjfStXf3PgbLPjPgoDnFucHwFAtdfn+M0fNxNytcwEAXp/Xzyo/XXJaU6iX\nfLgEj2943G++t19yu+7NQE24mRSBXDHhuHpCrR+uh1ZNdzVGM3Guv155wyI3SeJDrpgYYcStIUfL\ndaJVAgCAwsd9oO4AzrnOKcb6x2f/8MshV5/b7NOEuD/nHHMq56C5tU0lDo4RMl0A4NpFwIUbMfGd\niZpC7fK6kOXMUlj4DpujfSwDhGMVimJ3ySXC+27drHMZiFktVrki1P50s1a3+nu64gpyk3R0SNjD\npGJPBT4+9nFAt4YaLaGVb5u8ZjLGXjTWz52ilZ/eK7sX0h3pim317nrpmOW7l+OZzc+g0dMIQCjW\n9fAVD+ta7fJsFwWqnHTUjIL97jvRfKG2UKvnHk2MWNThunqsdEWobxRaVnes5kYkJuSKCQOzLg75\nMXLXyaEph1DwYgGaWgXLm4Eh3ZGOP936J4y/bDwApZtGvmjozoF3YuxFY3Gk/ojfQqJmTzO6vtBV\ncZPRyl6Ro86mAYDz0s+D+9U1qN97tWLfeG1DZtTNEs+LZqycWzxfJxEccsVEmVACplpumwkrJ0ii\nDgiLkppamzBp1ST85NKfgHMuuWlu6neTwqJetmsZ3t4n5OmprfLy6nJ4uRc22NCrUy/kZuYCCPw0\nIQ+qik8Bje5GXNvzSmzeq9y3tqEWQA8AwVM945FIWrbhiqlVc1M/vYgVLUnckxuy2EMk1CbPWrnh\nDa4G7K/br7n/wtEL4bA5JGt8dP/RWHdgncKitjM7OLhUe8Zpd6LshjKUVZWF1YRa/mRwT8bf8beZ\nt7dnaDiakPqLu3Hmz2+Ac276ySWSWB3cTLTzy+koDZ+TmVAsdhL2EAm2AMgoWu4SOemOdDjtTil3\nPM2Rhku7XQoA2FW7SxJzO7Pj8h6Xw8aEZZL5WfnYeGijYn7jB4/H2IvGBrSq5WmO8huX0+bEY93X\n45O3bsDXZ77GwYt/BVy4EQtvXgiH3aFbTyZcQrV8Y+l+iCcxjae5EKFBrpgoEiwPXAstd8VLH70U\n8Dhxmb6Ij/swYcgEuL1u7DzRniMn1msRRfWFf7+A4w3HFce9vud1rNy/Em6fG9XHqnFNwTWKucgz\nc2aOmqlwGbl9bvz+1Fgcfecoei4aCLTNeVblLKTaUw0HZ80QjhuBAogC8dJwg4guZLFHCb1A649W\n/Ahrv1zrl+8NCFZ4j8weYIwhLytP2p7mSMOx+mM4fEYo7C36z/Va3FXsqcCaA2tQ8UWFUIgfHC2t\nLeiU2gknpp+Q5nLfyvuw4rMVSLGn4Ia+N+B4/XHUNtTiSP0RaT63XHgLVh9YrRhfnl8f6pOLFsGs\nzXgNCsaTK0acTzx+T4QxyGKPY/QCraN6j8LxesGyPtFwQioXIDKm/xi8ctsr0vu6pjp0W9hNcSOo\nd9fjy/9YeemDAAAgAElEQVT7UtNKbvY0Y9LqSZIrR448376uqQ5/2/U3cHC4fW5U1VTh0JRD6Pu7\n9oqSXu7FuoPr/MbJdmbjovMukt4He3KxgmgHBc2Io9V57uFCTy8dD7LYo4A60JrhyFCkMor8aMWP\nsO5LQTjzs/JxovEEUuwpOF1yWhLt21+/HSv3r1QcF8hKXvLhEjy56UlF8TA5Oak5ODH9BO6uuFsx\nrtPuxC0X3YJV+1cpGoIwCIaDjdkkn76RZthynvnjNqx+dQB6ZPUIKHqRWjlqlnizwImOBdVjj1P8\nUhzbUhnlZXSbPc2oqqmCr+2/U82nkOZIg53ZpVK5dU11eGf/O4qxe2b2REF2AT4++rHfecXFRnqi\nDgjW/kv/fcmvTV/PrJ74+vTXyM/KR0ZKhvTqlNoJDpsDaY40qQZ7/879/WrR6LFqrQtljwzH7u09\ngpbjVXcoipWY6hUTSySiXbOdiC3kiokC8kBrbUMtjtUfg5d7FS6Z8upyNLrbV3u2eFuAtvpfYkDy\ngVUP+Pniv2v6DpnOTLy972289tlriqcArXZ6gJBpIxYT65LWBR/UfODXpu9E4wmccZ1RxAPEJw+P\nzwOPz4PSraXISc3BI+8+ghR7im5VS3nQ+Il5JwBPe0HvYNUH9dwIFBQ0DuWydzzIYo8CYtMNsemz\nDz64vW5F8wuX14UemT1g0/iViL7wFm8LUmwpis9aeSsa3Y2aTwEurwvZzmzF/jZmw3M3PYdlP16G\nZT9ehiOPH0GOMwcF2QVSt6fczFx4fV6/Bhxai6smrZ4kdITSadYh79J0uvk0DtUdCu1LVBFNa16r\n2UQi3USS4YmDMAcJexQJ1JFoyogpkugDgA02FOYVKppOv/uLd/Hsjc9KAlyYVwgGhlbeCgBobm3G\n0o+WSuNPGTEFje5G2JldGqswrxD17nqF2K49uBZ1LXWomlgl3HzcDfD4PH6dl9Tt/rpndJfKChvp\n0jTxnYlC8TBH+3eQkuoJWSSj1W4tXlxCBGEUcsVEkUC572rR98GHwbmDceuAWxW55vLiWgv+vUCR\nyw4As7fMxiMjHkGaIw3l1eWw24RVqUN6DJGya5Z8uETK0BFb8wFCtg4H161UKT+3uLBKjrqq5fLd\ny/H0+09L5RLWfLkGl117Geo7P4naTfcAAO6cVIPi4p+Z/i6jSaKnC5LbquNBwh4lKvZUoH/n/tjx\n0A7Nz0XRr22sxbFzx8DB8frnr2P1l6t1fdfbarb5bXN73SjfUY6Hhj/kV1Nm8Q8WI9Weqtj+zv53\nJL996dZSTLtmmqGFV+XV5YqFUwwMPTJ7SPs2e5rx4OoHFWV6W3krxg8ej+kPT5eNpCwsFm8kg386\n3tIvichDwh4FRD+zx+uB2+vG+MvH++1TMrIEU0ZMQc8Xe0ruGI+vvdG0OpVRzKKRc0HOBTgv4zy0\neFv8ngA4OCa+MxH/0+d//LaLNHmakO3M1r35yGlwN0jlC7TSHuUFyLKd2VIp4a01W6U+rYlAoPZ6\niQTlsncsLPGxM8Z+wBjbxxj7kjEWmyLcMaZiT4Vuyp/oZ270NOKB1Q/4+aHl+6mzWBo9jSh5rwRT\n350adN8j546gamIVSkaWSH1T5T1J3z3wLhrcDZKfvFd2L2SkZKBXdi+FL19+LXrXleXMQpojTZH2\nKHfTzN06F26vGz74cM59TkrjrKqp0r1+giCsIewFSowxG4AvAdwE4BiAjwHcxTnfp9ovKRYoadV7\nCVSXXb04CQAW3rxQ02oVKz8erDuouVJUbENXsacCK/etxBcnv1A0vFaPHUqhMvm1HJpyCP1+38/v\nuoJVttSq6S5iZcmBaECLk4hYE6sFSiMAHOCcH+acewC8DuDHFowbd8hT9+RWpzzzQ53yp2VZz6qc\npWm1lowswbb7t8HV6vL7DBACneIc1h5ci7sH3w27za7YRz623Gq3MzsGdR8UdLm/OotF/HnymsmY\ntmEaKvZUBMzuEc8b7KkgUaCMGCIRscLH3gvAt7L3RyCIfcSIVVMHrXov6lZy6uqGLq8LuZm5OHqu\nvSm1q9WFyWsmK2rAiNd0pP6I5GNXs3LfSizevliawweHP0BWShbqvHXSPmLwVMxiSbWnSiV15dUf\n1VTsqYDL69INrL6661UAwuKmp697OmCAVZ49k4gNONSQf5pINKxwxfwUQDHn/KG29/cAGME5n6La\nj8+ZM0d6X1RUhKKiItPnC6UdnRXouR9e3vFyUHeHvLmG2JVIXgNGfk3TrpmGlftWoq65DifqT6DZ\nq+wp6mAOKW89JzUHjZ5GeH1eDM0bKgUz7xx0J/p37g+X14VH3n0kaLMN8fyuVhdafa0BSxAA+q4k\nve8MiI8GHASRCFRWVqKyslJ6P3fu3JhUdzwKoLfsfUHbNj9KS0vDPlko7eisQM/9YKQuu9yClXcl\n0mpkne3Mxrb7t6Hniz3h9rkVy/9drS5J1AEo/PDjB4+XxFYU1BZPi8L6V+eZy6/N4/XAy73okdkD\nPbJ64ETDCZxuOQ0Hc+Cc+5xi/1lbZuGRq4Rc+UAWeax+VwSRyKiN3rlz55oewwqL3Q5gP4Tg6XEA\nHwG4m3O+V7Vf2MHTUNvRWYFWSzuzVQ31Gln3+30/xbaZo2Zi7lbhl6luTO3yujRrt6c50qQnAPHm\n4fa6kZeVh9zMXJxuPg0AeOiKhxRzDvSdNnuacd6C8xS56CILb16IX4/4teGgcTR/VwSRTMQkeMo5\n9wJ4BMBGAHsAvK4WdasIFrSLJGK9F/nLjKgDwOS1k1HvqpfeN3maMHHVRMU1NbobMadyDho9jYpl\n+vJVpFq0tLZg6X+XKnz+Hp8H9e56bLp3E+pa6lDXUoepVwVOm5R/p+XV5ZqZLYCQj24maBzN3xVB\ndHQsWaDEOV8P4GIrxgpEKO3o4oVmTzPe2PMGAKAwr1Dyh7e0tiiuqbahVtHSrsnThKX/XYqyqjIp\nqKnHlm+24PPvP/e/ebRlt7T6WjF5zWSMvWgsth/ZjmsKrgn4ne44ugOd0zrjVPMpAECKLQW5mbnI\ny8rDVb2uUgRan978tF/QOFF/VwSR6FCjjSix5MMlmPHeDADACze/oOtvlrt8RPfJwO4DFY2ptbDB\nhiE9huDz7z+Hj/vAOQdjDINzB2P/qf2K9MpOzk6od9f7tcaTI7pSzraclZ4S5O4UrVz1haMXYvq1\nibOqlCASAWq0EaeI7hGX1wWX16VZBVFEXuJXdJ9kOjOlPHB5XnhhXqG0stQHH3bW7oTH54HdZofd\nZkeaIw19O/dV1FoHhOYaHBwN7gZMXjNZcx6i20hdckB0p4gWeUGnAtja/tv6zVYrvi6CIMKEhN0i\nKvZUSAt41JRXlyvcI/Wu+qD+5vLqcjS5m1DfUo+39r6F0y2nUTWxCkceP4LGmY048vgR/HLIL/0W\nKAFCLrtYdnf9wfW45LxLFJ+LYu3lXry661XFqlgAUv9TcUWrWPZXvrhIXEzV4G5oLxfwLZULIIh4\ngITdAsSG0Ys/XOy3KhUQCmbJ4Zz7bVOPN/O9mXD73PDCC4/PgyZ3k591LVrNvbJ76Y4lim4gJr4z\nUfFe3anJbrP71YMBKEBKEPEKCbsFiAIn1jJXi1uWM0thWfvgQ6ojVXe8lz56yW9hktvnxrJdyxTW\ntWg11zXXqYdAii0FXdK6oGdWT+w7uc/vcznvHnhXuhk1e5r9+p/qlSJQN95ItHIBBJGskLCHSbOn\nGaWVpdJqTXXLO6C9rIAIAwvoj678plJzu1h6V45WSmKvrF6Yff1suFpdOHruqJSBo4cPPkWKo7r/\n6YQhExSWuljx0YoUUIIgrIeEPUzKq8ulWuMiah/6lBFTFD52Dq7rj272NGPL11t0z/fuQcG6Fn36\nHx/7GEPzhqIguz2IWdizEE67E02tTfBwD3IzchWBVhGx/Z7c0hZvQuJY8uYZgNAVacLKCZouJ4Ig\n4gMS9jBxeV3Iz8pHRkqG9MrLylOIYTDxl9c8L68uh9unn9bo4z689N+XJJ/+ugPrsOneTVK/VLHm\n+ezNs6VjTjafxN2D7/YfjMHPdz5lxBRFQLTeXS8tahK7IjW1NsHd6iZ/OkHEKdRBKUzkdWAA7WqG\novifbjktbeuS1gUt3hapDC8AjLt4HFxeFwrzCqX9TjScAADkZeVJ2z6o+UDh01evXq131StqtLe0\ntqDym0rkZeVpzkFOeXU5Gt2NirHEOi9LP1oqlRdoam3yq2RJEER8QMLehhXlZdUiLQqeWvzlyBtL\nl+8oD7iveI5uC7opfPrrD67HoO6DJF+6ulEHA4PNZsORx48EvQaX14UemT1wrP6YYLNzHxrcDWj2\nNGNO5RzFvo3uRiruRRBxCLlioN9AwyyBaqfonVdcli+vCxOIpR8tRVOrsnGHx+uRXCo7HtqBcReN\nQ7/O/aTXsPxhGNV7lKFrmDJiiuTWAYR4QKo9FeXV5X5zE/3vgdoCEgQRfUjYYV6QtQgm0lriF0oe\n+Ot7Xle8tzM7clJzJIFdvns51h5ci1PNp6SVq2IfVD3UPn6tjk9nWs4osmvszI5fXfErTBkxxZKb\nIkEQ1tHhXTHBOiAZRU+kxZK7Wi4arUJZHx37CBV7KjRdQnVNdfjk+CeKbT7ug5d78dCwh9Dv9/3g\nanWBgcHj84AxhhRbSkB3iZaPX93xye11Y9eJXbDb7PB621aj2uzIcmb51VwXF0slcsckgkh0Oryw\nBxJkM+hVM6zYU4E1B9ZoNpzo37k/ZoycIYng8t3L8fCah7HuwDrFDUDkgVUP+J1XHkD1eD1+tdPF\nvHq9m5VamEXLXl17vsXb4nd99e56LNq+SLopzqmcIzUF0Zo/QRDRocNXd7SigYYegSokcs4VTSo4\n5+j6Qle0eFuQmZIpNdiQjyV+rgUD063V7rA58NvRv/W7WYXbDENd4dHGbHDYHEixpeDOgXdi7EVj\nyXIniDAJpbpjh7fYg2WhhIO81ICI+ETAwRWWssvrkkRb9NF3z+gOp92JOwbdIawIldV8sTEbGJiU\n1qgW9RRbChw2B1paW3Rr0wR6WjGSJSR/ShF7ubq9bri9QvmDlftXkuVOEDGgwwt7pBB996I1a2d2\nXN7jctiYTdOF4Wp1KY5vcjfhgdUPIM2RJvm+5a4QMb89NzMXn574FBwcPbN6Ij87H4Dw1JFqT5X6\nq2Y5s/zmqOc+0osJqBFviqK7ac/3eyQfvLxuDqVDEkR06fCumEihdlM47U4suHkBpl49VdOFwTmX\nrG4GhpzUHDR6GuG0O/3cMurziOIt3y8cN4vemFrIG2df2v1SAMDu2t3SkwT1OiWI8KBGG3FEoMqH\n6s/ys/LRM7un9H5o3lC0eFukmup6+e3qFMsnNz0p7RdqSV2zufVi8FUsFqauEU+lfAki+pArJkIE\n8t0H8+uLFr2InktD3cDD7XNj8fbFeOq6p0LuOWomS0grVXT6NdOp1ylBxBgS9jjEqChrBUT/sfsf\neOq6p0IOCpu5IWjdBLKcWdjx0A7T5yUIwjpI2OMQo6Kc5cxCqj1VUWJg78m9ONNyBp3TOkf03IC5\nm4AVtXgIgjAGBU/jBLPCpw6Oyrn9ktvx1s/fsnR+4SDOFQAFUgnCJFEPnjLGFjDG9jLGPmWM/Ysx\n1imc8ToiYn0XvXoregW2tGq6iMjL7sYDVtTiIQjCOGFZ7IyxmwFs5pz7GGPPA+Cc86d09iWLXYVo\nyYr1Xbzci7sH341XbntF8TmgtHQr9lRg5b6V+OLkF4rUQgBwMAd+O8Z/lakVhOJOCXd1K0F0dKJu\nsXPO3+NcapD5IYCCcMbraIiWbHNrM5pam+DyuhQNq7UsXXHx0NqDa3H34LsVqYUA0Mpb8dT7T1le\naTHU0sahpl0SBBE6VuaxTwTwroXjJTXqVEERsWG1Xj65XOw/qPlA0SRbGru1GUv/u9TS+crPO3nN\nZMP11wPl8xMEERmCumIYY5sA9JBvAsABPM05X922z9MAhnHOfxpgHD5nTnsHnqKiIhQVFYU+8wQg\nkOtCvfpUTpo9DfNumIdntjyjWLk6/4b5+E3VbxRujWnXTMPLO17G943fo9XXKtWTGTdgHFbfvdqS\n61C7UxgYOqV2wonpJ0y7VCg7hiACU1lZicrKSun93LlzTbtiws6KYYzdB+BBADdyzl0B9utQPvZg\nmSBiVcnahlocrT+qKOJlZ3b88MIf4njDccUx+dn52PjVRr8yBQ8Nfwj5i/JxznXOr4qkFb5srZtQ\nii0FC0cvNOXLp+wYgjBPLLJifgDgCQC3BhL1jkiwTJCSkSXYdv82NHgaNMvtXl1wtdTqTnyN6j0K\nBdkF6Ne5n8KtEaiKpBWI7pTCvELYmeDT9/g8hlr5yaHsGIKIDuFmxRwA4ARwqm3Th5zz/9XZt8NY\n7EYzQfTcMfKCYVrjAu0Wr/pc8iqSVtWV15uv3jy1oOwYggiNqNdj55xfFM7xyYrReiuiJXyi4QRO\nt5wGAHRJ64K8rDzNAKO629HUq6f6nUssxhWJdMdQ688A1nWqIggiOFRSIAIYFUAjy/fFYOPYAWMV\nWTQz35+Jh694OCyxDXZOdYAznKYkkZgnQRDaUEmBOEbuepk5aqYiSwYAFt68ENNHTo/YOclVQhCx\nh+qxJxnqnPXLci9DQXYBbG3/ba3ZGtFzUoCTIBITstjjFHWwMcORgT/c8gc8tvGxiAUgKcBJEPEH\nWexJhF+wsbUJD615SFHgy+rl+bT8nyCSAwqexinyYGNtQy2O1R+Dj/uQn5WPHlntC4G1ApChru6k\nACdBJAfkiolzzLpHKPhJEMkFuWKSELPuEa3gp15Nd4IgkhNyxcQ5ZtwjWs2lfznkl3hwzYMAgHEX\nj7PcgqeiXgQRf5ArJonQWvJ/y4W3YNOhTQCAshvLLF3pSW4fgog85Irp4Khrnw/qPgjrv1rvV9Pd\nKijnnSDiE7LYEwwt14eeOyScol3BoJz3js0FF1yAw4cPx3oaSUWfPn3wzTff+G2PehEwIrqI7emA\ndn+51jaRSKYvUlGvjs3hw4dBhpq1MGZKuwNCwp5A6FV3VG8TCadoVzAo550g4hdyxSQIWq6PQ1MO\nod/v+5E7hIg6be6BWE8jqdD7Til4msRouT4mrppIJQAIgvCDXDEJgqbro7WF3CEEQfhBrhiCIEyT\niK6YZcuW4c9//jO2bdsW66loQq4YIi6h0gVEvGNl5kkw/vCHP+DKK69EWloaJk6cGLXzAiTsCU+8\niKmYdvngmgctXQRFJB+cc4z5+xhU1VTFeioRpVevXpg1axYeeOCBqJ+bhD2BiScxpVWohMhntZ9h\n3Gvj4OM+zc9Xf7kam7/ejEfXP6o7xpmWMzhw6kBI5z9y5Ah++tOfIjc3F927d8eUKVM093v00UfR\nu3dv5OTk4Morr0RVVfuN5uOPP8aVV16JnJwc5OfnY/p0oQWly+XCvffei27duqFLly646qqr8P33\n32uOf9ttt+HWW29F165dQ7qOcCBhT2DiRUzlxcciUbqASCymb5yOdw++i7f3vu33Gecc0zdOh5d7\nse/kPlR+U6k5xuQ1k3HDshvQ6ms1dW6fz4dx48ahb9++qKmpwdGjR3HXXXdp7jtixAjs3r0bp0+f\nxvjx43HnnXfC7RZWaU+dOhWPPvoozp49i6+++go/+9nPAAh++nPnzuHo0aOoq6vDyy+/jPT0dFNz\njAYk7AlKPIkpdV4iRD498Sm21WyDj/vwxKYn/Kz21V+uxrH6YwCARk8jpm/0b8b+Vd1XWLl/JU63\nnMaKz1aYOv9HH32E48ePY8GCBUhLS4PT6cS1116rue/48ePRuXNn2Gw2PPbYY3C5XNi/fz8AwOl0\n4uDBgzh16hQyMjIwYsQIAEBKSgpOnTqFL7/8EowxFBYWIisry9Qco4Elws4Ym8YY8zHGov/M0UGJ\nJzFVFx+7LPcySrvsoJRsKoHL6wIAfN/0vZ/V/sSmJ6Sy0gDwyfFPsPUbZVP2ZzY/g1ZvK5o8TZjx\n/gxTVvu3336LPn36wGYLLm2//e1vMXDgQHTp0gVdunTBuXPncPLkSQDAX/7yF+zfvx+XXHIJrrrq\nKqxduxYAcO+996K4uBh33XUXCgoKMGPGDHi9XsPzixZh57EzxgoAjAZAFYHCxExt83ha0h/J0gVE\n4rC7djc2HdoEDiFlr8HdgCc2PYGfXPoTKRtl4tCJ+K7xu/aDGJCbmSu9Fa31Vi6I+TnXOaz4bAXu\nHXKvoTmcf/75qKmpgc/nCyju27Ztw8KFC7FlyxYMHDgQANC1a1cp3bB///547bXXAAD/+te/cMcd\nd6Curg7p6emYNWsWZs2ahZqaGvzwhz/ExRdfjPvvv9/gtxQdrFigtBjAEwBWWTBWhyVQMS8tSEyJ\neCPVnor7ht6nyMXumt5VkWJYMirwv9llu5bB7XUjIyUDAOBqdeEPH//BsLCPGDEC+fn5mDFjBkpL\nS2G321FdXe3njmloaEBKSgrOO+88uN1uPP/886ivr5c+X758OYqLi9GtWzfk5OSAMQabzYbKykp0\n69YNAwcORFZWFlJSUnRvIF6vFx6PB16vF62trXC5XHA4HLDb7YauJSw45yG/ANwK4MW2n78G0DXA\nvpzQZ/H2xTyzLJNnlmXyJduXxHo6BBGQSP09u1pd/Nuz3ype51rOmRrj22+/5bfddhs/77zzePfu\n3fnUqVM555y/+uqr/LrrruOcc+71evnEiRN5p06deM+ePfnChQt53759+fvvv8855/yee+7hubm5\nPDs7mw8ePJivWrWKc875ihUr+MUXX8yzsrJ4Xl4ef/TRR7nX69WcR2lpKWeMcZvNJr3mzp2rO2+9\n77RtuyltDrrylDG2CUAP+SYAHMAzAGYCGM05r2eMfQ3gCs75KZ1x+Jw5c6T3RUVFKCoqCuFWlHxQ\nbXMi0UjElafxjvidVlZWorKyUto+d+5c0ytPQy4pwBgbDOA9AE0QxL4AwFEAIzjn32nsz+kfgjaR\nbIhBEJGAhN16rCwpELKPnXP+OYA82cm/BjCMc3461DE7KtEKhFLjaYLoGFhWBIwxdgiCK6ZO53Oy\n2GMINZ4mrIQsduuJyyJgnPN+eqJOxJ54WaVKEETkobK9HQAKzhJWQxa79cSlxU7EL/G0SpUgiMhD\nHZQ6APG0SpUgiMhDrhiCIExDrhjrIVcMQRCESZYtW4brrrsu1tOICiTsBEF0GKLVGs/tdmPSpEm4\n4IILkJOTg2HDhmH9+vVROTdAwk4QRBQ5dw74yU+AnBygb19g06ZYzygytLa2onfv3ti2bRvOnj2L\nZ599Fj/72c9QU1MTlfOTsBMEYRnV1UBhIZCXB9x1lyDkcn7+c2DdOmH7N98At90G7N2r3MftBjZs\nAN5+G2grj26KeGiNl5GRgdmzZ+P8888HAIwdOxZ9+/ZFdXW1+QsKARJ2giAs4cgR4IYbgE8/BWpr\ngZUrBetchHPBQne52rf5fMD777e/b2oCrrwSuOMOYMIEYMAAf+EPRLy2xqutrcWBAwcwaNAg4xcT\nBiTsBEFYwubNgniLuFzAli2CBQ4AjAFqDbTbBbeMyJIlwP79QEMDUF8PnDkDPPig8TnEY2u81tZW\n3HPPPbjvvvswYMAA4xcTBiTsBEFYQmam/za7HXDIVsssXgxkZLSLfO/egnUu8tVXSouec8CMWzre\nWuNxznHPPfcgNTUVL730kvELCRezBdxDfYEabRBE0qD199zczPnAgZynpXEOcJ6Rwfm8ef7HbtnC\n+ezZnL/0EueNjcrPli0TjhMknXOnk/Of/9z4vLZv38579Oih2fxC3mjjgw8+4Lm5uXzPnj3S5126\ndJEabcipqKjgaWlpvKmpSbH98OHDfODAgfyvf/2r7nzuv/9+ftNNN3GXyxV07noaiRAabZDFThCE\nJaSlAR99BMydC/z618A//gHMmuW/X1GRsM8jjwjWu5x77wUeeECw8lNSgCuuAMpNVL+Qt8ZramqC\ny+XCf/7zH7/91K3x5s2b59caT7Te1a3xPv/8c/h8vqCt8X71q19h3759WLVqFZxOp/GLsAASdoIg\nLCMzE3jySWDpUuD2280fzxjw+98DZ88C330H/PvfSh98MGw2G1avXo0DBw6gd+/eOP/88/HPf/7T\nb7/i4mIUFxdjwIAB6Nu3LzIyMqQMFgBYv349Bg0ahE6dOuGxxx7DG2+8gdTUVJw4cQJ33HEHcnJy\nMGjQINxwww24917/fqw1NTX44x//iE8//RQ9evRAdnY2OnXqhBUrVpj/UkKASgoQBGEaKilgPVRS\ngCAIgtCFhJ0gCCLJIGEnCIJIMkjYCYIgkgwSdoIgiCSDhJ0gCCLJoNZ4BEGYpk+fPlGrbd5R6NOn\nj2VjhZ3Hzhj7PwD/C6AVwFrO+Qyd/SiPnSAIwiRRz2NnjBUB+BGAyzjnlwH4bTjjxTOVlZWxnkJY\nJPL8E3nuAM0/1iT6/EMhXB/7ZADPc85bAYBzHkJZ/MQg0f9xJPL8E3nuAM0/1iT6/EMhXGEfAOB/\nGGMfMsa2MMausGJSBEEQROgEDZ4yxjYB6CHfBIADeKbt+C6c86sZY1cC+CeAfpGYKEEQBGGMsIKn\njLF1AF7gnG9te38QwFWc81Ma+1LklCAIIgTMBk/DTXdcCeBGAFsZYwMApGiJeigTIwiCIEIjXGF/\nBcBfGWOfAXAB+GX4UyIIgiDCIWr12AmCIIjoEJOSAoyxaYwxH2OsayzOHyqMsQWMsb2MsU8ZY/9i\njHWK9ZyCwRj7AWNsH2PsS8ZYSaznYwbGWAFjbDNjbA9j7DPG2JRYzykUGGM2xtgnjLFVsZ6LWRhj\nOYyxN9v+3e9hjF0V6zkZhTH2GGPsc8bYbsbYcsZYdPvThQBj7C+MsVrG2G7Zti6MsY2Msf2MsQ2M\nsaA9paIu7IyxAgCjARyO9rktYCOAQZzzoQAOAHgqxvMJCGPMBmApgGIAgwDczRi7JLazMkUrgMc5\n59HYfM4AAANXSURBVIMAXAPg1wk2f5GpAL6I9SRC5HcA1nHOLwUwBMDeGM/HEIyxngD+D8Awzvnl\nENzOd8V2VoZ4BcLfq5wZAN7jnF8MYDMM6E4sLPbFAJ6IwXnDhnP+Hufc1/b2QwAFsZyPAUYAOMA5\nP8w59wB4HcCPYzwnw3DOT3DOP237uQGCqPSK7azM0WbI3ALgz7Gei1nankiv45y/AgCc81bO+bkY\nT8sMdgCZjDEHgAwAx2I8n6BwzqsAnFZt/jGAZW0/LwNwW7BxoirsjLFbAXzLOf8smueNEBMBvBvr\nSQShF4BvZe+PIMGEUYQxdgGAoQD+G9uZmEY0ZBIxmNUXwEnG2CttrqQ/MsbSYz0pI3DOjwFYBKAG\nwFEAZzjn78V2ViGTyzmvBQRjB0BusAMsF3bG2KY2n5b4+qzt/7cCmAlgjnx3q88fLgHm/yPZPk8D\n8HDOX4vhVDsMjLEsABUAprZZ7gkBY2wsgNq2pw6GOPz3HgQHgGEA/sA5HwagCYJbIO5hjHWGYOn2\nAdATQBZjbHxsZ2UZQY0Ey8v2cs5Ha21njA0GcAGAXUyo91kAoJoxNoJz/p3V8wgVvfmLMMbug/Bo\nfWNUJhQeRwH0lr0vaNuWMLQ9RlcA+Dvn/J1Yz8ckIwHcyhi7BUA6gGzG2N8454mSFnwEwhP2jrb3\nFQASJQB/M4BDnPM6AGCMvQXgWgCJaIzVMsZ6cM5rGWN5AILqZdRcMZzzzznneZzzfpzzvhD+0RTG\nk6gHgzH2AwiP1bdyzl2xno8BPgZwIWOsT1tGwF0AEi0z468AvuCc/y7WEzEL53wm57w357wfhO9+\ncwKJOtoe/79tW3wIADchcYLANQCuZoyltRmSNyFBAr/wf7pbBeC+tp8nAAhq4MSy0QZH4j2avgTA\nCWBTW5OBDznn/xvbKenDOfcyxh6BkM1jA/AXznmi/OMGY2wkgF8A+IwxthPCv5mZnPP1sZ1Zh2IK\ngOWMsRQAhwDcH+P5GIJz/hFjrALATgCetv//MbazCg5j7DUARQDOY4zVQHBdPw/gTcbYRAjZhD8L\nOg4tUCIIgkguqOcpQRBEkkHCThAEkWSQsBMEQSQZJOwEQRBJBgk7QRBEkkHCThAEkWSQsBMEQSQZ\nJOwEQRBJxv8HRd3kaIbk8mkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x106e1e9b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline \n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "def GenerateSamples(cov=0, N=500):\n",
    "    y = np.random.randint(2, size=N).reshape((N, 1))\n",
    "    mean1 = [0, 0]\n",
    "    cov1 = [[1, cov], [cov, 2]]\n",
    "    mean2 = [4, 3]\n",
    "    cov2 = [[2, cov], [cov, 1]] \n",
    "    x1 = np.zeros(N).reshape((N, 1))\n",
    "    x2 = np.zeros(N).reshape((N, 1))\n",
    "    for i in range(N):\n",
    "        if y[i] == 0:\n",
    "            x1[i], x2[i] = np.random.multivariate_normal(mean1, cov1, 1).T\n",
    "        else:\n",
    "            x1[i], x2[i] = np.random.multivariate_normal(mean2, cov2, 1).T\n",
    "    X = np.concatenate((x1, x2), axis=1)\n",
    "    return X, y\n",
    "\n",
    "X, y = GenerateSamples()\n",
    "mat = np.concatenate((X[:,[0]], X[:,[1]], y), axis=1)\n",
    "class1 = mat[np.where(mat[:,-1] == 0)]\n",
    "class2 = mat[np.where(mat[:,-1] == 1)]\n",
    "plt.scatter(class1[:, 0], class1[:, 1], marker=\"^\", color=\"green\", label=\"class 1\")\n",
    "plt.scatter(class2[:, 0], class2[:, 1], marker=\"o\", color=\"blue\", label=\"class 2\")\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And next build the Naive Bayes classifier:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from scipy.stats import norm\n",
    "\n",
    "class NaiveBayes():\n",
    "    def train(self, X, Y):\n",
    "        mat = np.concatenate((X, Y), axis=1)\n",
    "        n = mat.shape[0]\n",
    "        self.pdfs = {}\n",
    "        labels = set(mat[:,-1])\n",
    "        for i in range(len(labels)):\n",
    "            subset = mat[np.where(mat[:,-1] == i)]\n",
    "            self.pdfs[i] = {\"mean\": np.mean(subset, axis=0),\n",
    "                         \"stdevs\": np.std(subset, axis=0), \n",
    "                           \"prior\": subset.shape[0]/n}\n",
    "    def predict(self, x):\n",
    "        def probability(sample, parameters):\n",
    "            prior = parameters[\"prior\"]\n",
    "            likelihood = norm.pdf(sample[0], parameters[\"mean\"][0], \n",
    "                        parameters[\"stdevs\"][0])* norm.pdf(sample[1], \n",
    "                        parameters[\"mean\"][1], parameters[\"stdevs\"][1])\n",
    "            return prior * likelihood\n",
    "        predictions = []\n",
    "        for i in range(x.shape[0]):\n",
    "            probs = []\n",
    "            sample = x[i, :]\n",
    "            probs.append(probability(sample, self.pdfs[0]))\n",
    "            probs.append(probability(sample, self.pdfs[1]))\n",
    "            predictions.append(probs.index(max(probs)))\n",
    "        return predictions "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train, then predict on another dataset drawn from the same distribution as the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.97\n"
     ]
    }
   ],
   "source": [
    "bayes = NaiveBayes()\n",
    "bayes.train(X, y)\n",
    "\n",
    "X, y = GenerateSamples(cov=0, N=200)\n",
    "predictions = np.array(bayes.predict(X)).reshape((200, 1))\n",
    "\n",
    "match = predictions == y\n",
    "print((sum(match)/len(match))[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the prediction accuracy is quite high given that assumptions to Naive Bayes hold! We would expect that the more our assumptions are violated, the lower our prediction accuracy. Let us test this by assessing the prediction accuracy as we generate independent datasets with increasing covariance values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/calvinchi/anaconda3/lib/python3.5/site-packages/ipykernel/__main__.py:17: RuntimeWarning: covariance is not positive-semidefinite.\n",
      "/Users/calvinchi/anaconda3/lib/python3.5/site-packages/ipykernel/__main__.py:15: RuntimeWarning: covariance is not positive-semidefinite.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEPCAYAAABRHfM8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xm8lVW9x/HPl0EFDLUcMhUsy4G6JCqI4tXjEKGpqJmB\nmXorI9OreV8W2r0llveKZRhqkgMOOIBDznkTp2NqDqCoZCDcDFIxysoSzWT43T/Wc2SLZ3jOYT/n\nOXuf7/v12q+zn/l39tln//Za61lrKSIwMzNrS4+yAzAzs9rghGFmZrk4YZiZWS5OGGZmlosThpmZ\n5eKEYWZmuRSaMCRNlbRU0rOt7HO+pIWSnpa0Y8X6UZLmS1ogaXyRcZqZWduKLmFcAXy6pY2S9ge2\niYiPAeOAn2brewAXZsd+HBgrafuCYzUzs1YUmjAi4mHgr63sMhqYlu37OLCBpM2AYcDCiFgcEcuB\nGdm+ZmZWkrLbMLYAXqxYfilb19J6MzMrSdkJY00qOwAzM2ter5Kv/zKwVcXyltm6dYABzaxvliQP\niGVm1k4R0a4v6Z1RwhAtlxxuB44GkDQceC0ilgKzgI9KGihpHWBMtm+LIsKPKjzOOOOM0mOop4df\nT7+eXfXREYWWMCRdBzQAH5D0e+AMUukhIuKSiLhL0gGS/g94A/g30saVkk4EZpKS2tSImFdkrGZm\n1rpCE0ZEHJljnxNbWP8LYLuqB2VmZh3S1Rq9rWQNDQ1lh1BX/HpWl1/PcqmjdVldiaSoh9/DzKyz\nSCK6YKO3mZnVAScMMzPLxQnDzMxyccIwM7NcnDDMzCyXukkYb71VdgRmZvWtbhLGCSeA76w1MytO\n3SSMWbNgypSyozAzq19lj1ZbNbfcArvvDp/4BOy5Z9nRmJnVn7opYWyzDVx9NYwZAy++2Pb+ZmbW\nPnWTMABGjoRTToFDD4V//KPsaMzM6kvdjSUVAUceCeusA1deCfIcfmZm7+GxpEgJYupUePZZOP/8\nsqMxM6sfdVfCaLJoEQwfDtOnw957lxOXmVlX5RJGha23huuuS9VTixaVHY2ZWe2r24QBsM8+MH58\nagR/882yozEzq211WyXVJAKOOQZWrIBrr3UjuJkZuEqqWRJcfDEsWACTJpUdjZlZ7ar7EkaT3/8e\ndt0Vpk2DT32qkwIzM+uiXMJoxYABcP31cNRR8MILZUdjZlZ7uk3CgDTG1He/C4ccAm+8UXY0Zma1\npdtUSTWJgC9/GZYtSyUON4KbWXfkKqkcJLjoIli8GM45p+xozMxqR7crYTR5+WUYNgwuuwz237+g\nwMzMuiiXMNphiy3ghhvg2GNh4cKyozEz6/q6bcIAGDECvve91Aj++utlR2Nm1rUVnjAkjZI0X9IC\nSeOb2b6hpJslPSPpMUmDKrYtytbPkfREEfGNGwd77AFHHw2rVhVxBTOz+lBoG4akHsACYF9gCTAL\nGBMR8yv2+QHwekR8X9J2wE8iYr9s2wvAzhHx1zau0+42jEr//Gcad2rUKPjOdzp8GjOzmtEV2zCG\nAQsjYnFELAdmAKPX2GcQcD9ARDwPbC1pk2ybOiFG1l0XbroJLrkE7rij6KuZmdWmoj+MtwAqZ9h+\nKVtX6RngMABJw4ABwJbZtgDukTRL0nFFBrr55nDjjamPxvz5be9vZtbddIVG74nARpKeAk4A5gAr\ns20jImIn4ADgBEl7FBnI8OEwcWJqBP/b34q8kplZ7elV8PlfJpUYmmyZrXtHRLwOfKlpWdLvgBey\nba9kP/8k6RZSFdfDzV1owoQJ7zxvaGigoaGhQwF/6Uvw5JPwxS/CrbdCj66QUs3M1lJjYyONjY1r\ndY6iG717As+TGr1fAZ4AxkbEvIp9NgDejIjlWbXTiIg4VlJfoEdELJPUD5gJnBkRM5u5zlo1eq/p\n7bdhv/3S1K5nnlm105qZdRkdafQutIQRESslnUj6sO8BTI2IeZLGpc1xCbADcJWkVcBzwJezwzcD\nbpEUWZzXNpcsirDOOqk9Y+hQ2HHHNGOfmVl3122HBslj9mw44ABobIRBg9rc3cysZnTF22pr2i67\nwLnnpkbw114rOxozs3K5hJHDN76Rpni94w7o2bOwy5iZdRqXMArywx/CW2+5F7iZdW9OGDn07p0m\nW7ruutQYbmbWHblKqh3mzIGRI+G++2Dw4MIvZ2ZWGFdJFWzIEJg8Od1m+5e/lB2NmVnncgmjA049\nFZ59Fu66C3oV3VfezKwALmF0kokTIQK+/e2yIzEz6zxOGB3QqxfMmJGGRJ8+vexozMw6h6uk1sKz\nz8K++8I996QhRMzMaoWrpDrZ4MHwk5+kRvBXXy07GjOzYrmEUQWnnw6PPw4zZ7oR3MxqQ0dKGE4Y\nVbByJRx4IGy/PZx3XmlhmJnl5iqpkvTsmXqB33knXH112dGYmRXDJYwqeu45aGiA//3fNNKtmVlX\n5RJGyT7+cbj4YvjsZ2Hp0rKjMTOrLieMKjvsMDjmGDjiCFi+vOxozMyqx1VSBVi1CkaPhq23hgsu\nKDsaM7P3cpVUF9GjB1xzTerQd/nlZUdjZlYdLmEUaP582HPPNFPfrruWHY2Z2WouYXQx228Pl10G\nhx8Of/hD2dGYma0dJ4yCHXwwHHdcunPq7bfLjsbMrOPaTBiSbpb0GUlOLh30X/8Fm24KJ51UdiRm\nZh2XJwlcBBwJLJQ0UdJ2BcdUd3r0gGnT4KGH4JJLyo7GzKxjcjd6S9oAGAv8J/AicClwTUSU3tug\nqzZ6r2nhQhgxAm69FXbfvexozKw7K6zRW9IHgGOBrwBzgMnATsA97YyxW/vYx+DKK+Fzn4OXXy47\nGjOz9mmzhCHpFmA74Grgyoh4pWLb7IgofdSkWilhNDn7bLjtNnjwQVh33bKjMbPuqJDhzSXtHREP\nrFVkBau1hBGRhg7p3z/ddqt2/cnMzNZeUVVSgyRtWHGRjSR9vd3R2TskuOIKeOIJmDKl7GjMzPLJ\nkzCOi4jXmhYi4q/AcXkvIGmUpPmSFkga38z2DbNbd5+R9JikQXmPrWXrr58av888E375y7KjMTNr\nW56E0VNaXWkiqSewTp6TZ303LgQ+DXwcGCtp+zV2+zYwJyI+CRwDnN+OY2vaNtukCZfGjIEXXyw7\nGjOz1uVJGL8Arpe0r6R9genZujyGAQsjYnF2++0MYPQa+wwC7geIiOeBrSVtkvPYmjdyJJxyChx6\nKPzjH2VHY2bWsjwJYzzwAHB89rgP+FbO829B6rPR5KVsXaVngMMAJA0DBgBb5jy2Lpx6arrl9mtf\nSw3iZmZdUa+2doiIVcCU7FGEicBkSU8Bc0n9PFa29yQTJkx453lDQwMNDQ1VCq94Ekydmjr1nX8+\nnHxy2RGZWb1pbGyksbFxrc6R57bajwFnk6qO1mtaHxEfafPk0nBgQkSMypZPS4fGOa0c8zvgX4BP\n5D221m6rbcmiRTB8OEyfDnvvXXY0ZlbPirqt9gpS6WIFsDcwDbgm5/lnAR+VNFDSOsAY4PbKHSRt\nIKl39vw44MGIWJbn2Hqz9dZw7bUwdmxKHmZmXUmehNEnIu4jlUYWR8QE4DN5Th4RK4ETgZnAc8CM\niJgnaZykr2a77QD8WtI80h1RJ7d2bP5frTbtuy+cdlpqBH/zzbKjMTNbLU+V1K+APYCbSHczvQxM\njIguM2ptvVRJNYmAY46BFStSicM9wc2s2oqqkjoZ6AucBOwMHEXqL2EFkeDii2HBApg0qexozMyS\nVksYWSe9cyLi1M4Lqf3qrYTR5Pe/T3OBT5sGn/pU2dGYWT2pegkja0fYY62isg4bMACuvx6OOgpe\neKHsaMysu8vThjGF1GHuRuCNpvURcXOxoeVXryWMJhdemGbqe/RR6Nev7GjMrB4UNbz5Fc2sjoj4\nUnsuVKR6TxgR8OUvw7JlqcThRnAzW1uFJIxaUO8JA+Ctt2CvvdLttqedVnY0ZlbriixhvGcnlzA6\n38svw7BhadKl/fcvOxozq2UdSRhtjiUF3FnxfD3gUGBJey5i1bHFFnDDDXDYYfDww2nAQjOzztLu\nKqlsnoqHI2L3YkJqv+5Swmhy8cVpkMLHHoP3va/saMysFhXVcW9NHwM27cBxViXjxsEee6Te4KtW\nlR2NmXUXbSYMSa9L+nvTA7iDNEeGlej88+EPf4D/+Z+yIzGz7sJ3SdWwV16BoUNhyhQ46KCyozGz\nWlJIlZSkQyVtULG8oaRDOhKgVdfmm8NNN6U+GvPnlx2NmdW7PLfVPh0RO66xbk5EDCk0snboriWM\nJpdfDj/4ATz+OGywQdv7m5kV1ejd3D55bse1TvKlL6V5NL74RTeCm1lx8iSM2ZImSdome0wCniw6\nMGuf886D116DM88sOxIzq1d5Esa/A28D1wMzgLeAE4oMytpvnXXgxhvhiivgllvKjsbM6pHvkqoz\ns2fDAQdAYyMMGlR2NGbWVRV1l9Q9kjasWN5I0t0dCdCKt8sucO65cMghqYrKzKxa8lRJbRwR73z0\nRMRfcU/vLu3oo1Mp48gjYeXKsqMxs3qRJ2GskjSgaUHSQJoZvda6lh/+MA2J/p3vlB2JmdWLPLfH\n/ifwsKQHAQH/CowrNCpba717p8mWhg6FIUPgc58rOyIzq3W5Gr0lbQwMzxYfi4hXC42qndzo3bI5\nc2DkSLjvPhg8uOxozKyrKGy02oh4NSLuBOYBx0t6riMBWucbMgQmT04z9f3lL2VHY2a1LM9dUh+S\ndIqkWcBz2TFjCo/MqubII1PCGDMGVqwoOxozq1UtVklJ+iowFtgCuCF73BYRH+688PJxlVTbVqxI\n07oOGZLGnTKz7q3aU7ReCDwKHBkRs7ML+FO5RvXqBTNmrG4EHzu27IjMrNa0ljA2Bz4H/EjSB0kl\njN6dEpUV4gMfgFtvhf32gx12gB13bPsYM7MmLbZhRMSfI+KnEbEXsC/wGrBU0jxJued5kzRK0nxJ\nCyS9Z6Y+Sf0l3S7paUlzJR1bsW2RpGckzZH0RPt+NWvO4MFw4YWpTePVLnWvm5l1de0eS0rStsCY\niPhejn17AAtICWcJMCs7dn7FPqcD/SPi9Oz23eeBzSJihaQXgJ2z3uWtXcdtGO10+ulp/oyZM1N1\nlZl1L4XdVlspIhbkSRaZYcDCiFgcEctJo92OXvOUwPuy5+8D/hwRTffyqCMxWtvOOgvWXRe++c2y\nIzGzWlH0h/EWwIsVyy9l6ypdCAyStAR4Bji5YlsA90iaJem4QiPtZnr2hOuugzvvhKuvLjsaM6sF\nXaEy4tPAnIjYR9I2pAQxOCKWASMi4hVJm2Tr50XEw82dZMKECe88b2hooKGhoRNCr20bbZQawRsa\nUiP4LruUHZGZFaWxsZHGxsa1OkfeoUG2AAZSkWAi4pc5jhsOTIiIUdnyaenQOKdinzuBsyPikWz5\nPmB80628FfudAbweEZOauY7bMNbCzTfDKafArFmwqcchNusWqt0Po+mk5wCfB34DNA2WHUCbCYPU\nyP3RbITbV0g9xNfsAbAY2A94RNJmwLbAC5L6Aj0iYpmkfsBIwBOQFuCww+Dpp9MAhffemwYuNDNb\nU5slDEnPA4Mj4p8duoA0CphMai+ZGhETJY0jlTQukbQ5cCWp3wek0sZ0SR8GbiElp17AtRExsYVr\nuISxllatgtGj4YMfhJ/+NLVxmFn96kgJI0/C+F/gc1mbQpfkhFEdf/tbmqmvf3+49lpYf/2yIzKz\nohSVMH4GfBK4D3inlBERJ3UkyCI4YVTP22/D178OTz4Jd9wBW25ZdkRmVoSiEsYxza2PiKvac6Ei\nOWFUVwRMmgTnnZfuovLdU2b1p5CEkZ14HVJjNMDzWSe8LsMJoxi33QbHHQcXXQSHH152NGZWTUXd\nJdUAXAUsIvW83krSMXluq7XaNno0DBiQfi5YkIYTUbveXmZWT/JUST1JGuL8+Wx5W2B6ROzcCfHl\n4hJGsZYsSUljhx3g0kvTkCJmVtuKGkuqd1OygDSWFB7mvFv50IfgwQfhzTfT0Oge5dase8qTMGZL\nukxSQ/a4FJjd5lFWV/r2hRtugD33hF13hXnzyo7IzDpbniqpdYETgD2yVQ8BF3W0I18RXCXVuaZN\nS6PcXnMNfOpTZUdjZh1R2F1SXZ0TRud76CE44gj47nfh+OPLjsbM2quqCUPSDRFxhKS5pOE53iUi\nBncszOpzwijHb38LBx4II0fCj37kiZjMakm1E8bm2dDiA5vbHhGLOxBjIZwwyvPaa6mk0asXzJiR\nhhUxs66vqndJRcQr2dOvZzPmvfMAvr42gVr92HBD+PnPYeBAGDECFi0qOyIzK0qeu6Saa9bcv9qB\nWO3q3Tv1Bj/uONh9d3j00bIjMrMitJgwJB2ftV9sL+nZisfvgLmdF6LVAglOOgkuuyx18ps+veyI\nzKzaWmvD2ADYCDgbOK1i0+sR8ZdOiC03t2F0LXPnwsEHw7HHpruoPJyIWddT1Gi1w4HnIuL1bLk/\nsENEPN7hSKvMCaPrWbo0za3x4Q/D1KnQp0/ZEZlZpaKGBpkCVE6etCxbZ9aizTaD++9PQ6Xvs09K\nIGZW2/IkjHd9fY+IVeQY5dasTx+47joYNSoNJzLXLV9mNS1PwnhB0kmSemePk4EXig7M6oMEZ5wB\nZ58N++4Ld91VdkRm1lF5EsbXgN2Bl4GXgF2BrxYZlNWfsWPThExf+Qqcf36qqjKz2uKxpKxTLV6c\nhhP513+FyZNTHw4z63zVHhrkWxHxA0kX0PxYUid1LMzqc8KoLX//O4wZAytWpCHTN9yw7IjMup9q\n3yXVNOPBbODJZh5mHdK/P9x+OwwaBLvtlgYxNLOuz1VSVqopU+DMM+HGG1M1lZl1jmpXSd1BM1VR\nTSLi4PaFVxwnjNo2cyYcdRScey4cfXTZ0Zh1D9VOGHtlTw8DPghcky2PBZZGxCkdDbTanDBq329+\nAwcdlNo2vv996JHn/j0z67CihgaZHRG7tLWuTE4Y9eHVV+HQQ1Mv8WnT0jziZlaMooYG6SfpIxUX\n+TDQr73BmbVl443h3nuhXz/Yay9YsqTsiMysUp6EcQrQKKlR0oPAA8A38l5A0ihJ8yUtkDS+me39\nJd0u6WlJcyUdm/dYqz/rrgtXXgmHHQbDh8OcOWVHZGZNct0lJWldYPtscX5E/DPXyaUewAJgX2AJ\nMAsYExHzK/Y5HegfEadL2hh4HtgMWNXWsRXncJVUHbrpJjj++NVzbJhZ9RRSJSWpL/BN4MSIeAYY\nIOnAnOcfBizMpnZdDswA1vzXD+B92fP3AX+OiBU5j7U6dvjhaeypE05Id1D5O4FZufJUSV0BvA3s\nli2/DJyV8/xbAC9WLL+Urat0ITBI0hLgGeDkdhxrdW7o0DTl67XXpilg33677IjMuq88w5RvExGf\nlzQWICLelKo6h9qngTkRsY+kbYB7JA1u70kmTJjwzvOGhgYaGhqqFqCVa6ut4KGH4AtfgE9/Gn72\nM3j/+8uOyqy2NDY20tjYuFbnyHNb7a9I7QiPRMRO2Yf69IgY1ubJ02x9EyJiVLZ8GhARcU7FPncC\nZ0fEI9nyfcB4UjJr9diKc7gNoxtYuRJOPx1uvRXuvBO23bbsiMxqV1G31Z4B/ALYStK1wH3At3Ke\nfxbwUUkDJa0DjAFuX2OfxcB+AJI2A7YlzbeR51jrRnr2hB/8AMaPT8OIPPBA2RGZdS+tljCyqqct\ngTeB4YCAxyLi1dwXkEYBk0nJaWpETJQ0jlRauETS5sCVwObZIWdHxPSWjm3hGi5hdDMPPJB6hf/3\nf6c5NsysfYrq6T03Iv5lrSIrmBNG97RgQZpbY/RomDgxlUDMLJ+iqqSekjS0gzGZFWbbbeGxx2D2\n7NTRb9mysiMyq295EsauwGOSfivp2aw39rNFB2aWx/vfD3ffDZtskto1Xnqp7IjM6leeKqmBza2P\niMWFRNQBrpKyCJg0Cc47D265JfXfMLOWVXt48/WArwEfBeaSGp1XrHWUBXDCsCa33ZYawadMST3F\nzax5HUkYrXXcuwpYDjwE7A8MYnUvbLMuafRoGDAg/VywIPXbqGo3U7NurLUSxjt3R0nqBTwRETt1\nZnB5uYRha1qyJCWNHXaASy9No+Ca2WrVvktqedOTrloVZdaSD30IHnwQ3nwT9tsvTc5kZmuntYTx\nSUl/zx6vA4Obnkv6e2cFaNZRffvCDTfAnnvCrrvCvHllR2RW23LNh9HVuUrK2jJtGnzzm3D11TBy\nZNnRmJWvqI57ZjXv6KPThEzHHJPuoDKz9nMJw7qV3/42DScyciT86EfQK88A/2Z1qJCxpGqBE4a1\nx2uvwRFHpGQxYwb07192RGadz1VSZjlsuCH8/OcwcCCMGAGLFpUdkVltcMKwbql3b7joojTt6+67\np2lgzax1rpKybu+uu+DYY2HyZBg7tuxozDqH2zDMOmjuXDjooJQ4zjjDw4lY/XPCMFsLS5fCIYfA\n1lvD5ZdDnz5lR2RWHDd6m62FzTaD++9Pz/fZJyUQM1vNCcOsQp8+cN11MGpUGk5k7tyyIzLrOlwl\nZdaC6dPh5JPhyivhgAPKjsasutyGYVZljz4Kn/0sjB8PJ53kxnCrH04YZgVYvDgNJ7LHHnD++akP\nh1mtc6O3WQEGDoRHHkmJ4zOfSUOLmHVHThhmOfTvD7ffDoMGwW67pUEMzbobJwyznHr1gh//OLVl\njBgBDz1UdkRmncsJw6ydjj8+Tcj02c+mn2bdhRu9zTroN79Jw4l8/vNw1lnQw1+/rIb4LimzTvan\nP8Fhh8Gmm6bpX/v2LTsis3y65F1SkkZJmi9pgaTxzWw/VdIcSU9JmitphaQNs22LJD2TbX+i6FjN\n2muTTeDee2H99WHPPWHJkrIjMitOoSUMST2ABcC+wBJgFjAmIua3sP+BwDciYr9s+QVg54j4axvX\ncQnDShUBEyem+cJvuw2GDCk7IrPWdcUSxjBgYUQsjojlwAxgdCv7jwWmVywLN8xbDZDg9NNh0qQ0\nX/htt5UdkVn1Ff1hvAXwYsXyS9m695DUBxgF/KxidQD3SJol6bjCojSrksMPTxMynXACnHtuKnmY\n1YteZQdQ4SDg4Yio7Ec7IiJekbQJKXHMi4iHS4rPLJehQ9MYVAcfnPpt9O+f2jj69UuPpufNrWtr\n+3rreTwrK0/RCeNlYEDF8pbZuuaM4d3VUUTEK9nPP0m6hVTF1WzCmDBhwjvPGxoaaGho6GjMZmtt\nq63giSdSI/gbb8CyZe/+uea6P/6x9e1N695+u/Uk09GE1K9f6pho9auxsZHGxsa1OkfRjd49gedJ\njd6vAE8AYyNi3hr7bQC8AGwZEf/I1vUFekTEMkn9gJnAmRExs5nruNHbuoUVK+DNN5tPLHkSU2vr\nevde+xJQc9tdKuqaOtLoXeh3iohYKelE0od9D2BqRMyTNC5tjkuyXQ8B7m5KFpnNgFskRRbntc0l\nC7PupFevVMXVv391zxsBb73VviTzxz/mS0zLl7+7JFOt6jmXijqfO+6ZWaFWrFidQNa2RNRWqaha\nJaTuUCpyT28z6zYqS0UdSTitbW+pVLS21XRdqVTkhGFmVgUdLRXl2d5V2oq6XBuGmVkt6tULNtgg\nPaqprVJREW1FLSWZDr0u1X05zMysJRL06ZMeG29c3XOveQddWwmpQ/HXQ1WOq6TMzNqnK44lZWZm\ndcIJw8zMcnHCMDOzXJwwzMwsFycMMzPLxQnDzMxyccIwM7NcnDDMzCwXJwwzM8vFCcPMzHJxwjAz\ns1ycMMzMLBcnDDMzy8UJw8zMcnHCMDOzXJwwzMwsFycMMzPLxQnDzMxyccIwM7NcnDDMzCwXJwwz\nM8vFCcPMzHJxwjAzs1wKTxiSRkmaL2mBpPHNbD9V0hxJT0maK2mFpA3zHGtmZp2n0IQhqQdwIfBp\n4OPAWEnbV+4TEedGxJCI2Ak4HWiMiNfyHGvV19jYWHYIdcWvZ3X59SxX0SWMYcDCiFgcEcuBGcDo\nVvYfC0zv4LFWBf6HrC6/ntXl17NcRSeMLYAXK5Zfyta9h6Q+wCjgZ+091szMiteVGr0PAh6OiNfK\nDsTMzN5LEVHcyaXhwISIGJUtnwZERJzTzL43AzdExIwOHFvcL2FmVqciQu3Zv+iE0RN4HtgXeAV4\nAhgbEfPW2G8D4AVgy4j4R3uONTOzztGryJNHxEpJJwIzSdVfUyNinqRxaXNcku16CHB3U7Jo7dgi\n4zUzs5YVWsIwM7P60ZUavdvNHfuqS9IiSc9kHSmfKDueWiNpqqSlkp6tWLeRpJmSnpd0d1b9am1o\n4bU8Q9JLWSffpySNKjPGWiJpS0n3S3ou6yB9Ura+Xe/Pmk0Y7thXiFVAQ9aRcljZwdSgK0jvx0qn\nAfdGxHbA/aTOqda25l5LgEkRsVP2+EVnB1XDVgD/EREfB3YDTsg+L9v1/qzZhIE79hVB1PZ7olQR\n8TDw1zVWjwauyp5fRWqvsza08FpCeo9aO0XEHyLi6ez5MmAesCXtfH/W8oeDO/ZVXwD3SJol6biy\ng6kTm0bEUkj/tMCmJcdT606U9LSky1y91zGStgZ2BB4DNmvP+7OWE4ZV34hsTK8DSEXWPcoOqA75\nLpOOuwj4SETsCPwBmFRyPDVH0vrATcDJWUljzfdjq+/PWk4YLwMDKpa3zNZZB0XEK9nPPwG3kKr9\nbO0slbQZgKQPAn8sOZ6aFRF/itW3dV4KDC0znlojqRcpWVwdEbdlq9v1/qzlhDEL+KikgZLWAcYA\nt5ccU82S1Df79oGkfsBI4NflRlWTxLvr2W8Hjs2eHwPctuYB1qJ3vZbZB1qTw/D7s70uB34TEZMr\n1rXr/VnT/TCy2+oms7pj38SSQ6pZkj5MKlUEqUPntX4920fSdUAD8AFgKXAGcCtwI7AVsBg4wuOl\nta2F13JvUt37KmARMK6p/t1aJ2kE8EtgLul/PIBvk0bQuIGc78+aThhmZtZ5arlKyszMOpEThpmZ\n5eKEYWZmuThhmJlZLk4YZmaWixOGmZnl4oRhVSdpZTb89K+zodL/o2LbzpJ+XGZ8WRzHSLqghW37\nZ+Np/VpsxjRJAAAFA0lEQVTSk5J+WHAsB0n6VpHXaOaafSS92tRZs2L9LZI+18pxe0m6o/gIrSsq\ndMY967beyMakQtLGwHRJ/SNiQkQ8CTxZbnjveE8nJEmfAC4A9o+IhZIEfLWoACT1jIg7gE79EI6I\nf0j6BXAocHUWS39gBDC2rcMLDs+6KJcwrFAR8SrpA/dEePc3VElDJf0q+xb/sKSPZev7SLo++4Z/\ns6THJDUloLGSns0e7/REl/S6pLOykUx/JWmTbP2B2fFPZhPFbNJGyN8EzoqIhVn8EREXZ+caKOm+\n7Br3ZJPS9Je0qCKOvpJ+L6mnpK9IeiIrZd0oab1snyskTZH0KHBOZWmnpXizyYOmSnpA0v9J+veK\nax6t1RNfXZWt21jSTZIezx67N/O7zuDdyeFQ0lTJb7X0t6mUxVRZepwraUD2/AvZdZ/KflcPS14H\nnDCscBHxO6BHxYd10zfUecAeEbEzaeiHs7P1Xwf+EhGfAL4DNCWLzYGJpCEjdgSGSjo4O6Yf8Kts\nJNOHgKbh2R+KiOHZNa4H2pqZ8RO0XAK6ALgiu8Z1wAUR8XdgjqS9sn0OBH4RESuBn0XEsIgYAswH\nvlxxri0iYreIOHWN12TNeCurqrYDPgXsCpyRJaWPk4Z4aMiuc3K272TSZEO7AocDlzXz+9wNDJG0\nUbY8BpiePW/pb9OaAFCamOfzwO5ZSXMV8IUcx1sX5yop6yzNfcPcEJiWfXttGsMKYA/gxwAR8ZxW\nT9M5FHggIv4CIOlaYE/SAGpvR8Rd2X5PAvtlz7eSdAOwOdAb+N1a/A67kb6FQ6rGOSd7fgPpA/JB\n0ofuT7L1gyV9P/s9+5E+oJvc2MI1Wov35xGxAvizpKXAZqTxlW6MiL8CVIwDtB+wQ8U3+/Ul9Y2I\nN5tOFhHLJd0OHC7pZlISboqxpb9Na5qutS8pyc/Krr8eaTwoq3EuYVjhJH0EWJENm17p+8D9EfEv\nwEGkD5ZmT9HC80rLK56vZPUH3AXA+RExGPhaK9do8mtglxa2tVR3fzswKvumvjNpqktI04x+Pbv2\n99a49hstnKu1eP9Z8bzyd2zuNRGwazbd7pCIGFCZLCo0VUsdDtyWlYwg399mBe/+DGnaR8BV2TSq\nQyJih4j4Xgu/r9UQJwwrQuWQ1JsAU0gfhGvagNVzmPxbxfpHSN/YkTSIVE0EaWTNPSW9X1JP0gdd\nYxux9AeWZM+PyRH7ucDpFe0pPSSNy7b9itV1/keRqr6IiDeA2aRqoDsq5mxYH/iDpN7kr5LJG2/T\na3w/qYTw/izepuqlmayunkLSJ1s4TyPwMVI14PSK9f1p/m9TaRGrqwt3Aj6crb8vi6mp/WWjprYN\nq21OGFaE9bLGzl+TPrh+0cI3zB8AEyU9ybvfixcBG2fHfw94DvhbNoXkaaQPuTnA7Ii4MzumpW//\nZwI3SZoFrFnCeY+ImAt8g3Rn13PAs6z+IDwJ+DdJT5MSwMkVh16frZtRse47pCT3EKlN4J3LtBJC\n3ngji/c3wH8DD0qaA/wo234ysEvWGP5rYFyzJ0nJ7Sbg/RHxYMWmH9L836bSz4APSJpLSjjPZ+ec\nB/wXMFPSM6T3wAdbOIfVEA9vbl2OpB5A74j4Z1addQ+wXVZ/b2YlcaO3dUV9gQeyqhyA450szMrn\nEoaZmeXiNgwzM8vFCcPMzHJxwjAzs1ycMMzMLBcnDDMzy8UJw8zMcvl/8Z0STPATbhkAAAAASUVO\nRK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x107b134a8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "covSet = [0, 5, 10, 15, 20]\n",
    "accuracies = []\n",
    "for cov in covSet:\n",
    "    X, y = GenerateSamples(cov=cov, N=200)\n",
    "    predictions = np.array(bayes.predict(X)).reshape((200, 1))\n",
    "    match = predictions == y\n",
    "    accuracies.append((sum(match)/len(match))[0])  \n",
    "plt.plot(covSet, accuracies)\n",
    "plt.ylabel(\"Prediction Accuracy\")\n",
    "plt.xlabel(\"Diagonal Covariance Value\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This concludes the Naive Bayes' tutorial!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}