2016_07_07_notes_on_advi

2016_07_07_notes_on_advi.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Notes on Automatic Differentiation Variational Inference\n",
    "\n",
    "Alp Kucukelbir, Dustin Tran, Rajesh Ranganath, Andrew Gelman, David M. Blei\n",
    "\n",
    "http://arxiv.org/abs/1603.00788"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pylab as plt\n",
    "import seaborn as sns\n",
    "import scipy.stats\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Variational Inference\n",
    "\n",
    "__What we want:__\n",
    "\n",
    "* the posterior distribution, or an estimate thereof, for some model over some dataset\n",
    "\n",
    "__What we have:__\n",
    "\n",
    "* a dataset $x_{1:N}$ of $N$ observations\n",
    "* a model, generally containing latent variables $\\theta$\n",
    "* (the model is often expressed as a probabilistic graphical model with likelihood $p(x \\mid \\theta)$ and prior $p(\\theta)$)\n",
    "* so we then have the join probability distribution $p(x, \\theta) = p\\left(x\\mid\\theta\\right)p(\\theta)$\n",
    "\n",
    "__Challenges:__\n",
    "\n",
    "* non-conjugate models (can be complex or no closed-form solutions)\n",
    "* MCMC can be slow, or some samplers may not converge for some models\n",
    "\n",
    "__A way to proceed:__\n",
    "\n",
    "* find a distribution that is approximates the posterior\n",
    "* find a cost function that represents how close the approximate distribution is from the posterior\n",
    "* minimize that cost function\n",
    "\n",
    "__Some constraints:__\n",
    "\n",
    "* only consider differentiable models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### K-L divergence (the 'cost function')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence\n",
    "\n",
    "Continuous divergence\n",
    "\n",
    "$$D_{\\mathrm KL}(q \\mid\\mid p)  = \\int_{\\theta} q(\\theta) \\log\\frac{q(\\theta)}{p(\\theta\\mid x)}\n",
    "            = \\mathbb{E}_q \\left[ \\log \\frac{q(\\theta)}{p(\\theta \\mid x)} \\right] \\geq 0,\n",
    "$$\n",
    "\n",
    "between an approximate distribution $q$ and the posterior $p(\\theta \\mid x)$.\n",
    "* not symmetric in exchange between $q$ and $p$\n",
    "* kindof like a distance function between distributions, but not a true metric\n",
    "* Bishop, 2006 gives a discussion by VI approaches generally use $D_{\\mathrm KL}(q \\mid\\mid p)$ instead of $D_{\\mathrm KL}(p \\mid\\mid q)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEKCAYAAAARnO4WAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl0ZGd54P/vrVVVqipJ3Sr1brfdtl+3DQZjgh0wawiZ\nEMySmQnhJAyQOCy/TDKG/HLAkyGzBTI5TJjAL79kEmxIwmTsMYTdjHEyA0zbLA6baeP2a7fdtntR\nt9TdUq1SrXf+uPeWqtWSaru3bl3p+Zzj4+66pVvP1a1+6q3n3QzTNBFCCLH5hfwOQAghxHBIwhdC\niC1CEr4QQmwRkvCFEGKLkIQvhBBbhCR8IYTYIiJ+ByA2F6XUpcAjWut022NvBv4/4APAx9qP9XDe\nLwOf0Vr/jVLqB8ArtNZ5t+IeRUqp/wIc0Fq/fp3j3wfiQNV+6G+11n88rPhE8EjCF15oTe5QSr0L\n+D3gZ4B8+7F+aa1fMOg5Rp1S6peAXwG+s87xJHAZkNVaN4YZmwguSfjCM0qpDwD/AniJ1vq43frv\n9md3AX8N7AJOANNtx5r2378M/LHW+nP2438IoLW+XSn168B7AAM4B/xLrfXjSqlPAduAy4GvAH8M\nfMr++zngDHBYa/0flFIHgT+xnx8GPq61/iul1MuBDwFPAc8BYsBvaq2/qZQax/o28xKgBnxRa/17\nSqko8EfAy+xz/RD4ba110f5QvEFr/U77Og4C/y/w74GfW+dX9CKgBHzV/l39A/CvtdbL3f6OxdYj\nNXzhBUMp9UdYSfHjWuvjfZzj/we+rbV+LvD/AKrtmPMt4RPAOwCUUiHgV4FPKKVehvVBc7PW+gbg\nI8Dn234+obV+rtb6duDjWCWoa4FfAl5sny8MfAZ4v9b6p4BXAL+rlHqRfY4XAR+xv218Evh39uP/\nEYhrrRVwPfBiO54PADWt9Qu11tcDs1gfAGit/6It2Y8DfwO8DShu8PtJA/8b+EXghcAlwB9u8Hwh\npIUvPDEOXAu8FrhHKfUtrfWPezzHq4HfAdBaH1NK/X3bMcP+/z3AR5RSM1hJ7wmt9VN2i/kA8C2l\nlPPcSaXUpP3nB9rO9VqsxIzW+rRS6rP241fZ5/hk2znG7Oc+BjyjtT5sP/4DrAQNVunqvfb5asAr\nAZRSHwEmlFKvsZ8Xxfo2sdqdWB+SR5RSN673y9FafxnrGw72+T8M/J3z2kKsRRK+8EIZeL3WummX\nWb6glNqw7q6U+qH9RxP4DaDJSmIHqK/+Ga11WSn1Gaxa909jtfjBKpl82m7BO+ffp7VeVErBhS3n\n2qrXcerhYWChvb9AKbUTWABuApbafsZsO0edC/swdgPL9vn+ldb6a/bj41gfIO2/gz3AzcBVSqn3\nYZWSMkqpr2itX7fqua8DclrrQ/ZDIftahFiXlHSEF0ytdRNAa/2fgJ8Ad2O934y1fkBrfb393wu0\n1t8H7gOcMsderJbzWu4A3o6V8P/Ofux+4C12gkYp9U77sbXcC/y6/bztwJuwErYGlpVSv9IWw8NA\npw7jfwDeppQylFJx4HNYdfv7gN9SSsXs8tNfAB9e9Ts4qbXea/8Orgd+Hzi0Otnb9mJ9uxmzy0/v\nxfodC7EuSfjCC6tH4rwNuBqrvp1USuXt/wr2/69d4xz/ErhWKfUTrDLHw2udX2v9A6yW7We11lX7\nsfux6uN/r5T6EVZt/03rxPY+4KBS6mGsmv3TQNkux7wBuNU+9jXg32itv93h2v+9Hc/DwPeBr2it\nvwD8AXAMq7P2Eax/e78D1kgmpdRfdjjv6uf9BfBNrHLSo0AB6/crxLoMWR5ZbGVKqfcAP9Baf1cp\nFQMOAb/vlF6E2Ey6quHbnWLfA16ttX687fFbgA9itWg+pbW+w5MohfDOo8Cf2mWRKHCPJHuxWXVs\n4SulIlijIa7B6oh7vO3xI8ANWB1YDwK/oLWe9zRiIYQQfemmhv+fgT8HTq16/CDWMLi8Xe98AKtz\nSgghxAjaMOErpd4OzGmt/56LR1dkgFzb3wvAhKvRCSGEcE2nGv47gKZS6meB5wN/o5R6vdZ6Dmtd\nlEzbc9PAYqcXNE3TNIw1R+YJIYRY38CJs+tROkqprwPvWlXD/wlwI9ZEm28Bt2itZzucypyfL/Qf\n8YjLZtPI9fUmXy3whaNf5eH5R/hX17+LSzJ7XT1/J5Vqg8ViBYBt28apLFVJJaJDjWEY5L0ZbNls\neuCE38tMWxNAKfUWYFxrfYc9G/B+rE+eO7pI9kK0NJoN/s/Jb3PvsftZqltrfh0+d2SoCd80TT54\n53c5m1tZcywSDvFH7/5pptLxocUhxDB0nfC11q+y//h422P3Ys1UFKJnd+nP8e3ZfyQRSfC6y36O\nrxz7GicKq8cGeOt8vsLZ3DI7tiVR+yaYz1U48vR5js3mmUpnhxqLEF6TmbbCN4+cO8JELM2/vel3\n+fnLfoZMLM3xwsmhxnBi3lpW56ev3cHbf/4g//xnrrzgcSE2E0n4whflWplCtcie9G7SsRQAe9O7\nWagsUqyVhhaHk9j3Zq0YLt2VsR8fXgxCDIskfOGL0+U5AHYmZ1qPXZLaAzDUss5JO7HvzY4DkJ1M\nkIhHOCktfLEJScIXvjhdshP++ErC35u2Ev4wyzon5ovEo2GmJxMAGIbBnuw4Z84vUavLzoFic5GE\nL3wxW7L2/tg1vqP12L70bmB4Cb/eaDJ7rszu6XFCbXND9mZTNE2T2XPlocQhxLBIwhe+WKuks31s\nG4nIGMeLw0n4Z86XaTRN9tjlHMeeaevv0nErNhtJ+MIXZ0pzpGMpktFk6zHDMNib2s18+RzL9Yrn\nMZxo1e9TFzzu1POl41ZsNpLwxdBVG1XOLy9e0Lp37EvvwcTkZNH7OXwrI3QubOHvnUldcFyIzUIS\nvhi6M+V5TEx2ttXvHfuG2HF7cp0W/vhYlKl0vHVciM1CEr4YutYInTVa+HtTdsftEOr4J+aLZJJR\nMuOxi47tyY6zUKhQWpZ9wcXmIQlfDF2rw3b84oS/I5klGop63sJfqtQ5m1tmz6rWvcNp9UsrX2wm\nkvDF0J22h2SulfDDoTB7UruYLZ2h1qx7FsOps1YiXz1CxyEjdcRmJAlfDN3p0hxj4TEmYpk1j+9N\n76ZpNpktnfYshtVLKlwUQ9bpuJUWvtg8JOGLoWo0G8wtnWXn+AzrbYTjLLHgZVlnvSGZjt3TSUKG\nIS18salIwhdDNb90jqbZXLPD1rHXnnHr5Zo6J+eLGKyUblaLRsLs2Jbg5HyJbjcJEmLUScIXQ7VR\nh61j9/hODIzW8gtuM02TE/MlspMJ4rHwus/bk02xVKlzPu/9JDAhhkESvhiqtRZNWy0ajpKKjrNY\nyXkSQ75UpbhUW7fD1uFMyDp5Vso6YnOQhC+GqjVCJ3nxpKt2k/EMuUrek3KKU79fb0imY8+0dNyK\nzUUSvhiq0+U5IqEI2xNTGz5vIj5BtVlr7XXrpvnFJQB2bkts+Lyd25MXPF+IoJOEL4amaTY5U5pj\nRzJLyNj4rTcRt4ZselHWWSxaNfmp1MablE+lrBm4uWLV9RiE8EPHTcyVUiHgE4ACmsC7tdaPth2/\nDbgVmLMfepfW+gkPYhUBt1jJUW3WNhyh45i0E36ummc3O92Nw07gk+mNE34iHiEWCbFQlE5bsTl0\nTPjALYCptb5ZKfVy4MPAG9uO3wC8VWv9Qy8CFJvH+eVFALYntnV87koLP+96HE4Lf2J844RvGAYT\nqVjr+UIEXceSjtb6i8A77b/uBxZWPeUG4Hal1CGl1AfcDU9sJvlqAYBMLN3xuZPxCQByHiT8XLFK\nLBoiEV9/SGYrjlScfKlKsylj8UXwdVXD11o3lVKfAj4G/O2qw3cB7wZeCdyslHqtuyGKzcJJ3k7r\nfSPOsgs5j2r4k6n4ujN9202m4pgm5MtSxxfB101JBwCt9TuUUu8HHlJKHdRaO0MXPqa1zgMope4F\nrge+utG5stnOLbwgk+tbW33WKo1cOrOj4znimT3wj7BE2dXfZ6PRJF+ucs2O7euet/3xXdkUPDaH\nEYlsivu6Ga5hI5v9+gbVTaftW4G9Wus/BJaBBlbnLUqpDHBYKXUQWAJeBdzZ6Zzz84VBYh5p2Wxa\nrm8dswtnAWiWIx3PYZomESPMXP68q7/PhUIF04TxeHjN866+vnjY+hZw7PgCE2OdS0CjTN6bwebG\nh1k3JZ3PAs9XSn0T+J/AbcAvKqVutVv27we+AXwTeERrfd/AUYlNqVXDj3d+4xqGwUQ84/qwTKcD\ndrLDkEyH87zFknTciuDr2MK3Szdv3uD43cDdbgYlNqdcJc9YOE48fPEOU2uZiE9wLPcMTbPZcdx+\nt3pP+FasiwVJ+CL4ZOKVGJp8tdBV694xEc9gYra+GbjBGYM/keryQ8dp4cvkK7EJSMIXQ1Fv1inW\nSutuerKW1uQrF4dm5vos6eRkLL7YBCThi6EoVK0VJ7sZg+9wPhzcnHy1UtLproWfiIeJRUPSwheb\ngiR8MRS5avdj8B0rk6/c67htLavQZQvfMAwmx+My21ZsCpLwxVDkK93PsnVMeFDSWSxUiEfDjG2w\n8clqk6kY+VKVRrPpWhxC+EESvhiKnN3x2lsL34OSTqnKZCrW1SzbVhzpOCaQL9Vci0MIP0jCF0OR\nt5N2by18u6RTdSfh1xtNCqVq1+UcR2ssvpR1RMBJwhdD0U8LPx6OkYiMuTb5Kl+qYtL9kEyH83xJ\n+CLoJOGLocg7nbY9tPDBauW7VcPPlXrrsHWsDM2UkToi2CThi6HIVQpEQhESkY23FVxtMpahXF+i\n2hi8fu7MlpWSjtiqJOGLochXC0zE0j11loK7Wx32OgbfMSklHbFJSMIXnmuaTWtZhR5m2TrcHJq5\n0OMYfMekLK8gNglJ+MJzpVqZptlkood1dBxuTr5qLavQYS/b1cZiYeLRsLTwReBJwheey7WGZPbf\nwl90YWhma+G08d5KOit720oLXwSbJHzhuZUhmf208N0r6SwWK8RjYRLxrjd6W4kjFacgs21FwEnC\nF57LD9DCd3Mz85y9l20/JlMxmW0rAk8SvvCc08LPxFI9/2w6msLAGHiUTr3RJF+uMdXjCB2HDM0U\nm4EkfOG5fB+zbB3hUJhMLDXwejr5krPxSb8tfDvhy85XIsAk4QvPDVLSAeuDIlfNY5pm3zGsLIvc\nbwvfHotfko5bEVwde6+UUiHgE4ACmsC7tdaPth2/BfggUAM+pbW+w6NYRUDlqgUMDNKx8b5+fiI+\nwbOFk5TqZVLR/s7R6162q0kLX2wG3bTwbwFMrfXNWIn9w84BpVQE+CjwauAVwDuVUlkP4hQBlq/k\nycRSfW9E7qyw6eya1Q8n4fe6cJpDFlATm0HHf4Fa6y8C77T/uh9YaDt8EHhCa53XWteAB4CXuR2k\nCC7TNMlVC2T6qN870nZnrxsJf2rQFr6MxRcB1tWAZK11Uyn1KeBNwD9rO5QB2odPFIAJ98ITQbfc\nWKbWrPW8SmY7p4xTrJX6PkevWxuulohHiMfCspm5CLSuZ6Bord+hlHo/8JBS6qDWegnIYyV9RxpY\n7HSubLb/f/xBINe34mTeStIzE9v7/r3sWZqGJ8CM1fo+R7naAODA/u0dJ16t9xrbM2Pky/3HMAqC\nHHs3Nvv1DaqbTtu3Anu11n8ILAMNrM5bgCPAFUqpSaCMVc75SKdzzs8X+g541GWzabm+Nk8vzAIQ\nb471/Xsxl6y36ez5c32f49zCErFoiGJ+iY0KQxtd3/hYhNlzJc6cyRMK9bbq5yiQ92awufFh1k0v\n2meB5yulvgn8T+A24BeVUrdqrevA+4D7gQeBO7TWswNHJTaNXGvz8sFr+IOUdPLlKplkfx22jkwy\nhmlCcVlm24pg6tjCt0s3b97g+L3AvW4GJTYPZz/aftbRcaTs4Zz9dtqapkmhXGPfTH9DOh3pZNSK\no1wb+MNDCD/IxCvhqXxrWYX+E/54JImB0XfCX642qDeapAdM0s7PF2TylQgoSfjCU8WqVYZJD5Dw\nw6Ew49Fk3yWdQtlK0E4LvV+tFv6SlHREMEnCF54q1KxWeb8zZB2p6DjFPlv4+bKVoAeu4dvr6Oel\nhS8CShK+8FSxWiQaihIPD1hOiaUo1cs0mo2ef3alhT9gDAmnhi8JXwSTJHzhqUK1RCo63vPm5aul\nWiN1yr3HYLfwBy7p2C1853xCBI0kfOEZ0zQp1oqtYZWDSLdm2/Ze1nGrhe+UhKSFL4JKEr7wTKVR\npdast4ZVDiI1wHo6zi5VmfHBWvipRBSDlT4BIYJGEr7wjDOqJh11o4Vvl3T6SPiFJbuFnxishR8K\nGYwnotLCF4ElCV94xmmNu9PCtydf9TE00xk3P2gN3zmH1PBFUEnCF55x6u2+t/DLNeKxMLFoeOA4\nMskYpaUajWaz85OFGDGS8IVnCvakq5QbnbZODb+PTltrHZ3BW/dgtfBNoLhUd+V8QgyTJHzhmZUW\nvoslnWpvJR1nHZ1BR+g4WkMzZfKVCCBJ+MIzTg3fjWGZyUiCkBHqeVjmUqVOo2m6ttiZTL4SQSYJ\nX3jGGaUz6LIKACEjRCo63vOwTGcIZcqlkk5reQXpuBUBJAlfeKboYg0fsBN+byUdpyXuWgtfJl+J\nAJOELzxTqBWJubCOjiMdS9l75HbfYepMunJjSCbQ6vyVFr4IIkn4wjPFasm11j207XzVQ1nHmXTl\nVgs/ZZ+nKC18EUCS8IUnTNOkUCu6Mgbf4fQF9DI0081JVyAtfBFskvCFJyqNCnWX1tFxrLTwu6/j\nr6yU6U4LfzwRxTCkhi+CSRK+8ISb6+g4nHP1MlIn79JuV46QYZBORKWFLwJpw03MlVIR4JPAfiAG\nfEhr/eW247cBtwJz9kPv0lo/4U2oIkjcXEfHsbImvn8tfOdci8WKa+cTYlg2TPjArwJntdb/Qik1\nBfwI+HLb8RuAt2qtf+hVgCKY3ByD70i3Ztv2UMMvV0nEw0Qj7n2ZTSejnDxbot5oEgnLl2QRHJ3e\nrfcAH2x77urvsTcAtyulDimlPuB2cCK43Jxl60hFe19Px81lFRzO5KuibGYuAmbDhK+1LmutS0qp\nNPAZ4PdWPeUu4N3AK4GblVKv9SZMETStSVeutvB767RtttbRcad+34ojIZuZi2DqVNJBKbUP+Bzw\np1rr/7Hq8Me01nn7efcC1wNf7XTObDbdR6jBIdcH9RNWMrxkxw6y29z5fZhmikgowrK51FUMhXKV\npmkyPZns6Z50eu7OGeuDJxyLBO5eBy3eXm326xtUp07bHcDXgN/UWn991bEMcFgpdRBYAl4F3NnN\ni87PF/qLNgCy2bRcHzCXWwCgVjSYb7j3+0hFx1ko57qKYfac9U0gHjG6vifdXF/INAE4firHnqlE\nV+cdBfLeDDY3Psw6tfBvByaBDyqlfh8wgU8A41rrO5RS7we+ASwD/0trfd/AEYlNobU0soujdKzz\npThTnu/quV6M0IGVyVey85UImg0Tvtb6NuC2DY7fDdztdlAi+IrVIrFwjJhL6+g4UtFxjjdOUmlU\nO67Rk2/NsnU3Bud8eZl8JQJGxpQJTxRqJVc2Plmtl/V0CvYoGrd2u2rFIC18EVCS8IXrTNOkWC26\nunCao7W3bReTrwoet/BleQURNJLwheuWGxXqZsOTFn6qh8lXKzV8d1v4ybEI4ZAhLXwROJLwhevc\n3vikXS/r6ayso+NuCz9kGKQSUanhi8CRhC9cV2htXu5Bwo91P9u24PLCaRfEkYxJC18EjiR84bqi\nBwunOXqZbVso10jGI56sd5NORlmq1KnVm66fWwivSMIXrvOyhd/Lejr5cpX0uLvlHIezno503Iog\nkYQvXLdSw/eihd9dp22zaVJccn8dnVYcCRmaKYJHEr5wnZct/Fg4Rjwc6zgOv7hcwzTd28t2tbS0\n8EUAScIXrvOyhQ/WB0mhwzh8r4ZkOmR5BRFEkvCF61Y2P3G/hQ9Wx22hWsS0FzFbi7OsglctfOe8\nOVkiWQSIJHzhukK1SDwcIxb2pnWdio3TMBss1ZfXj8EutWQ86rSVko4IIkn4wnXFWsmz1j20Tb7a\nYKROq4Xv1Sgdu6Qjk69EkEjCF64yTZNCtejq1oarOTN4Nxqpky97s3CaY2VYptTwRXBIwheuWqov\n0TAbnib8blbMLHi0rIIjHg0Ti4Rkm0MRKJLwhatam5dv8pKOYRikkzEp6YhAkYQvXJW3E35mCC38\njUs6VUKGQXKs47bNfcuMR8mXahuOFhJilEjCF65yWt1erJTpWEn464/FL5RqpMejhAzDuziSMeqN\nJsvVhmevIYSbJOELVxWG0MLvZj2dfLnq2Rh8R0a2OhQBIwlfuKpVw/c04SeB9Tttq7UGy9WGZyN0\nHK2ROiUZqSOCYcMCp1IqAnwS2A/EgA9prb/cdvwW4INADfiU1voO70IVQVCoFgBIx9KevUY4FGY8\nmly3ht9aVsGjDluHjMUXQdOphf+rwFmt9cuAnwf+1Dlgfxh8FHg18ArgnUqprEdxioBw1rjxcpSO\nc/71SjpOAva6pON8oMjQTBEUnRL+PVgteOe57d9dDwJPaK3zWusa8ADwMvdDFEFSqBYIGSGS0YSn\nr5OOpSjVyjSaF3eYej0k0yE1fBE0G5Z0tNZlAKVUGvgM8HtthzNAru3vBWDC7QBFsBSqRVLRcUKG\nt91DziigYq3MRPzC8lHew60N2znnlxq+CIqOg5SVUvuAzwF/qrX+H22H8lhJ35EGFrt50WzWu/ru\nKNjK11eslZhJTXv+O5jJTMEcRFNNspMXvlbT/rDZt2uirzi6/ZlI3Er4lUYzMPc8KHH2a7Nf36A6\nddruAL4G/KbW+uurDh8BrlBKTQJlrHLOR7p50fn5Qh+hBkM2m96y11dt1FiqL5MwEp7/DiINq5zy\n7Jk5krULv1jOzlmvbdYbPcfRy/1rNK39bM8ulANxz7fye3MzcOPDrFML/3ZgEvigUur3ARP4BDCu\ntb5DKfU+4H7AAO7QWs8OHJEIrJUhmd63sjaabTuskk44FCKViMqa+CIwOtXwbwNu2+D4vcC9bgcl\ngqnobG3o0U5X7ZxRQMU1dr4qeLz5yQVxJKOyYqYIDJl4JVyTb43B93ZIJmy8RHK+XGMsFiYWDXse\nRyYZo7hUa5V3hBhlkvCFa5y1bUahpDOM1j2sjMUvSitfBIAkfOGaYmtp5OGVdFZPvmqaJsWytXDa\nMEy0xuJLwhejTxK+cE2+ZpV0MkNo4SciY4SN8EXr6ZSX6zSa5hBb+PbyCtJxKwJAEr5wzTAWTnMY\nhkE6lrqopDOsWbYOmW0rgkQSvnCNk3y9XAu/XTo6flFJx+utDS+KIemsmCkJX4w+SfjCNYVqkURk\njGjIu12m2qViKSqNKtXGSrL1evPy1TJOSUdq+CIAJOEL1xSqxaGUcxxr7Xw19JLOuJR0RHBIwheu\naJpNirWS58sit1uZfLVS1hl2SScjJR0RIJLwhStKtTIm5lDG4DvWGos/7Bb+WCxMJBySFr4IBEn4\nwhXDHKHjWGu27bBr+IZhkBmPkpclkkUASMIXrhjmsgoOZ4JX+0idfLmKYcB4YjgJH6zyUaFcxTTN\nob2mEP2QhC9csTLL1o9O27YafqlKOhkjZBhDiyOTjFGtN6nULt59S4hRIglfuCJvt7Izfo/SKdeG\nVs5xyNBMERSS8IUrhj3pCiC1apROrd5gqVIfWoetozXbVkbqiBEnCV+4wkn4w2zhx8JRxsLx1msX\nWh22w034MttWBIUkfOEKP0bpgPWNwukwzg95DL5jpaQjCV+MNkn4whWFapFIKMJYeGyorzsRy1Co\nFmk0G62hkZkhLY3syMgSySIgJOELVxRqRdLRFMYQR8cATMYzmJgUasWhz7J1SElHBIUkfDEw0zQp\nVAtDL+cATMQzAOQq+aHPsnXIejoiKLpa1lApdSPwn7TWr1z1+G3ArcCc/dC7tNZPuBuiGHWVRoVa\ns+5rwl+s5FksWol3YsgJP52MYgCLRUn4YrR1TPhKqd8F3gpcvHko3AC8VWv9Q7cDE8GR92HSlWMy\nttLCXyha6/hMpuJDjSESDpEej7FYqAz1dYXoVTclnaPAm9Y5dgNwu1LqkFLqA+6FJYLEGQfva0mn\nmmexWCFkGENv4QNMpeIsFCuyvIIYaR0Tvtb680B9ncN3Ae8GXgncrJR6rYuxiYDI+zQkEy6s4S8W\nKkykYoRCw+04BphKx6nVm5Qr6/1TEcJ/g25N9DGtdR5AKXUvcD3w1U4/lM0ObwldP2y162ssWqWM\nS7I7hn7t6Zo1BHPJLLFYrHL5nszAMfTz87uyKX509CxGJDLS93+UY3PDZr++QfWS8C9oNimlMsBh\npdRBYAl4FXBnNyeany/08LLBks2mt9z1HT97BoBQJe7LtY+FxzidP0+9cQmpsehAMfR7/8Yi1j+P\np549TzIy/G8Y3diK783NxI0Ps14SvgmglHoLMK61vkMp9X7gG8Ay8L+01vcNHJEInMVKHrDGxPth\nIp4h58SQGn793npdq6N4QTpuxQjrKuFrrZ8BXmz/+a62x+8G7vYmNBEUi5VFYKWePmwT8QxnynNg\nNJlKD3eEjsN53YWiJHwxumTilRhYrpInHU0RCQ3aJdSfCXtophFdHvqQTMeknfBlLL4YZZLwxUBM\n02ShkmNybMK3GJxSkhGr+N7Cl7H4YpRJwhcDWaovUWvWfKvfQ1spKepfwk/GI8QiIanhi5EmCV8M\nZKGSA2Ai7l8Lf6LVwvevpGMYBpP25CshRpUkfDEQZ4TOlI8J3/l2ER2rkYj7048AVh2/UKpSbzR9\ni0GIjUjCFwNZGaHjYwvf7rSNJfxdj34qHcdEtjoUo0sSvhjIKLTwk2FrSYdw3N9yypSMxRcjThK+\nGMjislXD97PTtlRuYNaimNFl32KAlUlfkvDFqJKELwayWHUSvn8t/IViBbMWpx5a8i0GWBmLLx23\nYlRJwhcDWVzOMRaOMxYZ7l627RYKFczqGA2qVBr+1c9bY/El4YsRJQlfDCRXyfvaugdrspNZi9vx\n5HyLw6lCZqWoAAAUKUlEQVThy+QrMaok4Yu+VRs1SvWy/wm/WMWsOgk/71scE9JpK0acJHzRt8WK\n//V7cGr4VknJz4QfjYRIJaIsyHo6YkRJwhd9W0n4/o3QAbtFbbfwF6v+JXyw6viLBdnqUIwmSfii\nb62E7+PCaWDVzJ2x+H628MFK+JVag+Vqw9c4hFiLJHzRt9amIz6WdEzTZLFYYcL+0PE74ctGKGKU\nScIXfVtZOM2/kk65Uqdab7I9kcHAIDcCJR2QsfhiNEnCF31zhkBOxSd9i8FpSU+lE6RjqRFo4Vuz\nbWVophhFkvBF3xYqOcJGmPFo0rcYnMQ6lYq19rb1s8O01cKXhC9GkCR80Tdr0lWGkOHf28hJrJPp\nOBOxDNVmjaW6f2vqODV8mW0rRlFX/1KVUjcqpb6+xuO3KKUeUko9qJS61f3wxKhqNBvkKnlfl0WG\nlcQ6lY63+hL8rONLC1+Mso4JXyn1u8AngPiqxyPAR4FXA68A3qmUynoQoxhBhVoRE9PXZZGB1iSn\nyVS8NR9gYXnRt3hSiSiRsCEtfDGSumnhHwXetMbjB4EntNZ5rXUNeAB4mZvBidG1sOz/CB1oq+Gn\n40wntgNwdum8b/G0tjqUFr4YQR0Tvtb680B9jUMZoH2lqgLgb3NPDM3KCB2fW/iFCrFIiGQ8QraV\n8M/5GtNkOk6uVKXRlK0OxWgZZAPQPFbSd6SBrr5LZ7PpAV529G2F66stWB2j+7I7fb3ehWKF6ckE\nMzMZxjKXwvch18wNFNOg17N7OsXREzlCsSjZKf9GMK1lK7w3xfp6SfjGqr8fAa5QSk0CZaxyzke6\nOdH8fKGHlw2WbDa9Ja7v5Ll5AMLVmG/XW16ukS9V2b/Tisk0TcbCY5xaPNN3TG7cv0wyCsBjR+cx\n9m8b6Fxu2irvzc3KjQ+zXsbTmQBKqbcopW7VWteB9wH3Aw8Cd2itZweOSATCgrN5ecy/ks6ZBWuH\nq5nJBGDVz7OJbcwvnfN1LP6OqcQF8QkxKrpq4WutnwFebP/5rrbH7wXu9SY0McrOLS0QMkK+rpR5\nZqEMwI5tK2WT6eQ0x4unyFX925hlh13GceITYlTIxCvRl7mleaYT2wiHwv7FcN5qQTstaqDVcTtf\n9q/jdmab3cI/Ly18MVok4YueFWslSrUyMwl/p104LeiZthb+KIzUSSeiJOIR5hYl4YvRIglf9Gyu\nfBaAmeS0v3EsLBEOGWzPrMwJdMbiz/uY8A3DYMdUgrmFJZqyEYoYIZLwRc/mytYInZmk3y38JaYn\nE4RDK2/jUWjhA8xMJag3mpzP+7eujxCrScIXPXNa+Dt8TPil5RrFpdoF9XuwZv5GQhHml876FJll\npeNWyjpidEjCFz0702rh+1fSOdPqsL1wYlPICDE9to15H5dXANhhd9zOnZeROmJ0SMIXPZsrzxMP\nx5iIjcKQzMRFx7LJ7SzVlyjV/Eu20sIXo0gSvuhJ02wyv3SWmWQWw1g9+Xp4ztgt59UtfGjvuPWv\nrOPMDZiThC9GiCR80ZPz5UVqzTozCZ9H6NhDHmem1mjh27Gd9XEs/vhYhGQ8IpOvxEiRhC96cqpw\nBhiBETrnl4iEDbZnxi46NjJDM7clmF9cotmUoZliNEjCFz1ZSfh+j8Evk51MEApdXFbKjkDCB6vc\nVG+YMjRTjAxJ+KIns4U5wN8hmcWlGqXl+pr1e4DtY1OEjJDvCX9GFlETI0YSvujJ7Ai08J0O27Xq\n9wDhUJht8UnfJ1/JImpi1EjCFz2ZLcyRjqVIRNZOtsPQGpK5TsIHq46frxZYrvu31aAsoiZGjSR8\n0bVas85c+Zyv5RxYGerYvmjaaln7G8i5Zf8mYEkLX4waSfiia2ftjUX8XyXz4mWRV5tOWDtNzZf9\nG4ufSkQZH4vIWHwxMiThi67NjcCSCmDV8CPhENvWGJLpcMbi+91xu2NbkvnFJdnQXIwESfiiayvL\nIvvXwjdNkzMLS2QnxwhtMNM3OwKzbcHqWG40Tc7l/etLEMIhCV90zVk0zc8afmGpxlJl/SGZjpnk\nNBEjzPHCqSFFtjYnTllETYwCSfiia3PleQzDaNXH/dBaQ2eNRdPaRUIR9qb3cLI4S61RG0Zoa3L6\nGU5LwhcjoOMm5kopA/gz4HnAMnCr1vqptuO3AbcCc/ZD79JaP+FBrMJnc+WzzIxPEwl1fNt45ths\nAYBLZtIdn3tpZh9P55/lRPEUl01c6nVoa9q3w4rTiVsIP3XzL/eNQFxr/WKl1I3AR+3HHDcAb9Va\n/9CLAMVoKNXKFGpFrpj2J3E6njyZA+DAns5LM+/P7OObwNP5474l/F3bkyTjEZ48lfPl9YVo101J\n52bgPgCt9XeBF646fgNwu1LqkFLqAy7HJ0bEsdwzABzY5nPCP5UjnYySnew88Wt/Zh8Az+SPex3W\nukKGweW7M8wtLJEvV32LQwjoLuFngPbmSV0p1f5zdwHvBl4J3KyUeq2L8YkR8WTuaQCunr7CtxgW\nChXO5ysc2D3R1Vr82cQ0iUiCp/PPDiG69R3YMwHAUyfzvsYhRDclnTzQXjANaa3bBxV/TGudB1BK\n3QtcD3x1oxNms53rr0G2Ga/v2R8/i2EYXLn9MhLR9ce/e+nxU1Yd/Lqrsl3/jq+a3s/Dp4+QyIRI\nxce7+hm3798LrtnJFx84xuziEj/r83tjM74322326xtUNwn/QeB1wGeVUjcBh50DSqkMcFgpdRBY\nAl4F3NnphPPzm7cDK5tNb7rrqzVqHD33NHtTu0lEx3y7vh8cOQ3ArsnuY9g1tpuHOcL3jx3hmu2q\n4/O9uH/bk1EM4PAT876+Nzbje7PdVri+QXVT0vk8UFFKPQj8MfBepdRblFK32i379wPfAL4JPKK1\nvm/gqMRIebZwkrrZ4MDEfl/jePJUjpBhsH9n93vpOnV8P8s6ybEIu6fHeWo2LzNuha86tvC11ibw\nnlUPP952/G7gbpfjEiPkydwxAA5MXuZbDLV6k2dOF9g3kyIeC3f9c5eOQMctWKOKTp4tcWKuxKU7\npewg/CETr0RHTy4+DcDlPg1tBHj2TIF6w+xqOGa7TCzNtrEpns4fxzT922rwwG6r4/boSRmeKfwj\nCV9sqGk2eSr3NNNj25iMT/gWx8r4+95j2J/ZR7FW4tzygtthdc2JW8bjCz9JwhcbOl2ao1xf8rWc\nA3D0lDWk8Yo+Ev5KWce/Ov7O7UnGxyKtDy4h/CAJX2zIGX/ve4ftyRyZ8RjTE70PCd2fuQSwZtz6\nxZqANcH84jL5kkzAEv6QhC825NTvD0zu9y2G8/llFgoVDuzOdDXharV96T2EjJCvCR9WloOQVr7w\niyR8saGncscYjybZkZzxLYYnByjnAMTDMXaN7+B44QRVH1fOdOr4R6WOL3wiCV+sa2F5kXPLC1w+\nsb+vlrVbnjixCPTXYeu4dvvV1Jp1Dp991K2wenb5rgwGcPSEJHzhD0n4Yl2jUL9vNJt877E5EvEI\nl+3qf/z6i3a+AICHTv/ArdB6lohHOLBngqMncpzLLfsWh9i6JOGLdX3vjLXi9dXbrvIthkeeOs9i\nscpN1+wgGul+wtVqu8Z3sC+9h0fPawrVoosR9ubm63ZhAg8envUtBrF1ScIXazq/vMAjZx/j0vQ+\n9qV3+xbHoR9bifGlz9s18LletPMFNM0m3zvzo4HP1a+funqGeDTMA4dnafo4EUxsTZLwxZoePPld\nTExeuucm32LIlao8fPQs+2ZSXLpj8OUIXrjj+YSMkO9lnZ+6eoazuWUee8a/iWBia5KELy5Sb9Z5\ncPYhEpEEN+x4nm9xfPuR0zSaJi+9bpcrncaZWJqrt13Js4UTnC7Ndf4BjzjfVpxvL0IMiyR8cZGH\n539CoVrkpl03EAvHfInBNE0O/fgUkbDBTdfudO28N+7wv/P2ij0T7NyW5Pt6ntKyf8NExdYjCV9c\n5NDJbwPw0t3+lXOePJVn9lyZF1yVJZWIunbe67LXMhaO89DpH9A0/Vmq2DAMXnrdLuqNJt/5yRlf\nYhBbkyR8cYHZ0hmeWHyKq6auYMe4f5OtDj18CoCXXuduh3EsHOP52eeyUFlELxx19dy9ePFzdhIy\nDA79+JRvMYitRxK+uMChk98B8LWzdn5xiYeOzLE9M8bB/VOun/9m+9ruefwLVBv+rGszkYrzvCu2\n8+yZIoefOudLDGLrkYQvWp5YeIpDJ7/NZHyC501f60sM9UaT//rFR6jUGrzxpZcR8mCG72UTl/Cq\nfS9lrnyWzx/dcPtlT73+JZcRDhnc8ZVHWSxWfItDbB2S8AVgLaNw5yP/DYC3X/PLhEP9T3IaxGe+\n/iTHZgu85Dk7eclzBx97v57XX/5P2Dm+g/9z8lv85Jz27HU2cunONL/0qisolGv85Zd+QrMp4/KF\ntyThC2qNGp84/GkKtSL/9IpbuHLqgC9x/PDxef7+e8fZtT3Jr7zG29m90XDU+mAzwvztkXso1kqe\nvt56Xn3DXq6/cprHnl3kSw8e8yUGsXVIwt/immaTux//PM8UjnPjzht4+d4X+xLHs2cK3HnvEaKR\nEO95w3MYi3Xcbnlg+9J7eN1lryFXLXDH4U+zWBn+omaGYfBrv3CQ7Zkxvvzg03zvMf/mB4jNr+O/\nKqWUAfwZ8DxgGbhVa/1U2/FbgA8CNeBTWus7PIpVuOzp/LN89vEvcSz/LJek9/DL6heHvipmpdbg\nSw8e4/6HjtNomrz9569m70xqaK//6ktfzlP5Zzh89lE+9N2Pcmvjl7kqcfVQfw/jY1He/cZr+aO/\n/QF/9oVHeNHBGd7yM1cykYoPLQaxNRidNnZWSr0JuEVr/WtKqRuB27XWb7SPRYAjwA3AEvAg8Ata\n6/kNTmnOzxdcCX4UZbNpRvn6mmaTE8VTfOP4g3z39PcBuH7mOv75lW9gIt55+QK3rm+xWOGRp87z\npQePcTa3zPTEGL/6GsV1B7YPfO5emabJoZPf4fNHv0K1WeO509dw064XcvXUFYxFet9hq1/H54r8\nzX2P8eSpPMl4hNe9eD/Pv3KaHVMJVz6ARv29OagtcH0Dvwm6+d58M3AfgNb6u0qpF7YdOwg8obXO\nAyilHgBeBvzdeic7s7jIfN6/1Qq9VovUOZ93ux5srvM3E9MEkyZN08TEpGk2qDXr1Jt1as0axVqR\nQq1IoVrgZOkUTxeeodKwRoTsSu7k9ftfx+UTl0ETiksbz/o0TZNYsUKhXF2Jw3T+b9I0rec0myb1\npkm11qBWb7JUrZMrVlksVjiXW+bxEzlOnbV+RyHD4J/ceAlveMllxGP+dBQbhsHL9v40B7ddxd1H\nP8vhs49y+OyjhI0wByYvY8/4TibHJpiKT5COpYiGYsTDMaKhKCEjRMgwCBkhDMPAYOXfZPuf6eKf\n6tSUwW+/+WoeODzLV771NPccepR7DsFUKs6VeyeYnkgwkYoxMR4jEY8QDYeIRkNEwnYMIcOq0Tof\nDhf+j2q4xkK+7MavbCR5829vdGSzg68n1U3CzwDtxc26UiqktW6ucawAbLhLxW997faegxTuaS4l\naRb20sxv56nzO/kTTgAnhhpDLBriOZdv45pLt/H8K6fZuS051NdfTza5nX/3qvfxj08+yqPnHuOR\nc4/x+MJRHvdjgtZzIGH/cRk4DFbRdMH+T2w59xz484HP0U3CzwPtHy1OsneOZdqOpYHFjU52z5v/\n3L+tk4Towo1XPIcbr3iO32EI4bpuRuk8CLwWQCl1E3Zjw3YEuEIpNamUimGVc77tepRCCCEG1k2n\nrTNK5zr7oXdgddKOa63vUEr9AvBvsUqFd2qt/6uH8QohhOhTx4QvhBBic5CJV0IIsUVIwhdCiC1C\nEr4QQmwR3i9YYlNKXQ18B5jRWldXHfsN4J1YI40/pLW+d1hxDUoplQT+OzAFVIC3aa1nVz3nT4CX\nYM1TAHiD1nrkpwR2eW1BvncZ4L9hDS2OAr+jtf7OqucE8t5B19cX2PvnsFcD+Gda619Z41hg75+j\nw/X1dP+GkvCVUmngP2PNIVl9bAfwW8ALgCTwgFLqfq11UDb7/A3ge1rrP1BKvQ14P3DbqufcAPyc\n1vr80KMbzIbXtgnu3fuAf9Baf1wpdRVwF9a9ahfUewcdrm8T3D8nob8G+NE6Twny/dvw+vq5f8Mq\n6fwlcDuw1rzuFwEPaK3r9hINT7AyBHTkaa0/BnzI/uslrJoHaQ9rvRL4S6XUA0qpdww5xL51ujYC\nfu+AjwJ/Yf85irUeVEuQ751tw+sj+PcPrHlC71nrwCa4f7DB9dHH/XO1ha+U+jXgvVy4+MuzwF1a\n68P2DVht9fIMRTosz+CXVddn2P9/h9b6+0qpfwCeC/zsqh8bBz6O9Y8vAnxdKfWPWutHhhd5Z31e\n22a5dzuBTwO/verHAnHvoO/r2wz37zNKqZev82Ob4f5tdH093z9XE77W+pPAJ9sfU0o9Dvy6UupW\nYCdwP/CKtqf0vDyDX9a6vrZjr1ZKKeBe4Iq2Q2Xg41rrZQCl1P/GWmp6pN50fV5b4O+dUuq5WP0U\nv6O1fmDV4UDcO+j7+gJ//zoI/P3roOf753kNX2vd2rpIKXWMi1uJDwF/YC/NkACuZgRvyHqUUrcD\nJ7TWnwZKQH3VU64C7lZKXY/1+74Z+KuhBtmnLq4t6PfuGuAe4Je01ofXeEpg7x10dX2Bvn9dCPT9\n60LP929oo3RsztcVlFLvxVpa+StKqY8DD9jH/vXqUTwj7k7gr+2vZCHg7XDR9f011gilKvBXWusj\nfgXbo26uLcj37sNAHPiYXW5c1Fq/aZPcO+ju+oJ8/9a0ie7fmga5f7K0ghBCbBEy8UoIIbYISfhC\nCLFFSMIXQogtQhK+EEJsEZLwhRBii5CEL4QQW4QkfCGE2CIk4QshxBbxfwEutqzT5Jr8zwAAAABJ\nRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11307fb90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "t1 = scipy.stats.norm(-2.5, 0.1)\n",
    "t2 = scipy.stats.norm(-2.8, 0.1)\n",
    "\n",
    "x = np.linspace(-4,-1,100)\n",
    "\n",
    "plt.plot(x,t1.pdf(x))\n",
    "plt.plot(x,t2.pdf(x))\n",
    "plt.title(\"KL-divergence:\" + str(scipy.stats.entropy(t1.pdf(x), t2.pdf(x))));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEKCAYAAAARnO4WAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmUJFd94Ptv5Fa5Z3VXZe/aZV21hADRQsJCBrHNMGKE\nkfGYw2AGaywL/GzPyPj4CD0Pnjczxh4fxjzDY8zYCITtsSULDOZgsQgwMEgGBEgIrVctqdXqvZau\nyrVyj/dHZGRlZ1dVZlVGZkZk/j7n6KgrIjPy3oqsX978xe/eMEzTRAghxPjzjboBQgghhkMCvhBC\nTAgJ+EIIMSEk4AshxISQgC+EEBNCAr4QQkyIwKgbILxNKXUe8LjWOtG27R3A/wd8APho+75NHPdL\nwGe11n+llHoYuF5rnXWq3W6jlJoGvgPcrLV+uLntIuATQBoIAp/WWn9kjee+BPgecLBt8zu01gc7\nHysmmwR84YTWZA6l1HuB3wPeAGTb922V1voV/R7DzZRSNwD/L3Bex67PYAX5u5RSSeCHSqmHtdbf\n7njctcDfaK3fN/DGCk+TgC8co5T6APDvgFdrrY80R/+9Pnc38JfAbuAoMNu2r9H8+UvAn2itP9/c\n/kcAWus7lFK/Cvw6YACLwG9qrZ9RSt0FbAcuBP4R+BPgrubPi8Ap4DGt9X9VSu0H/rT5eD/wMa31\nZ5RSrwU+BDwPvAQIAb+htf6OUiqG9W3m1UAV+KLW+veUUkHgj4HXNI/1CPAftNb55ofiAa31rc0u\n/mbz93Z3x6/lTuDvmn3MKqWe5ewPBbAC/gVKqR9gfcD+sdb6Cz394sVEkRy+cIKhlPpjrKD4Ma31\nkS0c438C39NaXwH8X4Bq22d/S/gkcDOAUsoH/DLwSaXUa7AC5nVa6wPAh4H2gBfRWl+htb4D+BhW\nCupy4JewgiVKKT/wWeB2rfUrgeuB31VKXd08xtXAh5vfNj4N/D/N7f8NmNJaK+BK4Npmez4AVLXW\nV2mtrwROYH0AoLX+87Zgj9b6Bq31D7A+rGjb/pda61KzfW8Gfhb46hq/uzzwt1rra4BfAT6hlLpy\njceJCScjfOGEGHA5cANwr1Lqn7XWP93kMd4I/A6A1vqQUurrbfvsQHgv8GGl1A7gKuCg1vr55oj5\nIuCflVL2Y6ebeXGAB9qOdQNWYEZrfVIp9bnm9kuax/h02zHCzcc+DRzWWj/W3P4w8J7mv98A/Hbz\neFXgdQBKqQ8DKaXUv2g+Loj1bWLTlFLvwfoQe7vW+qxjaK1/s+3fTyul7gXeivWtQogWCfjCCUXg\nrVrrRjPN8g9KqQ3z7kopOxiZwK8BDc4c4dY6n6O1LiqlPgu8C2u0+8nmLj/w180RvH38c7TWy0op\nsEbAtmrH69TbjrHUfr1AKbULWAJeBay0PcdsO0aNM69h7AFKzeP9R63115rbY1gfIJuilPoT4BeA\nN7R94LTv97F6cbzQ3Gw0+ynEGSSlI5xgaq0bAFrr/w48AdyD9f4y1nqC1vrK5n+v0Fr/GCtVcSuA\nUmof1sh5LXdipS1+Fvj75rb7gXc2AzRKqVub29ZyH/CrzcfNADdhBWwNlJRS72prw6NAtwvG3wDe\no5QylFJTwOex8vZfBX5LKRVqBuU/B/6wy7HOoJT6GHAdcNVawR6g+Xt/K6u/u/OwPiD+fq3Hi8km\nAV84obMS5z3ApVj57ahSKtv8L9f8/+VrHOM3gcuVUk8An8IKtmcdv1myWAU+p7WuNLfdj5Uf/7pS\n6idYuf2b1mnb+4H9SqlHsXL2LwDFZjrm54Fbmvu+BvwnrfX3uvT9vzTb8yjwY+Aftdb/APwBcAgr\nrfI41t/a74BVyaSU+os1jtX+TWEf8BtYF6u/rpR6RCn1cDO9g1LqPqXUv24+/N8Cb1FK/RTrA+0/\naq11l3aLCWTI8shikiilfh14WGv9A6VUCPgu8Pt26kWIcdZTDr95kexHwBu11s+0bb8R+CDWCOcu\nrfWdA2mlEM55Evh4syonCNwrwV5Miq4jfKVUAKs64jKsC3PPtG1/CjiAdUHrQeAtWuv5gbZYCCHE\nlvSSw/8fWNO7j3ds349VFpdt5j8fwLpYJYQQwoU2DPhKqV8B5rTWX+fsaoskkGn7OQekHG2dEEII\nx3TL4d8MNJRSbwJeDvyVUuqtWus5rHVSkm2PTQDL3V7QNE3TMNas1BMu9uShRW7/+AOcuyvB//zd\n14+6OUJMor4D54YBX2v9WvvfSqlvAe9tBnuw8vcXN2czFrHSOR/u9oKGYTA/n9t6i10unU6MZf/u\n/94LALx4MsdPnjzB3nR8pO1xUrleIVO2vqxu3x6nnDOJBaMjbpXzxvW9aZuE/vVrMzNtTQCl1DuB\nmNb6TqXU+7EmuBjAnVrrE323SLhOo2Hyo6fnWj8/9NQcN41JwDdNkw/94CMslk63tgV8Af7Lz97O\n9JRkKMV46Tnga63t7/HPtG27D2uihxhjB48ukylUuOaynTxycIEfPj3H237uAsYhNbdUXmaxdJod\n0VkuTl3IYmUBvfg8h7NHmE5LwBfjRWbaiq4eesoa3V/30t28cv9OTp4ucmQu3+VZ3nA8fxKAq3e+\ngnft/0VuuuzNZ2wXYpxIwBcbqjca/EjPkYgGufTcaX7u5XsB+GFbisfL7MC+J74LgHNTVv+OFSTg\ni/EjAV9sSL+4TK5Y5YDagd/n48D+HUwF/fzwqTnGYVmOYwXrstOe2G4AZqLbiATCMsIXY0kCvtiQ\nnc65+tIdAIRDAV528QxzyyscPuX9iojj+ZOE/CFmItsAq4psd2wX8ysLVOuywrAYLxLwxbpq9QY/\n1nOkYiEuOWe6tf3q/TuB1Q8Dr6o36pwqzrM7thOfsfqnsCe+i4bZ4GRRVgkR40UCvljX04eXKJRq\nXKV24POtVuRcceF2wiHvp3VOFeepm3X2xHadsd3++XheqozFeJGAL9b17DFrMtLLLp45Y3sw4Oey\n87ezmC2xnK+MommOOF4484KtrRXw5cKtGDMS8MW6FjIlAHZsi5y1z9622HyMF7UqdDpG+Hvju87Y\nL8S4kIAv1rWYKWEA25Nn34p1NmVtW8iunLXPK443K3T2xnefsT0ajDI9lZIRvhg7EvDFuhYyJaYT\nUwT8Z79NZpofAl4f4SeCcRKhs5eJ2BPbxXI5Q7FaHEHLhBgMCfhiTfVGg6VcmZnU2aN7aBvhezTg\nl2olFktLZ+Xvbfb244VTw2yWEAMlAV+saSlXpmGarcDeyf4g8OoI/0QzkHfm721SqSPGkQR8sSY7\nkM+skb8HawJWPBL07Ai/c0mFTvZ2WWJBjBMJ+GJNdiBfb4QP1ih/MVvyZC3+sXVKMm27ojvwGT6p\n1BFjRQK+WNNiK+CfXZJpm02FqdYaZIveW4LgeP4EBtYyCmsJ+oOkI7OcKJz05AeaEGuRgC/WZI/w\n17toC6vpnoWMt0ozTdPkeOEkM5HtTPlD6z5uT3wXK7USS+Wud+4UwhMk4Is12UF8Jjm17mNmPXrh\nNlvJU6gW2bvO6N62NyYTsMR4kYAv1rSYLZGKhwgG/Os+xk73eC3g2xOudq+Tv7ftjssSC2K8SMAX\nZ2k0TE5ny8yuU6Fjm/FoLf7CinX/2h2R2Q0ftzOaPuPxQnidBHxxluV8mXrD3DB/D22zbbPeCvjZ\nchag603KU6Gk9fhKduBtEmIYut7EXCnlAz4JKKABvE9r/WTb/tuAWwB7cfT3aq0PDqCtYkgWeqjQ\nAYiGA8TCAc+N8DPNAJ6aSm74uEggTNAXJFOWgC/GQ9eAD9wImFrr65RSrwX+EHhb2/4DwLu11o8M\nooFi+BZ7qNCxzSTDnFwqYpomhmF0fbwb2AE8NZXY8HGGYZAKJSTgi7HRNaWjtf4icGvzx/OBpY6H\nHADuUEp9Vyn1AWebJ0ZhIdt90pVtJhWmUm2QX/FOLX6mkiPkCxL2d+9faipJtpKnYTaG0DIhBqun\nHL7WuqGUugv4KPA3HbvvBt4HvA64Til1g7NNFMO22CzJ7CXg22kfL6V1MuUsqalkT99IUlNJTExy\nlfwQWibEYPWS0gFAa32zUup24CGl1H6ttT3b5qNa6yyAUuo+4ErgyxsdK53e+Ku013m9f9liDYBL\nLpwlHDr7LdLev/P2puBHR6iahif6XW/UyVXy7E1ftG5727fvSs3AHPiiNdLb3d+/brxwjvox7v3r\nVy8Xbd8N7NNa/xFQAupYF29RSiWBx5RS+4EV4PXAp7odc34+10+bXS2dTni+fycW8iSjQXKZFTp7\n0tm/sN8aJT9/ZIlL9rj/j225nMHEJGrE1jxPnf0LNaxvMC+cOkmivn1o7RyEcXhvbmQS+tevXlI6\nnwNerpT6DvAV4DbgF5RStzRH9rcD3wa+Azyutf5q360SI9MwTRazJWa6VOjYvDbbdvWC7cYVOjb7\ncXLhVoyDriP8ZurmHRvsvwe4x8lGidHJ5CvU6t1r8G2rk6+8sZ7OpgN+SAK+GB8y8UqcYXETFToA\n0akAkSm/ZyZf2TX4yVBvX4/t0s2MTL4SY0ACvjjD6qJpvQV8wzCYSYZZyHhjXfxM2crxTm86pTO+\nuWExOSTgizMs9nDjk06zqQilSp1iuTaoZjmmldIJ9Rbww/4wIV9QRvhiLEjAF2fYSsBv5fGX3Z/W\naaV0ehzhG4ZBciopOXwxFiTgizP0cuOTTqs3QvFAwC9nCflDhP3rr/PfKRVKkqvkqTfqA2yZEIMn\nAV+cYTFbIhYOrDnhaj32t4HTHrhwm6lkmQ71NsvWNm3Ptq3KbFvhbRLwxRmyhQrT8d5HvwCpuHWb\nwEyhMogmOabeqJOvFHouybRJLb4YFxLwRUut3qBQqpGMrX+f17XYj8+6POBnKzlMzJ5LMm324yXg\nC6+TgC9a7ICdiAY39bxktBnwi+4P+ND7pCtba4RfkdJM4W0S8EVLrmgtcbzZEX445CcY8Ll+hL+8\nyVm2tmlJ6YgxIQFftNg5+NQmA75hGCSjIdeP8Ddbg29LyvIKYkxIwBct9gjdTtFsRjIWIluouHq2\nba+3Nuy0mtKRgC+8TQK+aLFH6IlNjvABktEgtbrJiotn22a3mNIJ+6cI+UOt5wvhVRLwRUt2iykd\naKvUKbr3VofL9gh/k1U69r1tl2WELzxOAr5osUf4W03pgLtLMzPlLFP+EOFA77OIbampJPlKQWbb\nCk+TgC9aWjn82ObKMq3nuD/gZ8u5TadzbKmQzLYV3icBX7RkCxUiUwGCAf+mn2ungdw627beqJOr\n5jddoWOT2bZiHEjAFy3ZQoXkJidd2RLNNFDOpaWZW510ZbOftywBX3iYBHwBQKNhklupbnrSlc3t\nKZ1WSeZWR/jN52Xlwq3wsK5LIiqlfMAnAQU0gPdprZ9s238j8EGgCtyltb5zQG0VA5RfqWKam59l\na3N7Smez97LtJCkdMQ56GeHfCJha6+uwAvsf2juUUgHgI8AbgeuBW5VS6QG0UwzY6gXbrQX8aDiA\n32e4drbt6izbzZVk2lKygJoYA10Dvtb6i8CtzR/PB5badu8HDmqts1rrKvAA8BqnGykGL9NHSSaA\nzzCIR4PuTek4NMKXWnzhZT3d5UJr3VBK3QXcBPxi264kkGn7OQeknGueGJZ+R/gAqWiIU8srTjXJ\nUZk+L9qGA2Gm/CGycjNz4WE939ZIa32zUup24CGl1H6t9QqQxQr6tgSw3O1Y6fTWvlZ7hRf71zBO\nAXDO7mTX9q+3f3ZblBfn8iRSkU3dMWsYVp4sAHDRnj2EgxtPvFqvf9sj0+SqOU+eX5uX296Lce9f\nv3q5aPtuYJ/W+o+AElDHungL8BRwsVJqGihipXM+3O2Y8/PjO0pKpxOe7N/xuWaba40N279R/8JB\nK0P4/OHTpKcjjrexH4uFDCFfkNxylRzrL/+wUf+i/ign8/OcmsvgM7xX4ObV92avJqF//erlXfs5\n4OVKqe8AXwFuA35BKXWL1roGvB+4H3gQuFNrfaLvVomh62eWrc3NpZm5Sp5EKN7XMRKhOCYmhWrR\noVYJMVxdR/jN1M07Nth/H3Cfk40Sw5ctWKPexBYv2oJ773xlmib5aoG9sd19HScejAGQrxb6/vAQ\nYhS8971UDES2UCEU8BEObX5ZBZv97cBtI/xSvUytUSMRivV1HDvI5yqyno7wJgn4ArBG5clYCMMw\ntnwMt6Z08hXrgm082N+o3H5+vlrou01CjIIEfIFpmtY6On2UZEJbSqfgrjXx7RUu+8/hW98QZIQv\nvEoCvqBYrlFvmFuedGWzPzAyLsvh55sBOt5nSqc1wpeALzxKAr5wpEIHIBENYgA5l6V0WiP8PlM6\nrRy+pHSER0nAF47MsgXw+3zEIkHXVem0cvgOXbSVEb7wKgn4orXCZb8pHbBWzXTbRVunRvixYBQD\nQ+56JTxLAr4g17zxeL8jfLDSOoVSjVq90f3BQ+LUCN9n+IgFo63jCeE1EvCFoyN8+0PD/hBxA7uq\npt+yTOsYMRnhC8+SgC8cy+G3H8NNaZ18tcCUP0TI399FabDy+MXqCvVG3YGWCTFcEvCFowHfjXe+\nylXyfefvbfFgzFpPpybr6QjvkYAvyBYr+H0GsXD/SxqvTr5yR8C319GJO7T2jSyvILxMAr4gW6hY\nNfR9LKtgS8TctYDaSq1E3az3vY6OrbWAmly4FR4kAV+01tFxQsplOXz7AqsTF2yhffKVjPCF90jA\nn3ClSo1KteFYwHfbEsn2SNyp5YzjrclXMsIX3iMBf8LZI/GUAyWZ4L4lkldH+M6kdBLN48gIX3iR\nBPwJl23WyyccGuEHA34iU37XBHx7GQTnR/gS8IX3SMCfcDkHJ13ZEtGQayZe5Vpr4Ts1wpcF1IR3\nScCfcLkV+9aG/U9KsiWiQfIrVRqm6dgxtyrv0Fr4tmgwgoEhI3zhSRLwJ5ydeunnXradktEQ9YZJ\nsVRz7JhbtbqsgjMjfJ/hk+UVhGdtONNGKRUAPg2cD4SAD2mtv9S2/zbgFmCuuem9WuuDg2mqGAQ7\n9eL0CN86doV4xLnjboV9O0KnJl5Zx4qRKWcdO54Qw9JtauUvAwta63+nlNoG/AT4Utv+A8C7tdaP\nDKqBYrByK4PJ4YP1YbJ7xrHDbkmukifsDxP09T+L2JYIxjlROEW9Ucfv2/pN34UYtm4pnXuBD7Y9\ntvNK3AHgDqXUd5VSH3C6cWLwcq2UjpMjfDvgj75SJ18tODbL1ta6EYpcuBUes2HA11oXtdYFpVQC\n+Czwex0PuRt4H/A64Dql1A2DaaYYlFyxylTQTyjo3Eh1NaUz2kqdhtmw1tFxaJatLS43Mxce1fV7\nrlLqHODzwMe11n/XsfujWuts83H3AVcCX+52zHQ6sYWmeoeX+pcv1ZhOTG2qzd0ee87uFQDqhjHS\n30W+XKBhNphJTDvav51zM3AU/NGGp841eOu9uRXj3r9+dbtouxP4GvAbWutvdexLAo8ppfYDK8Dr\ngU/18qLz87mttdYD0umEZ/pnmiaZfJlzd/be5l7616ha1Tkn5/Mj/V2cLFi1BKHGlKP981WtP5uj\n8/Ps9u/rr5FD5KX35lZMQv/61W2EfwcwDXxQKfX7gAl8Eohpre9USt0OfBsoAd/UWn+17xaJoVkp\n16k3TJIO5u/BPTl8O8fuVA2+zZ58JTl84TUbBnyt9W3AbRvsvwe4x+lGieGwA7KTNfjW8dyRw2/V\n4Dt80TYua+ILj5KJVxNsEDX4AAG/j8hUwAUj/OYsW4cv2toLqOVl8pXwGAn4E2xQI3yAZDTYWpht\nVAY/wpeUjvAWCfgTLFt0vgbfloiGyBdHu55OK4fv8Ag/EgjjM3wywheeIwF/gtkpHaduftIuEQ3S\nMEe7ns6gRvit9XQkhy88RgL+BBtUDt865ugrdfIOL43cLhGKS5WO8BwJ+BOslcOPDGaEb73G6PL4\nuWqeSCBCwMF1dGzxYIyVWolqY/QrggrRKwn4Eyw3wBx+6962I7zzVa6Sd3wdHVtC7nwlPEgC/gTL\nFqtMhZxdR8fWGuGvjGaE3zAbFKpFx9fRscVbpZmS1hHeIQF/guWKFcdn2drse+SOKodfqBYxMR2f\nZWtLyOQr4UES8CeUaZrkitWB1OADJJo3PskVRjPCb934ZAAXbEGWVxDeJAF/Qq2Ua9QbZiswO61V\npbMymhF+rmItopUc0AjfnnyVrYzvYl1i/EjAn1CtkswB1ODDag5/VBdtV2vwB5vSyctsW+EhEvAn\n1CBr8MFaTyc6FRjZRVt72YNkaDDro9spHcnhCy+RgD+h7GUVnLyXbadELDSyOvxca+G0wZZl5mR5\nBeEhEvAn1CBr8G2JaHBk6+nYI+9BVelM+UMEfUEZ4QtPkYA/obKtlM4AR/iR0a2nM+gcvmEYJEJx\nCfjCUyTgT6jcEFI69qJso7hwm6vk8Rk+ooHIwF4jEYyTq+YxR7giqBCbIQF/QuUHfNG2/dijmHyV\nq+ZJBGP4jMG9xROhGLVGjVK9PLDXEMJJEvAn1CDXwretrpg5/Au3+Up+YOkcm9zqUHiNBPwJlStW\nCYf8BAPOr6NjG9UIv1KvUqqXHb/xSSe75FNuhCK8YsN1Y5VSAeDTwPlACPiQ1vpLbftvBD4IVIG7\ntNZ3Dq6pwkm5YmWgo3sY3Qi/dS/bAY/w7ZJPGeELr+g2wv9lYEFr/RrgXwEft3c0Pww+ArwRuB64\nVSmVHlA7hYPsdXQGecEW2pZIHvIIf9AlmbbV5RUk4Atv6Bbw78UawduPbR+q7QcOaq2zWusq8ADw\nGuebKJzWWkdnwAF/VDdBGVbAlzXxhddsmNLRWhcBlFIJ4LPA77XtTgKZtp9zQMrpBgrnZYdQoQMQ\nj4wmh98K+APO4beWV5AcvvCIrvd+U0qdA3we+LjW+u/admWxgr4tASz38qLp9GDWN3ELt/dvPm8F\n4J2z8S21dTPPiUeCFCv1of5OGgvWB9q+dHqg/QvGdwFQMcquP+c2r7Rzq8a9f/3qdtF2J/A14De0\n1t/q2P0UcLFSahooYqVzPtzLi87Pj++Ssul0wvX9O3Lc+mLmx9x0Wzfbv1gkyHK2NNTfycmlRQAa\nK/6B9q/esCZcLeSWXH/OwRvvzX5MQv/61W2EfwcwDXxQKfX7gAl8Eohpre9USr0fuB8wgDu11if6\nbpEYuGEsnGZLRoPMLRVpmCY+wxj460HbsgoDTun4fX5iwahU6QjP6JbDvw24bYP99wH3Od0oMViD\nXhq5XSIawjShsDK4u2t1GtZFW1hdXkEIL5CJVxNodaXMwQfg1o1Qhlipk6vmCfunCPmH8IEWilOo\nFqk36gN/LSH6JQF/AtmLmQ1jhG+njXJDXEBtGMsq2OzXyVeLQ3k9IfohAX8C2QE/OaDbG7ZLxa3X\nyAwp4DfMBrlqYeAlmbZkK+BLWke4nwT8CZQpVIhHggT8gz/9qdhwA36xtkLDbAwlfw+rtfhyM3Ph\nBRLwJ1AmX2mNvActFZuyXrMwnCWE80O8YAuyYqbwFgn4E6Zaq1Ms11oj70FLNj9YsvnhjPCHWaHT\n/jqyvILwAgn4E8ZOrQwr4Keiw03p5KoFYPDLKthWl1coDOX1hOiHBPwJsxrwp4byelMhP+GQf3gB\nvzXCjw3l9RKS0hEeIgF/wtiplWFU6NhSsdAIAv5wUzoS8IUXSMCfMK0R/pAu2oIV8HPFCo3G4G/2\nnWtWyyRCw1lEK+yfIuALSMAXniABf8IMO4cPkIxPYZqQWxn8bNth5/ANw5DlFYRnSMCfMKMI+K1a\n/PzgSzNzlTwGBtFgZOCvZUuEYuQqeUxz8N9ghOiHBPwJYwfdVHw4F21hNeBnh5DHt5ZViOEzhvfW\njofiVBtVyvXh3uhFiM2SgD9hsoUKfp9BNNz13jeOGeZs21w1P7R0ji0ZtK4XyPIKwu0k4E+YTKFC\nMhYa2tr0MLz1dKr1Kiu1EskhXbC1JeRm5sIjJOBPENM0WwF/mJKtHP5gA36+ecE2PqQafJv9elKp\nI9xOAv4EKVXqVGuNoV6wheGtpzPsGnybnUKS5RWE20nAnyCjqNCBtpugDDilY5dGDjuH35p8JTl8\n4XIS8CfIaoXOcAN+wO8jHgkOPIc/shG+zLYVHiEBf4IMex2ddql4aOA5fAn4Qmysp9o8pdQ1wH/X\nWr+uY/ttwC3AXHPTe7XWB51tonDKqFI69msemy9QrdUJBvwDeY1MJQsw/CqdYBwDo/X6QrhV14Cv\nlPpd4N3AWsOXA8C7tdaPON0w4bxh3tqwU3st/mxqMLNgl8tWwE1NJQdy/PX4fX7ioVjr9YVwq15S\nOs8CN62z7wBwh1Lqu0qpDzjXLDEIdkpl2Dl8aK/UGVxaJ1PO4jN8Qx/hA0xPpciUM7K8gnC1rgFf\na/0FoLbO7ruB9wGvA65TSt3gYNuEw0aZ0rG/VQzyzleZcoZkKDHUZRVs01NJqo0aK7WVob+2EL3q\nd379R7XWWQCl1H3AlcCXuz0pnR7+CGyY3Nq/QrlGOOTnnL3b+jrOVvp3zm4rzVL3+Qby+zFNk0wl\nx/nT+/o+/laevys1y2MLYERrpKfdef7Bve9Np4x7//q1mYB/xlx8pVQSeEwptR9YAV4PfKqXA83P\n5zbxst6STidc27/FzArJaKiv9m21f0ajAcCxk9mB/H5ylTy1Ro2YPz6S/k01ogAcOnmCSHW41xB6\n5eb3phMmoX/92kzANwGUUu8EYlrrO5VStwPfBkrAN7XWX+27RWIgGg2TXKFKeu/wlg1uN+gF1FoX\nbEOjCbb2hWK5cCvcrKeAr7U+DFzb/PfdbdvvAe4ZTNOEk/IrVRqmOZL8Pawuxzyo2baZcgawcumj\nYL+u3Q4h3EgmXk2IUV6wBYiGA/h9xsDW08k0R9bTU6mBHL8b+3WXpRZfuJgE/AlhB9pRBXyfYZCM\nDW627XJzZD3sGnybjPCFF0jAnxCrNfjDX1bBloqFyBYqA6lVXx7xCD8SiBD0BSWHL1xNAv6EGOUs\nW1sqFqJSa1Cq1B0/tr2swahG+IZhkJpKtr5pCOFGEvAnxKhz+DDYO18tlzNM+UNEAmHHj92r6akk\n+UqBesNuJTDXAAAUsUlEQVT5DzQhnCABf0K4IeAn7eUV8s5fuM2UsyNL59imp1KYmGQr41sLLrxN\nAv6EsIPsqFM64PwIv9qoka8WSI044K/W4ktaR7iTBPwJkSlUiIUDBPyjO+WDCvjZEU+6sk2HZPKV\ncDcJ+BMiW6iMtEIH2hZQczjgr1bojDbg298wZIQv3EoC/gQoV+sUSjWmR7AscrvphPWBczrrbA5/\nuTXLdvQ5fFidBCaE20jAnwCnsyUAZpKjq2AB2J6Ywmhrj1NGvayCbVrW0xEuJwF/Aixm3BHwA34f\nqXiIRYcD/nKrBn+0I/ykzLYVLicBfwIs2CP81GgDvt2GpVyZRsO52bYZl+Twg74A8WCM5YoEfOFO\nEvAngFtG+HYb6g2TZQdr8ZfLGQyMkdzasJM12zYrtzoUriQBfwIsumyED7CQcS6ts1zOkgjF8fv8\njh1zq6anUlTqFUr1wawKKkQ/JOBPgMVMCcOAbYnRlmUCzDa/ZTiVxzdNsznL1h13mZJVM4WbScCf\nAIvZEtPxqZFOurLZI/xFh0b4K7UVqo3qyC/Y2lZr8aVSR7jP6COAGKh6o8FSruyKdA6sXkdwaoTf\nurWhW0b4IVleQbiXBPwxt5QrY5qrqZRRc3qE35p0FXLLCF9q8YV7ScAfc60KHZeM8MOhALFwwPER\nvnty+DLbVrhXTwFfKXWNUupba2y/USn1kFLqQaXULc43T/Rr0SWzbNvNpMIsZkuOlC6O+l62nVYD\nvqR0hPt0DfhKqd8FPglMdWwPAB8B3ghcD9yqlEoPoI2iD24b4YP14VOpNsivVPs+lj3JyS05/Fgw\nSsDwS0pHuFIvI/xngZvW2L4fOKi1zmqtq8ADwGucbJzon1tH+ODMhVu3rKNjk1sdCjfrGvC11l8A\namvsSgLt7+oc4I7v1aLFTbNsba1KHQcu3C6XswR9QSKBSN/HckpqKkW2kpNbHQrXCfTx3CxW0Lcl\ngOVenphOj34K/CC5qX/LhQqJaIh9e6cdO2a//bvgnG0AlOr9H2u5kmE2uo0dO5wb4ffbpl2pWZ7P\nvEAg3mA25tzv3Qluem8Owrj3r1+bCfhGx89PARcrpaaBIlY658O9HGh+fnzv+ZlOJ1zTP9M0mVta\nYc9MzLE2OdG/INbF2sPHl/s6VrG6Qq6c59z4Plf1L+mzvug+ffQwanvQiWY5wk3vzUGYhP71azNl\nmSaAUuqdSqlbtNY14P3A/cCDwJ1a6xN9t0g4JlesUq01XHXBFpyrxZ9fWQAgHZnpu01Ostsz12yf\nEG7R0whfa30YuLb577vbtt8H3DeYpol+ufGCLUAiEiQU8PV90Xa+2Az40VknmuWYHc322O0Twi1k\n4tUYc2NJJliVLDOpcN8jfHsEvSPiroCfbrZHRvjCbSTgj7EFF1bo2GaSYQqlGqXKWgVgvZkrLgKr\nI2q3iAdjRAJh5lcWR90UIc4gAX+M2SmTWZeN8MGZPP7CygJ+w8+2KXdVwhiGQToyy8LKIg2zMerm\nCNEiAX+MuTWlA86smjm3ssBMZJsrbnzSKR2ZodaosVSSCVjCPSTgj7HFbImpoJ9YuJ/pFoOxOtt2\na3eGKlaLFKpF1+Xvba0Lt5LHFy4iAX+MLWZKzKTCGEbnFIrR63e2rX1B1G0VOrbWhVup1BEuIgF/\nTK2UaxTLNbYnR39bw7X0m9KxA6mM8IXonQT8MdW6YOvCCh2A6UQIn2FseYTv1hp8W1oCvnAhCfhj\nys0XbAH8Ph/bElNbHuHbJY9pl47wY4EokUCkVToqhBtIwB9Tp04XAUhPu2cVyU47tkVYypW3VIs/\nt7JAwPCzPeyukkybYRjsiMyyKKWZwkUk4I+po/MFAPam4yNuyfr2zsYAOLZQ2PRz54sLzERm8Bnu\nfQunozPUzDpLpZ4WkRVi4Nz71yL6cnQ+T8BvsHObe0f4+3ZYH0bH5jcX8PPVAsXaCjui7lo0rdMO\nWWJBuIwE/DHUaJgcXyiwZyZGwO/eU7w3bY3wj87lN/W81gVbl+bvbWlZRE24jHujgdiy+eUVKrWG\nq9M5YKV0DKxvI5sx55WALyN84TIS8MfQkeaIed+O2IhbsrFwKEB6OsLR+QKmafb8PLtCx22LpnWS\nZZKF20jAH0P2iHmfy0f4YKV18itVMoVKz89ZvfGJuwN+LBglFojKqpnCNSTgjyH7IqgXAr7dxs2k\ndeaKCwR8AbaFU4NqlmPS0VkWVk7LDc2FK0jAH0NH5/PEwgGm46FRN6Uru1Ln6FxvlTqmaTK/ssBs\neLurSzJt6cgMdbPOUllKM8Xouf8vRmxKuVpnbmmFfem4KxdN67SvWalzrMcRfr5aYKVWcu2SCp3s\ndsoiasINJOCPmeMLBUy8kc4Ba7ZtwO/jSI8B3+2LpnXaIatmChfpulC6UsoA/gx4GVACbtFaP9+2\n/zbgFmCuuem9WuuDA2ir6IFd077X5RU6Nr/Px97ZGMcWCtQbDfy+jccgh3NHANiX2DOM5vXNbqfd\nbiFGqZc7Y7wNmNJaX6uUugb4SHOb7QDwbq31I4NooNicox66YGvbl45x+FSOuaUVds9s/EF1KHMY\ngAuS5w2jaX3bGU0TCURa7RZilHpJ6VwHfBVAa/0D4KqO/QeAO5RS31VKfcDh9olNsqtd7HVqvGBv\nq1Kn+4XbQ5kXiQdjzEa2D7pZjvAZPs5PnsP8yiK5yuYmmAnhtF4CfhJovzFnTSnV/ry7gfcBrwOu\nU0rd4GD7xCYdm88zmwoTmXLfbQ3XY08Q67bEwnI5w1J5mQtS53nigrTtgpT1beSF7IsjbomYdL1E\nhSyQaPvZp7VuX+/1o1rrLIBS6j7gSuDLGx0wnU5stNvzRtW/pVyJbLHKNZfPDLQNTh/7ZVNB4FHm\nMqUNj/3cEevS0BV7LvFU/66sX8qXD32dk9UTvD59jaPH3iz525tsvQT8B4F/DXxOKfUq4DF7h1Iq\nCTymlNoPrACvBz7V7YDz87mttdYD0unEyPr3xAunrTakpgbWhkH0zzRN4pEgzx9b3vDYPznyNAA7\n/Ds91b9t5iwGBk+cOMj87tG990f53hyGSehfv3oJ+F8A3qSUerD5881KqXcCMa31nUqp24FvY1Xw\nfFNr/dW+WyW25Nicd5ZUaGcYBvvSMZ5+cZlSpUY4tPbb8lDmRXyGj3OT5wy5hf2JBCLsiu3gcPYI\n9UYdv88/6iaJCdU14GutTeDXOzY/07b/HuAeh9sltsCLFTq2fek4T7+4zLGFAhftOXvJhGqjxpHc\nUfbGdzPld/8M4k4XJM/jROEUxwsnOSexd9TNERNKJl6NkeeOZwgFfOzc7t6bnqzn3J3W19XnjmXX\n3H80d4yaWfdMOWYn+8Lt81KeKUZIAv6YmF9e4cRikf3nbes6ecmNLr/AKrN87Pm1V5Zs1d+nzh1a\nm5x0YbPdUo8vRsl7kUGsyQ6UL73I3bf9W8+2xBTn7IijX1yiXDl7ZcnnmyWNF6bOH3LLnLEjmiYq\nE7DEiEnAHxM/fc4K+Fdc6M2AD9aHVa1u8tThpbP2HcocJhGKMxPeNoKW9c9n+Dg/dS4LpdMyAUuM\njAT8MVCp1nn68BJ7ZmPMTnsvf2+zP6x+2pHWWSots1zOcGHSWxOuOl2YlDy+GC0J+GNAH1mmUmt4\nNp1ju2hvklg4wGPPLZxxy8NDzXSOfeHTq+z2S1pHjIoE/DFgp3Ne6uF0DlgrZ15+wXYWs2WOL6yu\nq/P88guA9wP+eclzMDB4PvPCqJsiJpQEfI8zTZOfPrdAOOTn4n3uv+VfN/a3FPtDrN6o8/DcT4kE\nwpyX2DfKpvUtEghzQepcns8c5nTp7OsUQgyaBHyPO3m6yPxyicsv2E7A7/3T+ZILZjBYDfhPnX6G\nTCXLVTuvJOgPjrZxDnjV7qswMfn+iR+NuiliAnk/Qky4x8YknWNLxkKcvzvJwaMZiqUa/3zihwBc\nu/uVI26ZMw7seBkhf4jvn/gRDbPR/QlCOEgCvsfZFS0vGZOAD1Zap2Ga/Pi5Izy28CR747vHZjmC\ncCDMK3a8lMXSEs8sPTfq5ogJIwHfw4qlGs8cWebcnXG2JaZG3RzH2Hn8//PiD2mYDa7dfbWnyzE7\nXbv7agC+1/z2IsSwSMD3sPt/+CK1usnV+3eOuimOOm9XgvR0mGONp/Abfl6568pRN8lRF6bOY2c0\nzU/mH6dYLY66OWKCSMD3qGyhwtceOkIyFuINr/B29Uonn2Fw3TVhjEiBRPUcYsHoqJvkKMMw+Nnd\nr6TWqPHQKbkVtBgeCfge9Y/fe4Fytc6N157PVGj81lfPhK389tyzs6379I6Tq3cdwGf4+P5xSeuI\n4ZGA70ELmRW+/cgxZlNhXvvyPaNujuMWVk7z47lHifuT1LMzfP47z4+6SY5LTSV4ycx+juSP88Si\nHnVzxISQgO9BX3zgELW6yU0/d+FY1N63qzVqfPrxv6FSr3DTJW/mZ/ZN85NnF3j2aGbUTXPcv7rg\nDfgNP3/15D1kymvfB0AIJ41XtJgAR+fz/PPjJ9mXjnHNZeN1sRbgH577ModzR7hm1wFetfsq3v7a\niwD43HeeO2N9nXFwbmIfN138FvLVAp954m6pyxcDJwHfQ04sFvjTzz6KacIvvPYifL7xKVUEeHT+\nCb515AF2RnfwS5e8DYBLzpnm5RfP8syRZe768tPUG+MVFK/f92peNns5zyw/x1cOfWPUzRFjTgK+\nRxw+meOP/vfDnM6WeftrL+TlF8+OukmOOpo7zl8/dS9BX4Bffcm7CAdW5xXcfMOlXLA7wQOPneAT\n//AE1drZN0jxKsMw+OX9/4bt4W185YVv8sjcY6NukhhjRrevyUopA/gz4GVACbhFa/182/4bgQ8C\nVeAurfWdXV7TnJ/P9dVoN0unEzjdv8cPLfJnX3iccqXOu9+suP7lo5t16nT/KvUKXz70Db555P/Q\nMBv820vfzqv3XHPW41bKNT7++cd46vAS+8/bxvt+/nISUedvZj6I89eLQ5kX+dOHP0HNrHNgx8t4\n+8+8ldRUwtHXGFXfhmUC+tf3V/peAv5NwI1a63+vlLoGuENr/bbmvgDwFHAAWAEeBN6itZ7f4JAS\n8HtQrtZ56MlT/NMjxzh8MoffZ/BrN1428klWTvUvU87y5Oln+Mqhb7BYOs1MeBvvUDdx+cyl6z6n\nWqvzv774BI8cXCDg93H1/h287hV7uXB30rGZuKMMGsfyJ7j76b/nUPZFIoEIbz7/9Vwxexk7IrOO\n9G8CAuK4928oAf9PgB9ore9t/nxUa72v+e8rgD/WWt/Q/PkjwINa679f73inlpfN+cXxq6u2f4/b\nt8dYPF3o2AmYpvU/oF5vUG+Y1Bom5WqNUrlOsVwjW6hwcrHIidNFji8UKFfrGFh3gnrDVfu4YHdy\nyL3qYMLMbJzFhXzzR7P1f9MEkwamadIwG9TMOtV6lWqjSqlWJlPJkilnOV1a4tnMC5wsnAKsW/+9\n/pyf44YL3sSUv/uIvd5o8E8/PsY/PXyUU0srAMymwuydjbFnNsau7VFikSDRqQCRqQChoA+/z8Dv\n8+HzGRiGlUYxANr+fOx/zszEWVzMj2wph4bZ4AenHuIrL95PuV4GIBVKcVHqQmamtpMMJUiGkoQD\nYYK+IEFfkIDhxzB8+DDwGT7A6qfdK7sn27bHWDo9vjN7t2+Pcbrzb2+MvOSifX2/KQM9PCYJtNfE\n1ZRSPq11Y419OWDDRdl/62t3bLqRE2UK2A2+3WDfrPAgcPBZ4NnRNctJIV+Qy7Yr1PaLuWL2MnZG\n0z0/1+/z8aZXnsMbrtrHU4eX+PYjxzh4NMOjzy3y6HOL3Q/gFYFr8W+bw5dcZDm5yMMVmZE76e69\n6BN9H6OXgJ8F2pOJdrC397UPOxPA8kYHu/cdnxiv0hIxMjt3JLn+ld6+C5YQw9RLlc6DgJ2yeRXQ\nXkbwFHCxUmpaKRUCXgN8z/FWCiGE6NtmqnRe2tx0M9ZF2pjW+k6l1FuA/4yVKvyU1vp/DbC9Qggh\ntqhrwBdCCDEeZOKVEEJMCAn4QggxISTgCyHEhOilLNMRSqlLge8DO7TWlY59vwbcirU8w4e01vcN\nq139UkpFgb8FtgFl4D1a6xMdj/lT4NVY8xQAfl5r7fopgT32zcvnLgn8b6zS4iDwO1rr73c8xpPn\nDnrun2fPn625GsAvaq3ftcY+z54/W5f+ber8DSXgK6USwP/AWounc99O4LeAVwBR4AGl1P1a6+ow\n2uaAXwN+pLX+A6XUe4Dbgds6HnMA+Jda69NDb11/NuzbGJy79wPf0Fp/TCl1CXA31rlq59VzB136\nNwbnzw7o/wL4yToP8fL527B/Wzl/w0rp/AVwB7DWvO6rgQe01jWtdRZrYulL13icK2mtPwp8qPnj\nucBS+/5mWevPAH+hlHpAKXXzkJu4Zd36hsfPHfAR4M+b/w5irQfV4uVz17Rh//D++QNrntCvr7Vj\nDM4fbNA/tnD+HB3hK6X+PfDbQHut54vA3Vrrx5onoFPn8gx5uizPMCod/TOa/79Za/1jpdQ3gCuA\nN3U8LQZ8DOuPLwB8Syn1Q63148NreXdb7Nu4nLtdwF8D/6HjaZ44d7Dl/o3D+fusUuq16zxtHM7f\nRv3b9PlzNOBrrT8NfLp9m1LqGeBXlVK3ALuA+4Hr2x6y6eUZRmWt/rXte6NSSgH3ARe37SoCH9Na\nlwCUUv+EtdS0q950W+yb589dcwHAv8XKbz/QsdsT5w623D/Pn78uPH/+utj0+Rt4Dl9rfYn9b6XU\nIc4eJT4E/EFzaYYIcCkuPCHrUUrdARzVWv81UABqHQ+5BLhHKXUl1u/7OuAzQ23kFvXQN6+fu8uA\ne4Ff0lqvdecRz5476Kl/nj5/PfD0+evBps/f0Kp0muyvKyilfhs4qLX+R6XUx4AHmvv+784qHpf7\nFPCXza9kPuBX4Kz+/SVWhVIF+IzW+qlRNXaTeumbl8/dH2KtT/rRZrpxWWt905icO+itf14+f2sa\no/O3pn7OnyytIIQQE0ImXgkhxISQgC+EEBNCAr4QQkwICfhCCDEhJOALIcSEkIAvhBATQgK+EEJM\nCAn4QggxIf5/LZUbwUWklC8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1131b0f10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "t3 = scipy.stats.norm(-3, 0.1)\n",
    "t4 = scipy.stats.norm(-2.5, 0.1)\n",
    "\n",
    "x = np.linspace(-4,-1,100)\n",
    "\n",
    "plt.plot(x,t3.pdf(x))\n",
    "plt.plot(x,t4.pdf(x))\n",
    "plt.title(\"KL-divergence:\" + str(scipy.stats.entropy(t3.pdf(x), t4.pdf(x))));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ELBO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* We want to minimize the KL divergence, but since it explicitly includes the posterior, we need a proxy measure that approximates the divergence. For this, we'll use the Evidence Lower BOund (ELBO):\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "    \\log p(x) &= \\log\\int_{\\theta} p(x,\\theta) \\\\\n",
    "              &= \\log\\int_{\\theta} p(x,\\theta) \\frac{q(\\theta)}{q(\\theta)} \\\\\n",
    "              &= \\log\\left(\\mathbb{E}_q\\left[ \\frac{p(x,\\theta)}{q(\\theta)} \\right]\\right) \\\\\n",
    "              & \\left. \\geq \\mathbb{E}_q[\\log p(x,\\theta)] - \\mathbb{E}_q[\\log q(\\theta)] = \\mathrm{ELBO} = \\mathcal{L} \\right.\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "which can be related to the KL divergence:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "    D_{\\mathrm KL}(q \\mid\\mid p) &= \\mathbb{E}_q \\left[ \n",
    "        \\log \\frac{q(\\theta)}{p(\\theta \\mid x)} \n",
    "    \\right] \\\\\n",
    "        &= \\mathbb{E}_q \\left[ \\log \\frac{q(\\theta)}{p(\\theta,x)/p(x)} \\right] \\\\\n",
    "        &= \\mathbb{E}_q \\left[\\log q(\\theta)\\right] \n",
    "          -\\mathbb{E}_q \\left[\\log p(\\theta,x)\\right]\n",
    "          + \\log p(x) \\\\\n",
    "        &= -(\\mathrm{ELBO}) + \\log p(x) \\\\\n",
    "        &= -\\mathcal{L} + \\log p(x)\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Since $D_{\\mathrm KL}(q \\mid\\mid p) \\geq 0$ and $\\log p(x) = \\mathrm{constant}$, maximizing the ELBO minimizes the KL divergence. As such, finding a good approximate posterior comes down to finding a good $q$, as well as a optimization problem to maximize the corresponding ELBO."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The importance of support-matching"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* The gradient $\\nabla_{\\theta} \\log p(x,\\theta)$ is valid within the support of the prior:\n",
    "\n",
    "$$\n",
    "    \\mathrm{supp}(p(\\theta)) = \\left\\{\n",
    "        \\theta \\mid \\theta \\in \\mathbb{R}^{K} \\,\\mathrm{and}\\, p(\\theta)>0 \n",
    "    \\right\\}\n",
    "    \\subseteq \\mathbb{R}^{K},\n",
    "$$\n",
    "\n",
    "where $K$ is the dimension of the latent variable space.\n",
    "\n",
    "* This is due to the fact that if \n",
    "\n",
    "$$\n",
    "    \\mathrm{supp}(q) \\subsetneq \\mathrm{supp}(p) \n",
    "        \\implies \n",
    "    D_{\\mathrm KL}(q \\mid\\mid p) = \\mathbb{E}_{q}[\\log q] - \\mathbb{E}_{p}[\\log p] = -\\infty\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ADVI\n",
    "\n",
    "### Transformation to unbounded real coordinate space\n",
    "\n",
    "* Need to ensure that the support of the prior is unbounded, so we don't have to worry about mis-matched support\n",
    "* introduce a coordinate transformation, $T : \\mathrm{supp}(p(\\theta)) \\rightarrow \\mathbb{R}^{K}$, that maps the latent coordinates to the unbounded real space\n",
    "* variables in this mapped space are $\\zeta = T(\\theta)$\n",
    "\n",
    "$$\n",
    "    p(x, \\zeta) = p\\left( x, T^{-1}(\\theta) \\right)\n",
    "    \\left| \n",
    "        \\det J_{T^{-1}} (\\zeta)\n",
    "    \\right|\n",
    "$$\n",
    "\n",
    "* the Jacobian ensures that the transformed density integrates to one\n",
    "\n",
    "### Mean-field Gaussian\n",
    "\n",
    "* one approach is to try a factorized Gaussian solution in the real-valued coordinate space\n",
    "\n",
    "$$\n",
    "    q(\\zeta, \\phi) = \\mathcal{N}(\\zeta; \\mu, \\mathrm{diag}(\\sigma^2))\n",
    "    = \\prod_{k=1}^{K} \\mathcal{N}(\\zeta_k; \\mu_k, \\sigma_k^2)\n",
    "$$\n",
    "\n",
    "where $\\phi = (\\mu_1, \\dots, \\mu_K, \\sigma_1^2, \\dots, \\sigma_K^2)$\n",
    "\n",
    "* however, these parameters live in the set of $\\Phi = \\left\\{ \\mathbb{R}^{K}, \\mathbb{R}^{K}_{\\gt 0} \\right\\}$\n",
    "* reparameterize as $\\omega = \\log(\\sigma)$, then\n",
    "\n",
    "$$\n",
    "    q(\\zeta, \\phi) = \\mathcal{N}\\left(\\zeta; \\mu, \\mathrm{diag}(\\exp(\\omega)^2)\\right)\n",
    "$$\n",
    "\n",
    "* where the parameters $\\phi = (\\mu_1, \\dots, \\mu_K, \\omega_1, \\dots, \\omega_K)$ are unconstrained in $\\mathbb{R}^{2K}$\n",
    "\n",
    "### Full-rank Gaussian\n",
    "\n",
    "* another approach is to use a full-rank Gaussian,\n",
    "\n",
    "$$\n",
    "    q(\\zeta; \\phi) = \\mathcal{N}(\\zeta; \\mu, \\Sigma)\n",
    "$$\n",
    "\n",
    "where $\\phi = (\\mu, \\Sigma)$ are the mean values, $\\mu$, and the covariance matrix $\\Sigma$\n",
    "\n",
    "* to ensure that $\\Sigma$ is positive semidefinite, it can be reparameterize as $\\Sigma = LL^{T}$, using the non-unique definition of the Cholesky factorization, such that $L$ is in the unconstrained space of lower-triangle matrices with $K(K + 1)/2$ real-valued entries. So then\n",
    "\n",
    "$$\n",
    "    q(\\zeta; \\phi) = \\mathcal{N}(\\zeta; \\mu, L L^{T})\n",
    "$$\n",
    "\n",
    "where $\\phi = (\\mu, L)$ are unconstrained in $\\mathbb{R}^{K + K(K+1)/2}$\n",
    "\n",
    "* the full-rank Gaussian is a generalization of the mean-field approximation, and the off-diagonal terms in the covariance matrix capture correlations in the posterior distribution\n",
    "* however, it is computationally more costly to compute"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variation inference in the real-coodinate space\n",
    "\n",
    "* after employing the transformation function, the ELBO becomes\n",
    "\n",
    "$$\n",
    "    \\mathcal{L}(\\phi) = \n",
    "    \\mathbb{E}_{q(\\zeta;\\phi)} \\left[\n",
    "        \\log p \\left(\n",
    "            x, T^{-1}(\\zeta)\n",
    "        \\right)\n",
    "        +\n",
    "        \\log \\left| \n",
    "            \\det J_{T^{-1}} (\\zeta)\n",
    "        \\right|\n",
    "    \\right]\n",
    "    +\n",
    "    \\mathbb{H} \\left[\n",
    "        q(\\zeta; \\phi)\n",
    "    \\right]\n",
    "$$\n",
    "\n",
    "where $\\mathbb{H}$ is the entropy\n",
    "\n",
    "* the ELBO can then be optimized with respect to the variational parameters $\\phi$ without worrying about the constraining the support:\n",
    "\n",
    "$$\n",
    "    \\phi^{*} = {\\arg\\max}_{\\phi} \\mathcal{L}(\\phi)\n",
    "$$\n",
    "\n",
    "* which can be tackled via gradient ascent:\n",
    "\n",
    "$$\n",
    "    \\begin{align}\n",
    "        \\mu^{(i+1)} &\\leftarrow \\mu^{(i)} + \\alpha \\nabla_{\\mu} \\mathcal{L}, \\\\\n",
    "        \\omega^{(i+1)} &\\leftarrow \\omega^{(i)} + \\alpha \\nabla_{\\omega} \\mathcal{L},\n",
    "            & \\mathrm{(mean \\, field)} \\\\\n",
    "        L^{(i+1)} &\\leftarrow L^{(i)} + \\alpha \\nabla_{L} \\mathcal{L},\n",
    "            & \\mathrm{(full \\, rank)}\n",
    "    \\end{align}\n",
    "$$\n",
    "\n",
    "where $\\nabla \\mathcal{L}$ is the gradient of the ELBO."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ELBO gradients\n",
    "\n",
    "* we would like to compute the gradients of the ELBO using automatic differentiation methods, but the ELBO has the unknown expectation $\\mathbb{E}_{q(\\zeta;\\phi)}$\n",
    "* to proceed, we'll apply another transformation that will ultimately allow us to take the gradients inside the expectation\n",
    "* specifically, the transformation $S_{\\phi}$ absorbs the variational parameters $\\phi$, and converts the variational approximation Gaussian into a standard Gaussian\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "    \\eta = S_{\\phi}(\\zeta) &= \\mathrm{diag}(\\exp(\\omega)^2)^{-1} (\\zeta - \\mu)\n",
    "        & \\mathrm{(mean \\, field)} \\\\\n",
    "    \\eta = S_{\\phi}(\\zeta) &= L^{-1} (\\zeta - \\mu) \n",
    "        & \\mathrm{(full \\, rank)}\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "The variational density then becomes:\n",
    "$$\n",
    "    q(\\eta) = \\mathcal{N}(\\eta; 0, I) = \\prod_{k=1}^{K} \\mathcal{N}(\\eta_k; 0, I),\n",
    "$$\n",
    "\n",
    "and the optimization problem becomes:\n",
    "\n",
    "$$\n",
    "    \\phi^{*} = \\arg\\max_{\\phi}\n",
    "    \\mathbb{E}_{\\mathcal{N}(\\eta; 0, I)} \\left[\n",
    "        \\log p \\left(\n",
    "            x, T^{-1}\\left(S_{\\phi}^{-1}(\\eta)\\right)\n",
    "        \\right)\n",
    "        +\n",
    "        \\log \\left| \n",
    "            \\det J_{T^{-1}} \\left(S_{\\phi}^{-1}(\\eta)\\right)\n",
    "        \\right|\n",
    "    \\right]\n",
    "    +\n",
    "    \\mathbb{H} \\left[\n",
    "        q(\\zeta; \\phi)\n",
    "    \\right]\n",
    "$$\n",
    "\n",
    "* since the expectation no longer depends on $\\phi$, we can push the gradient inside of the expectation:\n",
    "\n",
    "$$\n",
    "    \\nabla_{\\mu} \\mathcal{L} = \n",
    "    \\mathbb{E}_{\\mathcal{N}(\\eta)} \\left[\n",
    "        \\nabla_{\\theta} \\log p(x, \\theta)\n",
    "        \\nabla_{\\zeta} T^{-1}(\\zeta)\n",
    "        +\n",
    "        \\nabla_{\\zeta} \\log \\left|\n",
    "            \\det J_{T^{-1}}(\\zeta)\n",
    "        \\right|\n",
    "    \\right]\n",
    "$$\n",
    "\n",
    "and\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "    \\nabla_{\\omega} \\mathcal{L} &= \n",
    "    \\mathbb{E}_{\\mathcal{N}(\\eta)} \\left[\n",
    "        \\left(        \n",
    "            \\nabla_{\\theta} \\log p(x, \\theta)\n",
    "            \\nabla_{\\zeta} T^{-1}(\\zeta)\n",
    "            +\n",
    "            \\nabla_{\\zeta} \\log \\left|\n",
    "                \\det J_{T^{-1}}(\\zeta)\n",
    "            \\right|\n",
    "        \\right)\n",
    "        \\eta^{T} \\mathrm{diag}(\\exp(\\omega))\n",
    "    \\right]\n",
    "    + 1         & \\mathrm{(mean \\, field)} \\\\\n",
    "    \\nabla_{L} \\mathcal{L} &= \n",
    "    \\mathbb{E}_{\\mathcal{N}(\\eta)} \\left[\n",
    "        \\left(        \n",
    "            \\nabla_{\\theta} \\log p(x, \\theta)\n",
    "            \\nabla_{\\zeta} T^{-1}(\\zeta)\n",
    "            +\n",
    "            \\nabla_{\\zeta} \\log \\left|\n",
    "                \\det J_{T^{-1}}(\\zeta)\n",
    "            \\right|\n",
    "        \\right)\n",
    "        \\eta^{T}\n",
    "    \\right]\n",
    "    +\n",
    "    (L^{-1})^{T} & \\mathrm{(full \\, rank)} \\\\\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "* the gradients inside the expectation can now be computed using automatic differentiation\n",
    "* the expectation itself can be estimated using MC integration\n",
    "* that is, we can sample from the standard Gaussian and evaluate the emperical mean of the gradients in within the expectation\n",
    "* this procedure gives noisy unbiased gradients of the ELBO of any differentiable probability model, and we can further use those gradients for a stochastic optimization for the variational inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stochastic gradient ascent\n",
    "\n",
    "* armed with the noisy gradients, the optimization of the ELBO can proceed via the gradient ascent approach discussed above\n",
    "* the stepsize, $\\alpha$ can be made to be made to vary, and a procedure outlined in the paper discusses the calculation of the stepsize at each iteration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Complexity\n",
    "\n",
    "* ADVI has complexity $\\vartheta(NMK)$ per iteration, where $N$ is the number of datapoints, $M$ is the number of MC samples, and $K$ is the number of latent variables\n",
    "* scaling to large datasets can be done by subsampling. For a minibatch of size $B \\ll N$, the complexity per iteration is $\\vartheta(BMK)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate a small dataset dataset with fixed probability `p=0.2` of an entry being a 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 1 0 0 0]\n"
     ]
    }
   ],
   "source": [
    "data = np.random.binomial(1, 0.2, size=20)\n",
    "print data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll make a simple, but not trivial, model to try out the ADVI methods in `pymc3` and `Stan`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## pymc3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I think `pymc3` currently only supports the Mean-Field version of ADVI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Couldn't import dot_parser, loading of dot files will not be possible.\n"
     ]
    }
   ],
   "source": [
    "import pymc3 as pm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Applied log-transform to alpha and added transformed alpha_log to model.\n",
      "Applied log-transform to beta and added transformed beta_log to model.\n",
      "Applied logodds-transform to p and added transformed p_logodds to model.\n"
     ]
    }
   ],
   "source": [
    "with pm.Model() as model:\n",
    "    alpha = pm.HalfCauchy('alpha', 20)\n",
    "    beta  = pm.HalfCauchy('beta', 50)\n",
    "    \n",
    "    prior = pm.Beta('p', alpha=alpha, beta=beta)\n",
    "    likelihood = pm.Binomial('binomial', n=len(data), p=prior, observed=sum(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For `pymc3`, `accurate_elbo=False` means that the number of MC samples per iteration is set to $M=1$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 0 [0%]: ELBO = -3.9\n",
      "Iteration 400 [10%]: Average ELBO = -28.8\n",
      "Iteration 800 [20%]: Average ELBO = -24.09\n",
      "Iteration 1200 [30%]: Average ELBO = -20.25\n",
      "Iteration 1600 [40%]: Average ELBO = -18.58\n",
      "Iteration 2000 [50%]: Average ELBO = -17.92\n",
      "Iteration 2400 [60%]: Average ELBO = -15.24\n",
      "Iteration 2800 [70%]: Average ELBO = -12.94\n",
      "Iteration 3200 [80%]: Average ELBO = -11.75\n",
      "Iteration 3600 [90%]: Average ELBO = -10.62\n",
      "Finished [100%]: Average ELBO = -10.56\n",
      "CPU times: user 3.51 s, sys: 369 ms, total: 3.88 s\n",
      "Wall time: 10.8 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "advi_fit = pm.variational.advi(model=model, n=4000, accurate_elbo=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAERCAYAAACzejr1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcXGWd7/FP9b5Vd6fT1ensnYTkl5UkhKwkgQABYtgC\nAxMWkU0WI0oGUNC5yMRxwREHvSquuY7jqFev43LHOxIUx0FkQEFHGMcHEANJWNJZuzvpdCedvn+c\nqqS6uqqrqvvU0tXf9+uVV6qfOnXOr05Vnd85z3mWQG9vLyIiIn4oynUAIiJSOJRURETEN0oqIiLi\nGyUVERHxjZKKiIj4RklFRER8U5KrDZvZUuDjzrk1ZjYN+BpwHHjBObcpvMw7gVuAo8BHnHM/zlW8\nIiKSXE6uVMzsHuDLQHm46FPAB5xzZwJFZnaJmY0B7gCWAxcAHzOz0lzEKyIiqclV9dfLwIaovxc5\n554IP/5XYC2wBPilc+6Yc64NeAk4NbthiohIOnKSVJxz3weORRUFoh63A7VAEDgYVd4B1GU+OhER\nGax8uVF/POpxEDgAtOEll9hyERHJUzm7UR/jOTNb7Zz7d2Ad8Djwa+AjZlYGVAIzgReSrai3t7c3\nEAgkW0xERPry5cCZL0nlbuDL4Rvx/w38H+dcr5l9Bvgl3pv9gHOuO9mKAoEAra3tmY3WB6FQUHH6\naDjEORxiBMXpt+EUpx9yllScc68CK8KPXwLOirPMV4GvZjcyEREZrHy5pyIiIgVASUVERHyjpCIi\nIr5RUhEREd8oqYiIiG+UVERExDdKKiIi4hslFRER8Y2SioiI+EZJRUREfKOkIiIivlFSERER3yip\niIiIb5RURETEN0oqIiLiGyUVERHxjZKKiIj4Jl+mE07IzALA54H5wBHgZufcK7mNSkRE4hkOVyqX\nAuXOuRXAfcCnchyPiIgkMBySykrgJwDOuaeB03MbjoiIJJL31V9ALXAw6u9jZlbknDseb+E39x7i\nid/t4us/cSfKHrhhMY8/t4vqyhJ+9cKbNNZWMHZ0NWMaKnl550GO9hyn7VA3O1sPce15M5gzpYFf\n/O51TpsR4s9vtFFXXcZv/ribV99qZ+Wp45g+vo6e3l5Ki4s40NHFke4ejvf2UhQIMLqugu1vtFFd\nWcp/vPAmc6aOprgoQGfXMdoOdbNjdwfnLJpAw6iDPPrUdkbXVrBkVhOTxgT50y4vlu6jXjyzJo/i\nsd/sYMbEeva2HeFgRzdnzBvLy7sO0txQySuvt1FTWUpH51FG11ZQVBRgzpQGtnzt10wbV8dFZ7TQ\nc7yX1/ccAqCupoyJTUF2vNXOr154k7WLJ/LYb3bwF2dO44U/7yNUX8m40VUUFxfxx1f3s7+9i85j\nx7l4+WQAfvfSHoLVpVSUlfD8n/bSfayH5/+0lzuvnM+Tz7/Jz3+7izsum8cLr+wlEAiwYHojddVl\nPPdiK93HjrN8zhgef24Xf9p1kJqqMqaOrcW9tp/6YDnnnj6RYGUpr77Vzlv7DxOqr6SxrpLd+w+z\ne38nx3t7qa4opau7h2BVKROaatixu4Ntz7xG19Hj7O/o4vzFE3l510Gmja/jnEUTeGnHAXbtOURT\nfSUV5SW41/bTUFvB7JYGjh7tYcfuDmqry9i15xCjayuYMraWHbvbGdtYzS9+9zp11WUsnT2Gl3ce\npLmhip//dhcTQtXsb++iurIU99oBQvXed6mXXtxrB1gxt5npE+p5/pW97GztoLPrGNMn1DOqppxf\nv7SHxpoyGusq6OruYfeBTirLSxhdW8Gf32jjWE8vYxoqee7FVlqaa6koK2ZcYzWtBzopCgQoLyvm\nD9v3MWdKA/+9fT9FRQGWzGqitKSIA+3d1NWU0X20h9YDR3hxxwG6j/UwdWwtE5pq6Og8yiuvt/HK\nG22sPnUcpSVFvLTzABNCNfT2QnVlCeWlxZSVFvONbz3H7En1LJjeSMfho7y57zC9vb1MHVfH7v2H\nKSoKMKahip27O/jjq/tZOqeZitJi3I4DjGusoqenl/bDR5ncHOTJF95g8pgg7rUDnLNoPHvbuvjt\nS61Magpyyvg6CMDBji6aRlXx8s6D1FaXcqS7h57jvfT09HL0WA9dR48zdXwtpcVFdHYdY1SwnJ2t\nhzhw5BjlATh67Djth7spKSniQHsX08bXAVAUCBAIwPY329mxu4ODHV10HT3O7JZRzJhYT9uhboJV\nZexvP0KwqozWA52MrqvgyeffZM3CcZSWFAOwr+0InV3HaKit4OChbv6wfR+LrIlgVSlv7TvMvvYu\naipKmdwcZF/bEV7b3cG8qQ0c7Oims+sYoVCQna0dhOoqKSkJ0HrgCGUlRdTXlHOku4fWA500jark\n9T2HaKyvpK66jLf2HyZYWcarb7bxxr7DLJ7ZxOt7DjF1XB29vb0ABAIB9hzspLqylKryEra/2U59\ndRnFxUX09vZSUlxEbXUZvb3e51FRVkxJSdGJ1x86coxDnUdpCFawp+0IoVDQlwN2ILKBfGVmDwFP\nOef+T/jv15xzkxItf9FdP8zvNyQikof+70OXBPxYz3Co/noSeBuAmS0Dns9tOCIikshwqP76PrDW\nzJ4M/31DLoMREZHE8j6pOOd6gdtzHYeIiCQ3HKq/RERkmFBSEREZxmoqS9N+zekWykAkHiUVERlR\n7vrLBbkOwXfpJIlls8dw26Vz+cx7V7HlpiW+x6KkIiIJnblgXL+ypvrKHETin9kto3Ky3fLS4gGf\nnz6hbtDrPn9Jwl4WfcycVM8tF8+hKBCgprKUiiQxDYaSiiT0oesXxy2vqy7LciSSKaOC5QM+v/Gc\n6SceRzoxnD6zqd9yn71zlZ9hZcycKQ0EAgG+8r41SZe97nzjvMUTB72tWy6efeLxzRfOYvOV8wdc\nfs1p4we9rVQVF/c95BcV+dI1pe86fV+jFIzJzUE+fOvyfuWXrpqSg2gkEx7adAYXn9GS8PmiwMmD\nTnGx97iX/v2LqypKh3QA9sNVUQkwkfdcPg9I7WC6yEJ9kupARtf2T87LZjefeLxi7liSdTQvKcr8\n4bg45n3H/u0HJRVJ2+I4Z6qF5hO39U+mhWqgY13f41z4ANQLq+f3rxbLtL+5cQkXrWhJ+PzU8bUA\nLJ8zJuEyxWkcuAOB1A+416+bldb6Nm2Yx8p5Y5k3dXRUbEM4wMe89N2XzevzGUWq1ladOrbPcrXV\nZZx7+gQ2bZg3+G3HUFKRtKXzY/PLGXObky/ko8b6ShZOb8zqNqPNmnyy3n/p7MQHyWgzJ9UnfO72\nS+cmfC7elUdEUSDAFWumcf6SiUR/7Dn4ClBWWsSG1VPZePYp/Z6b3Bxk2rg6PnHbcm5aPzvOqz1+\nxX3HZScPwossxJwpDUlfc8r4OpbNGcOdV8xnkYW4cf0sxo6uOhlbUYBbLkoce6pOnTaa02aEuH7d\nzBNlq+eP43ObV7PI+p4QBgIBrj53Bot8bA2mpJJjufhxDsVFK1ooKc5u0KdOG93vxxBRVd63/25s\nFcgHr1s06O3euH4W1RWD6x88aUwNl66cwhVrpnHqtNHJXxDjnqsWnmgqWpViDDe8Lf7ZcnND1YBX\nlwNdqQQCAdYtncxfnp1aNVC06LPwVFWUpXfj+Ev3nMW1583gno0LAe9kILpq6wPXLurz3od6QnTJ\nyinccdk8LCqBTwjVALDlpoGvpIqKAtxy0ZyE34cAsGxO/JOnZFGPbTiZnG69eE7/1wegsjw7fd2V\nVHJs3dLJGV3/BUtTaxWSquaGKkpLilkyKz+qwJbGVnXE/PqmjUveombu1L5nmX//7jMAqK4oZXWc\n1k/JvOvSudy9cSEXr5zCuqWTufOKgW/QJhKpgy9KekjxhOored9VC/l4TNVduiOsfuD6xWy992y2\n3nt2n/JIFL3A1HG1Sdez8Zz+VxTJlJUkPyRFv5+S4iLOPm1CwsQ7dVztgFdpAOuWTeKeqxb2K4/k\nnw+8fRHvvmweX33/Gi5ZOYWFM0JEf9EijyaEatiwemqfdVyxZhrvvCTx9qNzXOR9vf1867NMfU0Z\nj9x15oDvoaqilC/efRZfff+auMkjkOJ3yA9KKj5ZcMrgqkoWz2yK23kp1SqPZPy+ERepKlmbxZuy\nAaBlbPxhuWMPuIN5t3Nb+iaVupqTN12jb55esWYa40PVSdd3eoLPNJHoz/q8xRO584pTgagriDTe\n1MzJoxhVE3PTOMGlyKfCyfN4zPOp3Hc4Y95YrlgzrU/ZipgqyrGjk++raFPH1XLH5aee+Pv8JSe/\nY7MmjyJU5zVlTmtg9RT23Yq5Y7GJ/asOIy89ZXwdp80I9bnKSfWCZ93SyVy8elrC58eMqupXtmbh\neN4VlQivXHMKZSk0/S0tKUp8JZbFygUlFZ/Enl2kI151UgYaZfgi2Q/az7rZiEAgQH1NOZeu7N/q\nLPpK5ZKVUwZVvXHuAAkyOimPbahO/7Q/7JxFE5jVEr/ePTrijedM59Rp3gnKiZyS5luKXT5RyPWR\n5JPue+r17rVMn9D3QDxpzMDzcYxp6H8AjfbX151+Yi6UaOWlxdxz1cJBNX9N5RWBhAumuL0h/FZX\nzR+bdJlEVWLpyObhREklDeMaq7n2vBkn6lCjDbaqNhCIX8+bi5vh8aQ73Y5fUV+44mS1YGRXlJb2\n/7pOHXuyGib6NeC1FkpFURr7OvasPlXXrJ3BJ+6I35djRrh+PrZF1TVrvfsYK+clP/AMKEnIKb+j\n2N2U5q6479rTeE/UlUgiN6ybyXUXJD5Jy8QcUPG+AQN9Lfo0WhjCdtNpjTYkulLJT411FZx92gRu\nuyTOjTCftxX9pW2I0wY+YjA3gTPKp2S44JQ4VzzxjiV9WiQF+pzNTmzqn/yHJAC1Vf53/LSJ9Xzi\ntuX9DqQr5o7lq+9fk/QKIJlkiTCV+04Ad1x+KmMaqjj39AnAwK3G4qmtKmNBCi3qVs0fx1kLEncE\nTKv2K/x9vO6CgTsypnsSF+j7xUvrtakYqHFGWfjkqjaNTsi6p5JnJo3xDk6Rj6U0hZuJQxV99hyZ\n1nQw/D6pS7a+uhQOuucumpB0mfKoq5JUf/ABMnEPqa+bL5zNWQv97/ncWF8Z94ppMFes6b7ktBmN\nferwK8rjf9/mtDTwsVuW0VBbAfj73Vq/PPUGK4O5UjlrwfiUOzJGDLgfM3yMnjV5FJefOZUP37y0\n33P3XbOIjedM57QZqVc1Z7PiQ0klRtze4uHvcLIf+Iw4N/uSSfT78OvyOtsqB9kEN1Z0/Xuq7z8Q\nGPh8bPHMprhVl+kYXVfBdXHun0WacU5I4UZ+psXuhWSNSAKBAKfPbOKDb1/EpaumMG9a9vvnXH5m\n4pvZ2XLOogkDVrtFCyR47JdAIMD65S2Mb+z/fQrVV3De4olZTRTpyPtJurIt3ucUe9yPf09v8BeY\n8c68BtPSJO66B3tneZDr8+t7XhQIML6xml17DqW1vYGiW3BKI2/sfS2tOKI/m4YBxsk6ddpo7r/+\n9LitebLhsuimrFE75YPXLWJyitVn08bXMW18XcpXR+lcMWTi6s5v16ydAcDXf+Jysv10Tp5Sddsl\nc/h/T72a1WrynCUVM9sA/IVz7prw30uBTwNHgcecc1vC5fcD68Plm51zv852rL0nrlQGXi7R80tm\nNfHMf+9Oa5up3jyOXmrF3GZuWj+Lmx78eVrbAi/22uoyDnZ0D7xgZF9k+vopwImOh35d/Qw15GT3\nNlqak/fdyJQLozrdRb/NVO+XDEY6tVBXDaLPih/bzYZUetOnItnbKi4K0HO8l5Li1CuYlswaw5JZ\n/nRPSFVOkoqZPQycB/wuqvgLwAbn3HYz+7GZzcernlvtnFtqZhOB7wH+TwCQVGrf4kRnEMkSRNy1\nx9yATmWba0+f2H/ZFH+AH75pKV/80X8lTyphLc1BFs0I8eyLrTHxpLa9ZAJ49y9+9Kvt/MVZqVeN\nDLR532/cR283T6siMimde4sDfYeTjZQcK89yClPG9j2Z+Nity+jq7vF9O5969xm0HT6alXu6Q5Gr\n6J4kat55MwsCZc657eGiR4G1wEpgG4BzbgdQbGZZb+4UGQrkRE/VOL+PAF5flVMm1PXptAWDO+BE\nd+pL9fXBqvRngIs2KYWDbuQHXVQUYNNl/gxCN2ZU//k5AoEAjfWV3Pi2WSdbXMXb7zE7Z6ADTnOS\nfhLZkM4N6cHKVnP0qeNqWbdsEh94e/KhcHwNyadLlcggi/WxnUVT2MxA+3jMqKpBtdhLtouCVWVx\n77Hkm4xeqZjZjcBmvN96IPz/Dc6575pZ9LgDtUBb1N/twFSgE9gbVd4B1MWU+Sre9+imC2fzgyde\n4YokZ8zNDVV84NpF/Pt/vh7zTJKvS5yN2qR6fvbczlRefUKkVU6SVSd09doZPPnCm33K0unDkXF5\ndIpaWlLE0WPHT/ydanXg5WdO48LlLZSUBHjnJ/4tQ9FlRyAQ4IqzUqvW8rO6NNIaMt1xwmK9/+rT\n6Ow+1q/57sazT+EPr+4fcP2lJUVcs3YG43w8yOfR13tIMppUnHNbga0pLNqGl1gigsB+oDv8OLr8\ngG8BxlFd3f+sZfb0JmZPPznWVW9x/y9bY2PNieE9gsG+B/eKAe4JjBpVFben8AUrp/L5H7wAQMkA\nTYrLo8b5CYX6nx1VVabWlr2hoZqJMWdXoVCQUQ3VLJ3TzNP/5SWbmpqKuNuJiLf/YlVEDWESrCqj\nKKaO+J8fvDBuM+p46w6Fgnz9gfPp7DpGqLGmz76PjTMUClISp+og3nIRVVXlccu//bdv4/jxXq74\nwI8BaAzVUFGW/OcUb9+NGlU94D4dSGV5MUvnjk34+oHWO9jnUhW9jqamYMKz++KiQNztVYa/u4FA\n33Vdfu4M3jzQyWVnnZI0zsG8j2vW9++HFs/GC5IPd59OHHV1lWnFW1nh/Y6KEuy/XMmL1l/OuXYz\n6zKzKcB24HzgAaAHeNDMHgImAgHn3L5k6/vYrcu474v/MahYDh3q6lfW2tre5++9Bzv7LbNnbwfd\nnd79iPb2I32eO3LkWMLt7d9/mJ7j/c9R9uzpOPG4p+d4v+cjurtPrjs2ToDDh1O7R7Jv3yEqYo63\nra3thEJBbr1o9omk0t7eGXc7ybZ3y8Wz+dKP/gBA5+GjJ8o/fusy/uZrfdteHNh/OO46BvpsxoWC\ntLa299n3sXHu2dPOsWP992VrazsfvG4RH/n6s/1eF73Ngd73nj0dSaeLDYVjjLV/f/99n6rP3rma\nQCDQb73vv3ohdTXlA8ac6LlEcaYreh3R3+eICaFqdrYeoud4b9ztdYZ/T729/WO9/nxLKU4/3sdQ\npbo/Dx4c+LcVq/OI9zs6nmD/pcuvxJQXSSXsNuCbePd5tkVaeZnZE8BTeLVAm1Ja0xCuI1PpPOfn\npXwgQNI64mRbu+qc6bR3ppY8hirZro2N9bN3rmZv2xEmNtWcSCoR1RUl3n2q8ErnTR3dpxOe3+J9\nbpGOmC3NA/+gkn0G2aokjNQhn/g7wdm/TRp4HvZQff+q0ky4a+MC3toX/yRhePXAklTlLKk4534B\n/CLq72eAftPthZsWb0ln3dVR1Ssbz5nOt3/2UsqvrUuh+iaewfdRGfrKBxoxeLD9VC4/c2r8J9LM\nKlUVJVRVpNbqqrqyhPIh1pMPKM5+jIwnluhEId/quc9ZNIGfPrtzSOv44t1nZm3MqTktDcyJGUhz\nzpQG/uvP+wjVV7Cztf8VzEiV7jHkzPnj+Plzu7j63PTnusmkfLpS8cVn71lDVdSov/k62m+0pMfp\nHNwsX7+8JWPrjk10kb/9eJeDXscQL0Wy9REtnN7IlWefwi1/929pv3bzlfNpO9Q9pGF//LD5yvkc\n6TrGt3/2ckrL+92BN1+l+y4njQny1fevyZvBZyMKLqlMbq7tU7+Y7geVyueT7c9wwCGIkvWB8Xvs\nryTPp7NrIrGfjHHoO3ag+AY3lFN+HNAiHd9qqsrS6vwWbTAzMaarqb6S3Qf633OMVhQIUFVRmjxR\nD+H7UF5aTNdR//uK5Jt8SyhQgEklV4by4SY78OfV9yYm2Pdcfiqf+d7vT/w9ui47dfWDkWon0rjP\n5/iuykdvWcbLuw5mtAOnHxbPauLHT73KjAmZ68mfik+/ZyVHB2jgko/y6Wc+FPndNdMP+XGiOUQD\nHAyzFEFkoMTYpBE9lPn162aybHbyCYUy+ZEk2x8XndGSme0O4YOoSmHu8FB9Jct9mKwp0y4+o4Vb\nLprN5isX5DSOstJiqiuG1hk42wriUMUISCq5+KBib0wC3HLRbN77F8knKIonE0Nwz2kZxQevS94T\nOuLuqxZy+6VzB6xCWT1/XGqz86UwMvNgJfu8F89s4ivvW3OyCimHv+RPvmsFd21c0Gf64uGutKSY\nZXOafWlwMVLupRSawq/+ysHoc6PrKvjK+9Zw7xefYs/BIwQCQ5sSNBNXI+9YN5PGuv7DoyRSW1XG\n4plNyRdMQySJZPojeuCGxexrO9nfJJ1paTMZW0NtRdxREFJxuoXyuqpR0lco1V8Fn1Ryda4zmPm0\nEymL6lQ3Z0oD65ZO4pPf9sbiHMykVLXVZYwe5MEsno3nTGfHW0PvfJWpH9WkMcHEYzGluNG8uq8F\nvGuDP+Ou5dKclgZ++fs3WDY7u6PoSmYVfFLJhPhzyg9+fcnmpXjHupnc+4WnALjrL7266g/ftIRv\n/eylpGMvxVv1zetnpTUQYzIDTdOaGlVzjERLZjUxuTlIU5wBRWX4Kvx7Kikcrz50/eLMB5KCM+bG\nryJrqu//oxsfquHujQt9rwK57gLj5gtTH9NoMBJ+JHl2NQD5N3dHIQkEAjQ3VCUctDSb86rng0L5\nqhV8UklFfZrzOWRKKgMSpivdm51nLRjPirljfY8j2vI5XnXHpSunAPnzYyopDnBKgqawg52gTWSk\nKfjqr2y1IPHlmOLTgWnauFr+9Hpbwuf9HK57MKZPqOdL95zVrxNfrs9Mv3DXWf2SQ7Lvz4euX8zu\nA51ZG/ZkJJk23hu43O8GIvmqUM5LCv+XkMPT4FsumsPY0VVJ+0b4XcWyIroaLc66B9viyE/RCSVf\nqpiKigJpd2Kd3BwcMQe9bDttRoj7rz+dd1wwM9ehZEWe/AyGrOCTSi+wYXWCwRF91f9gdMqEOj7y\nzmWMGZXajIOFcqaSrmvWzgC8wRKzrlB+yQUoEAjQ0lw76GFpJDdGxKd10YqWQb92TBamoM3kcS2X\nx8yb1qd2w/+0GSG23ns2k5MMP5+KVBNz+vdARmrKl2wplG9YwSeVZM11k1m/rP+c4hm7KevTegd6\nxzesy15VQi5uXqf6aaf8tdCVjEhaCj6pxDM5UUe4OFI9MA7tAJrBI1fMqlfNH5e5bQ1Hw7Tzo0i+\nKvjWX/G0jA3yqg89wP0SOWuObf208Zzp1NekNsd8IuNDuWvplevWXH7QhYpkS6F817KeVMysFvgG\nUAuUAn/lnHvazJYBDwNHgcfCMz5iZvcD68PlmyPTDKcqzvTvQ5ax2q+YFQ+2p3p01c7q+eMIVpXy\nue+/MOBrMjKZ2fDPKScU0FsRyahcVH/9FfBT59xZwA3A58PljwAbnXOrgKVmNt/MFgKrnXNLgauA\nz/kRQL4dIDJ5hlJUFGCRJW7yumnDXFbMbaY5Cw0ShionZ3KFcvooeS/fjkuDlYuk8ingi+HHpUCn\nmQWBMufc9nD5o8BaYCWwDcA5twMoNrP0pq+Lc0c2344TkQH1pudgYqNF1sTNF87Oyxnk8op2j2RY\nvh2XBiuj1V9mdiOwGW9/BcL/3+Cce9bMmoF/BN6DVxUW3QW8HZgKdAJ7o8o7gLqYspSMD1Wzq/UQ\nZaV982hxUYDqiqHvhqEck69ZO4M1C8fn/ax+6fL7OJyR43qh/JJF8kRGk4pzbiuwNbbczOYB3wTu\ncs79MnylUhu1SBDYD3SHH0eXH0i23VDo5EuqqssJhYLMbGlgV+sh6mrKqQjPCFcfLOfrHzq/z1l6\nbUxv89rayj7rAygu7z+jXGNjkMoUZvCLqKsp42BHNzatkWBVGWOb+16lxG4zWXm0mqhJn2KXT+X1\ng9lmPME4+24o66uu7j9GW/S6QqEgwWBF3OdiRe4hVVWVDbjchrOn89Nnd/JXVy0adNyJ4s1nitNf\nqcRZXxf/9zLc5OJG/WzgO8CVzrnnAZxz7WbWZWZTgO3A+cADQA/woJk9BEwEAs65fcm20dp6smVX\nR0cXra3tHDlyFICenuMnHh8/3suePR19XtvWdqTP3+3tnX3WB3Cgo4tYe/a0pzUg5IdvXUGgp4cj\nh7o4cujk+prqKyFAv23Ge2+JtLeffA+xy6fy+mihUDDt15yMo/++G0wMER2H+u/3yLoicQ703qNF\nGnAcPtw94HKlwJfuOSvp+lIxlH2ZTYrTX6nGefBg/N9LtviV0HLRpPijQDnwaTMLAAeccxuA2/Gu\nXoqAbZFWXmb2BPAUXu3HJj8D8bM6Jd3ms4FAIO4c2h+9dZmq77NJO1vEV1lPKs65SxOUPw0sj1O+\nBdgy2O3F9qiPPobEq05P5d5IJgdATDS3hOj4LzIcjLge9cnyweTm2iRL5L98uffsd+fHfHlfIpJY\nwSeVga4q4h3yxjdW89CmM6KWSXUcj/TiEhEpRAU7TEuk/XJsTknl2D8qT2aCjFZVXjLs5vJWTZ5I\n6grlSrxgk8rU8bX8aVcbjTFzuPcmeDxUmT5+fubOVcPmYujSlVP4wS//jE2s93W96qcikv8KNqnc\ncfmp/PbFVs6Y15x84WEgrRv4OT5QXrxyChee0eJ7owNfTwKGS4aWEaNQvpIFm1Rqq8o4c8H4fuWB\nBI/PPm08x3qOZzyukUKt2ETSU1FWnOsQfFGwSSWRRNVf155nqa8jzt3/fDqGzm4ZBcAFSyblOBJ/\n+bmLM9ksXCQdW25awn++vIcZPlcX58qISyppy6NkkarxoRoe+aszKS+QM5+IjOSBYfj5SmGZEKph\nQqhwxv0r+CbFsRJVfxWaQksoIjI8jLikkjmFnKJERFIzoqu/VK2eXQ+/ZyXdR3t8WVdLc5CdrYd8\nWZeI+GfntmnBAAAUsUlEQVREJxU/5dON+nxVW1Xm27r+xztO9+dmu84sRHw1opNKKnlAuSI/BQKB\nISVynQSIZMaIuaeybI7XCfKCpZNPlA32JFXNUUVE4hsxSWVOSwNfuOtMzlk0IdehSB7QiYFIZoyY\npAJQVtq3ma1qQERfAhF/5WI64Sq8GR5HAV3AO5xzb5jZMuBh4CjwWHhyLszsfmB9uHxzZEZIERHJ\nP7m4Unkn8Bvn3JnAPwHvC5c/Amx0zq0ClprZfDNbCKx2zi0FrgI+l/VoU51ORWe8IiLZTyrOuU8D\nHwn/OQk4YGZBoMw5tz1c/iiwFlgJbAu/bgdQbGajsxtxf71qh5q3xo6uAmDa+OE/g6fIcJTR6i8z\nuxHYjNfQKjJv1g3OuWfN7KfAPLzkUQu0Rb20HZgKdAJ7o8o7gLqYsrzg99S5fjvdQhzrKfxkOH1C\nPfdecxoTm1IcS6nwd4lIVmU0qTjntgJbEzx3rpkZ8GNgAV5iiQgC+4Hu8OPo8gOZiTZ1leXebotk\nyeHgXRvm5TqErElltFdVV4pkRtKkYmZlwBXA4nDRr4HvOue6B7NBM7sP2Omc+0fgEHDMOddhZl1m\nNgXYDpwPPAD0AA+a2UPARCDgnNuXbBuhUHDA56++YBbPv7KPOzcuTLpsbbCy3zIh4BPvXsWY0VW8\n428eBaAxFKS4KPUjVXlpMaHG6pSXz6Vk+yhbqqtPTvMcL6Z04ox8UlVVZVl9f/myL5NRnP4aLnH6\nYcCkEr5/8W/AYeDfgTLgvcC9ZnaWc24w1VBfBf4hXDVWBFwfLr8dr1VYEbAt0srLzJ4AnsI7DmxK\nZQOtre0DPl8KfPJdK1Jatr39SNxlGmtK6ek6euLvPXvaU5qY6kPXL+blXQcZ21iddNv5IBQK5k2c\nhw51nXgcG1O6cUauMA8f7s7a+8unfTkQxemv4RSnH5JdqXwC+IZz7sHoQjP76/BzN6W7QefcbmBd\nnPKngeVxyrcAW9LdTr6a3BxkcvPIOWvJV+r8KJIZyVp/LYlNKADOub/Fa5klkjVzWhoAOG/xRP9W\nqnsrIr5KdqVSOsBz/oxhXiB0bMq8yc1BPnvnqhMNJUQk/yS7UtllZmtiC83sHOC1zIQkklhVRSkB\nNd0SyVvJTvnuBX5oZl8AngkvvwK4Aa+FVsFL9filA90wpXsrIr4a8Eol3ALrXGAK8EngY0ATsNI5\n9/vMhyeSGToHEMmMpJXTzrk/4F2ZiIiIDChZP5VyvITyJvA48B286q/ngFuccy9mPEIRERk2kt2o\n/yre2Fy3Ar8AfofXlPhHwBczG5pI5qifikhmJKv+WuCcmxseqmWXc+7ecPnvwz3iRYY33VsR8VWy\nK5WjAOFxvnbGe05ERCQiWVLpTfA43t8iIjLCJa3+MrNIz/lA9GOUVEREJMaAScU5l4vphkWyR6dG\nIr4adNIws+f9DCRfqad8YdLHKpIZQ7kSafErCBERKQxDSSqqOJBhS/1URDJD90xkZFM1mIivkg3T\ncpz4VyRDbv1lZjOB/wCanHPdZrYMeBiv/8tj4RkfMbP7gfXh8s2RaYZFRCT/JLtSebdzrtg5VwzM\njzwOtwr77GA3amZBvFGPj0QVPwJsdM6tApaa2XwzWwisds4tBa4CPjfYbQ6WTmRFRFKXLKncHPX4\n6zHPrRrCdr8E3AcchhNJpsw5tz38/KN4Y46tBLYBOOd2AMVmNnoI2xURkQxK1vkxkOBxvL/7CY8P\ntpm+VWWvAd9yzj1vZpF11AJtUcu0A1OBTmBvVHkHUBdTJjJ4umEv4qt0JvtOe5gW59xWYGt0mZm9\nCNxkZjcDzXhXIhfhJZaIILAf6A4/ji4/kEbMInGpn4pIZiRLKr6fxznnZkQem9mfgbXOuaNm1mVm\nU4DteFMVPwD0AA+a2UPARCDgnNuXbBuhUDDZIimrra1MaX2D2aafcWZSIcYZySlVVWVZfX+FuC9z\nSXHmn2RJZY6ZvRJ+PD7qcQAY68P2ezn5+74N+CbefZ5tkVZeZvYE8FR4uU2prLS1td2H0DxtbZ0p\nrS/dbYZCQV/jzJRCjTNytnT4cHfW3l+h7stcUZz+8ivxJUsqM5I8PyTOualRj58BlsdZZguwJZNx\nyMijzo8imZFsQMlXsxWISE7o3oqIr9SjXkREfKOkIiIivlFSSUJNTwuc7q2I+EpJRUYknSyIZIaS\nioiI+EZJRUREfKOkIiOS+qmIZIaSioxsurci4isllaR01BERSZWSSgK11WUA1NeU5TgSEZHhI52h\n70eUD12/mJd2HmDa+LpchyKZpHsrIr7SlUoCo4LlLJk1JtdhSIaon4pIZuhKZYge2nQGXUd7ch2G\niEheUFIZolHB8lyHICKSN1T9JSIivlFSkRFJnR9FMiMn1V9mthN4MfznU865D5rZMuBh4CjwWHjG\nR8zsfmB9uHxzZJphEV/ohr2Ir7KeVMxsGvCsc+6SmKceATY457ab2Y/NbD7eldRq59xSM5sIfA9Y\nkuWQRUQkRbm4UlkETDCzx4HDwGbgTaDMObc9vMyjwFqgC9gG4JzbYWbFZjbaObc3+2FLQVI1mIiv\nMppUzOxGvKTRi1fR0AtsAj7qnPuemZ0B/BOwAWiLemk7MBXoBKITSAdQF1Mmkjb1UxHJjIwmFefc\nVmBrdJmZVQLHws8/aWZj8RJKbdRiQWA/0B1+HF1+INl2Q6FgskXyguL0VzpxRnJKVVVZVt9fIe7L\nXFKc+ScX1V/3A/uAvwvfN9nhnGs3sy4zmwJsB84HHgB6gAfN7CFgIhBwzu1LtoHW1vZMxe6bUCio\nOH2UbpyRWq/Dh7uz9v4KdV/miuL0l1+JLxdJ5ePAN8zsbXhXLNeHy28Hvol3c35bpJWXmT0BPIV3\ncrkp69GKiEjKsp5UnHMHgYvilD8NLI9TvgXYkoXQZARRPxWRzFDnRxnZdMNexFdKKiIi4hslFRnZ\nVA0m4islFRmR1E9FJDOUVERExDdKKiIi4hslFRER8Y2SioiI+EZJRUYkdX4UyQwlFRnZ1ApMxFdK\nKjKy6YpFxFdKKjIiqZ+KSGYoqYiIiG+UVERExDdKKiIi4hslFRER8Y2SioxI6qcikhlZn/nRzIqA\nTwGLgDLgfufco2a2DHgYOAo8Fp7xETO7H1gfLt8cmWZYxBdqBSbiq1xcqbwdKHHOrQI2ALPC5Y8A\nG8PlS81svpktBFY755YCVwGfy0G8Ush0xSLiq1wklfOB183sX4AvAT80syBQ5pzbHl7mUWAtsBLY\nBuCc2wEUm9no7IcshUb9VEQyI6PVX2Z2I7CZvueDrUCnc+5CM1sNfA24GmiLWqYdmAp0AnujyjuA\nupgyERHJExlNKs65rcDW6DIz+xbwL+Hn/93MpgMHgdqoxYLAfqA7/Di6/ECy7YZCwWSL5AXF6a90\n4oxcqFRVlWX1/RXivswlxZl/sn6jHvgl8Dbg+2Y2H3jNOddhZl1mNgXYjldF9gDQAzxoZg8BE4GA\nc25fsg20trZnKnbfhEJBxemjdOOMXDofPtydtfdXqPsyVxSnv/xKfLlIKl8GHjGzp8J/3xb+/3bg\nm3j3ebZFWnmZ2RPAU3gnl5uyHKuIiKQh60nFOdcN3BSn/GlgeZzyLcCWLIQmIiJDpM6PMiKp86NI\nZiipiIiIb5RUZERSPxWRzFBSERER3yipiIiIb5RURETEN0oqIiLiGyUVERHxjZKKjEjqpyKSGUoq\nIiLiGyUVGZHUT0UkM5RURETEN0oqIiLiGyUVERHxjZKKiIj4RklFRER8o6QiIiK+yfrMj2b2fuAC\nvGnCRwFjnHPjzGwZ8DBwFHgsPOMjZnY/sD5cvjkyzbCIiOSfXEwn/CDwIICZ/V/g7vBTjwAbnHPb\nzezHZjYf70pqtXNuqZlNBL4HLMl2zCIikpqcVX+Z2WXAPufcz8wsCJQ557aHn34UWAusBLYBOOd2\nAMVmNjoX8YqISHIZvVIxsxuBzXhVXYHw/zc4554F7gU2hhetBdqiXtoOTAU6gb1R5R1AXUyZiIjk\niYwmFefcVmBrbLmZzQL2O+deCRe14SWWiCCwH+gOP44uP5Bsu6FQMNkieUFx+iudOCOjtFRVlWX1\n/RXivswlxZl/sn5PJexc4F8jfzjn2s2sy8ymANuB84EHgB7gQTN7CJgIBJxz+5KtvLW1PRMx+yoU\nCipOH6UbZ2SQ4sOHu7P2/gp1X+aK4vSXX4kvV0llBvBYTNltwDfx7vNsi7TyMrMngKfwTi43ZTNI\nERFJT06SinPujjhlzwDL45RvAbZkIy4RERkadX6UEemU8XUANI2qzHEkIoUlV9VfIjl1+6Vzef5P\ne1k6e0yuQxEpKEoqMiLVVJayfG5zrsMQKTiq/hIREd8oqYiIiG+UVERExDdKKiIi4hslFRER8Y2S\nioiI+EZJRUREfKOkIiIivlFSERER3yipiIiIb5RURETEN0oqIiLiGyUVERHxTdZHKTazSuBbwCig\nC7jWObfbzJYBDwNHgcfCk3NhZvcD68PlmyMzQoqISP7JxZXKdcAfnXNnAt8B7gmXPwJsdM6tApaa\n2XwzWwisds4tBa4CPpeDeEVEJEW5SCpHgIbw41rgqJkFgTLn3PZw+aPAWmAlsA3AObcDKDaz0dkN\nV0REUpXR6i8zuxHYDPQCgfD/7wbuNbP/wqsCW4WXXNqiXtoOTAU6gb1R5R1AXUyZiIjkiYwmFefc\nVmBrdJmZfRH4lHPuy2Y2D/hnvCuS2qjFgsB+oDv8OLr8QCZjFhGRwcvFdMLVwMHw41Yg6JxrN7Mu\nM5sCbAfOBx4AeoAHzewhYCIQcM7tS7L+QCgUTLJIflCc/hoOcQ6HGEFx+m24xOmHXCSVDwJfNrN3\nA8XAzeHy24Fv4t3n2RZp5WVmTwBP4VWfbcp+uCIikqpAb29vrmMQEZECoc6PIiLiGyUVERHxjZKK\niIj4RklFRER8k4vWXxlhZgHg88B8vF77NzvnXslxTM9ysvn0n4GPAl8DjgMvOOc2hZd7J3AL3vhm\nH3HO/ThL8S0FPu6cW2Nm01KNzcwqgG8ATXidVt/hnMtIh9SYGBcA/wK8GH76Eefcd3MZo5mV4PXF\nagHKgI8AfyDP9mWCOHeQf/uzCPgyYHj77za8MQK/Rn7tz3hxlpFn+zMq3ibgN8C5eF01vkaG9mch\nXalcCpQ751YA9wGfymUwZlYO4Jw7O/zvpnBMHwiPe1ZkZpeY2RjgDmA5cAHwMTMrzUJ89+D9KMrD\nRenEdjvwe+fcauAfgf+RpRgXAQ9F7dPv5jpG4FpgT3g7FwCfJQ/3ZUyc68Jxnkb+7c+LgF7n3Mrw\nNj5Kfu7PeHHm4/czckLxBeBwuCij+7OQkspK4CcAzrmngdNzGw7zgWoze9TMfho+4z7NOfdE+Pl/\nxRvfbAnwS+fcMedcG/AScGoW4nsZ2BD196IUY5tP1L4OL3tutmIE1pvZL8zsy2ZWkwcxfoeTP7Ri\n4Bipf865irMI72x0EXBhPu1P59wP8c6WASbjjayRd/szJs6WcJx5tz/DPok3YO/reP39Mro/Cymp\n1HKyqgngWPgSNVcOA3/nnDsfL9v/E94HGtGOF3OQvnFHxjfLKOfc9/EOgBHpxBZdHlk2GzE+DdwT\nPsN6BfgQ/T/3bMd42Dl3KDwo6nfxOvfm476MjfOvgWeAu/Npf4ZjPW5m/wv4DF6H6LzbnzFxfhrv\n9/00ebY/zex6YLdz7jFO7sfo46Lv+7OQkkobfccJK3LOHc9VMHj1qv8E4Jx7CW8QzDFRz0fGMWuj\n/7hnuRjfLHpfDRTbfvru62zG+wPn3G8jj4EFeF/4nMZoZhOBx4F/cM59mzzdl3HizMv9CeCcuwGY\nAXwFqIyJJy/2Z5w4t+Xh/rwBWGtmP8e78vg6EIqJx9f9WUhJ5UngbQDhCb+ez2043AA8BGBm4/A+\nsG1mdmb4+XXAE8CvgZVmVmZmdcBM4IUcxPucma1OMbZfEd7X4f+fiF1ZhvzEzCLVmucAz+Y6xnBd\n9KPA+5xz/xAu/m2+7csEcebj/ny7md0X/vMI3k3l36Txu8lVnMeBfzazxeGyvNifzrkznXNrnHNr\ngN8Bbwf+NZPfz4IZpiWq9VfkfsQNzrkXB3hJpuOJbm3TC7wP72rlK0Ap8N/AO51zvWZ2E3Ar3uXp\nR5xzP8hSjJOBbznnVpjZdLyb4kljM2/2zn8AxuK1zLnaObc7CzHOx5uorRt4E7jFOdeRyxjN7GHg\nSuCPnJze4b3A/ySP9mWCOO/DO/HJp/1ZidcyqRmvderHwjGn9LvJcZyv4R2D8mZ/xsT8OF4rtV4y\n+FsvmKQiIiK5V0jVXyIikmNKKiIi4hslFRER8Y2SioiI+EZJRUREfKOkIiIivlFSERmAmR0P/19r\nZt/3cb2PRz1+zq/1iuSakorIwCIduRrwhrnwy1mRB86503xcr0hOFcx8KiIZ9mlgnJl9zzl3uZld\nh9dzPoA3HMcm51y3mbXizVsxBm/k188Dc8J/O+By4EEAM3vKObfczI4754rCvZe/jJe8evCGUf9H\nM3sH3nDkDcBUvDGmNmXvrYukTlcqIql5D/B6OKHMBm4GloevMlqBu8PLjQY+Gi5fDnQ5584ApgNV\nwDrn3HsBnHPLw6+JXA39Dd6cJ/Pwxo56wMzmhp9bjjcNwKnARWY2J4PvVWTQdKUikr41wCnAf4TH\nnCvFu1qJeAbAOfeEme01s3fhDdB3ClCTZL03hl+718x+gFdN1g78yjl3GMDMXsG7ahHJO0oqIukr\nBr7jnLsTwMyqOflb6nXOdYXLL8a7+vh7vMFFG+k7N0is2JqDoqj1Hokq702yHpGcUfWXyMAiB+9j\nnDzA/xuwwcxC4SuVR/Dur0QvD14V1v92zn0d2A2sxktI0HcSuchrHgduAjCzRuCS8LZEhg0lFZGB\nRe53vAXsMLOfOed+D2zBSwLP4yWFj8csD95N96vN7Bm8OcJ/CEwJP/cj4D/NrDzqNVuA0Wb2e7xk\n8rfOud8NEJNI3tHQ9yIi4htdqYiIiG+UVERExDdKKiIi4hslFRER8Y2SioiI+EZJRUREfKOkIiIi\nvlFSERER3/x/L1vCyL3kpioAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1188befd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(advi_fit.elbo_vals)\n",
    "plt.xlabel('Iteration')\n",
    "plt.ylabel(\"ELBO\");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ADVIFit(means={'alpha_log_': array(2.7251985780730963), 'beta_log_': array(3.750899891216531), 'p_logodds_': array(-1.0869779295409336)}, stds={'alpha_log_': 0.61737582209059227, 'beta_log_': 0.64250377477510057, 'p_logodds_': 0.57563120930079181}, elbo_vals=array([ -3.90476844,  -2.82537291, -42.19240793, ..., -11.15347832,\n",
      "        -3.4346194 ,  -4.77228724]))\n"
     ]
    }
   ],
   "source": [
    "print advi_fit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On the other hand, with `accurate_elbo=True`, $M=100$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration 0 [0%]: ELBO = -35.23\n",
      "Iteration 400 [10%]: Average ELBO = -26.89\n",
      "Iteration 800 [20%]: Average ELBO = -18.48\n",
      "Iteration 1200 [30%]: Average ELBO = -12.9\n",
      "Iteration 1600 [40%]: Average ELBO = -9.64\n",
      "Iteration 2000 [50%]: Average ELBO = -7.33\n",
      "Iteration 2400 [60%]: Average ELBO = -5.98\n",
      "Iteration 2800 [70%]: Average ELBO = -5.18\n",
      "Iteration 3200 [80%]: Average ELBO = -4.76\n",
      "Iteration 3600 [90%]: Average ELBO = -4.58\n",
      "Finished [100%]: Average ELBO = -4.53\n",
      "CPU times: user 25.3 s, sys: 576 ms, total: 25.9 s\n",
      "Wall time: 33.1 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "advi_fit = pm.variational.advi(model=model, n=4000, accurate_elbo=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY8AAAERCAYAAACD9ivUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8XHW9//HXZN/TtJ3ubbp/SxdaWqAbSwsFBAHBCj9E\nUfZV8aIo4PUqFy8ICggqolepogIqF0ERsexboaWUpQvwpbQNpXvatE3aNOvM748zM5nJNjPJ7Hk/\nHw8ezFnmzCcnzfnMd3d5vV5ERESikZXsAEREJP0oeYiISNSUPEREJGpKHiIiEjUlDxERiZqSh4iI\nRC0n2QH4GWNcwC+B6UADcKm1dmNyoxIRkc6kUsnjLCDfWjsPuAm4O8nxiIhIF1IpeRwD/BvAWrsC\nODK54YiISFdSKXmUAfuDtluMMakUn4iI+KTSw7kWKA3azrLWepIVjIiIdC1lGsyBZcDpwP8ZY+YA\na7o72ev1el0uV0ICExHJIDF5cKZS8ngcOMkYs8y3fVF3J7tcLqqr6+IfVS+53aWKM4YUZ2ylQ5zp\nECOkV5yxkDLJw1rrBa5KdhwiIhJeKrV5iIhImlDyEBGRqCl5iIhI1JQ8REQkakoeIiISNSUPERGJ\nmpKHiIhETclDRESipuQhImnL6/Xi9XoD280t4afDq61v4m+vbGTr7oOdHt++5yCNTa0hnwHQ6vGw\neWcdnqDP8/N4vezYc5B31lcH9m3cVssHn+ztcG5Lq4fqfYd4fe129h9soqXVw4Zt+zlwqLnTeA4c\nauZQY0vYnyv4c4Pj7yzeWEiZEeYi0nMej5esLGfKop1768nNzqJ/WUHgeE1tAxu31fLJzjoWHz+u\n02us3rCHex59j1PnjGLx8ePI8s0d99jLG3j53W3cdc18wIvL5SIn2/neua6qhh176pk+fgADywup\nqW3gnfW7OXHWCNZtquEXj6/hiydO4JhpQ3ngqfd5Y91O/vvio6kozcdu3sd9j6/h0tMPY+ZEN8+8\n+SlPvLaJRbNGsHlnHcPcJbhc8OYHu/jul2fS0upl0/Zatu0+yDMrP+WYaUN5bc32Dj/HhBHlrN+y\nn5OPGskzKz/t8p798/UqALKzXLR6Oj5gPzu3klaPl3+v2Nzh2LypQ3h97Y4ur93emfNH09DU2m08\nfheeOonfP/0hAFefNZVfPrE24s+JxJN3fS4m13F545SVEsCbLvPIKM7Y6UtxVu2oZdfeQxx92ODA\nPrt5Ly0eL+XFeRQX5FJUkOM8pP+2hq99fhozJ7q5+PYXALj+vBlMHNmPjdtquf2htwPXuOua+ZQV\n5/Lm+7sYMaycYf3yeXf9Hu57PHQu0orSfPbWNXYa2/EzhvHh5n3srKnv1c8oiffkXZ/LuIkRRfqM\nQ40tbKk+wN66xkByWPLUBxQX5rD4+HFkZ7m45fdvATBzohuPx8udf3mXj7fs7/Ka9z+xlsMqKwLb\nd/75XQAWzBgWct637ltGJLpKHAAvv7stomtI5lLJI8760jflREiVOD1eL3gJVBX5NTa18vHW/eys\nbWDpG1V889wZDCgvoLnFwzU/fQVwHuYvtXv4XvLZw3jgqQ8SFb6kqIHlBeze39Dl8cmjK8jLyebd\nj3d3evxLJ03koWc/6vYzYlXyUPKIs1R52IWTCXF6vV4ONrRQUpgb1TW9Xi+tHm+gHt9vz/4GKkrz\nQxLE1t0HWbtxD7v2HuLFd7Zy77XHUFqUFzjurzKSxJg6tj9rN9aE7CvIy2b4wGL6lxWw8sNdXHXW\nVKaO6c/3H1jB4gXjGDagmPVb9lOQlx2SsIsLcrj4tMPoV5rP/oNNjHAX86u/r2NgeQFvfrALgCU3\nnkBDUwsfVO1lUmUFjc2t/NdvV3CwwWnQzs3JYtywMrbX1PP548bywSd7Wb5uJ+5+BVTvc5LCjPED\nufYLhwc+d9P2WtZs2MMZ80fjcrlYv2Uf5SX5eD1esrNd9CvJ7/Bvs3rfIfbWNTJxZD8AdtTU4wIG\n9y+i1eOhpdVLfm42AP/75DoaGlsDn+l2lyp5pPvDLpWkapwbtu1n2IBiCvOdGtbu4vz90x/yynvb\nuOWSoxnhLon4M3744Fts2l7Lf331SMYMLaPV4+H5t7bw5xc+BuDWy2bzt1c2csrRo7jtj6s6vcYt\nFx/Ng//+kA3baqP8CfuGuVMG88a6nYHtk48aybqqGs5ZMJ7ag00s+ZfzEJ8zZTC52Vl89TOTeOT5\n9by7fjeXnT2NvzzzIRNG9OOZlZ9y5vzRnDBrBPUNLQzpX0RzSyubttdx91/f5aufmcTcKUOiiq2h\nqYWc7KwOD+hgzS2t5GRn0d0CdJ3922xsbmXNhj3MmDCQT3cd4PFXN3LZ6ZNDvnAkmpKHkkdMpWKc\nm7bX8sMH32L88HK+e8EsoPs4/d/6LzjFsPCI4by6ehsDyws5rLKC9Vv2UbWjjkeeW8+U0RUUFeSy\n8sNdXLv4cH722OrANeZNHUJLqyfwTbMvi6Qq7f5vHc/19y3jYEMLZx83lhNmDqe4IJcX39nKH5da\nPn/cWE6fNxpwurp++5evs+9AEw/csDDkQVxT20BWlvMtuz3/79zj8bJpRy2jh5SSnZV6owxS8W+o\nM7FKHmowl5S1fY/TD//jrV03EoNT7RT8INq9/xDrqmr43b+c7o4lhbkhfejXVbX1vQ9OHEBU3S/T\nyajBJZy/aGJIr6tzFo7j0Rc3BLaHDSzmxFkjWLpiMx6vl/nThgaSx48un8PBhhaGDyym7lAT37n/\nDQDyc7O548p5HDjUxKCKosC1Fh4xnAUzhoX8XrKzsvjJ1fNobvF0+AYf3K24K1lZLsYNK+/ZDZCY\nU/KQtPH2R9WMb2ylLD+bhqYWNm6rpa6+mV//Y11IHfLTyzfz9PK2vvldDb7qK2YZN9ecPQ2AO66c\ny6//sY4vnTSRgeUFPPriBhYcMZwvnzwxMK7j+BnDoF2FxOD+bYkhL7eAuVOGMH38AACKCnIoKuj4\nKOmsiic7K4vsvNQrNUj0lDwkpnbW1PP3ZZs478QJlIWp111XVcPIQSVdnuei7eGzs6aeX/zNGYew\n5MYTuPruV0LO/dn/hZYgMlnwILIlN54AwIat+7nV1x4zsLyA0+ZU8uxbn7J9Tz15OdmB97r7FfK9\nrxwZ2H78x2ewtyZ0pHWWy4X/1p+zYByHgkYrg5MULjtjcsx/LkkvSh59kMfj5b2PdzOpsiLQEN1b\n7328m937G3htzXY+2VHHgUPNfPPcGV2ev7X6AHf9+V0qSvN9I5fhoWc/Ys/+BooLcljWrvropv9d\nHni9a2/fGJh29VlTKcjL5u6/vhfYd8P5R2BGVdDU3MqufYcC+8cNL+ecBeN49KUNzJzoZsERw5lU\nWcFjL23gCws6H1EOdNtIDHDqnMre/yCSkZQ8+qBla7bzu6c/ZOZEN1/7/LSYXPNe3zf/Qf0KAVi7\nsYadNfUh1R3BanwD0PbWNfL4KxuZPLqC51dtieizbvz18vAnpZD2bS7t5WS7uOfrx1KQl82lP34R\ncL7xHzlpUMh5X/mMwYxyBgEuOnJkh+ucdNRIRg4qCZwzpH8R18To9yvSnpJHH7R9j/PNfe3GPXH9\nnBfe3kpdfRPHTR/GpMoKPF4vP/3Lu+w/2MyW6gOB8558vYonffMMZRr/lCFPvl7F469sZM6UwZQU\n5vLcW22J0uVyBdoM7v/m8ezcW8+owaUdrpXdTTdRcEoRU8cOiO0PINIFJY8+KDvbeQh1NhlcV3bU\n1LN7/yGmjml7OLXv5QQQ1EzBs285k8Atf38n1583g6oddSE9ndLRzRcdxc2/WxnRud/7ypGMHVYG\nOJPsmZH9GDO0rMNkfvOnto1LyM/L7pA48nOzaWxupaggusGPIvGk5NEHZWd1nTweeOp9xg4tY+HM\nESH7v+trc/j19Qv42WOrWbfJGdX7xUUTOCmoCmXX3kN0xj/PUrobNrA48PrX1y/gijtfCmyfPq+S\nzx83js0765gwZiCN9W1zQ2W5XIHRwFNGt80/NXpIKeefNLHbz/z+hUfy1oe7OGLiwBj9FCK9l7Q+\nc8aYs40xDwVtzzbGLDfGvGqM+X6y4uoLuhol2+rxsGzNDv74TNdz43y0ZV8gcQA88tz6kLED6eyw\nygq+++VZIftGuNuShQunaui6c6dz62Wzyc3J4pyFbY3R/pHNowaXUlbcdU+zQRVFlBXl+q5fErbR\neuiAYs6YPybQlVYkFSQleRhj7gFuJaSSg18B51lrjwVmG2OmJyO2vqCrR5AnqCQSPHtr8Ou7OilB\nfPTpvpjFliznLBjHt794BONHlHPc9KGB/Td8aWZgptpfXHccANPGDmDoACepnDq7kiU3nsADNywM\n7ItEtj9hKB9ImkpWtdUy4HHgCgBjTCmQZ62t8h1fCiwC3uv03RK15pZWfvjgKk6YOZwDDW09f976\ncBf9ywpwu0tDqrFu+9MqyorzOGrSoIh7QaWL9osEXbv48MCAt2DlJc6aGdefNwMvdPvNv7s5j0Qy\nUVyThzHmYuA6nPGqLt//L7LWPmqMOT7o1DIgeEa5OmBMPGPrazZtr2NL9QH+sNSG7PevUvbkXcM7\ntIHUHmzKuMQxdlgZ5504gRnjB7J190FOnDUi7HtcLlfMCwjzpw3ln69XMU29oyRNxTV5WGuXAEsi\nOLUWJ4H4lQJh60Lc7o7dGVNRT+Nct3EPT7z8Md/60iwK8nr3qwq3ANCHVTW4Kwp79Rmp6B93nsmD\nT73PYy86M+Tm5GThdpd2+zspLHTaK7KzXL36N9bdey///OGctXBCl+NgEikd/o7SIUZInzhjISV6\nW1lr64wxjcaYMUAVcApwc7j3pckMlj2O88b7XgPgqVc2cNz0Yd2ee7Chmc07DzB+eDk/e2w1J8wc\nzhET3IHj3a0KB/Dtn7/aoxhTxQh3ScjYEYCrzprK7t0H+OzsUdhPali7sYbmZk/Y38ehQ02A0xut\np7+7SH7vWST/33A6zASbDjFCesUZC6k0Q9mVwMPAcuBta21knen7gNZWT4d9H3yyl/qGZvxT6t/+\np7f5ySPv8M/Xq1i3qYafP7amw3sy2f87YXzIdllRLke1G6HtiHxsi1oxRLqWtJKHtfZl4OWg7TeB\nucmKJ5W1f9w9vfwTHn3JmUp7YHkBP7joKLbudia3q94fOs7izQ92drvudaZoP0fXuOGhU3f7Wy2i\nWb4mbVe6EUmAVCp5SBeCH3jNLZ5A4gDYvb+Br9/TVuXUvmn3V39fx3MZ0Oj9m+8sCAxu7MyYoW1F\n8cvPnMwln+35rK/+kdzlSVztTSTVpUSbhzgam1rJz8vu9pyWTqqwgr2xrm02Wrs5vacCAbjs9Mm0\nerxkZ2UFkuhn51ZSlJ8TkkSDu8rOmdxxGdJoetKePreSxqZWTpk9qsdxi2Q6lTwSoKa2gfseXxMy\nhXZ7r67exlV3v8wqW93hmCeo6BHNgLw7Hn4nukBTxJGmraHfjOrHMYc7g/a8vooklyu0SulHl8+J\n+NqRVEUVFeRywSkmMEOwiHSkkkecNDa3smXXAdzuUh55bj2rPqrmQH0zN3xpZsh571fVsKOmnlfe\n3QY406XPCnp4AuB1Gsh/8kh6JoNoXX32tMB65CFtFIHXrkBHAWhb5e6Wi48OX8JQQ4ZITCh5xMl9\nf1vD2k013FZSQFOLU9XU1NK2IltjcysejzcwYeAIdwkAW3cf6HAtL2RM4hg1uITCvBwOHzcgUO10\n/qIJzJkyhJfe2cq8qR2rnPxOmDWC51dtYXJlBfsOOl2PK4e0tXWMGFTS5XsXzRrB6g17OG2uFjcS\niQUljzhZ65s8sGp7bafHr77r5ZAvwbW+h2H1vgb21jVSUZrfdjCaLkIpauiAIq44cwojB5Xgcjkl\nh/5lBUyqrKDcN4ng6fNGd3ifN+gufXHRBL582mS8zS14vF5aWrwcPi6yEdpTxw7wNbqrplYkFpQ8\n4szbRT1J+7219W3zTT2/aktI8vjzCx/HI7SE+o9zpuMOakNwuVzMnjy4y/PLS/LYf6CJovy2NSyy\nXC4G9iukurqOLJcr0BYSKSUOkdhR8kiAxqYWAA42tPD4KxvDzqf0r+WfJCKsuPvp14+h7mATG7bt\nD0kckbjtsjnUHmwKrLAnIqlFf5lx1trq5SPfIL1dew/x5OtVHabRyFTlxXmUF+d12xbRlcL8nA4D\n/0Qkdagc3436hhZ+8+T7bI3iYb//QCN3/bmtcfv11ds6nFNT2/08U+lm9JC+MxmciDiUPHyeeHUj\nT7y6MWTfMys388a6Hdz918iXFXny9aqQdbprahs6nNNVO0gqOX5G6ESMt1/R+VgKM7If37/wqESE\nJCIpRMnD5x/LqvjHsqqQfS+/55QaDhxyJiB8+NmPWLtxT7fXaWkNnxg270z9aqsz54cupzKoovOp\nw79z/hEAfOMLhzM4A6d0F5HOKXl0Y/8BZ2purxe276nnuVVbIiiFhCYPjyf1Sxntfeu8GVSU5nPh\nqZO6Pa+8OC8wLcj08QNDpoAXkcym5BERb4dV9vYfbOKtD3dx4FAz37pvGX9Yamlp9fDKe9tDztu9\nv2O1VSozI/sxZXR/gLBriNz9tfkh2ycfPZLKwaWcPq+SaxcfHrcYRST51J0lQsHTYTS3tPKDJW9S\ne7CJEe5i9tY18tI7WykIM6lhOhg9NPLG7/brdvcryecHF6n9Q6QvyMjk4fV6OzzYene90O0r7gws\nQ8K23fWB1/9esTlmn5ksn507OoJzKjVpoEgfl3HVVq+8t41L7niR3d3MYOsXbnpzP4/H2+UMIZ4M\nmDokWElhbthzFh8/jmPDVGmJSGbLuOTx+6c/BGCl3dXteY+9vIHLf/IS1REkGS+wdGX6lipO1boU\nIhJjGZc8/NqvqNfeU284U4B8GOGCScvX7ex1TMmSne0KWWnv9iu12q+I9E5Gtnn4eTxe3li3g+nj\nB0ZUHRNs+56DcYoq8bxeqBxSxqbtdQBdtlf8xzkde0hdd+50crMz9juGiPRQRj8VXl+7gwee+oCf\nP7Y64vd88MledtTU86Cv+isTdNYuM3VM/5DtgrxsDh83sMN508YOYFJlRdxiE5H0lNElD/+yrx9v\n3R/xezJl0aVg5UV5TK4s4aV3trL4+LEAjBpcGlhz5NjDh3L1uUdw6ED3Y1LmTx3Czr3h24hEJPNl\ndPJoz+v10tLqJTcnowtcIb6wYBwLZ44gNyeLX/zHsRQVONV3wfNrffnkiZQU5oZNHpecPjmusYpI\n+siYp6jX6+1QPbNlV+gcUrc/9DZX3PkSrZ7IuuhmgtPmVAaSpT9xAIFZVHKys8jNSf/BjSKSWAkv\neRhjyoA/AWVALvBNa+0KY8wc4B6gGXjWWntLNNf93m9X0NzSlhSWv7+jwwSE633rajy/amtvfoSU\n8cVFE3jkufWdHjv7uLHMm9L1euD+NJsVu7GUItKHJKPk8U3gOWvtAuAi4Je+/fcD51lrjwVmG2Om\nR3PR7XvqQ+aR6m7m2k1B64qH69Kbyto3egc70rgZUF7Q5fHAdCvp++OLSBIlo83jbsC/GlIucMgY\nUwrkWWurfPuXAouAyBfS6IbXCzZoPEfwPFVevNTWN8XiYxIuP7djddOgfoXc9OWZlJfkd/KONm25\nQ9lDRKIX1+RhjLkYuA6nlsTl+/9F1tpVxpghwB+Ba3GqsGqD3loHjCGG7ni4815Uv/tX+nbJLSvO\nC7yeMrqCdVV7cWW5wiaOEModItIDcU0e1tolwJL2+40x04CHgW9Za1/zlTzKgk4pBfaFu77b3bPl\nT/Py0r+T2TVfmM7gQW23rKK8ENhLTnZWRPdloG9xpwFlBYHze3o/E01xxlY6xJkOMUL6xBkLyWgw\nnwz8FTjXWrsGwFpbZ4xpNMaMAaqAU4Cbw12rurquRzE0NLb06H3JVl6SF1ig6vDRFezZ09au0+j7\nmTweT0T3Zf6UwWzffYCTjxxJdXUdbndpj+9nIinO2EqHONMhRkivOGMhGQ3mtwH5wL3GmBeNMY/7\n9l+FUxpZDrxtrV0ZtwjSdCbc4KVh249VifYnKszP4YKTDYP7d768rIhIdxJe8rDWntXF/hVAQmbs\nS8/UAZNHO9OEdLroVJomRBFJT+lf+d+HDK4o4rpzpzPCXdLh2ClHj+ItW805C8YnITIR6Wv6ZvJI\n4y/p08YO6HT/uOHlPHDDwpiuoCgi0pWMmZ4kGqs+qk52CHGhxCEiidInk4eIiPSOkoeIiEQtbZPH\noTQdqxHswlMnce7Czhu4T5g5PMHRiIhELm2Tx7nffSowX9XqDXuSHE3PDOlfxNABnY+z+NJJE7nv\nuuPCXuP682Zw7eKOy8eKiMRTWve2WmWrMaMquOfRmMyfmHDdtW+7XC4K88P/eiaP7npmXRGReEnb\nkkcmyM3JCuk1fO0XVIIQkfSg5JFEQ/oXhYw5UUdbEUkXSh5JcvVZUynIywlZS3ygb/GmIZpvSkRS\nXFq3eeCC11ZvT3YUPZLtX/81qOQx3F3CjV+ayXB3cXKCEhGJUHonD2DJvz5Idgi90n6mlIkj+yUl\nDhGRaKjaKsk0Ga6IpKO0Th5NzZ5khxCVE2eNCHTPLSpwCn2DKwoBqBzS/QItF502Ka6xiYhEI62r\nrRqbW5MdQlT6leTxP5fOZs3GmkD11IhBJXzvK0d2OVjQb+qYzmfTFRFJhrROHt40q/PJynIxdEAx\nQweENoiPHVbWxTtERFJTWldbJSp3HGncgeql3nBpJIeIZIi0Th6JUlSQ0+nqfdHypvMqVCIiQdI6\neXgSVPRIhdoxrfMkIqkkrZPHKpuYFQG9xGblWlVbiUimSOvkkTBhMsecyYO7PPbDK+bGJASlHRFJ\nJUoeESgqyOlxz66SwrwYRyMiknxKHhE4Y/5ohg3ser6p7tJKzBrJ1eghIikk4eM8jDFFwMNABdAI\nfNVau90YMwe4B2gGnrXW3pLo2LpSXJDLGfNG89Qbn0T9Xo/Hy5Wfm8JDz37EnCldV2+JiKSTZJQ8\nLgPestYeDzwEfMe3/37gPGvtscBsY8z0JMQW4tvnzeCnX5sPQF5udpfndVel5fXC0YcN5t5rj6Vf\nSX6PY1G5Q0RSScKTh7X2XuBW3+YoYJ8xphTIs9ZW+fYvBRYlOrZpY0OnADlsdH/Ke/HAh9To5isi\nEmtxrbYyxlwMXIfTLODy/f8ia+0qY8xzwDTgJKAMqA16ax0wJp6xdSYvt2e5tLsE0duxKJVDSvlk\nR11E65mLiCRKXJ9I1tolwJIuji0yxhjgKWAGTgLxKwX2xTO2zhS16xnldnc/061ffpgHe6TX6cy9\n31pIY1MLRQW5Pb5GpHoTZyIpzthKhzjTIUZInzhjIRkN5jcBW6y1fwQOAi3W2gPGmEZjzBigCjgF\nuDnRsbm8oVO8V1fXRfS+hsaWLo95vN6Ir9Odg3UNvb5Gd9zu0pjEGW+KM7bSIc50iBHSK85YSEZd\nyAPAg74qrSzgQt/+q3B6YWUBz1hrVyY6sJzs2DUBZWe5aPV4KS3SOA8RyTwJTx7W2l3AqZ3sXwHE\nZjh2onXSrvGjy+fw8db9jB5alhbfRkREohE2eRhj8oBzgKN8u1YCj1prm+IZWLob2K+Qgf16P427\niEgq6raexhgzAFgFXIszeM8FfANY5TuWUXo6caF644pIXxOu5PFj4E/W2juCdxpjvuc7dkm8Aksr\nyh4i0seEayE+un3iALDW/g9wTHxCSp7cnB6O84hxHCIiqS7c07K7wQWtsQwkFZw2tzLZIYiIpIVw\nyWOrMWZh+53GmBOBzfEJKXlKCns2EK+n07WLiKSrcG0eNwJ/N8b8CnjTd/484CKcgXwZo19J9OMx\nslyuhC2FKyKSSrotefgG6i3CmWfqTuBHwCDgGGvt6viHlzinzQlfZTW4oq3r7SzjDlnj44bzj2D8\n8PK4xCYikmrCjvOw1r6PU9LIWHdePY/+ZQVhz7vt8jl877cr2L6nHheh6zOZURVcc/ZUrvvFsvgF\nKiKSIrpNHsaYfJzEsQN4AfgrTrXV28Dl1tqP4h5hHC2cOZzFx42jqCCygfYulwtXJyv6qeZKRPqa\ncA3mD+BMmX4F8DLwLk4X3X8Av45vaPGXneXqkDg+M3tUxO+/+LTDGOEu4ZyF42IdmohISguXPGZY\naxcDnwNGWGtvtNauttbeDbjjH158ZWd1LEWcu3B8t+9ZdOQIAOZOHULlkFJuueRohg7oen1zEZFM\nFK6+phnAWttkjNnS2bF0NnVs9DOsLJgxnDmTB1OQp8WZRKTvClfy8HbxurPttDNldP8evU+JQ0T6\nunBPwRnGGP9IclfwazIgecSaboiI9BXdJg9rbexWRxIRkYzR4+RgjFkTy0Dibfq4AcyfNiTZYYiI\nZITeVN6PjlUQibB4wThGuEtYtmZHskMREUl7vamWUhW/iEgflTHdhgaUFbCntiGwXVaUS219z3oT\nHzVpEG4tISsi0qVw05N46LyEkXK9rT4zexQPPds2W8oPL53NN372Wpfnd7fg7FVnTe1RDIW+Lrxl\nxdHP0Csikk7ClTy+Zq39JYAxZqq1dq3/gDHm3rhGFqX2a2qUFoV5gPdsufJu5edlc8eVc3u8LoiI\nSLoI1+ZxadDrP7Q7dmyMY+mVziYsvPtr87s+Px7ZA3D3K6QwP2NqA0VEOhUuebi6eN3ZdsrpV5Lf\n5bFOco2IiEQomq/IMZ2exBgzCVgODPLNnTUHuAdnzqxnrbW39Ob67SlXiIjETjRzW8WMMaYUZ2XC\nhqDd9wPnWWuPBWYbY6bH8jNTqnVfRCTNhSt5TDHGbPS9Hh702gUM7cXn/i9wE/B3CCSTPGttle/4\nUpzlb9/rxWd0q7Pp2EVEJDLhksfE3lzcGHMxcB2hX/w3A49Ya9cYY/xP8DKgNuicOpx102PG/0Gn\nHD2SpW9+qnosEZFeCDcx4ie9ubi1dgmwJHifMeYj4BJjzKXAEOAZ4AycBOJXCuyL5rNK2zWOu92l\nIdsV/Ytxu0spLHS68Ga5XB3OiZdEfU5vKc7YUpyxkw4xQvrEGQsJ71NqrQ2UZowxm4CTrLXNxphG\nY8wYoAoBtDdvAAANOElEQVQ4Bbg5muvWHWgM2a6urgvZ3ltzkKJsF/X1TYBTFGp/Tjy43aUJ+Zze\nUpyxpThjJx1ihPSKMxaSPSDBS1sF0pXAwziN+M9Ya1cmLSoREelWUpOHtXZs0Os3gblx+7B2AzvU\n5CEi0nMZs9hT2EF/XnXWFRGJlYxJHpFSDhER6b2+kzx8RROvr9ewpicREem5jEkeEZcoAucpe4iI\n9FTGJA8REUmcjEkeqoYSEUmcjEke4aqt/LnF225bRESilzHJI5z2uUUlFRGRnuszySNAXXVFRHot\n45OHf+r14oJkz8QiIpI50vaJOtxdzNbqg2HP+/FV89ix52C3S9KKiEh00rbkMaC8sMtjv75+QeB1\nRWk+h43uH9j2qt5KRKTX0rbk0ZmB5QUMGVBEbk7XOTHQ20ot5iIiPZZRyeOOK+cqKYiIJEDaVlt1\nRolDRCQxMip5RERNHiIivdbnkkdbm0dSwxARSWt9Lnn4KXeIiPRc2iYPLeokIpI8aZs8ekxZR0Sk\n1/pe8vBTo4eISI/1ueShcoeISO+lbfLo6TQj/lorlTtERHoubZOHiIgkT1KmJzHGbAE+8m2+Ya39\nT2PMHOAeoBl41lp7SzJiExGR8BKePIwx44BV1trPtTt0P3C2tbbKGPOUMWa6tfa9rq7T805TqrcS\nEemtZJQ8ZgEjjDEvAPXAdcAOIM9aW+U7ZymwCOgyefSU2jxERHovrsnDGHMxTnLw4jyvvcA1wG3W\n2seMMfOBh4Czgdqgt9YBY+IZm4iI9Fxck4e1dgmwJHifMaYQaPEdX2aMGYqTOMqCTisF9kXzWSUl\n+bjdpWHPKyjIBSA7Oyui82MhUZ/TW4ozthRn7KRDjJA+ccZCMqqtvg/UAD8xxkwHPrXW1hljGo0x\nY4Aq4BTg5mguWlfXQHV1XdjzDjU0A+DxeCM6v7fc7tKEfE5vKc7YUpyxkw4xQnrFGQvJSB63A38y\nxpyGUwK50Lf/KuBhnO7Dz1hrVyYhNhERiUDCk4e1dj9wRif7VwBze3rdiBeC0hBzEZFey5hBgt4I\n++76R6ZraisRkZ5L2+QRabIQEZHYS9vkUVKYF7Kt9ctFRBInbZPH1V84PGQ74pKIBgmKiPRa2iaP\nAeWFvbyC0oeISE+lbfLoKbWUiIj0Xp9LHn5qIhER6bk+lzzUSUtEpPf6XPIQEZHey5jkMWfKkAjP\nVNFDRKS3MiZ5lBTmRnRefp4zI0tRQVIWURQRyQh97gl69rFjaGxq5cxjRic7FBGRtNXnkkdpUR6X\nnTE52WGIiKS1jKm2EhGRxFHyEBGRqCl5iIhI1JQ8REQkakoeIiISNSUPERGJmpKHiIhETclDRESi\nlhHJY0BZfrJDEBHpUzIieZw2pzLZIYiI9CkZkTxERCSxEj63lTEmC7gbmAXkAd+31i41xswB7gGa\ngWettbdEek1Nsi4ikljJKHlcAORYa48FzgYO8+2/HzjPt3+2MWZ6EmITEZEIJGNW3VOAtcaYf/q2\nv26MKQXyrLVVvn1LgUXAe5FcUMuRi4gkVlyThzHmYuA6QmuWqoFD1trTjTHHAb8Hzgdqg86pA8bE\nMzYREem5uCYPa+0SYEnwPmPMI8A/fcdfMcZMAPYDZUGnlQL7wl0/K8uFx+NlQP9i3O7S2AUeY6kc\nWzDFGVuKM3bSIUZInzhjIRnVVq8BpwGP+9o1NltrDxhjGo0xY4AqnKqtm8Nd6PtfPZKX39vG5JHl\nVFfXxTPmHnO7S1M2tmCKM7YUZ+ykQ4yQXnHGQjKSx2+A+40xb/i2r/T9/yrgYZxG/GestSvDXWjU\n4FIuONnEJ0oREelSwpOHtbYJuKST/SuAuYmOR0REoqdBgiIiEjUlDxERiZqSh4iIRE3JQ0REoqbk\nISIiUVPyEBGRqCl5iIhI1JQ8REQkakoeIiISNSUPERGJmpKHiIhETclDRESipuQhIiJRU/IQEZGo\nKXmIiEjUlDxERCRqSh4iIhI1JQ8REYmakoeIiERNyUNERKKm5CEiIlFT8hARkagpeYiISNRyEv2B\nxpgbgM8AXqACGGytHWaMmQPcAzQDz1prb0l0bCIiEpmElzystXdYaxdaa08AtgAX+A7dD5xnrT0W\nmG2MmZ7o2EREJDJJq7YyxnweqLHWPm+MKQXyrLVVvsNLgUXJik1ERLoX12orY8zFwHU4VVQu3/8v\nstauAm4EzvOdWgbUBr21DhgTz9hERKTn4po8rLVLgCXt9xtjDgP2Wms3+nbV4iQQv1JgXzxjExGR\nnnN5vd6Ef6gx5utAtrX2nqB9bwOLgSrgn8DN1tqVCQ9ORETCSlabx0RgY7t9VwIPA8uBt5U4RERS\nV1JKHiIikt40SFBERKKm5CEiIlFT8hARkagpeYiISNQSPrdVbxljXMAvgelAA3Bp0HiRZMW0Ctjv\n29wE3Ab8HvAAa6211/jOuwy4HGf+rluttU8lKL7ZwO3W2oXGmHGRxmaMKQD+BAzCGYvzVWvtngTF\nOQOny/ZHvsP3W2sfTWacxpgcnHFLo4E84FbgfVLsfnYR56ek3v3MAn4DGJz7dyXQSArdzy5izCPF\n7mVQvIOAt3Bm6GgljvcyHUseZwH51tp5wE3A3ckMxhiTD2CtPcH33yW+mL5rrT0eyDLGfM4YMxj4\nOjAXZ2LIHxljchMQ37dx/vHn+3ZFE9tVwGpr7XHAH4H/SmCcs4C7gu7roykQ55eB3b7P+QzwC1Lz\nfgbHeaovzpmk3v08A/Baa4/xfcZtpN797CzGVPy36f/S8Cug3rcrrvcyHZPHMcC/Aay1K4AjkxsO\n04FiY8xSY8xzvm/PM621r/qOPw2cBBwNvGatbbHW1gLrgcMTEN/HwNlB27MijG06Qffad2485xvr\nECfwWWPMy8aY3xhjSlIgzr/S9keVDbQQ+e86WXFm4XzDnAWcnkr301r7d5xvwACVwF5S7H62i3G0\nL8aUu5c+d+JMMLsNZzqouN7LdEweZbRVEQG0+IqWyVIP/MRaewpO9n4I5xfnV4cTcymhcR8AyuMd\nnLX2cZyHnF80sQXv95+bqDhXAN/2fWvaCPyAjr/7hMZpra231h70TeT5KPCfpOD97CTO7wFvAten\n0v30xeoxxvwO+BnOIOFUvJ/+GO/F+fteQYrdS2PMhcAua+2ztN3D4OdizO9lOiaPWpwf1C/LWutJ\nVjA49Z4PAVhr1wN7gMFBx/3zdKXK/F3B96q72PYSeq8THe8T1tp3/K+BGTj/uJMapzFmJPAC8KC1\n9s+k6P3sJM6UvJ8A1tqLcGad+C1Q2C6elLif7WJ8JgXv5UXAScaYF3FKEn8A3O3iiem9TMfksQw4\nDcC3gNSa5IbDRcBdAMaYYTi/mGeMMcf7jp8KvAqsBI4xxuQZY8qBScDaJMT7tjHmuAhjex3fvfb9\n/9X2F4ujfxtj/FWSJwKrkh2nr754KfAda+2Dvt3vpNr97CLOVLyfFxhjbvJtNuA08L4Vxd9O3OPs\nJEYP8DdjzFG+fSlxL621x/vWSVoIvIuzTtLT8fy3mXbTkwT1tvK3F1xkrf2om7fEO57gni1e4Ds4\npY/fArnAB8Bl1lqvMeYS4AqcYuWt1tonEhRjJfCItXaeMWYCTsN02NiMMYXAg8BQnF4w51trdyUo\nzunAfUATsAO43Fp7IJlxGmPuAc4FPqRtiYFvAD8nhe5nF3HehPMlJ5XuZyFOb6AhOD0/f+SLOaK/\nnUTE2UWMm3GeQSlzL9vF/AJOrzAvcfxbT7vkISIiyZeO1VYiIpJkSh4iIhI1JQ8REYmakoeIiERN\nyUNERKKm5CEiIlFT8hDxMcZ4fP8vM8Y8HsPrvhD0+u1YXVckmZQ8RNr4Bz31x5niIVYW+F9Ya2fG\n8LoiSZN263mIJMC9wDBjzGPW2sXGmK/gjCR34UxFcY21tskYU42zdsJgnNlKfwlM8W1bYDFwB4Ax\n5g1r7VxjjMdam+Ub0fsbnCTVijPF9x+NMV/FmSq7PzAWZx6laxL3o4tERiUPkY6uBbb5Esdk4FJg\nrq/UUA1c7ztvAHCbb/9coNFaOx+YABQBp1prvwFgrZ3re4+/dPPfOGtuTMOZH+lmY8xU37G5ONPT\nHw6cYYyZEsefVaRHVPIQ6d5CYDyw3DevWi5O6cPvTQBr7avGmD3GmKtxJpsbD5SEue7FvvfuMcY8\ngVO9VQe8bq2tBzDGbMQphYikFCUPke5lA3+11v4HgDGmmLa/G6+1ttG3/0yc0sRPcSbKHEjo2hTt\ntS/1ZwVdtyFovzfMdUSSQtVWIm38D+kW2h7kLwFnG2PcvpLH/TjtH8Hng1P19Bdr7R+AXcBxOIkH\nQhcs87/nBeASAGPMQOBzvs8SSQtKHiJt/O0RO4FPjTHPW2tXA7fgPOzX4Dz8b293PjiN3+cbY97E\nWUf678AY37F/AO8ZZ717/3tuAQYYY1bjJI3/sda+201MIilFU7KLiEjUVPIQEZGoKXmIiEjUlDxE\nRCRqSh4iIhI1JQ8REYmakoeIiERNyUNERKKm5CEiIlH7//qG6IqI7z5UAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11adb8990>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(advi_fit.elbo_vals)\n",
    "plt.xlabel('Iteration')\n",
    "plt.ylabel(\"ELBO\");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ADVIFit(means={'alpha_log_': array(2.0496932971415927), 'beta_log_': array(3.6787438305091418), 'p_logodds_': array(-1.6862302056890575)}, stds={'alpha_log_': 0.35386626709018154, 'beta_log_': 0.3674171620327436, 'p_logodds_': 0.33450676412542907}, elbo_vals=array([-35.22716538, -23.25196236, -37.7726848 , ...,  -4.28355296,\n",
      "        -4.70956574,  -4.35644914]))\n"
     ]
    }
   ],
   "source": [
    "print advi_fit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get a good look at the approximate posterior, we can use the `sample_vp` function, which draws Gaussian samples in the real-space, then transforms the samples back to latent variable space:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "advi_trace = pm.variational.sample_vp(advi_fit, draws=5000, model=model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare with the NUTS sampler:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " [-----------------100%-----------------] 5000 of 5000 complete in 14.9 sec"
     ]
    }
   ],
   "source": [
    "with model:\n",
    "    step = pm.NUTS()\n",
    "    nuts_trace = pm.sample(5000, step)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAecAAAHwCAYAAACc3qd3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3XuUZHV57/93V9+mu7p6eqa7MeJdSR5JgqOCAcdhADPE\nEEHGc3AlnFwxBIN4C5EoejAXj5ccdIxKkMRRlCxzfuIcB5SRAYmoMBoSkbMckvEZdOIFGJjpS/Wt\n+r7r98eu7qmuqb5VV1d9q/vzWqvXqu/+7l31VHXvfmrv/d3Pty6bzSIiIiLhSFQ7ABEREZlLyVlE\nRCQwSs4iIiKBUXIWEREJjJKziIhIYJScRUREAtOwlJXM7Gzgw+5+gZm9CPgcEAGPuvs1qxifiCyR\nmdUBNwNbgDHgSnc/UrBOK3Av8EZ3P5xb9m7gdcT/D25y99sqGriInGTRI2czuw74NNCcW7QLeI+7\nnwckzOzSVYxPRJZuJ9Ds7luB64n31VlmdibwLeCFecvOA16Z2+aC/D4RqZ6lnNb+EfD6vPaZ7v5A\n7vHdwI6yRyUipdgG7Adw94eAswr6m4gT+A/zlr0GeNTM7gC+kvsRkSpbNDm7+15gKm9RXd7jIWBj\nuYMSkZK0AwN57Skzm93H3f277v4Ec/fhLuBM4DLgauCfKxGoiCxsSdecC0R5j1NAerENstlstq6u\nbrHVRGRu4lyuQeJ9ckbC3aP5Vs7pBQ65+xRw2MzGzKzL3Xvm20D7s8iSlbyjlJKcv29m293928BF\nwDcW26Curo7jx4dKeKnK6e5OKcYVCj0+CD/G7u7U4ivN7wBwMbDHzM4BDi5hmweBtwEfM7NTgVbi\nhD0v7c8rF3p8EH6MoccHK9ufS0nO7wQ+bWaNwCFgT8mvLiLltBe40MwO5NpXmNnlQNLdd+etNzvb\njbvvM7NzzezfiL/lv9ndNRuOSJUtKTm7+0+BrbnHjwHnr2JMIlKCXFK9umDx4SLrvbqg/e7VjEtE\nlk9FSERERAKj5CwiIhIYJWcREZHAKDmLiIgEppTR2iIiSxJFEel0/5xlHR2bSCR0XLDWFfvdr9R6\n+ttRchaRVZNO97PnvoO0trUDkBke5LIdZ7B5c2eVI5PVVvi7X6n19rej5Cwiq6q1rZ22VEe1w5Aq\nqObv/gtf+Dy33/5/2LPnqzQ2NvLBD/417j9k48aNTE1N0dGxibe97Vra2zfyO7/zem6//U42bNgw\nu/0b3/i7vP/9f8uHP/x+rrvuPTz3uc+raPzr4/yAiIisK/feu58dO17DfffdM7vsmmvezic+cQs3\n37yb3/7t3+WGG95Fa2srr3rVdu6//77Z9dx/SCq1kWc969nVCB1QchYRkTXmkUce5tnPfjY7d/53\nvvzlLxVdZ8uWl9LQ0MgTTzzOJZdcyv79+2b79u27k0sv/W+VCrcoJWcREVlT7rrrDi6+eCfPec5z\naWxs5D//89Gi623atJmBgTS//Mu/ytDQIMePH2NycpKHH/53tm8/v7JBF9A1ZxERWTOGhob47ne/\nQ39/mj17vsjIyAj/9//eTn19/UnrPvXUUU455RkAvPa1l7J//9c49dRTedWrttPQUN30qOQsIiKr\nIjM8WPHnuueefVx88aW8+c1vA2B8fIw3vOFSXvziXyabPTGny7//+7/S0tJCV1c3AL/xGxdx7bVv\noauri7e85c/KFneplJxFRKTsOjo2cdmOM8r+nIvZt+8r3HDD38y2m5s3cN55r2bfvjs5duxpvvCF\nz1NXlyCZTPLXf/2h2fVSqRTPe97z6evrmzMQrFpzl9flf5NYRdlamHdTMa5M6PFB+DF2d6eq859g\neZa8P/f19fK1f/3p7O00w0Npfuuc5636vao18HsOOj4IP8bQ44OV7c8aECYiIhKYNXlaWyUDRUSk\nlq3J5KySgSIiUsvWZHIGlQwUEZHatWaTs8h6Y2Z1wM3AFmAMuNLdjxSs0wrcC7zR3Q/nLT8F+B6w\nI3+5SKk0K9XKKDmLrB07gWZ332pmZwO7cssAMLMzgVuAZ+VvZGYNueWZCsYqa1w63c8dB+8i2d5W\nlucbGRxm5xkXr5vLk0rOImvHNmA/gLs/ZGZnFfQ3ESfrfypY/hHgU8D1qx6hrCvJ9jbaNqYq9nqP\nPPIw11//5/zTP91Od/cpANxyy00873nP55ZbPsmdd56YBOOhh77Lv/zLvVx44W9y222fBeDRR3/A\nGWdsAeAtb3kHyWQbH//4R5iamiaTGWHLlpdx9dVvrch7UXIWWTvagYG89pSZJdw9AnD378Ls6W9y\nj/8IOObuXzez91QyWJHV0NjYxAc/+Nd87GN/X9BT/JbjV7zibF7xirMBuPTS3+QTn7hltu9977ue\nyy77HX7t184B4L3vvY4HHvgm5557/mqEPse6SM5RFNHfr1urZM0bBPIPU2YT8wKuACIzuxB4KXCb\nmb3O3Y8ttFF399KOhhKJCVpbm0gmmwGIppvo6krR2bn6R1NLjbFaQo8PVhZjIjFBy/EmWnO/+5Wa\nnhw/6W+nML6Ojla2bn0l2WyWe+/9Cr/7u79La2sTqdQGEom6Oetv3NjChg2Nc5YVrvOsZ/0C3/jG\nfp75zE5e8pKX8KlP/X3RGt2rYV0k59HMEPsO9LC5Kz7NoVurZI06AFwM7DGzc4CDi23g7ufNPDaz\n+4E3LZaYgSVXZurrGyKTmSBRPw5AJjNBT88QUdS0pO1LFXr1qNDjg5XH2Nc3xGhmgvrG8bLEM1rw\nt1MsvnQ6w/j4FNde+y6uuuoP+ZVfeTmZzARDQ2NEUXbO+gMDo4yPT81ZVrjOH//xNezdu4e//dsb\nOXLkx7zyla/iz/7sL2hrW9p19BV9uSl5yxrTmoxvrWpLdcze/yyyxuwFxs3sAPBR4M/M7HIzu7Jg\nvflq9laklq/Iamtvb+etb72W//W//nJ2sovCM6Wjoxmamxc+qv/+97/HG97wO9x00z/y5S/vo6Wl\nhc9//jOrFne+dXHkLLIeuHsWuLpg8Um3Rbn7q+fZvuhykVKNDA5X7ble9apz+fa37+drX/sqb37z\n23jmM0/l+9//Hi9/eTxO8qGHvstLXvLSgq3mfj/9+7//OM3NzWzZ8jI2bNjAc57zXAYGBqgEJWcR\nESm7jo5N7Dzj4rI/53K8/e1/zve//z0A3vWu/8lHPvIh/vEfbyaKIn7lV87gNa/5rYIt5g4ae//7\nP8THPnYjw8NDNDQ0cOqpz+ad76zMTQ1KziIiUnaJRKLi43pe9rIzednLzpxtt7Ym+dKXvjLb/uQn\n/2HB7e+8c/+c9nOf+/wio74rY91ccxYREakVSs4iIiKBUXIWEREJjJKziIhIYNbEgLDC2U/6+/tn\n720TERGpNSUl51xt3t2AAdPAn1Rzmrl0up899x2cLS7S89TjtG3sIqVaIyIiUoNKPa39G0DS3bcB\n7wc+WL6QStPadqICWEtb+DVrRURE5lNqch4DNuaOoDcCE+ULSUREZH0r9Zrzg0AL8EOgk7jYvoiI\niJRBqcn5L4AD7v5eM3sWcL+Z/aq7z3sEvZrToxVOS9fS0kR9Q+O87fmmrVvrU7hVQujxQW3EKCLr\nW6nJuY0Tk7qnc8+z4CSXqzk9WuG0dKOjE9TX1zEyUrxdbNq69TCF22oLPT4IP0Z9cRARKD053wjc\namYP5J7jencfLV9YIiIi61dJydnd08DryxyLiIiIoAphIiIiwVFyFhERCcyaKN8pIrOV+24GthDX\nIrjS3Y8UrNMK3Au80d0Pm1kD8Fng+UAT8AF3/2pFAxeRk+jIWWTt2Ak0u/tW4HpgV36nmZ0JfAt4\nYd7i3wN63H07cBFwU4ViFZEFKDmLrB3bgP0A7v4QcFZBfxNxAv9h3rLbgRtyjxPA5CrHKCJLoNPa\nImtHOyfqDwBMmVnC3SMAd/8uzJ7+Jrcsk1uWAr4EvLdy4YrIfJScRdaOQSC/islsYl6ImT0H+DJw\nk7t/cSkvtNRiKYXV++arzrcaQi/oEnp8EH6Moce3EkrOImvHAeI693vM7Bzg4GIbmNkzgHuAa9z9\n/qW+0FKrrBVW7ytWnW811EIluJDjg/BjDD0+WNmXByVnkbVjL3ChmR3Ita8ws8uJp3fdnbdeNu/x\n9UAHcIOZvS/Xd5G7j1ckYhEpSslZZI1w9yxwdcHiw0XWe3Xe43cA71jl0ERkmTRaW0REJDA6chaR\niomiiP7+/jnLOjo2kUjoOEEkn5KziFTMaGaIfQd62Nx1CgCZ4UEu23EGmzd3VjkykbAoOYtIRbUm\n22lLdVQ7DJGg6VySiIhIYJScRUREAqPkLCIiEhglZxERkcAoOYuIiARGyVlERCQwSs4iIiKBUXIW\nEREJjJKziIhIYJScRUREAqPkLCIiEhglZxERkcAoOYuIiARGs1KJrBFmVgfcDGwBxoAr3f1IwTqt\nwL3AG9398FK2EZHK05GzyNqxE2h2963A9cCu/E4zOxP4FvDCpW4jItWh5CyydmwD9gO4+0PAWQX9\nTcTJ+IfL2EZEqkDJWWTtaAcG8tpTZja7j7v7d939CaBuqduISHXomrPI2jEIpPLaCXePVmEburtT\ni60SP1ligtbWJpLJZgBaWpqob2icbUfTTXR1pejsXNrzLcdSY6yW0OOD8GMMPb6VKDk5m9m7gdfl\nnuMmd7+tbFGJSCkOABcDe8zsHODgKm3D8eNDRZdHUUQ63T/b7u/vZ2RknET9OACjoxPU19cxMhK3\nM5kJenqGiKKmpbzsknV3p+aNMQShxwfhxxh6fLCyLw8lJWczOw94pbtvNbMkcF3JEVRBFEX09/fP\nWdbZmaxSNCJlsxe40MwO5NpXmNnlQNLdd+etl11om5UEkE73s+e+g7S2tQPQ89TjtG3sItW+kmcV\nWX9KPXJ+DfComd1BfEqsppLzaGaIfQd62Nx1CgCZ4UH+tCtFPF5GpDa5exa4umDx4SLrvXqRbVak\nta2dtlQHACPDA4usLSLFlJqcu4DnEp8OeyHwFeDF5QqqElqTJ/6BiIiIhKTUUZm9wD3uPuXuh4Ex\nM+sqY1wiIiLrVqlHzg8CbwM+ZmanAq3ECXte5RxVd/I143Gmp0aJpuPT0tH0GK3J1nlHiBYbMVru\nGFdL6DGGHh/URowisr6VlJzdfZ+ZnWtm/0Z8z+Sbc9eu5lXOUXV9fb1zBp387MeH6E3+nE113QD0\nZp7mBRMvJ9UetwtHiBYbMVruGFdD6KMTQ48Pwo9RXxxEBFZwK5W7v7ucgSxX/qCTlmQbzclWWlJt\nAGwYGYTxakYnIiJSOlUCEhERCYySs4iISGCUnEVERAKj5CwiIhIYJWcREZHAKDmLiIgERslZREQk\nMErOIiIigVFyFhERCUzJFcJClo0iMiNDDA+lgXhKyGSqs8pRiYiILM2aTM7jo2McrT/E1PQgAL3Z\np3lB5uVVjkpERGRp1mRyBmhubVGtbRERqUk1mZyjKGJkeGC2nRkZItu64KRYIiIiNaMmk/PAwABH\nxh4m1bIJgCezP6Ftsr3KUYlUl5nVATcDW4Ax4Ep3P5LXfwlwAzAJ3Oruu3Pb7AYMmAb+xN0PVzx4\nEZmjZkdrz0wR2ZJqozm5odrhiIRgJ9Ds7luB64FdMx1m1pBr7wDOB64ys27gN4Cku28D3g98sNJB\ni8jJajY5i8hJtgH7Adz9IeCsvL7TgcfcfdDdJ4EHgO3ER9gbc0fQG4GJyoYsIsUoOYusHe3AQF57\nyswS8/QNEyfjB4EW4IfAPwCfqECcIrKImrzmLCJFDQKpvHbC3aO8vvyBGSkgDfwFcMDd32tmzwLu\nN7NfdfcFj6C7u1NFlycSE7S2NpFMNgPQ0tJEfUPjvO1ouomurhSdncWfbyXmizEUoccH4ccYenwr\noeQssnYcAC4G9pjZOcDBvL5DwGlm1gFkgHOBG4EzOXFEnSb+n1C/2AsdPz5UdHlf3xCZzASJ+vje\nxdHRCerr6xgZKd7OZCbo6RkiipqW+VYX1t2dmjfGEIQeH4QfY+jxwcq+PCg5i6wde4ELzexArn2F\nmV1OPOBrt5ldC9wL1AGfcfejZnYjcKuZPUD8/+B6dx+tSvQiMkvJWWSNcPcscHXB4sN5/fuAfQXb\npIHXr350IrIcGhAmIiISGCVnERGRwNTEae0oikin+2fbAwNpslmV6xQRkbWpJpJzOt3PHQfvItke\nT2Tx8//6GVNTVQ5KRERkldREcgZItrfRtjEelt6SbIH00geUan5nERGpJTWTnFdC8zuLiEgtWRfJ\nGTS/s4iI1A6N1hYREQmMkrOIiEhglJxFREQCs26uOecrHL09MjxAFEWLbCUiIlIZ6zI5F47eHhrr\nJ50+nY6OX6hyZCIiIitMzmZ2CvA9YIe7H15s/ZDkj96emNDQbRERCUfJ15zNrAG4hXhuWBERESmT\nlQwI+wjwKeDJMsUiIiIilJiczeyPgGPu/nXiidtFRJYtiiL6+/vp6+ud/dHgTJHSrzlfAURmdiHw\nUuA2M3udux+bb4Pu7lSJLwWJxAQtx5toTTYD0NLaRMPQBI2NcfgNiXrqGxMltxsbEiuOsVJCjzH0\n+KA2YlwvRjND7DvQw+auU4C47v1lO85g82bVvpf1raTk7O7nzTw2s/uBNy2UmAGOHx8q5aUA6Osb\nYjQzQX1jPHBrNDPB1PQ0k5Px1FRT0TR1k/UltyenohXHWAnd3amgYww9Pgg/xpV8cTCzOuBmYAsw\nBlzp7kfy+i8BbgAmgVvdfXdu+buB1xH/P7jJ3W8rOYgStCbbaUt1VPIlRYJXjiIkmlhZJAw7gWZ3\n3wpcD+ya6cgN4NwF7ADOB64ys24zOw94ZW6bC4AXVjxqETnJiu9zdvdXlyMQEVmxbcB+AHd/yMzO\nyus7HXjM3QcBzOwB4Dzg5cCjZnYHkAKuq2zIIlLMuixCUiibzdLf308UNc4u6+jYRCKh6qZSU9qB\ngbz2lJkl3D0q0jecW9YFPBe4mPio+SvAiysTrojMR8kZmJqa4CvfdDqfMQpoUIrUrEHio98ZM4l5\npq89ry8FpIFe4JC7TwGHzWzMzLrcvWehF5rv2ngiMUFraxPJmcGbLU3UNzQuuR1NN9HVlaKzc+WD\n9kIf+Bd6fBB+jKHHtxJBJucoikin+2fb/f39ZLOre2m7pa1Ng1Kk1h0gPgLeY2bnAAfz+g4Bp5lZ\nB3HhoHOBG4lnNn8b8DEzOxVoJU7YC5pvUF1f3xCZzASJ+tzgzdEJ6uvrGBlZWjuTmaCnZ4goalru\ne5+jFgb+hRwfhB9j6PHByr48BJmc0+l+7jh4F8n2uLzmsSeeJrW5ndScL/4iUmAvcKGZHci1rzCz\ny4Gku+82s2uBe4lrE3zG3Y8C+8zsXDP7t9zyN7u7BnmKVFmQyRkg2d5G28b4W8fI4HCVoxEJXy6p\nXl2w+HBe/z5gX5Ht3r1aMUVRxPjoIMMtG4D4klEypctFIosJNjmLSO0bywzzOIcYnO4GoDf7NC/I\nvLzKUYmET8lZRFZVc7J1dga4DSOD8VVuEVmQ7hUSEREJjJKziIhIYHRaG8hGEZnRIYaH0gCMDA9o\nZhwREakaJWdgfHSMn/EfDE/3ATA01s/AwHPo6uqucmQiIrIeKTnnNCdbZgetTExoxIpIKaIoYmT4\nRJXQzMgQ2VbdNi2yXErOIlI2AwMDHBl7mFTLJgCezP6EtskTxYOyUURm5MQlJN33LFKckrOIlFX+\nrVPNyQ1z+sZHxzhaf4ip6UFA9z2LzEfJWUQqqrm1Rfc9iyxCt1KJiIgERslZREQkMErOIiIigVFy\nFhERCYySs4iISGA0WltEqqbwvmeVzhWJKTmLSNUU3ves0rkiMSVnkTXCzOqAm4EtwBhwpbsfyeu/\nBLgBmARudffdeX2nAN8Ddrj74UrGnX/fs0rnisR0zVlk7dgJNLv7VuB6YNdMh5k15No7gPOBq8ys\nO6/vFiBT6YBFpDglZ5G1YxuwH8DdHwLOyus7HXjM3QfdfRJ4ENie6/sI8CngyeW+YBRF9PX1zv4M\nDKTJZjXRhchK6bR2EdkoYmAgTV9f7+yyjo5NJBL6LiNBawcG8tpTZpZw96hI3xCw0cz+EDjm7l83\ns/cs9wXT6X7uOHgXyfb4tPTP/+tnTE2V/gZEJKbkXMR4ZpRvPfEdHqv7KQAjg8PsPONiNm/W7DkS\ntEEgldeeScwzfe15fSkgDbwNyJrZhcBLgdvM7HXufmypL5psb6NtY/yyLckWSI+u4C2ICCg5F5XN\nZqlvqKe+Kf54Eo31RJFO1UnwDgAXA3vM7BzgYF7fIeA0M+sgvra8HbjR3b88s4KZ3Q+8aSmJubs7\nTsaJxAQtx5toTTYD0NLaRMPQBI2N8b7TkKinvjGx5HZjQ4LOzrbZ51+JcjzHago9Pgg/xtDjW4ma\nSM7ZbJax8TFGR+Nv5OMT47CK17Wmpif5r6MDjNTHp+oG+/o49xlpurq6Vu01RcpgL3ChmR3Ita8w\ns8uBpLvvNrNrgXuBOmC3ux8t2H7JO9Xx40MA9PUNMZqZoL4xHmU9mplganqaycn43PZUNE3dZP2S\n25NTEb29w3R0DJXw9k/o7k7Nxhii0OOD8GMMPT5Y2ZeHmkjOY2Nj/OTYAP2TjQA8+UQfLW1tq/qa\njY2NNDVtyD1uXtXXEikHd88CVxcsPpzXvw/Yt8D2r16l0ERkmYJIzlEUkU73z7b7+/tPGvHZ0NQ0\nmywbmpoqGp+IVIYGY4rEgkjOhSM+jz3xNKnN7aTmjF+pHv3DEKkMDcYUiQWRnGHuiM+RweEqRzOX\n/mGIVE5LW+vs/wKR9aqk5JyrKPRZ4PlAE/ABd/9qGeMKjv5hiIhIpZR6Xvb3gB533w5cBNxUvpBE\nRETWt1JPa98OfCn3OEFcSF9ERETKoKTk7O4ZADNLESfp95YzKBERkfWs5AFhZvYc4MvATe7+xcXW\nX+hm7MIqQxtaGmloaihb1aGltIH5+xsStGw4Ed/05DhdXSk6Oyt/DTr0ijihxwe1EeN6lc1mmRgb\nny04NDo6qup8si6VOiDsGcA9wDXufv9StlmokkthlaGx0UkSUxGZkfJUHVpKu4HG+funIkbHJubE\n09MzRBRV9n7r0CvihB4fhB/jev/isFh1vsKaCKDbGmVtKvXI+XqgA7jBzN5HXPbvInfXTOkisiIL\nVecrrImg2xplrSr1mvM7gHeUORYRkUXl10QQWauCKUIiIlKosDpfsdK+ImuRkrOIBKuwOl9opX1F\nVouSs4gELb86X2ilfUVWi4Y4ioiIBEZHziWIooj+ft3OISIiq0PJuQSjwxnu6f0Gnd3x7Ru6nUOk\nOop9UQbo7ExWIRqR8lFyLlGrZqmSdaqvr4/+/riQSzqdruro6cIvyhB/Wb6y63LiCfNEapOS8xIU\nKynY2qBv5hIWM6sDbga2AGPAle5+JK//EuAG4olqbnX33aVM//rFh+9kKoofH3/yWNVHT+uLsqxF\nuki6BDMlBR97Is1jT6T50RO9jE+oGJoEZyfQ7O5biav47ZrpyCXhXcAO4HzgKjPrpoTpX5Mb20h1\ntJPqaKe1rbX870JEdOS8VHNKCjbodJkEaRuwH8DdHzKzs/L6Tgcec/dBADN7ENjOGpz+NYoi+vr6\niKLG2WUasCm1RslZZO1oBwby2lNmlnD3qEjfELBxLU7/OjqcYe8P7qY1FZ9q14BNqUVKziJrxyCQ\nf/F1JjHP9OVfGE4BaVj+9K/AnOldG5sbV29614LpWgunky1s5y/rfmY8k9VQa1PVpnhdSC3MQBZ6\njKHHtxJKzmWg+54lEAeAi4E9ZnYOcDCv7xBwmpl1ABniU9o3ljL9KzA7ferY6CST09nVm961YLrW\nwulkC9szy9qaGqo+xetCQp+6FMKPMfT4YGVfHpScy0D3PUsg9gIXmtmBXPsKM7scSOZGZl8L3AvU\nAbvd/aiZ/R2a/lUkOErOZaLbOaTa3D0LXF2w+HBe/z5gX8E2mv5VJEA67yoiIhIYJWcREZHAVOW0\ndhRFpNMnBlCttQnUNUBMJBzaH6UWVSU5p9P93HHwLpLtbcDam0BdA8Rkvchms4yNj82Wth2fGIcy\nftEuR+ncwv1xKD3IBc89l02bNs2uo2QtoanagLBke1vNTqC+lH8YGiAm68H4+Dg/OTZA/2RcjevJ\nJ/poaWsr2/PPlM4dqZ/5It+LvXD5/7by98eRwWHu+bGStYRNo7VLUK5/GCJrQUNT02xp24am8t9L\nvBqlcxdK1jrTJSFQRimRam2LrB060yWh0XkbERGRwAR55Lzag0xERERCFmRyHhsbW9VBJiIi89Gt\nVxKCIJMzrP4gk0rSzi5SO3QrpIQg2OS8lmhnF6ktGiAm1abkXCHa2UVEZKmUnMugHFWMRGRxIexr\nheWHoygC6kgk6maX6bKVrJSScxkstyiJrkGLlKYaBYAK99f+/n6++fMHZ8+EHXviaeqbGlRxTMpK\nyblMllOUZCm1fkE7tEgxlS4AVLi/zswFkF9hLNFYr4pjUlZKzlWyUPnAmWXaoUXCULi/Lmd9kVKU\nlJzNrA64GdgCjAFXuvuRcga23hTuzMVOfXd26jq2zG+x/dLMLgFuACaBW919d63vy4XXoCG+Dp1s\n31DFqOYq3Jfja9Rj9PWNzC5b6CxZ4TXuxdaXtaHUI+edQLO7bzWzs4FduWXC6k1zd1niIqKocXYd\n7aBSYN790swacu0zgVHggJndCWybb5taUHgNGuDY4z00bEhQnzv1Xe0BmsVOi7e1t9CaiqfILbys\nVTjArPAat65prw+lJudtwH4Ad3/IzM5aaOUf+EF+8vhTs+366bn9a61c50mDVh7v4QXPmibVEe+M\nS/1nUXgqbe8P7p53hwbtoLLgfnk68Ji7DwKY2QPAecArF9imJuRfgwagro4f/ayPvolmIIxZ4wr3\n5WR7C8n24pe1CgeYFbvGvdCUl8VGjxcuW2yEeRRF9Pb20tc3VLRfVl+pf7HtwEBee8rMEu4eFVv5\nyLGfkW4bm21P/GyE3one2WT81NGneKJ3iI1PxX8Ix37+czYkkzTmdrjMwCANzU0M9PauWnt6coLG\nxvI+/+gc6LjyAAAgAElEQVRQfNoqMzjEf4wMczQ9AUDf0ad54XM7qauLd4y+nj7qm+rnfGaFy/p6\n+kimNlDXEB85p/vSfPGpPWzc1AHAWGaUC55/Lhs3dhT7FVREOt1Gb2/Yc3OHHmN398tWsvlC+2Vh\n3zCwEUgtsM2SjKQHaGyME2El9tWF2jPLmls3zO5/EyNjDPT2s6E5/n9SbN+qdHt8fAOZzMSc/tmD\nk/Fx6rNTc9pTvX0nPd9Mf+H/gv5jvSQa62fbxZYVtgv/fwwMpPnuU/8KiYai/SEIfV+Gle3PpSbn\nQeKdesaCO/POc19bN2fB2SW+qgTvF3+x2hEsrhZiLNFC++UgcYKekQL6F9mmqD/4tTec2J+1L69Z\nr+GCaoewqDW8L5c8ZeQB4LcAzOwc4GDZIhKRUi20Xx4CTjOzDjNrAs4Fvgt8Z4FtRKRK6rIlXNvN\nG+H5ktyiK9z9cDkDE5HlKbZfEg8AS+ZGZr8W+EugDviMu9+ifVkkTCUlZxEREVk9GnonIiISGCVn\nERGRwCg5i4iIBEbJWUREJDBlLZtTSm3fcr5+GeK7HHh7Lr6D7v7mSsa3lBjz1vsHoNfd31PhEJfy\nOb4C+Giu+QTwB+4+GVB8rwfeA0TEf4e3VCq2gjjPBj7s7hcULK/qfjKfEOtw58qSfhZ4PtAEfAD4\nT+BzxL/fR939mmrFl8/MTgG+B+wApgkoRjN7N/A64pxwE/FteZ8jnPjqgN2AEX92f0JAn2H+vmxm\nLyoWl5n9CXAV8X79AXfft9BzlvvIeba2L3A9cZ3emeBnavvuAM4HrjKz7jK//kri2wD8DXCeu58L\ndJjZxRWOb8EYZ5jZm4BfrXRgeRaL8R+BP3L37cC/AC8ILL6Zv8NtwJ+b2cYKx4eZXQd8GmguWB7C\nfjKfRf82q+D3gJ7c39pvEieWXcB73P08IGFml1YzQJj9vd4CZHKLgonRzM4DXpn7vV4AvCik+HJ+\ng/iWwG3A+4EPEkiMRfblk+Iys2cAbyUul/ubwIfMrLHoE+aUOznPqe0LFK3tmzuKehDYXubXX0l8\n48BWdx/PtRuIjw4qbaEYMbNXAq8A/qHyoc2aN0Yz+yWgF7jWzL4JdFThvtkFP0NgAtgEtOTa1bif\n8EfA64ssD2E/mc9in2s13E58lgGgHpgCXu7uD+SW3U38RafaPgJ8CniS+D7zkGJ8DfComd0BfCX3\nE1J8EP8v3pg7gt5IfPQZSoyF+/KZBXFdCPwa8KC7T+Xq2z/GidoCRZU7ORet7TtP3xDxh1xJ88bn\n7ll3Pw5gZm8l/pZ2X4XjWzBGM/sF4iISbyHewatlod9zF/G3w08Q7yw7zOz8yoa3YHwQn3J/mLga\n1l0zk0FUkrvvJU4khULYT+az2Odace6ecfcRM0sBXwLey9x9o+qfn5n9EXDM3b/OidjyP7dqx9hF\nXKzmMuBq4AuEFR/EX1JbgB8SH5h8gkB+z0X25cK42jm5hv1Mbft5lXvHWm5t33SZX38xC9YRNrM6\nM7sR+HXgv1U4thkLxfgGoBP4GvBu4H+Y2R9UOD5YOMZe4Efuftjdp4iPtCp9hDVvfGb2HOLTS88j\nvk75DDP77xWObyEh7CfzWXYd7krI/U6/AXze3f8/4mt9M0L4/K4ALjSz+4mv198G5F+qqHaMvcA9\nuaO6w+SOUvP6qx0fwF8AB9zdOPEZNuX1hxDjjGJ/f8ver8udnJdT23c7cW3fSlqsJvg/El9T25l3\nervS5o3R3T/p7q9w91cDHwb+2d1vCylG4AjQZmYvzLXPBf6jsuEtGN8G4m+54+6eBY4Rn+KulsIz\nICHsJ/MJrqZ+7lrePcBfuPvnc4sfMbOZSwEXAQ8U3bhC3P08d78gN/Dv/wG/D9wdUIwPEl8HxcxO\nBZLAv+SuRUP14wNo48SRZ5r4suMjgcU44/tFfrf/Dmwzs6bcGJcXA48u9CTlnuR0L/E3xAO59hW5\nEdAztX2vBe4l/oe0292Plvn1S46P+DTnFcADuW+4WeDj7n5nKDGGMmqXxX/Pfwz8HzMD+I673x1Y\nfLcB3zGzUeDHxCMrqyULs3cKhLKfzOekz7WaweRcD3QAN5jZ+4g/z7cDn8wNuDkE7KlifPN5J/Dp\nEGJ0931mdq6Z/Rvx39zVwE+A3SHEl3MjcGtuHvIG4jOHDxNWjDNO+t26e9bMPkH8RaiOeMDYxEJP\notraIiIigVEREhERkcAoOYuIiARGyVlERCQwSs4iIiKBUXIWEREJjJKziIhIYJScRUQCZ2bPM7P/\nWmSdv8zd6y1rgJKziEhtUFGKdaTcFcKkBuRK3v1P4ko1zwYeIp6bt2JzLotIcWZWTzyD1a8CpwAO\n/Hle/63EM6u9nLhG8/vd/Qu57rNzFdxOBT7n7n+dmxTkM8Czcsu/7e5/WKn3I6XRkfP6dQ5wlbu/\nmHi2lyAmpBcRthLXft8K/CLQSq6meZ5nEU9D+OvAR83slNzyU4DziCebuc7MksBrgUfc/VXALwFb\nzexlq/82ZCWUnNev+9z9SO7xPwGvrmYwIhLLzQX8KTN7M/Bx4DTiiR/yfTo3ze0TxPWat+WW352b\nXaoXOA5szs3UdZ+ZvR34JLC5yPNJYJSc16/pvMcJis8tLCIVZmavI55TeRj4LPGsRj8tWC1/f63P\na+cvzwJ1ufnp/zfwNPE8yIeo7nzwsgRKzuvX+Wb2DDNLAH8AVHrmKBEp7teBL+amgz1GPG1ofcE6\nl0M8ipv49Hax6RJnEvAO4B9yR9B1wEuLPJ8ERsl5/XqS+Nv5o8DPgVCmoxRZ7z4N/I/cFI63AHcC\nFzB3tHabmX0P+CrwJ+7eX+R5Ztb/O+CvzOy7wA25bV6wWsFLeWjKyHUoN1r7Xe5eOMhERAKXG619\nt7vfXu1YZPXoyFlEpLboiGod0JGziIhIYHTkLCIiEhglZxERkcAoOYuIiARGyVlERCQwmvhCpMaY\nWR1wM7AFGCOetORIXv8lxPezTgK3uvvuXLGZTwMGRMCfuvt/mtlLgbuAw7nNP+XuX6rcuxGRYpSc\nRWrPTqDZ3bea2dnArtwyzKwh1z4TGAUOmNmdxJMpZN19W+4+9w/mtjkT+Ki7f6wK70NE5qHT2iK1\nZxuwH8DdHyKegWjG6cBj7j6YmwL0QWC7u98JXJVb5/nATEWpM4HXmtm3zGx3bhYjEakyJWeR2tMO\nDOS1p3KnrYv1DQEbAdw9ylWX+jhx6VaI5/K+zt3PA44Af7WKcYvIEik5i9SeQSCV1064e5TX157X\nlwLSMw13v4J4Tt/dZtYC3OHuj+S69xJPirCgbFy5SD/60c/iPyXTNWeR2nMAuBjYY2bnAAfz+g4B\np5lZB5ABzgVuNLPfB57t7h8iHkQ2TTwwbL+ZvdXdv0c8G9LDi714XV0dx48PlfUNlVt3dyroGEOP\nD8KPMfT4II6xVErOIrVnL3ChmR3Ita8ws8uBZG5k9rXAvcTTA37G3Y+a2R7gc2b2LeL9/u3uPm5m\nbwJuNrMJ4ClOXJcWkSpSbW0RWa5sLRyxhBxj6PFB+DGGHh9Ad3eqbvG1itM1ZxERkcAoOYuIiARG\nyVlERCQwSs4iIiKBUXIWEREJjJKziIhIYJScRUREAqPkLCIiEhglZxERkcAoOYuIiARGyVlERCQw\nSs4iIiKBUXIWEREJjJKziIhIYJScRUREAqPkLCIiEhglZxERkcAoOYuIiARGyVlERCQwDdUOQESk\nlkVRRDrdP2dZR8cmEgkd+0jplJxFRFYgne5nz30HaW1rByAzPMhlO85g8+bOKkcmtUzJWaTGmFkd\ncDOwBRgDrnT3I3n9lwA3AJPAre6+28wSwKcBAyLgT939P83sRcDncssedfdrKvpm1ojWtnbaUh3V\nDkPWEJ13Eak9O4Fmd98KXA/smukws4ZcewdwPnCVmXUDlwBZd99GnLg/kNtkF/Aedz8PSJjZpRV7\nFyIyLyVnkdqzDdgP4O4PAWfl9Z0OPObug+4+CTwIbHf3O4Grcus8H0jnHp/p7g/kHt9NnNRFpMqU\nnEVqTzswkNeeyp22LtY3BGwEcPfIzG4FPg58IddfV2xdEakuXXMWqT2DQCqvnXD3KK+vPa8vxYmj\nZNz9CjN7F/BvZvbLxNeai667kO7u1OIrVVmlYkwkJmhtbSKZbAYgmm6iqytFZ+fCr6/PcOVCj28l\nlJxFas8B4GJgj5mdAxzM6zsEnGZmHUAGOBe40cx+H3i2u3+IeBDZdO7n+2a23d2/DVwEfGMpARw/\nPlS2N7MaurtTFYuxr2+ITGaCRP04AJnMBD09Q0RRUxDxlSr0GEOPD1b25UGntUVqz15g3MwOAB8F\n/szMLjezK919CrgWuJc4iX/G3Y8Ce4CXmtm3iK8tv93dx4F3An+Te67G3HoiUmU6chapMe6eBa4u\nWHw4r38fsK9gm1Hgt4s812PEo7pliQqLjvT395PNZqsYkaxFSs4iIstQWHSk56nHadvYRap9kQ1F\nlkHJWURkmfKLjowMD8zpi6KI/n6V85SVUXIWESmj0cwQ+w70sLnrFEDlPKU0Ss4iImXWmlQ5T1kZ\nnWcREREJjI6cRUQWoNHZUg1KziIiC9DobKkGJWcRkUUsNDpbZDXomrOIiEhgdOQsIlJBURTR29tL\nX9+JutC6D1oKKTmLiFRQOt3Pvu84dYkNgO6DluKUnEVEKizZ1k6ivrXaYUjAdB5FREQkMErOIiIi\ngdFpbRGRPOUuOlI4EUb8fCsKUdYBJWcRkTzlLjpSOBFGz1OP0/3MZ5Js0zVnmZ+Ss4hIgXIXHcmf\nCENFTGQpdM1ZREQkMErOIiIigVFyFhERCYySs4iISGA0IEykxphZHXAzsAUYA6509yN5/ZcANwCT\nwK3uvtvMGoDPAs8HmoAPuPtXzeylwF3A4dzmn3L3L1XszYhIUUrOIrVnJ9Ds7lvN7GxgV24ZuSS8\nCzgTGAUOmNmdwGuBHnf/AzPbBPw/4Ku59T7q7h+rwvsIQrnvaxYpByVnkdqzDdgP4O4PmdlZeX2n\nA4+5+yCAmT0IbAduB2aOiBPER9UQJ+dfMrOdwGPA2919ZPXfQjjKfV+zSDnomrNI7WkH8m+WnTKz\nxDx9Q8BGd8+4+4iZpYiT9Htz/Q8B17n7ecAR4K9WNfJAzdzX3JbqoKUtVe1wRHTkLFKDBoH8DJJw\n9yivL/+YLwWkAczsOcCXgZvc/Yu5/jvcfSaZ7wU+sZQAurvDT2BLjTGRmKC1tYlkshmAlpYm6hsa\nV7UNzLaj6Sa6ulJ0dob3mYb+ew49vpVQchapPQeAi4E9ZnYOcDCv7xBwmpl1ABniU9o3mtkzgHuA\na9z9/rz195vZW939e8CvAw8vJYDjx4fK8DZWT3d3askx9vUNkclMkKgfB2B0dIL6+jpGRlav3ZZq\nmm1nMhP09AwRRU1levflsZzPsBpCjw9W9uVByVmk9uwFLjSzA7n2FWZ2OZDMjcy+FrgXqAN2u/tR\nM/s7oAO4wczeB2SBi4A3ATeb2QTwFHBVpd+MiJxMyVmkxrh7Fri6YPHhvP59wL6Cbd4BvKPI0/2A\neICZiAREA8JEREQCo+QsIiISGCVnERGRwCg5i4iIBEYDwkREAlJYThSgo2MTiYSOpdYTJWcRkYAU\nlhPNDA9y2Y4z2Ly5s8qRSSUpOYuIBGamnKisXzpPIiIiEhglZxERkcAoOYuIiARGyVlERCQwSs4i\nIiKBUXIWEREJjJKziIhIYJScRUREAqPkLCIiEhhVCBMRqaIoiujvP1FLu7+/n2w2W8WIJARKziIi\nVTSaGWLfgR42d50CQM9Tj9O2sYtUe5UDk6pSchYRqbLW5Ila2iPDA1WORkKga84iIiKBUXIWEREJ\njE5ri9QYM6sDbga2AGPAle5+JK//EuAGYBK41d13m1kD8Fng+UAT8AF3/6qZvQj4HBABj7r7NZV8\nLyJSnI6cRWrPTqDZ3bcC1wO7ZjpySXgXsAM4H7jKzLqB3wN63H07cBFwU26TXcB73P08IGFml1bs\nXYjIvJScRWrPNmA/gLs/BJyV13c68Ji7D7r7JPAgsB24nfhoGuL9fjL3+Ex3fyD3+G7ipC4Bi6KI\nvr7eOT9RFFU7LCkzndYWqT3tQP6Q3ikzS7h7VKRvCNjo7hkAM0sBXwLem+uvK1x31aKWskin+9lz\n30Fa2+J7rTLDg1y24ww2b+6scmRSTkrOIrVnEEjltWcS80xf/h2yKSANYGbPAb4M3OTuX8z1Txdb\ndzHd3anFV6qy+WIsLPqRSEzS0tJEMtkMQEtLE/UNjavaBpa8/tRkA4nEJInExGy8nd1dtG/cBMDQ\nYBNdXSk6O8v/Own99xx6fCuh5CxSew4AFwN7zOwc4GBe3yHgNDPrADLEp7RvNLNnAPcA17j7/Xnr\nP2Jm293928TXor+xlACOHx8qw9tYPd3dqXlj7OvrnXPkOVP0o76hFYDR0Qnq6+sYGRlftXZbqmnJ\n6/f29HLbV548qUjJTLyZzAQ9PUNEUVO5Pj5g4c8wBKHHByv78qDkLFJ79gIXmtmBXPsKM7scSOZG\nZl8L3Et8ynq3ux81s78DOoAbzOx9QJY4Gb8T+LSZNRIn9j2VfjPV0NpWW0U/VKRk/VFyFqkx7p4F\nri5YfDivfx+wr2CbdwDvKPJ0jxGP6haRgGi0toiISGCUnEVERAKj5CwiIhIYJWcREZHAKDmLiIgE\nRslZREQkMErOIiIigVFyFhERCYySs4iISGBUIUxEZBmiKGJ8dJDhlg1APCtUMhXOjFBRFJFO95+0\nvKNjE4mEjsdqhZKziMgyjGWGeZxDDE53A9CbfZoXZF5e5ahOKJxSEjStZC1SchYRWabmZCstqTYA\nNowMwniVAyqQP7GH1CYlZxGRMgr9tLfUBiVnEZEyCv20t9QGJWcRkTIL/bS3hE/JWURkASedph4Z\nItuarXJUstYpOYuILKDwNPWT2Z/QNtm+yFYiK6PkLCKyiPzT1M3JDVWORtYD3ZEuIiISGB05i4jU\nsCiK6O8/URGsv7+fbFbXxGudkrOISA0bzQyx70APm7tOAaDnqcdp29hFSpfFa5qSs0iNMbM64GZg\nCzAGXOnuR/L6LwFuACaBW919d17f2cCH3f2CXPulwF3A4dwqn3L3L1XkjQRitYuGZKOIzMgQw0Pp\n2edvTbaWNb7W5ImKYCPDA2WKXKpJyVmk9uwEmt19ay7Z7sotw8wacu0zgVHggJnd6e7Hzew64PeB\n4bznOhP4qLt/rKLvICCrXTRkfHSMo/WHmJoenH3+ZCZJqr27pPh6pp/iF47/0myCVwWytUkDwkRq\nzzZgP4C7PwScldd3OvCYuw+6+yTwILA91/cj4PUFz3Um8Foz+5aZ7Taz5OqGHqaZ0dgtqTY2JFvK\n//ytLSt6/vz46hJwtP4QP5l+hJ9MP8LPsv/BWGZ48SeRmqLkLFJ72oH8c5dTZpaYp28I2Ajg7nuB\nqYLnegi4zt3PA44Af7UaAUt5rTTZS/h0Wluk9gwCqbx2wt2jvL78oUApIL3Ac93h7jPJfC/wiaUE\n0N2dWnylKpsvxkRigtbWJpLJZgA2tDTR0FBPY2P877Chvp4NrQ3z9yfqqW9MzLbr6+qIolGi6QwA\nUTROfX1i3vUb6usBZp+/paWJ+obGeduLvX5hvIXbA0TTTXR1pejsXN7vLfTfc+jxrYSSs0jtOQBc\nDOwxs3OAg3l9h4DTzKwDyBCf0r6xYPu6vMf7zeyt7v494NeBh5cSwPHjQ6XGXhHd3anZGKMoIp2e\ne6vRyMg4ifq44PXY6ARTiWkmJ+OTClPT04yNTzEyMk9/NE3dZP1se2R4hKGpgwwP9QHwZOYntDW3\nz7v+1PQ01DP7/KOjE9TX183bXuz1C+Mt3B4gk5mgp2eIKGoq6TMMUejxwcq+PCg5i9SevcCFZnYg\n177CzC4Hku6+28yuBe4lTsK73f1owfb5N8G+CbjZzCaAp4CrVjn2ikun+9lz30Fa2+ITCqtxq9HM\naWZYvIJYNorIjM4dvb3aA7oK74UG6OjYRCKhK5uhUnIWqTHungWuLlh8OK9/H7Bvnm1/CmzNa/+A\neIDZmtbaFs6tRuOjY/yM/2B4Oj7SrsSUkoX3QmeGB7lsxxls3qxR3qFSchYRyXPSfcmrMAtVc7Kl\n4lNK5t8LLeFTchYRyVN4X7JmoZJqUHIWESmwnGvIIqtBowFEREQCoyNnEZGAnFRLexWueWv0dviU\nnEVEAlJYS3uxa97FJtZY7Nas0cwQ9/z4+3R2x+uNDA6z84yLNXo7IErOIiKBmamlHT9e+Jp3sYk1\nlnJrVmtbK20b126FrVqn5CwiNauw+teMzs7amb+jHLdu5Q9gq9StWbK6lJxFpGal0/3ccfAuku1t\ns8tGBoe5sutyoHipykpc010O3bolxSg5i0hNS7a3Lev07HKv6VaCbt2SQkrOIrLuLOeabq0pPE0O\nlanfLeWl5Cwia0oURfT19RFFjUA8C1UURYtstXYUniaHytTvlvJSchaRNWV0OMPeH9xNa27aqd7j\nvdRnfpH2jZurHFnl5J8mBw0Sq0VKziKy5iTbkyTb4+vQIyMjDDw+uKoTWYiUm5KziKxpo8MZfpZ9\nnJHpHiCMAWDrTbFb3lSRbGFKziKypmSzWUZHx0g0jgIwPjHOhtYNa3YAWC0ovOVNFckWp+QsImvK\n2NgYjx8dor0zPnX95BN9tLS1LbLV2lasxGdTtrKn9pd7y9t6p+QsImtOQ0MzTU3xEXJDU/FiJOtJ\n4Qju4+NPcPrEqVWOShai5Cwisg7kj+BuatnAxNg4o6Pxqf/R0VGiSIPkQqLkLCKyzkxNT/JfRwcY\nqY+T9WBfH+c+I01XV1eVI5MZSs4iUjMKR/329/eTrfC107WisbHxxKn/+kYGBtL09fXO9udPHqLR\n1pWn5CxSY8ysDrgZ2AKMAVe6+5G8/kuAG4BJ4FZ3353XdzbwYXe/INd+EfA5IAIedfdrKvU+SlE4\n6vfYE0+T2txOCt0atRLjmVG+9cR3eKzupwAMpQe5LHHRnCpr3/z5g7MDujTaevXpa49I7dkJNLv7\nVuB6YNdMh5k15No7gPOBq8ysO9d3HfBpoDnvuXYB73H384CEmV1akXewAjOjfts2pkimamdqyNC1\n5OZ3btuYIpFIsPcHd/P1n36Tr//0m3zt0L3UNzWc+Nzb1/fo90pQchapPduA/QDu/hBwVl7f6cBj\n7j7o7pPAg8D2XN+PgNcXPNeZ7v5A7vHdxEldhGR7Ul+CqkjJWaT2tAMDee0pM0vM0zcEbARw973A\n1ALPO7uurC/ZbHZ29PbMj67lV5euOYvUnkEgv5pDwt2jvL78C7ApIM388qdrWmzdWd3d1SkmkUhM\n0HK8idZkfGZ+Q0sjDU0Ns22AltYmGJqgsTH+99aQqKe+MRFUG6jY6y1pnWzEz48PMdE8BMDTT6Vp\n29jKKacW/5wnx0dJJCZJJCZmX2PTpvkHiBX+3qYnx+nqStHZubK/o2r9HVaCkrNI7TkAXAzsMbNz\ngIN5fYeA08ysA8gQn9K+sWD7urzHj5jZdnf/NnAR8I2lBHD8+FCpsa9IX98Qo5kJ6hvjKZbGRidJ\nTEVkRk5MuTSaiRPG5GR8kmAqmqZusj6odgONFXu9pW7T2NBEXSIeAFZXF3+BmPlcCz/nvmNp/vnn\nX6GzOx4QttgAscLf22hmgp6eIaKo9AIx3d2pqv0dLtVKvjwoOYvUnr3AhWZ2INe+wswuB5LuvtvM\nrgXuJU7Cu939aMH2+ecr3wl82swaiRP7nlWOXdaI1twAMlkdSs4iNcbds8DVBYsP5/XvA/bNs+1P\nga157ceIR3UHSfc114Yoiujv133Q5aTkLCLB0n3NtWF0OMM9vd9Y8mluWZySs4gELX82o5HB4SpH\nsz5ks1nGxsfm1N5ubVj4diqd5i4vJWcREZljanqSH/2sj76JeHT1sSd6sRcqXVSSPm0RETlJQ9OJ\n2tuNDas77aZqd59MyVlERKqqcGzBUHqQC557Lps2bZpdZ70layVnERGpusKxBff8eOkDzNbikbeS\ns4iIBGc5A8wKj7zXwmhxJWcRqVmFo4oBxifGyc4pgibrQf6R91qg5CwiNWtsbIyfHBugf7JxdtmT\nT/TRtkn3QYessGiJisucTMlZRGpaQ1PT7KjimbaErbBoyWLFZYpVIOvsnP++67VQsUzJWUREKi7/\nmvJixWWKVSC7sutyoPgXsbVQsUzJWUREgrfcCmS1XrFMyVlEgqGJLsKUzWaZGBtfVjlPWRklZxGp\nmmLJ+Js/f3D2iEcTXYRhanqS/zo6wEj9zAQkC5fz1ICvlVNyFpGKWWoy1kQX4WlsPFHOs6G+ccEj\n6eUO+JKTKTmLSMXMNwWkknFtWcqR9HIGfMnJlJxFpKI0BeTakH8kvdoTY6xHSs4iIiKLqHT9biVn\nERGRRVS6freSs4iI1JQoiujr6yOK4rKtpYwGX+xIuNjgxdZUsmL3Tis5i4hITRkdzrD3B3fTmopH\nf5cyGnyxI+H5Bi9WasS5krNIjTGzOuBmYAswBlzp7kfy+i8BbgAmgVvdffd825jZS4G7gMO5zT/l\n7l+q3LuRtaAaRUqS7UmS7UsbWFis1vZSjoQXGrxY+JxRFAF1JBInZkTr7i79KFvJWaT27ASa3X2r\nmZ0N7Motw8wacu0zgVHggJndCWybZ5szgY+6+8eq8D5kjVhukZJKK7zvGlZ+JFzsXu76poY59bzf\nZW8pOeZwPj0RWapt/P/t3XuMnFUZx/HvzM7M7na7bdcAokQjAXzkH4yWKFXK2ih4ARUTY2IiwSpg\nCBISBQIYLkYRFeUPNIihSFEhBjElmIaCQUzaYlAQkUJ5qGLl0jale6ezl9mLf5x3utPZ2e3sbebM\n9vdJNp33PfPOPnPpPPuec97nwBYAd3/KzE4vaTsV2OXu/QBmthXoBNaUHbM6uf9q4L1mdj6wC7jC\n3cT8HwMAAAnUSURBVA/W5mnIUhL7pVXltbYX4jK+8mu509mmBRuTbpz1s0SkaAXQV7I9ambpadre\nAlYC7WX7x5JjngKucvdO4BXgpsUKWkSqpzNnkcbTT0i2RWl3Hy9pK+2nawd6pjvGzB5y92LS3gTc\nXk0A042lVRrb6+iYnAGbTo/Q+maOZW3NALS0ZsnkMnPebl2WIzMwQjY7+VWWSTcBHNqXSTfRlE1H\ntV3L+OoRY1MmRSo1Tio1FgJIjdHa2rJg73tLa5ilPdfjK+0rDA+SThdIp0cASKcLtLRm5/w7xgrD\nzIeSs0jj2Q6cBzxoZmcAz5e07QRONrNVQB5YC9yatFU6ZouZXe7uTwMfB56pJoA33xyouL+7u+uw\nGa4Dvf2se/daOjo6gDAJJ39wmKZs+OIaGiyQHh0nf3Bu24P5EUbHxigURg/FMDo+RobsoX2j42Ok\nCk1RbdcyvnrEODQ0zEuvHKB7JCSq/a/u4cQTOmhKur17uvtZ1t5G2xzf96HBAstzmTl/birt697f\ny/2vPTylHngmiXkun835UHIWaTybgLPNbHuyvd7Mvgy0JTOzvwU8BqSAu919r5lNOSb59xvAHWY2\nAuwDLplvcOUzXB/9jxZAOBqVjkGTSkU9Yawopnrg8b06IjIjd58ALi3b/XJJ+2ZgcxXH4O7/Ikww\nWzQxfeFJ/cw0YWxiYoKh4SGtF11CyVlEROpqaGiI3fv76CmEseRYz6xr6eh+9iLSUMrPsIZHhmGW\nZRslTplcLupLsWpNyVlEGkb5GdaeN7ppXb68zlGJLDwlZxFpKKVnWJmczrAaUXm5T/WATKXkLCIi\nNVVe7lM9IFMpOYtItDTGvHSVzt5WD8hUSs4iEi2NMcvRSslZRKKmMWY5Gik5i8icjY+P09s7WUu7\np6eHCXU7i8ybkrOIzFlvb89htbRVnlNkYSg5i8i8lNfSFllo5RMD8/k86Syks2FhjaVY7lPJWURE\nojZlYuDufbS0tfK240P7Uiz3ubSejYiINLxKRUoyZZdeZXLTL6SxFCg5i4hIVGZbpKQ8mS+Fbm4l\nZxERic5sipSUJ/Ol0M3d2NGLiIgw83rRjUjJWURElrTy2d4Qf9e3krOIzMrm7VvI5wsAdHd1M9Je\nqHNEIjMrn+0NU7u+yxN4vZO3krOIzEpXdoDxFWkAenr7aRrV14jE5UizvWFq13d5At//+gFOPGGM\n9lWhoE6tk7X+V4lINLQKlSyEamZ7HzGBp1J1nWSm5Cwi0dAqVLJQjjTbu5oEfthjNGUPS+b5fJ5M\nc3bRusGVnEUajJmlgDuA9wNDwEXu/kpJ+2eB64ECcI+7b5juGDM7CdgIjAM73P2ymj6ZCrQKldTK\nfC7X2rN7H5nmZnpGk27wBT6zTi/YI4lIrZwPNLv7R4BrgduKDWaWSbY/AXwMuMTMjp3hmNuA69y9\nE0ib2edr9ixEGkwxmedyLWRyucO2F/ryLSVnkcZzJrAFwN2fAk4vaTsV2OXu/e5eALYCnRWOWZ3c\nf7W7b01uP0JI6jVTOsY8ODioMWZpWKVj2MWf+VC3tkjjWQH0lWyPmlna3ccrtL0FrATay/aPmVkT\nkCrZN5Dcd0av7/wfhdGQQHsOdLPsuPZDbd0HumnKNVW9vW/vPt7oGmDlvgEA9r/2Gi1tbWSTrsZ8\nXz+Z5hx9XV1VbRf3jRVGyGarO6Ye27WMrxFirHV8ixFTf3cXL/R1s7d3BICDff0wj34oJWeRxtNP\nSLZFxcRcbCtdTLkd6JnmmDEzGy+7b++Rfvl1F16ZOtJ9RGR+1K0t0ni2A58BMLMzgOdL2nYCJ5vZ\nKjPLAWuBvwJPTnPMP8zsrOT2pwnd4CJSZ6kJje+INJSSmdenJbvWE8aQ25KZ2ecCNxK6rO929zsr\nHePuL5vZKcBdQJaQ2C92d30piNSZkrOIiEhk1K0tIiISGSVnERGRyCg5i4iIREaXUolIVY5UNrRO\nMWWAXwHvAXLAzcCLRFaSFMDMjgOeJhR6GSOiGM3sGuBzhJzwc8IVARuJJ74UsAEwwmt3MRG9hmb2\nYeCH7r5uupK4ZnYxcAmhrO7N7r55psfUmbOIVGvasqF19BXggLufBXyKkFiiK0ma/BFxJ5BPdkUT\no5l1AmuS93UdcFJM8SXOIVyNcCbwPeAHRBKjmV1FuOKhOdk1JS4zeztwObCG8Dm9xcyyFR8woeQs\nItWaqWxovTxAWOQDoAkYBT5Yz5Kk0/gJ8AtgD+ESt5hi/CSww8weAh5OfmKKD0JPzcrkDHol4ewz\nlhj/DXyhZLu8JO7ZwIeAbe4+6u79wC4mL2usSMlZRKpVsWxovYIBcPe8ux80s3bg98B3mENJ0sVk\nZl8F9rv7n5iMrfR1q3eMxxCuk/8icClwH3HFB7ANaAVeAn4J3E4k77O7byL8UVhUHtcKppbPLZbV\nnZaSs4hUa6ayoXVjZu8C/gzc6+6/I4z1FVVVknSRrQfONrMnCOP1vwaOLWmvd4xdwKPJWd3LJGep\nJe31jg/gamC7uxuTr2HpMlAxxFhU6fNXqazujPEqOYtItWYqG1oXyVjeo8DV7n5vsvvZmEqSunun\nu69z93XAP4ELgEciinEbYRwUM3sn0AY8noxFQ/3jA1jO5JlnL2Hi2rORxVhUqSTu34EzzSxnZiuB\n9wE7ZnoQzdYWkWptIpwBbk+219czmMS1wCrgejO7AZgArgB+lky42Qk8WMf4pnMlcFcMMbr7ZjNb\na2Z/I3TJXgrsBjbEEF/iVuAeM9tKyFvXAM8QV4xFU95bd58ws9sJfwilCBPGRmZ6EJXvFBERiYy6\ntUVERCKj5CwiIhIZJWcREZHIKDmLiIhERslZREQkMkrOIiIikVFyFhGJlJl1mtkfZ3H/m8zso4sZ\nk9SGkrOISNxmU4yik7AAiDQ4FSEREYlUUp7yp0A3cDzwJPBNwgpM3yVUy/ovYZ3gcwnrbe8lrJJ0\nDPB9woIRHYQSp3+o8VOQOdKZs4hI3E4GLnL30wgLUlwH3AKc4+6rgceAH7n7b4Cnga+7+wvAZcnt\n04GLgBvrEr3MiWpri4jE7XF3fzW5fT+wkdDV/USyvnGasLJUUXHJwguA88zsS8AZhAUtpEEoOYuI\nxK18reAUsNXdzwcws2YOX8qzaBvwOPCX5N/7FjdMWUjq1hYRids6M3uHmaWBC4HbgDVmdkrSfj3w\n4+T2KJAxsw5Cd/gN7r4FOAdNFGsoSs4iInHbAfwWeA54FbgZ+BrwgJk9B3wA+HZy3y3AnYABG4AX\nzWwb8BbQYmatNY5d5kiztUVERCKjM2cREZHIKDmLiIhERslZREQkMkrOIiIikVFyFhERiYySs4iI\nSGSUnEVERCKj5CwiIhKZ/wPT2S6dJSrMuQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11dd6f990>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(8,8))\n",
    "plt.subplot(2,2,1)\n",
    "\n",
    "plt.hist(advi_trace['p'], range=[0,1], bins=50, normed=True, label='ADVI', alpha=0.5)\n",
    "plt.hist(nuts_trace['p'], range=[0,1], bins=50, normed=True, label='NUTS', alpha=0.5)\n",
    "plt.xlabel(\"p\")\n",
    "\n",
    "plt.subplot(2,2,2)\n",
    "plt.hist(advi_trace['alpha'], range=[0,100], bins=50, normed=True, label='ADVI', alpha=0.5)\n",
    "plt.hist(nuts_trace['alpha'], range=[0,100], bins=50, normed=True, label='NUTS', alpha=0.5)\n",
    "plt.xlabel(\"alpha\")\n",
    "plt.legend()\n",
    "\n",
    "plt.subplot(2,2,4)\n",
    "plt.hist(advi_trace['beta'], range=[0,100], bins=50, normed=True, label='ADVI', alpha=0.5)\n",
    "plt.hist(nuts_trace['beta'], range=[0,100], bins=50, normed=True, label='NUTS', alpha=0.5)\n",
    "plt.xlabel(\"beta\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### pystan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from pystan import StanModel, stan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model_code = \"\"\"\n",
    "    data {\n",
    "        int<lower=0> N;\n",
    "        int<lower=0,upper=1> y[N];\n",
    "    }\n",
    "    parameters {\n",
    "        real<lower=0> alpha;\n",
    "        real<lower=0> beta;\n",
    "        real<lower=0, upper=1> p;\n",
    "    }\n",
    "    model {\n",
    "        alpha ~ cauchy(0, 20);\n",
    "        beta ~ cauchy(0, 50);\n",
    "        p ~ beta(alpha, beta);\n",
    "        for (n in 1:N)\n",
    "            y[n] ~ bernoulli(p);\n",
    "    }\n",
    "\"\"\"\n",
    "\n",
    "stan_data = {\n",
    "    'N':len(data),\n",
    "    'y':data\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### pystan NUTS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, for comparison"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "stan_nuts = stan(model_code=model_code, data=stan_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inference for Stan model: anon_model_2df7ffd52b91f10700e84d4a94d52b93.\n",
      "4 chains, each with iter=2000; warmup=1000; thin=1; \n",
      "post-warmup draws per chain=1000, total post-warmup draws=4000.\n",
      "\n",
      "        mean se_mean     sd   2.5%    25%    50%    75%  97.5%  n_eff   Rhat\n",
      "alpha  33.27   11.63  47.95   1.84   7.87  15.29  30.69 190.23   17.0   1.22\n",
      "beta  176.84   36.37 241.25  12.47  43.91  84.18 182.95  895.4   44.0   1.09\n",
      "p       0.16  3.9e-3   0.07   0.04   0.11   0.15   0.21   0.33  370.0   1.02\n",
      "lp__   -4.51    0.08   1.37  -7.95  -5.29  -4.23  -3.45  -2.82  279.0   1.01\n",
      "\n",
      "Samples were drawn using NUTS at Wed Jul 13 10:46:45 2016.\n",
      "For each parameter, n_eff is a crude measure of effective sample size,\n",
      "and Rhat is the potential scale reduction factor on split chains (at \n",
      "convergence, Rhat=1).\n"
     ]
    }
   ],
   "source": [
    "print stan_nuts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### pystan Mean-field\n",
    "\n",
    "Note: although Stan's ADVI feature has been around for a couple months, the PyStan integration for ADVI was pushed literally yesterday, so no guarantees on the bleeding edge."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "stan_model = StanModel(model_code=model_code)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:pystan:Automatic Differentiation Variational Inference (ADVI) is an EXPERIMENTAL ALGORITHM.\n",
      "WARNING:pystan:ADVI samples may be found on the filesystem in the file `/var/folders/ks/ny8mvlt11hbfnk6vcdd8ghx4b6qs2l/T/tmpYzZ6_3/output.csv`\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OrderedDict([('args', {'algorithm': 'MEANFIELD', 'random_seed': '89366828', 'tol_rel_obj': 0.01, 'eval_elbo': 100, 'sample_file': '/var/folders/ks/ny8mvlt11hbfnk6vcdd8ghx4b6qs2l/T/tmpYzZ6_3/output.csv', 'output_samples': 10000, 'init_radius': 2.0, 'chain_id': 1, 'iter': 10000, 'adapt_engaged': True, 'init': 'random', 'eta': 1.0, 'grad_samples': 1, 'elbo_samples': 100, 'append_samples': False, 'enable_random_init': False, 'method': 'VARIATIONAL', 'adapt_iter': 50}), ('mean_pars', [])])\n"
     ]
    }
   ],
   "source": [
    "results = stan_model.vb(data=stan_data, iter=10000, output_samples=10000, algorithm='meanfield')\n",
    "print(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model samples are currently being written to disk, so we'll import them back in:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "meanfield_samples = pd.read_csv(\n",
    "    results['args']['sample_file'],\n",
    "    skiprows=7,\n",
    "    names=['alpha','beta','p']\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAecAAAHwCAYAAACc3qd3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3XmcXHWd7/9XV2/ppTqddDcI7rh8ZEYMAgpmwioRGQHj\nqNdhrqPiRTTiNigjy+C4jMsVDYqIKBEU74w/IRJQIgkgoBCd+EOZO0HjJ0EgSASSdHf1Wr2eun+c\n6k51pXqvrjpV/X4+Hnk8+pzvOac+VZ3Tnzrf8z2fb0UqlUJERESiI1bsAERERGQiJWcREZGIUXIW\nERGJGCVnERGRiFFyFhERiRglZxERkYipmslGZnY88CV3P9XMjgauBkaAQeBd7r5vAWMUkRkwswrg\nWmAFMACc7+6PZW1TD9wFvNfdd6b3WQ8YMAq8z913FjZyEck27ZWzmV0MXA/Upld9DbjQ3U8DNgKX\nLFx4IjILa4Bad18JXAqsy2w0s2OBXwBHZKx+A9Dg7quAzwFfKFCsIjKFmXRrPwq8JWP5He6+Pf1z\nFZDMe1QiMhergM0A7r4NOC6rvYYwgf8xY90AsDR9Bb0UGCpAnCIyjWmTs7tvJOzCHlt+FsDMVgIX\nAlctWHQiMhtNQFfG8oiZjZ/j7v5rd98DVGRs8yBQR5iwv014y0pEimxG95yzmdk7CLvN/tbd26fb\nPpVKpSoqKqbbTEQmJs7Z6gbiGcsxdw+m2eefga3ufrmZPRe4z8xe6e6TXkHrfBaZsTmfKLNOzmb2\nTuAC4BR3T8xkn4qKCvbt65ntSxVUW1tcMc5T1OOD6MfY1haffqPJbQXOAjaY2QnA9mm2B2jkwNV2\ngvBvQuVUO0x1PgdBQCLROWFdc/MyYrHCPhhSCr/nKMcH0Y8x6vHB/M7nWSXndBfZ14HdwEYzSwG/\ncPfPzDkCEcmXjcBqM9uaXj7PzM4lHPC1PmO7zNlurgRuNLMHCP8eXOrucx5Hkkh0ctv2O2hoagSg\nr7uXNUedxfLlLXM9pMiiNKPk7O67gZXpRZ1lIhHk7ilgbdbqgx6LSj9pMfZzgokDPuetoamRxqXz\n6gEQWfRUhERERCRilJxFREQiRslZREQkYpScRUREImZOzzmLiEhh5Ho8bb6K8XibzI6Ss4gsmCAI\n6Ows/nPPpSyR6GTDPdupb2zKy/H6e7t52+lH6fG2iFNyFpEFk+ztZ0v7vbS0hYlAzz3PTX1jE43x\n5oK+5m9+8xvWrl3LD35wM21thwBw3XXX8MIXvojrrvsGt9++ZXzbbdt+zc9/fherV7+Rm266AYBH\nHvlvjjpqBQAf+tDHaGho5Otf/wojI6P09/exYsWrWbv2wwV9T6VEyVlEFlR9Y72eey5R1dU1fOEL\nn+Gqq76Z1ZK7KuVrXnM8r3nN8QC8+c1v5Oqrrxtv+9SnLuVtb/t7XvvaEwC4/PKLeeCB+znxxFMW\nIvSSp74lERHJ6ZhjjqOpqYkf//jmeR9r+fIWfvazn7J9+/9lZGSEz372S0rMU1ByFhGRnCoqKvj4\nxy/lllt+yJ49T0277VQ+9KGP8dd/fRTf/vY3OeecM/jCFz5Db29vPsMtK0rOIiIyqaamJj784Yv4\nt3/7V1KpsCx79oC+ZLKf2traKY/zu989xNvf/vdcc813uPXWTdTV1fH97393weIudWV5zzkqM+OI\niORDf293UY/1N39zIr/85X387Gc/5YMf/AiHHXY4v/vdQxxzzHFAOCDsVa86Omuv1ISlb37z69TW\n1rJixatZsmQJz3/+C+jq6kJyK8vkrJlxRKRcNDcv422nH5X3Y87WRz/6cX73u4cA+OQn/4WvfOWL\nfOc71xIEAX/910dxxhl/m7XHxG7uz33ui1x11ZX09vZQVVXF4Yc/j0984tK5voWyVxbJOftKubOz\nk/p4g0aIikjJi8ViRbmweO1rX8uLX3zk+HJ9fQO33PKT8eVvfOPbU+5/++2bJyy/4AUvyjHqWyZT\nFsk5+0p5755niS9vIk5+HtoXEREppLJIzjBxDtm+bo0AlMXHzCqAa4EVwABwvrs/lrVNPXAX8F53\n35ledwlwDuHfg2vc/aaCBi4iB9EIKZHysQaodfeVwKXAusxGMzsW+AVwRMa6k4HXpfc5NbNNRIpH\nyVmkfKwCNgO4+zbguKz2GsIE/seMdWcAj5jZbcBP0v9EpMjKpltbRGgCMp9NGTGzmLsHAO7+axjv\n/h7TCrwAOIvwqvknwCsKE67MhGalWpyUnEXKRzeQ+YjCeGKeQjuww91HgJ1mNmBmre6+f6qd2tpy\nPwkRiw1Rt6+G+oawIMWSumqqaqrGl0eHB2ltjdPSsvBPUkwWY1TMNL729na2/Omu8QGv89XX3cv/\nPP6ttLRMPwK8XD7DUqTkLFI+thJeAW8wsxOA7TPY50HgI8BVZnY4UE+YsKe0b19PzvUdHT0k+4eo\nrB4EYCA5TGwkoL8vXE72D7F/fw9BUDOD0OaurS0+aYxRMJv4Ojp6iFXVUFk9dQWumYpVTf87ePjh\n3/KRj3yAT3/6C7z+9avH17/73X+P2ZFcdtm/5iWWTL/85f1ce+3VvP3t7+Dhh3/Lv/3blyeN7bbb\nfsy1135jwmd41VVf5tRTT+foo485aJ9PfvKfAPjf//uq8XVvf/s5HHroc6ioqGB4eBizV3DhhR/j\nkUf+mxtu+A7XXPOd8W07OztZu/a9/PCHt7JmzRsnzMg1lfl8eVC/hkj52AgMmtlW4KvAP5nZuWZ2\nftZ246Wb3H0T8LCZ/Qa4Hfigu6eQRe+II47g5z+/a3z5scceZWBgYMFeb+vWX/KRj1zEW9/6jkkT\n85jp6nhnevbZZxgYGKCvr4+nn/5L5lH42teu5Rvf+DbXXXcDLS2tXH/9tzjmmOPo6GjnmWeeHt9y\ny5ZNnHnmWenXnflrz4eunEXKRDqprs1avTPHdqdlLV+ykHFJaXrFK17Brl1/or+/j/r6BrZsuZM3\nvOFMnn32GQDuvfcebr75P6isrORVrzqa97//Qvbt28tXvvJFhoeHaW/fz/vet5ZVq07m3e8+l1e/\n+hgefXQXsViML33pq9TXN4y/1oMP/pL//M+tuP+RpqalXH75J7j99i386U+P8vWvfwWApqalXHbZ\npybEeNttG/jJTzaybFkLAwNJTj319IPex6ZNP+HEE0+mtnYJt956Cxde+NHxtrFa4QDveMf/5J3v\nfDsXXvhR3vSmc9i8eRPveU/4vXbz5p/x1a9enb8PdwZ05SwiIjmdcspp/OIX9wGwY8fvOeqoFQB0\nd3dzww3f4etf/xbf/Ob17N37LA899Bt2736Cc8/9R9atu4aLL76MW2+9BYD+/j5Wrz6Ta675Dq2t\nbfz617+a8DqrVp3E8cev5MILP8IrX3kUY1enX/7y5/n4xy/h6quv44QTVvJ//s/3x/fp6Ojg5pt/\nyHe+832uvPJrOa+mU6kUd9+9mTPOeBOnnXY6P//5XQwNDY21Tti2trZ2vO1v//bs8V6DHTt+z2GH\nHU5LS+v8PsxZ0pWziIgcpKKigtWr38iVV36Rww47nBUrXj1+pblnz59JJDq5+OKPkkqlSCaT7Nnz\nFK961dF8//vf5Y47bgdgZGRk/Hgve9nLATjkkEMZGhrkxz++mfvv/zkVFRV86lOfAyCVdUNl9+7H\n+epXvzR+rOc97/kZbbt50YuOoKoqTGOvfOWrDnoP27b9mmQyyWc+c/n4se++ezNvetM5B20b9hDU\nA7Bs2XJe+MIX88gj29m8+Q7OOects/785kvJWUQk4vJZ9XA2xzrssMMZGEiyYcOP+MAHPjQ+p/Nh\nhz2XQw99Dldd9U0qKyu5447bOfLIv2b9+m9xzjl/x/HHv46f/eyn3HnnHePHyr6yfetb/wdvfev/\nmOSVw0z6ghe8iH/5l89wyCGH8l//9Tu6uw/MqPXCF76Qxx//E4ODg9TU1LBjx+854YSVE45yxx23\nccklV4yv3779/3LVVVfmTM7//u838frXv2F8+Zxz3sLmzZv4wx8eyZqgozBDMmaUnM3seOBL7n6q\nmb0E+B4QAI+4+4ULGJ+IyKLW3LyMNUedlfdjztTrX7+aLVvu5HnPe/54cm5ubuYd7/gHPvSh9zE6\nGnDYYYezevUbOfXU07nmmqu45ZYf8ld/9Uq6u8ceuz+QmGc2mCvc5uMfv4TPfe5TjI6OEovFuOSS\nK9i3by8Ay5cv513vei9r1/4vli5dSmXlxHTW2dnBH/7wez772S+NrzvqqBUMDw/xyCPbgQr+6Z8u\nJBaLEQQBL3uZTbgf/ZrXHM+6dV/mjW+cerathVKRyu5HyGJmFwP/CPS6+0ozux34irs/YGbfAja7\n++3TvE5qIR9r6Oho5+7d94/X1n72z08Tq66k7TmHANDb1cPqF54y5cwuUX/0AqIfY9Tjg+jH2NYW\nL8yZPz+Tns/5OBfzoQR+z5GOD6IfY9Tjg/mdzzMZEPYokNnhfqy7P5D++U7g4OFxIiIiMmfTJmd3\n3wiMZKzK/CbQAyzNd1AiIiKL2VwGhGWWA4wDiZnstJBl1vJVMrAUSsFFPcaoxwelEaOILG5zSc6/\nM7OT3P2XwJnAvTPZaWHvOc+/ZGCJ3L+IdIxRjw+iH6O+OIgIzC05fwK43syqgR3AhvyGJCIisrjN\nKDm7+25gZfrnXcApCxiTiIjIoqbynSIiIhGj5CwiIhIxSs4iIiIRo9raImXCzCqAa4EVwABwvrs/\nlrVNPXAX8F5335mx/hDgIeD0zPUiUhy6chYpH2uAWndfCVwKrMtsNLNjgV8AR2StrwKuA/oLFKeI\nTEPJWaR8rAI2A7j7NuC4rPYawgT+x6z1XwG+BfxloQMUkZlRchYpH01AV8byiJmNn+Pu/mt330NG\nCV4zew+w193vplDT7YjItHTPWaR8dBOW1B0Tc/dgso3TzgMCM1sNHA3cZGbnuPveqXaarJJZvkrp\n5kPUq61FPT6IfoxRj28+lJxFysdW4Cxgg5mdAGyfbgd3P3nsZzO7D3j/dIkZJi/Hm49SuvlQCmVa\noxwfRD/GqMcH8/vyoOQsUj42AqvNbGt6+TwzOxdocPf1GdtNNon71JO7i0jBKDmLlAl3TwFrs1Yf\n9FiUu582yf4514tI4WlAmIiISMQoOYuIiESMkrOIiEjEKDmLiIhEjJKziIhIxCg5i4iIRIySs4iI\nSMQoOYuIiESMkrOIiEjEKDmLiIhEjJKziIhIxCg5i4iIRIySs4iISMQoOYuIiESMkrOIiEjEzGk+\nZzOrANYDBowC73P3g+aNjYogCOjs7Jywrrl5GbGYvptI+Uifl9cCK4AB4Hx3fyxrm3rgLuC97r7T\nzKqAG4AXATXA5939pwUNXEQOMtfs9Aagwd1XAZ8DvpC/kPIv2dvPlj/dy9277+fu3fdz2/Y7SCQ6\np99RpLSsAWrdfSVwKbAus9HMjgV+ARyRsfqdwH53Pwk4E7imQLGKyBTmmpwHgKXpb+pLgaH8hbQw\n6hvraVwap3FpnIamxmKHI7IQVgGbAdx9G3BcVnsNYQL/Y8a6m4Er0j/HgOEFjlFEZmBO3drAg0Ad\n4UneApyVt4hEZK6agK6M5REzi7l7AODuv4bx7m/S6/rT6+LALcDlhQtXRCYz1+T8z8BWd7/czJ4L\n3Gdmr3T3Sa+g29ric3yp6cViQ9Ttq6G+oRaAJXXVVNVUTbo8OjxIa2uclpaJMS1kjPkS9RijHh+U\nRoxz1A1kvrnxxDwVM3s+cCtwjbv/aCYvNNlnON25ONm5txCi/nuOenwQ/RijHt98zDU5N3LgG3oi\nfZzKqXbYt69nji81vY6OHpL9Q1RWDwIwkBwmNhLQ35d7Odk/xP79PQRBzfgx2triCxpjPkQ9xqjH\nB9GPcZ5/bLYS9mJtMLMTgO3T7WBmhwJbgAvd/b6ZvtBkn+F052Kuc28hlMLvOcrxQfRjjHp8ML/z\nea7J+UrgRjN7IH2MS909OecoRCQfNgKrzWxrevk8MzuXcPDm+oztUhk/Xwo0A1eY2afSbWe6+2BB\nIhaRnOaUnN09Abwlz7GIyDy4ewpYm7X6oEcc3f20jJ8/BnxsgUMTkVnSg74iIiIRM9du7aIKgmDC\nc8qdnZ2kUqkp9hARESkdJZmcE4lObtt+x/jzynv3PEt8eRNxmoocmYiIyPyVZHIGaGhqpHFpOBKu\nr7u3yNGIiIjkj+45i4iIRIySs4iISMQoOYuIiESMkrOIiEjEKDmLiIhEjJKziIhIxCg5i4iIRIyS\ns4iISMQoOYuIiESMkrOIiEjElGz5ThGZyMwqgGuBFcAAcL67P5a1TT1wF/Bed985k31EpPB05SxS\nPtYAte6+ErgUWJfZaGbHAr8AjpjpPiJSHErOIuVjFbAZwN23AcdltdcQJuM/zmIfESkCJWeR8tEE\ndGUsj5jZ+Dnu7r929z1AxUz3EZHi0EkoUj66gXjGcszdgwXYR0QWmAaEiZSPrcBZwAYzOwHYvkD7\n0NYWz7k+Fhuibl8N9Q21ACypq6aqpmp8eXR4kNbWOC0tuffPp8lijIqoxwfRjzHq8c2HkrNI+dgI\nrDazrenl88zsXKDB3ddnbJeaap+ZvNC+fT0513d09JDsH6KyehCAgeQwsZGA/r5wOdk/xP79PQRB\nzUzf05y0tcUnjTEKoh4fRD/GqMcH8/vyoOQsUibcPQWszVq9M8d2p02zj4gUme45i4iIRIySs4iI\nSMQoOYuIiESMkrOIiEjEzHlAmJldApyTPsY17n5T3qISERFZxOZ05WxmJwOvS9fjPZWJtXpFRERk\nHuZ65XwG8IiZ3UZYXeji/IUkIiKyuM01ObcCLyCsLHQE8BPgFfkKSkREZDGba3JuB3a4+wiw08wG\nzKzV3fdPtkM+y6xNVyJwuuXJSgiWQim4qMcY9figNGIUkcVtrsn5QeAjwFVmdjhQT5iwJ5XPMmvT\nlQicbjlXCcFSKQUX5RijHh9EP0Z9cRARmOOAMHffBDxsZr8Bbgc+mC4DKCIiIvM050ep3P2SfAYy\nG0GQIplMUlkThp9MJqmvaihWOCIiInlVkhNfdHUl8Cf30bR8FIC9e9qxI0ryrYiIiBykZDNadXUt\nNTVLwp+rFnb6ORERkUJS+U4REZGIKdkrZxGZyMwqgGuBFcAAcL67P5bRfjZwBTAM3Oju69P7rAcM\nGAXe5+4HzQEtIoWlK2eR8rEGqE2X1b0UWDfWYGZV6eXTgVOAC8ysDXgD0ODuq4DPAV8odNAicrCy\nTM6pVIqBwQGSyeT4v1RKT3pJ2VsFbAZw923AcRltRwK73L3b3YeBB4CTCK+wl6avoJcCQ4UNWURy\nKctu7YGBAZ7Y20XncDWg0dyyaDQBXRnLI2YWc/cgR1svYTK+DagD/gi0EJbkFZEiK9uMVVVTo9Hc\nsth0E05EM2YsMY+1NWW0xYEE8M/AVne/3MyeC9xnZq909ymvoCerZDZdad3JSucuhKhXW4t6fBD9\nGKMe33yUbXIWWYS2El75bjCzE4DtGW07gJeaWTPQD5wIXAkcy4Er6gTh34TK6V5oshKo05XW7esd\nYNeuJ9m//8D+zc3LiMXye4etFMq0Rjk+iH6MUY8P5vflQclZpHxsBFab2db08nlmdi7hgK/1ZnYR\ncBdQAXzX3Z82syuBG83sAcK/B5e6e3KhAkz29rOl/V5a2loA6OvuZc1RZ7F8ectCvaRISVJyFikT\n6fr2a7NW78xo3wRsytonAbxl4aM7oL6xnsal5dsdKZIPZTlaW0REpJQtiivnVCrF0MAgyWTYW5dM\nJgkCPVolIiLRtCiS88joMI8/3UVfZSMA3R0dnHhogtbW1iJHJiIicrBFkZwBqqurxx+tqqqspqsr\nQUdH+3h7S4umnBQRkWgoi+Sc3W09ODQIU1QEG+xP8os9v2JXxW4gHDF6fuu5gJ6HFhGR4iuL5Jzd\nbf2XPR3UNTZOuU+dRoyKiEhElUVyhqxu6xpdAYuISOnSo1QiIiIRo+QsIiISMWXTrT0buZ97DqbZ\nS0REpDAWZXLO9dxz4q8SNDc/p8iRiYiILNLkDBMHkFVX1xY5GhERkQN0z1lERCRiSuLKOQgCEonO\n8eWursRUNUZERERK2rySs5kdAjwEnO7uO6fbfq4SiU423LOd+sYmAJ78kzN6yMhCvZxISTKzCuBa\nYAUwAJzv7o9ltJ8NXAEMAze6+/r0+kuAcwj/Hlzj7jcVOnYRmWjOydnMqoDrgP78hTO5+sYmGuPN\nANQ1NNJL+zR7zFwqlaKzs5MgqB5f19y8jFhMvf5SUtYAte6+0syOB9al142dr+uAY4EksNXMbgf+\nCnhdep8G4OLihC4imeZz5fwV4FvApXmKpWhGRob4yf1Oy6Hho1X9vd287fSjWL68pciRiczKKmAz\ngLtvM7PjMtqOBHa5ezeAmT0AnAwcAzxiZrcBcZScRSJhTpeGZvYeYK+73w1U5DWiIqlrbKQx3kxj\nvHm8+1ykxDQBXRnLI2YWm6StN72ulfBq+m3AWuA/ChCniExjrv225wGrzew+4GjgpvT9ZxEpnm7C\nq98xMXcPMtoyv3XGgQTQDmxx95H0uJEBM9NE5yJFNqdubXc/eezndIJ+v7vvnWqftra5zwAViw1R\nX19DQ0P4PPKSuhqqqiqprg7Dr4pVUlkdm/tyZSXA+PGD0RpaW+O0tERv1qr5fI6FEPX4oDRinKOt\nwFnABjM7Adie0bYDeKmZNROOEzkRuBIYBD4CXGVmhwP1MP2Ajsk+w1hsiLp9NdSPn6vVVNVUTbo8\nOjy4YOda1H/PUY8Poh9j1OObj3w8SjWjh5r27euZ8wt0dPTQ3z9ErHIQgIHkECOxUYaHwxHbI8Eo\nFcOVc18eHYVK6OsLj9/fP8T+/T0EQbRmt2pri8/rc1xoUY8Poh/jPP/YbCTs0dqaXj7PzM4FGtx9\nvZldBNxFeCvqu+7+NLDJzE40s9+k13/Q3ac9pyf7DDs6ekj2D1FZPXauDhMbCejvy72cXKBzrRR+\nz1GOD6IfY9Tjg/mdz/NOzu5+2nyPISLzl06qa7NW78xo3wRsyrHfJQscmojMkp4VEhERiRglZxER\nkYhRchYREYkYJWcREZGIUXIWERGJGCVnERGRiFFyFhERiRglZxERkYhRchYREYkYJWcREZGIUXIW\nERGJmHxMfCEiAkAQpEgmk1TWhH9akskk9VUNRY5KpPREMjkHQUAi0Tm+3NnZSSo1o8mvRKSIuroS\n+JP7aFo+CsDePe3YEZH8MyMSaZE8axKJTjbcs536xnBu+P3PPEXj0lbiTdPsKCJFV11dS03NEgCq\nKqsZGhgkmUwCupIWmalIJmeA+sYmGuPNAPT1dhU5GhGZi5HRYR5/uou+ykZAV9IiM6WzRKRMmFkF\ncC2wAhgAznf3xzLazwauAIaBG919fUbbIcBDwOnuvpM8qq6uHr+Srq6qyeehRcqWRmuLlI81QK27\nrwQuBdaNNZhZVXr5dOAU4AIza8touw7oX+gAU6nUeDf32L8g0HgSkWy6chYpH6uAzQDuvs3Mjsto\nOxLY5e7dAGb2IHAS8GPgK8C3CBP6gsru5u5ub+dVS54gFqsY36a5eRmxmK4bZHFTcs4hCAI6Ozsn\nrNMfDCkBTUDmAI0RM4u5e5CjrQdYambvBva6+91mdlkhgszs5qaigru2Pc7zngnPrf7ebt52+lEs\nX95SiFBEIkvJOYdkfw+btu5neeshgP5gSMnoBuIZy2OJeawt83mHOJAAPgKkzGw1cDRwk5md4+57\np3qhtrZ4zvWJRCPVVTGqq8M/LVWxSiqrp1iurGR56zIOfc6hAPR019DaGqelJffxZ2OyGKMi6vFB\n9GOMenzzoeQ8ifqGA6PFRUrEVuAsYIOZnQBsz2jbAbzUzJoJ7y2fBFzp7reObWBm9wHvny4xA+zb\n15NzfXt7L8MjAcPDIwCMBKNUDFdOvjw6ysDgCH19gwD09w+xf38PQTC/gWNtbfFJY4yCqMcH0Y8x\n6vHB/L48lERyDoKAwWQ3vXVhV1h/Xw+p+vwNIkkFAf3JHnp7EuHxe7tpiOsqWUrORmC1mW1NL59n\nZucCDe6+3swuAu4CKoD17v501v4amSUSESWRnAf6e3mKHXSPtgHwl9QTNA7nryLJYHKAJ/k9vaMd\nALSnnuXF/cfk7fgiheDuKWBt1uqdGe2bgE1T7H/abF8zu5pfV1cCFfMTmb+SSM4AtQ311MUb0z8v\nWYDj140ff0lfNwweaNMAMZHcsqv5PfknZ/SQkSJHJVL6SiY5F5MGiIlMLrOaX11DI720FzkikdKn\n5DxDGiAmcrAgCCaU153veBD1UomE5pSc0xWFbgBeBNQAn3f3n+YxLhEpAV1dXTw28FvidcuA+Y8H\nUS+VSGiuV87vBPa7+7vMbBnwX4CSs8gilO/xIOqlEpl7cr4ZuCX9c4ywkL6IiIjkwZySs7v3A5hZ\nnDBJX57PoERERBazOQ8IM7PnA7cC17j7j6bbfqpKKdmDQGKxYerqamhoqAVgSV0NVVWVMy8JOIdl\nYEJJwSX1VeOvX1dXQ2VV9fhyMJq/EoOzFfVydVGPD0ojRskt+7lq0IAxKU9zHRB2KLAFuNDd75vJ\nPlOVWevoaJ/wrOT+Z56icWkrlVX1AAwkhxiJjc68JOAclquonrSkYDI5RGVlRd5LDM5W1MvVRT0+\niH6M+uIwUfYX987OTu556M80xJcCGjAm5WuuV86XAs3AFWb2KcKyf2e6++DUu00u81nJzEczRGTx\nyh69PfbFXQPGpNzN9Z7zx4CP5TkWEVlkUkFAf9/Ude0zR2/ri7ssFipCIiJFM5gc4OnKHYyMdgOq\nay8yRslZRIqqtn7yuvYii5WS8xyoxKCIiCwkJec5UIlBERFZSErOc6QSgxI1ZlYBXAusAAaA8939\nsYz2s4ErCCv63eju61UnXySalJxFyscaoNbdV5rZ8cC69LqxyWrWAccCSWCrmd0OvIkSrpOf6xYT\nQEtLQxGiEckfJWeR8rEK2Azg7tvM7LiMtiOBXe7eDWBmDwInUeJ18rNvMUF4m+kDrXHCjgCR0qTk\nLFI+moClJjoZAAAgAElEQVTMB4FHzCzm7kGOth5gaTnUydctJilHSs4i5aMbyKz/OZaYx9oyJ1qO\nAwmYfZ18OFBmNJFopLoqj3Xup6lrP90yhLXvM2OMqqjHB9GPMerxzYeSs0j52AqcBWwwsxOA7Rlt\nO4CXmlkz0E/YpX3lXOrkw4Fa+e3tvQyPBPmrcz9NXfvpliGsfZ8ZYxRFvcY7RD/GqMcH8/vyoOSc\nw0xKCopE0EZgtZltTS+fZ2bnAg3pkdkXAXcBFcB6d3/azL5GnuvkF1sQBHR0dBAE1ePLUEEsVjG+\njeoSSNQpOeegkoJSitw9BazNWr0zo30TsClrn7Krk5/s7+GWu/9AXUN4H3r/M08Rq6pRXQIpKUrO\nk5hNSUFVDJPF5M57/5O+3nBQ95///DijFcE0exRefWOchsYDk2VUVtZq0JiUFCXnPFDFMFlMnu6u\nZjSoA2BvbwWphugl56noy7SUgqIk5yAISCQmTqCeSqWKEUre6HEOkdKgL9NSCoqSnBOJTjbcs536\nxvDJjrEJ1ONN0+woIpIH+jItUVe0bu36Rk2gLiLFp25uiSLdc14AOtlFSkd2N3dvd4LVr3kBy5Yt\nA/QolhSHkvMC0D0tkdKS2c3d19vFpq2Pjp+/ehRLikHJeYHonpZI6cpO1noUSwpNybkA1M0tUj50\nPkshKDnPwHzLeaqbW2RmSqF07nT3qEHJWuYvksk5CAIGk9301i0BoL+vh1R98Z6Dzkc5T3Vzi0yv\nVErnTnWPerZfvrPrPoCSu0Q0OQ/09/IUO+gebQPgL6knaBwu7kPQsynnKSJzV4rnWmaynm23d3bd\nB/WsCUQ0OQPUNtSPn6C1DUuKHE1+6Z6VSPma2aNZA3R09AFhhcS6hvick7sURqF7OCKbnMvZTO5Z\ngU5IkVI13aNZDfHGCbNmZVZI1BiVaCp0D8eckrOZVQDXAiuAAeB8d39ssu23/2EnT+zeN75cGSut\nQvkLYaqTF6b/tg1K3jLRdOelmZ0NXAEMAzem53ie1blcaKUwQGwmss/3+sbGCbNmTbW9LIy5XAln\nVrZcaHO9cl4D1Lr7SjM7HliXXpfTrt37aU82ji+Pdj8BFaX7H28h/mBkn4zTfdvWt2nJYdLz0syq\n0svHAklgq5ndDqyabJ8oKJUBYoU0XVLJ1Z5d5Wyhq55lxxDFKmtRv9c/1+S8CtgM4O7bzOy4/IUU\nfYX6gzHVt+1sGvEpTH1eHgnscvduADN7ADgZeN0U+0RCKQ4Qy6fse9CdnZ3c89CfaYgvBQ7uZctu\nh4OrnGUv57q11tLSMCGGqZJt9nJ2DNO9Xr6/PMzky0H2vf7Zyv695HqNtrb4nI4Nc0/OTUBmX8yI\nmcXcfcb91fufeWq8O6f92b8Qq6phdDQ86zr2PUPfSIKqqjC8/q5uqmpr6GpvX7Dl0eEhqqtnt//Q\nUBjv6PAQ7Ymnqa2rzfl+Zrs82TYDfQ30NvQCkOzt4YknDvzn6OpKsPlXO6mrD0+oZH8fb1z5cpYu\nLVwPRSLRSHt7b8Feby6iHmNb26vns/tU52V2Wy+wFIhPsU9OQRCk/xBBEKToTRTuXO1LdNHeN/m5\nNrYu81yZ7/m4EMuzje8HO4dZ2hwmso79z9IQXwbpHNCx72l+cPuTk7ZD+DhqrKqG2iW1OZezj5Hs\n7+PcN72aIKgGDv770rH/WSqraia8ZvZyZgzTvd50x8v192yqc3m6eDNjXNbSGr5G1t/UXMfc/8xf\nJuSt7N9LdsxXfeb9OY81E3NNzt2EJ/WYKU/mvztzZcXENSvm+LIylTPOOLXYIfCylxU7gumVQoxz\nNNV52U2YoMfEgc5p9snp/L97Tcb5PK8vE1JCovD3JdtU5/JCxHvGGXk/5KTm2ue5FfhbADM7Adie\nt4hEZK6mOi93AC81s2YzqwFOBH4N/GqKfUSkSCpSqdlX3soY4fmq9Krz3H1nPgMTkdnJdV4SDgBr\nSI/MfhPwr4Sdjd919+t0LotE05ySs4iIiCwcDeUVERGJGCVnERGRiFFyFhERiRglZxERkYjJ68QX\nc6ntm8/Xz0N85wIfTce33d0/WMj4ZhJjxnbfBtrd/bIChziTz/E1wFfTi3uAd7n7cITiewtwGRAQ\n/j+8rlCxZcV5PPAldz81a31Rz5PJRLEOd7os6Q3Ai4Aa4PPAH4DvEf5+H3H3C4sVXyYzOwR4CDgd\nGCVCMZrZJcA5hDnhGsLH8r5HdOKrANYDRvjZvY8IfYaZ57KZvSRXXGb2PuACwvP68+6+aapj5vvK\neby2L3ApYZ3eseDHavueDpwCXGBmbXl+/fnEtwT4LHCyu58INJvZWQWOb8oYx5jZ+4FXFjqwDNPF\n+B3gPe5+EvBz4MURi2/s/+Eq4ONmtpQCM7OLgeuB2qz1UThPJjPt/80ieCewP/1/7Y2EiWUdcJm7\nnwzEzOzNxQwQxn+v1wH96VWRidHMTgZel/69ngq8JErxpb2B8JHAVcDngC8QkRhznMsHxWVmhwIf\nJiyX+0bgi2ZWPdVx852cJ9T2BXLW9k1fRT0InJTn159PfIPASncfqwNYRXh1UGhTxYiZvQ54DfDt\nwoc2btIYzezlQDtwkZndDzQX4bnZKT9DYAhYBtSll4vxPOGjwFtyrI/CeTKZ6T7XYriZsJcBoBIY\nAY5x9wfS6+4k/KJTbF8BvgX8hfA58yjFeAbwiJndBvwk/S9K8UH4t3hp+gp6KeHVZ1RizD6Xj82K\nazXwWuBBdx9J17ffxYHaAjnlOznnrO07SVsP4YdcSJPG5+4pd98HYGYfJvyWdk+B45syRjN7DmER\niQ8xoXJuwU31e24l/HZ4NeHJcrqZnVLY8KaMD8Iu998SVsO6Y2wyiEJy942EiSRbFM6TyUz3uRac\nu/e7e5+ZxYFbgMuZeG4U/fMzs/cAe939bg7Elvm5FTvGVsJiNW8D1gL/TrTig/BLah3wR8ILk6uJ\nyO85x7mcHVcTB9ewH6ttP6l8n1izre2byPPrT2fKOsJmVmFmVwKvB/6uwLGNmSrGtwMtwM+AS4B/\nMLN3FTg+mDrGduBRd9/p7iOEV1qFvsKaND4zez5h99ILCe9THmpmby1wfFOJwnkymVnX4S6E9O/0\nXuD77v7/Ed7rGxOFz+88YLWZ3Ud4v/4mIPNWRbFjbAe2pK/qdpK+Ss1oL3Z8AP8MbHV348BnWJPR\nHoUYx+T6/zfr8zrfyXk2tX1PIqztW0jT1QT/DuE9tTUZ3duFNmmM7v4Nd3+Nu58GfAn4D3e/KUox\nAo8BjWZ2RHr5ROD3hQ1vyviWEH7LHXT3FLCXsIu7WLJ7QKJwnkwmcjX10/fytgD/7O7fT69+2MzG\nbgWcCTyQc+cCcfeT3f3U9MC//wL+EbgzQjE+SHgfFDM7HGgAfp6+Fw3Fjw+gkQNXngnC244PRyzG\nMb/L8bv9/4FVZlaTHuPyCuCRqQ6S19HawEbCb4hb08vnpUdAj9X2vQi4i/AP0np3fzrPrz/n+Ai7\nOc8DHkh/w00BX3f326MSY1RG7TL97/l/AT80M4BfufudEYvvJuBXZpYE/kQ4srJYUjD+pEBUzpPJ\nHPS5FjOYtEuBZuAKM/sU4ef5UeAb6QE3O4ANRYxvMp8Aro9CjO6+ycxONLPfEP6fWws8AayPQnxp\nVwI3puchryLsOfwt0YpxzEG/W3dPmdnVhF+EKggHjA1NdRDV1hYREYkYFSERERGJGCVnERGRiFFy\nFhERiRglZxERkYhRchYREYkYJWcREZGIUXIWEYk4M3uhmT0+zTb/mn7WW8qAkrOISGlQUYpFJN8V\nwqQEpEve/QthpZrnAdsI5+Yt2JzLIpKbmVUSzmD1SuAQwIGPZ7TfSDiz2jGENZo/5+7/nm4+Pl3B\n7XDge+7+mfSkIN8Fnpte/0t3f3eh3o/Mja6cF68TgAvc/RWEs71EYkJ6EWElYe33lcDLgHrSNc0z\nPJdwGsLXA181s0PS6w8BTiacbOZiM2sA3gQ87O5/A7wcWGlmr174tyHzoeS8eN3j7o+lf/4BcFox\ngxGRUHou4G+Z2QeBrwMvJZz4IdP16Wlu9xDWa16VXn9nenapdmAfsDw9U9c9ZvZR4BvA8hzHk4hR\ncl68RjN+jpF7bmERKTAzO4dwTuVe4AbCWY12Z22Web5WZixnrk8BFen56b8MPEs4D/IOijsfvMyA\nkvPidYqZHWpmMeBdQKFnjhKR3F4P/Cg9HexewmlDK7O2ORfCUdyE3du5pkscS8CnA99OX0FXAEfn\nOJ5EjJLz4vUXwm/njwB/BqIyHaXIYnc98A/pKRyvA24HTmXiaO1GM3sI+CnwPnfvzHGcse2/Bnza\nzH4NXJHe58ULFbzkh6aMXITSo7U/6e7Zg0xEJOLSo7XvdPebix2LLBxdOYuIlBZdUS0CunIWERGJ\nGF05i4iIRIySs4iISMQoOYuIiESMkrOIiEjEaOILkRJjZhXAtcAKYIBw0pLHMtrPJnyedRi40d3X\np4vNXA8YEAAfcPc/mNlLgO+l1z3i7qqxLhIBunIWKT1rgNr0xAiXAuvGGsysKr18OnAKcIGZtQFn\nAyl3X0WYuD+f3mUdcJm7nwzEzOzNBXsXIjIpJWeR0rMK2Azg7tsIZyAacySwy92701OAPgic5O63\nAxekt3kRkEj/fGx6ogUIS7ievsCxi8gMKDmLlJ4moCtjeSTdbZ2rrQdYCuDuQbq61NcJS7fCxAkQ\nxrcVkeLSPWeR0tMNxDOWY+4eZLQ1ZbTFOXCVjLufZ2afBH5jZn9FeK8557aTSaVSqYoKTWokMgNz\nPlGUnEVKz1bgLGCDmZ0AbM9o2wG81MyagX7gROBKM/tH4Hnu/kXCQWSj6X+/M7OT3P2XwJnAvdO9\neEVFBfv29eT1DeVbW1s80jFGPT6IfoxRjw/CGOdK3doipWcjMGhmW4GvAv9kZuea2fnuPgJcBNxF\nmMS/6+5PAxuAo83sF4T3lj/q7oPAJ4DPpo9Vnd5ORIpMtbVFZLZSpXDFEuUYox4fRD/GqMcH0NYW\nn3O3tq6cRUREIkbJWUREJGKUnEVERCJGyVlERCRilJxFREQiRslZREQkYpScRUREIkbJWUREJGKU\nnEVERCJGyVlERCRilJxFREQiRslZREQkYpScRUREIkbJWUREJGKUnEVERCJGyVlERCRilJxFREQi\nRslZREQkYqqKHYCISDkJgoBEonPCuubmZcRiuhaSmVNyFhHJo0Sik9u230FDUyMAfd29rDnqLJYv\nbylyZFJKlJxFRPKsoamRxqXxYochJUz9LCIiIhGj5CwiIhIxSs4iIiIRo+QsIiISMUrOIiIiEaPR\n2iIlxswqgGuBFcAAcL67P5bRfjZwBTAM3Oju682sCrgBeBFQA3ze3X9qZkcDdwA707t/y91vKdib\nEZGclJxFSs8aoNbdV5rZ8cC69DrSSXgdcCyQBLaa2e3Am4D97v4uM1sG/Bfw0/R2X3X3q4rwPkqS\nioxIISg5i5SeVcBmAHffZmbHZbQdCexy924AM3sQOAm4GRi7Io4RXlVDmJxfbmZrgF3AR929b+Hf\nQulSkREpBH3VEyk9TUBXxvKImcUmaesBlrp7v7v3mVmcMElfnm7fBlzs7icDjwGfXtDIy8RYkZHG\npfHxJC2ST7pyFik93UBm+amYuwcZbU0ZbXEgAWBmzwduBa5x9x+l229z97FkvhG4eiYBtLVFv/rV\nQsUYiw1Rt6+G+oZaAEaHB2ltjdPSEp9R+0LHl09RjzHq8c2HkrNI6dkKnAVsMLMTgO0ZbTuAl5pZ\nM9BP2KV9pZkdCmwBLnT3+zK232xmH3b3h4DXA7+dSQD79vXk4W0snLa2+ILF2NHRQ7J/iMrqQQCS\n/UPs399DENTMqH2h48uXqMcY9fhgfl8elJxFSs9GYLWZbU0vn2dm5wIN6ZHZFwF3ARXAend/2sy+\nBjQDV5jZp4AUcCbwfuBaMxsCngEuKPSbEZGDKTmLlBh3TwFrs1bvzGjfBGzK2udjwMdyHO6/CQeY\nyRwFQUBn54HR252dnaRSqSJGJOVAyVlEZB6Svf1sab+XlrZwtPbePc8SX95EfMKtf5HZUXIWEZmn\n+sb68Ski+7p7ixyNlAMlZxGRBZTd7Q3Q0tJQpGikVCg5i4gsoOxu777uXs5vPZewiqpIbkrOIiIL\nLLPbW2QmVCFMREQkYnTlLCJSRJpIQ3JRchYRKaAgCOjo6CAIqoHwuej7//zghNHemkhDlJxFRAoo\n2dvPxv++k/p4+Bz02HPRuictmZScRUQKrKGpgYYmPRctk9NNDRERkYjRlbOIyBSyB2ypdrYUgpKz\niMgUEolObtt+Bw1NjYBqZ0thKDmLiEyjoalRtbOloHTPWUREJGKUnEVERCJG3doiIhk0AEyiQMlZ\nRCSDBoBJFCg5i4hk0QAwKTbdcxYREYkYJWcREZGIUXIWERGJGCVnERGRiFFyFhERiRglZxERkYjR\no1QiJcbMKoBrgRXAAHC+uz+W0X42cAUwDNzo7uvNrAq4AXgRUAN83t1/amYvAb4HBMAj7n5hId+L\niOSmK2eR0rMGqHX3lcClwLqxhnQSXgecDpwCXGBmbcA7gf3ufhJwJnBNepd1wGXufjIQM7M3F+xd\niMiklJxFSs8qYDOAu28DjstoOxLY5e7d7j4MPAicBNxMeDUN4Xk/nP75WHd/IP3znYRJXYooCAI6\nOzvp6Ggf/xcEQbHDkgJTt7ZI6WkCujKWR8ws5u5BjrYeYKm79wOYWRy4Bbg83V6Rve2CRS0zkuzt\nZ0v7vbS0tQBhhbI1R53F8uUtRY5MCknJWaT0dAPxjOWxxDzWllkEOg4kAMzs+cCtwDXu/qN0+2iu\nbaW46hvrx8uHyuKk5CxSerYCZwEbzOwEYHtG2w7gpWbWDPQTdmlfaWaHAluAC939voztHzazk9z9\nl4T3ou+dSQBtbdFPHHONMRYbom5fDfUNtQAsqaumqqYqr8vAjLcfHR6ktTVOS0vhP/Oo/56jHt98\nKDmLlJ6NwGoz25pePs/MzgUa0iOzLwLuIuyyXu/uT5vZ14Bm4Aoz+xSQIkzGnwCuN7NqwsS+YSYB\n7NvXk993lGdtbfE5x9jR0UOyf4jK6kEABpLDxEYC+vvyt9xYUzXj7ft6B9i160n27z/wfpqblxGL\nLeyQofl8hoUQ9fhgfl8elJxFSoy7p4C1Wat3ZrRvAjZl7fMx4GM5DreLcFS3RJTuQS9OSs4isqgE\nQUAi0TlhXSGuROdD96AXHyVnEVlUEolObtt+Bw1NjYCuRCWalJxFZNFpaGrUlahEWnT7cURERBYp\nJWcREZGIUXIWERGJGCVnERGRiNGAMBFZ1MYmmhjT2dlJKpUqYkQiSs4isshlF/nYu+dZ4subiE8o\nUS5SWErOIrLoZRb56OvuLXI0IrrnLCIiEjlKziIiIhGj5CwiIhIxSs4iIiIRowFhIiIlJPvRL4j+\nrFoye0rOIiIlRPM7Lw5KziIiJUbzO5c/JWcRkRKSSqUYGBwgmUwCkEwmCQJVNCs3Ss4iIiVkYGCA\nJ/Z20TlcDUB3RwcnHpqgtbW1yJFJPik5i4iUmKqaGmpqloQ/V1bT1ZWgo6N9vF0DxEqfkrOISAkb\n7E/yiz2/YlfFbkADxMqFkrOISImryxgglutRK9DVdKlRchYRKSPZj1qBrqZLkZKziEiZ0aNWpU99\nHCIiIhGj5CwiIhIx6tYWKTFmVgFcC6wABoDz3f2xjPazgSuAYeBGd1+f0XY88CV3PzW9fDRwB7Az\nvcm33P2WgrwREZmUkrNI6VkD1Lr7ynSyXZdeh5lVpZePBZLAVjO73d33mdnFwD8CvRnHOhb4qrtf\nVdB3UEJyVeSqr2ooclRS7pScRUrPKmAzgLtvM7PjMtqOBHa5ezeAmT0InAT8GHgUeAvwg4ztjwVe\nbmZrgF3AR929b+HfQuEEQUAiceDRos7OTlKpmZe7zK7ItXdPO3aE/nTKwtI9Z5HS0wR0ZSyPmFls\nkrYeYCmAu28ERrKOtQ242N1PBh4DPr0QARdTItHJbdvv4O7d93P37vvZ9IctJLoSJJPJ8X/TJeux\nilw1NUuorqqZ1etnXnnP9PVE9PVPpPR0A5nPycTcPchoa8poiwOJKY51m7uPJfONwNUzCaCtLfqP\n6YzFGIsN0fqc5cSbw4+lr6eLXXva6U2F5S+ffSbBK5fWU99QC8CSumqqaqrGl+vqa6jqGaK6Ovxz\nWVUVo25JzaTbZy9XxAJ27+mc8HqNS+s55PCZ7T/feABGhwdpbY3T0jK731vUf89Rj28+lJxFSs9W\n4Cxgg5mdAGzPaNsBvNTMmoF+wi7tK7P2r8j4ebOZfdjdHwJeD/x2JgHs29cz19gLoq0tPh5jR0cP\nyf4hKqsHAUj2D0FFFRWxsJu6oqKS5MAQ/X1h+0BymNhIML6c7B9iZHSU4eGw02FkJJhy+1z7Z78e\nMOP95xvP2D779/cQBDO/6s/8DKMo6vHB/L48KDmLlJ6NwGoz25pePs/MzgUa3H29mV0E3EWYhNe7\n+9NZ+2f2qb4fuNbMhoBngAsWOHYRmQElZ5ES4+4pYG3W6p0Z7ZuATZPsuxtYmbH834QDzGSGUqkU\nQwODJT16O3uQHKj2dtQoOYuIzMLI6DCPP91FX2UjUJqjtxOJTjbcs536xvA+fH9vN287/SjV3o6Q\n0vofJSISAdXV1ePzKc929HZU1Dc20RhvLnYYMgn1YYiIiESMkrOIiEjEqFtbRCSPsst9Dg4NQpGL\njgRBQGfnxCppQRBMsYcUm5KziEgeZZf7/MueDuoaG4saU7K3ny3t99LSFg74at/XTmX/y2hauryo\nccnklJxFRPJsrNzn2M9RUN9YT+PSsChGX18fXU9109sTFo/r6+3SlXTEKDmLiCwyyd5+nkw9Rd/o\nfgB6Bjrp6no+ra1tRY5Mxig5i8iiVuiiItNNQVmoKSqXNNRRFw+72wcHB+jqStDR0T7e3tJSWoVV\nyo2Ss4iUtSBIkUwmqawJ/9xlD9AqdFGRkdFhHn2yg46h2pyvV4wpKkdGhrhr2+M875nwAZ7+3m4+\n0BoHotElvxgpOYtIWevqSuBP7qNp+SiQe4BWoYuKVNVM/XqZ96wLVeSkvj6uoiQRouQsImWvuro2\ncgO0xmR3q2df2U/XLuVJyVlEpIiyu9Wzr+yna8+Wfc8aZn/fOggCOjo6CILq8XWaGKOwlJxFRIos\ns1s915X9dO2Zsu9Zw+zvWyf7e7jl7j9Q1xB2c2tijMJTchaRshIEAe3t7XR09ADhPedUGXcD5+r2\nrspI5jC3+9b1jXEaGnUPuliUnEWkrCQSnWz5013E0gnpz48/ycjIzPcvtXu8M+n2LrX3JErOIlKG\nGpoaqawOH1Wqa6iDRHKaPQ6Y7T3eKJiu27sU39Nip+QsIpJlNvd4S+WqdDbvSYpPyVlEZB50VSoL\nQclZRGSedFUq+aaH1kRERCJGV84iUrKCICCR6JywrrOzs6wfnZLFQclZREpWItHJhnu2U9/YNL5u\n/zNPcfgxA1RlPOcrU0sFAf19PePzO/f3dlPfUF/kqBY3JWcRKWn1jU0TJmzo6+0CBooXUAkaTA7w\ndOUORka7AWhPPUtDfwPxpnB+5yAI6Oyc2EOhcp4LS8lZRESorT8wv/OSvu4Jbcn+HjZt3c/y1kMA\nlfMsBCVnERGZVn1Dk6aULCAlZ5ESY2YVwLXACsL+2/Pd/bGM9rOBK4Bh4EZ3X5/RdjzwJXc/Nb38\nEuB7QAA84u4XFup9SPnKNVBP3eCzo09KpPSsAWrdfSVwKbBurMHMqtLLpwOnABeYWVu67WLgeqA2\n41jrgMvc/WQgZmZvLsg7WEBBEJBMhlMmJpPJyFbsirJUENDfGw4Q6+1J0N/bTRAEM95/bKDez/5z\nNz/7z91suGf7QclapqbkLFJ6VgGbAdx9G3BcRtuRwC5373b3YeBB4KR026PAW7KOday7P5D++U7C\npF7SBvp7+ePu/ezak2DXngSP7+lgdGS02GGVlMHkAE8Gv+eJ0Yd5YvRhnkz9noH+3lkdY2ygXmO8\necJoepkZdWuLlJ4moCtjecTMYu4e5GjrAZYCuPtGM3vhFMcd37bUVVfXqmLXPNU2ZA0QG8zfsafr\n9p5vt3g5dKsrOYuUnm4gnrE8lpjH2jIvU+JAYopjZfZVTrftuLa2+PQbLYDsR3pisWHq6mpoaDjQ\nU7+kLkzG1dXhn7eqWCWV1bFILRcyvrzEWFnJkvqq8c85GK2htTVOS0vu/wex2BD19TWTbt/e3s6m\nXzkN6Svqvt5u3n3OcbS0tMyofcxk/w9nuv9Ucj0+tmxZ4RK8krNI6dkKnAVsMLMTgO0ZbTuAl5pZ\nM9BP2KV9Zdb+FRk/P2xmJ7n7L4EzgXtnEsC+fT1zjX1eOjraJxQd2f/MUzQubaWy6kDBjIHkENTA\n8HA4ifNIMErFcGWklquoLtjr5SXG0VEGBkfo6wsvn/v7h9i/v4cgyN0r0dHRQ3//ELHK3Nt3dPRQ\nEVtCrDL8vVXEZtcOYWKe7P/hTPafTvb/tbk8PjafL7FKziKlZyOw2sy2ppfPM7NzgQZ3X29mFwF3\nESbh9e7+dNb+maOjPgFcb2bVhIl9wwLHPm+ZRUfCgiNSaIulKEl2gZtCUnIWKTHungLWZq3emdG+\nCdg0yb67gZUZy7sIR3VHQjncKyxH2eU92/fuYXP1b2k9pBWAvu5e1hx11qRXldnJfL71z4MgoL29\nnY6OA1fO5fb/RMlZRCIju1a2KlFFQ3Z5z32Deziy5nAal86s2za7wtjY7Yj4HAdxJxKdbPqVUxEL\nB/2V4/8TJWcRiZRidiXK5DLLe9bW1816/8wKY/m4HdHQ+P/au9cYuc6zgOP/nb3Ze7EdYZOGiwRq\nykO/tFBHNClOjQUJlzQQJISERAWBJCgKFRLQqBelF0EoUBqkgNKguk3aUoTaVCkFq24gFGQ7KJC2\nlIYf26gAAAzHSURBVKQ1r1MlTogcJ/au1+v1rr2XWT6cs+vJ7OzFe5l5x/7/JMtzzpkz+8zuaJ89\n73nf59kyf0/5UmRyltS2qtUq5ydGGdt8oQPV+NkzzG6x6Ijam8lZUts6Nz7GSxxmdGbH/L5js0fZ\nOu2Vt9qbyVlSW+vt75sfbi227eOs9mdylpSt9Z7lq/UxOzvL5LnzTExMADAxMUG12rqfSys+Jxu9\nssDkLClb6z3LV+tjemaK518+zdnOYsRidHiY668cYfv27S2JpxWfk41eWWBylpS19Z7lq/XR3d19\noX55Vw+nT48wPDwEtGaEoxWfk41cWWByliStyfT0JI89+Tw/cLwY0m2HEY7cC96YnCVJa9bXNzh/\nFXlm9BTjYxeWuI2PjdI/mFeBkNwL3picJUnrqn6J29DsK/zw+FtaHNVCORe8MTlLktZd7RK39e4H\nfTnIY3BdkiTNMzlLkpQZh7UlSZe0Rv2nL3a5V7MLnZicJUlrUt/vefzsGWb78qnkVl+kBBYu91ou\n+Ta70InJWZK0JvX9no/NHmVgKq9FzrVFSmBhoZKVJN9mFjoxOUuS1uw1/Z7btPlITtXoTM6SpA21\nYNg7w6IkuTE5S5I2VP2wd65FSXJicpbUNqrVKucnaspCZjbxSIurHfauL0qy3GSsxsc3PuZWMjlL\nahv1ZSFznHiki7fcZKxGx3dcdRX9A32tCnnDmZwltZXaspDtOvFICy03GSunyVrNYIUwSZIy45Wz\nJKmpnL29PJOzJKmp6mdvn5w5zutO/Ah9/cU9ZJO1yVlqOxHRATwAvBk4B9yWUnqu5vjNwD3AFPBQ\nSmnvYudExI8B/wQcKU//eErpC816L9VqlZGR5tUrVj5qZ293vIpLreqYnKX2cwvQm1J6W0S8Fbiv\n3EdEdJXbO4EJ4FBE/AOwa5FzdgIfSyn9ZQveByMjp3jkX56mb6CYlrvR9YqVr6WWWl2OnBAmtZ9d\nwH6AlNKTwDU1x94IPJtSGk0pTQEHgN0NztlZPn8ncFNE/HtE7I2I/ia9h3l9A8Us3IHBbWweGGz2\nl5eyZHKW2s8WoHYtyXREVBY5NgZsBQbr9s+U5zwJvDultBt4DvjQRgUNxTD28PDQ/D+HsaXGHNaW\n2s8oRbKdU0kpVWuO1Q4KDwKnFjsnIr6UUppL2o8C968kgB07VneFOzQ0xL4nEv3lMParx19icOv3\n0N/fC8DmzT10dnUvur1pcw9dXZ10dxe/uroqnXR2V+a35/YBiz4nh+1mxtcOMS7Y7uxkU1/Xkp8L\nYMWfm/rt1Zyz3Pb0VBeVyhSVyiTrweQstZ9DwDuARyLiWuDpmmOHgasjYhswDlwPfLQ81uic/RHx\nrpTSU8BPA19fSQAnTpxZVeDDw2foqGyi0lnMyu2o9DIxPsXZs8UNxomJSTo7OxbdPjcxyXRlhqmp\naQCmqzN0THXOb8/t66J70efksN3M+NohxgXbMzOcOz+95OdiYLBnxZ+b+u3VnLPc9tDJIT7z5WPz\nVczGx0Z5/103s1omZ6n9PArcEBGHyu1bI+LXgP5yZvbvA48BHcAnU0ovR8SCc8r/fwd4ICImgePA\nHc17G9Klpb5n9FqYnKU2k1KaBe6s232k5vg+YN8KziGl9D8Uk8UkZcQJYZIkZcbkLElSZkzOkiRl\nxnvOkrJVrVY5PzHK2OaiNeT42TPM9rku+nKz4HMwNjpfh/tSZXKWlK1z42O8xGFGZ3YAcGz2KANT\n1va81NV3rRo+8TJDff/H6EyxTOnkzHE6Xq3S0dENXJqNMkzOkrLW2983X3O5t39Ti6NRM9R3rTo2\ne5SBni2vaZTxYvXbjM0MA5dmVyuTsyQpO7WNMBr9Udbbv3hXq0shWZucJUlt71JrQWlylrRq9f2Y\nq9Uq0EGl0tFw20YXapZ2b0Fpcpa0ao36MVe6eubrCzfatl+ztDyTs6Q1mevHDHB27DSdnb1Lbkta\nnkVIJEnKjFfOkqTLTqPCJjnN6DY5S5IuafVFTWBhYZPcZnSbnCVlw3Kd2gj1RU1gYWGT3GZ0m5wl\nZcNyndootUurYO3V5jZ6WNzkLCkrlutUO6j/Q3K9h8VNzpKky179femzoyNUOnvZtMSVce0fkus9\nLG5yliRd9hY02xg/SndvN2Mz3w80f8KYyVmSJBY22+ju6WnZhDGTsyRJy6gf9t7olQQmZ0mSltGw\nx/QGriQwOUuStAJL9ZheMKFsjXXkTc6SJK1R/ZX1mXOngF9c9euZnCVJWge1V9aTk2ubPWZylnRR\nPvvFxzl/vgrA0MlXYPOVq34ty3VKjZmcpTYTER3AA8CbgXPAbSml52qO3wzcA0wBD6WU9i52TkS8\nHngYqALPpJTuWu7rT3VuY3ZT8atjsjJO10x11e/Fcp1SY/ZzltrPLUBvSultwHuB++YORERXuf0z\nwE8Bd0TEjiXOuQ94X0ppN1CJiF9q2rsozVVZ2jw4YLlOqWRyltrPLmA/QErpSeCammNvBJ5NKY2m\nlKaAA8DuBufsLJ+/M6V0oHz8FYqkvmGq1SrjY6OMnRlh7MxIMYw96zC2VM9hban9bAFq12lMR0Ql\npVRtcGwM2AoM1u2fiYhOoKNm35nyuUt64cg3mJoqHo+cOkHf1quYmSkmvwy9coxKV8+i28de+C4n\nel9gC1cAcGL0JfoqW+ju7gZg/PQoXb09nB4aWtX23L6ZqUm6u1f3Gs3YbmZ87RBjs+NrRoxjIxd6\nR6+GyVlqP6MUyXbOXGKeO1Z703YQOLXIOTMRUa177rK/UT589+0dyz1H0to4rC21n0PALwBExLXA\n0zXHDgNXR8S2iOgBrgf+A3hikXO+ERFvLx//PMUwuKQW6/B+j9ReamZev6ncdSvFPeT+cmb2TcAH\nKYasP5lSerDROSmlIxHxBuATQDdFYr89peQvBanFTM6SJGXGYW1JkjJjcpYkKTMmZ0mSMuNSKkkr\nslzZ0BbF1AV8CvghoAe4F/gOF1mStBki4nuBpygKvcyQUYwR8R6KFkpdwF9TrAh4mHzi6wD2AkHx\nvbudjL6HEfFW4E9TSnsWK4kbEbcDd1CU1b03pbRvqdf0ylnSSi1aNrSFfh04mVJ6O/BzFIml5SVJ\n65V/RDwIjJe7sokxInYD15U/1z3A63OKr3QjxWqEXcAfAX9CJjFGxLspVjz0lrsWxBURVwLvAq6j\n+Jx+JCK6l3pdk7OklVqqbGirfJ6iyQdAJzANvKWZJUlX6C+AjwPHKJa45RTjzwLPRMSXgC+X/3KK\nD4qRmq3lFfRWiqvPXGL8LvDLNdv1JXFvAH4COJhSmk4pjQLPcmFZY0MmZ0kr1bBsaKuCAUgpjaeU\nzkbEIPAF4P2soiTpRoqI3wReTSn9Mxdiq/2+tTrG7RTr5H8FuBP4HHnFB3AQ2Az8L/A3wP1k8nNO\nKT1K8UfhnPq4trCwfO5cWd1FmZwlrdRSZUNbJiJ+EPhX4NMppb+nuNc3Z0UlSTfYrcANEfE1ivv1\nnwF21BxvdYxDwFfLq7ojlFepNcdbHR/A3cChlFJw4XvYU3M8hxjnNPr8NSqru2S8JmdJK7VU2dCW\nKO/lfRW4O6X06XL3N3MqSZpS2p1S2pNS2gP8N/BO4CsZxXiQ4j4oEfF9QD/weHkvGlofH8AAF648\nRygmrn0zsxjnNCqJ+1/ArojoiYitwI8Czyz1Is7WlrRSj1JcAR4qt29tZTCl9wLbgHsi4gPALPB7\nwF+VE24OA4+0ML7F/CHwiRxiTCnti4jrI+I/KYZk7wSOAntziK/0UeChiDhAkbfeA3ydvGKcs+Bn\nm1KajYj7Kf4Q6qCYMDa51ItYvlOSpMw4rC1JUmZMzpIkZcbkLElSZkzOkiRlxuQsSVJmTM6SJGXG\n5CxJmYqI3RHxjxfx/A9FxE9uZExqDpOzJOXtYopR7KZoAKI2ZxESScpUWZ7yY8Aw8DrgCeB3KTow\nfZiiWtbzFH2Cb6Lot/0yRZek7cAfUzSMuIKixOkXm/wWtEpeOUtS3q4GbkspvYmiIcX7gI8AN6aU\ndgKPAX+WUvos8BTw2ymlbwN3lY+vAW4DPtiS6LUq1taWpLw9nlJ6sXz8d8DDFEPdXyv7G1coOkvN\nmWtZ+E7gHRHxq8C1FA0t1CZMzpKUt/pewR3AgZTSLQAR0ctrW3nOOQg8Dvxb+f/nNjZMrSeHtSUp\nb3si4qqIqAC/AdwHXBcRbyiP3wP8efl4GuiKiCsohsM/kFLaD9yIE8XaislZkvL2DPC3wLeAF4F7\ngd8CPh8R3wJ+HPiD8rn7gQeBAPYC34mIg8AYsCkiNjc5dq2Ss7UlScqMV86SJGXG5CxJUmZMzpIk\nZcbkLElSZkzOkiRlxuQsSVJmTM6SJGXG5CxJUmb+H3BUsR3qSnrcAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11dd6f3d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(8,8))\n",
    "plt.subplot(2,2,1)\n",
    "\n",
    "plt.hist(stan_nuts['p'], range=[0,1], bins=50, normed=True, label='NUTS', alpha=0.5)\n",
    "plt.hist(meanfield_samples['p'].values, range=[0,1], bins=50,\n",
    "         normed=True, label='Mean-field ADVI', alpha=0.5)\n",
    "plt.xlabel(\"p\")\n",
    "\n",
    "plt.subplot(2,2,2)\n",
    "plt.hist(stan_nuts['alpha'], range=[0,100], bins=50, normed=True, label='NUTS', alpha=0.5)\n",
    "plt.hist(meanfield_samples['alpha'], range=[0,100], bins=50,\n",
    "         normed=True, label='Mean-field ADVI', alpha=0.5)\n",
    "plt.xlabel(\"alpha\")\n",
    "plt.legend()\n",
    "\n",
    "plt.subplot(2,2,4)\n",
    "plt.hist(stan_nuts['beta'], range=[0,100], bins=50, normed=True, label='NUTS', alpha=0.5)\n",
    "plt.hist(meanfield_samples['beta'].values, range=[0,100], bins=50,\n",
    "         normed=True, label='Mean-field ADVI', alpha=0.5)\n",
    "plt.xlabel(\"beta\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### pystan Full-rank\n",
    "\n",
    "We can also look at the output of the Full-Rank flavour of ADVI:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:pystan:Automatic Differentiation Variational Inference (ADVI) is an EXPERIMENTAL ALGORITHM.\n",
      "WARNING:pystan:ADVI samples may be found on the filesystem in the file `/var/folders/ks/ny8mvlt11hbfnk6vcdd8ghx4b6qs2l/T/tmpIbhBl1/output.csv`\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OrderedDict([('args', {'algorithm': 'FULLRANK', 'random_seed': '1131162010', 'tol_rel_obj': 0.01, 'eval_elbo': 100, 'sample_file': '/var/folders/ks/ny8mvlt11hbfnk6vcdd8ghx4b6qs2l/T/tmpIbhBl1/output.csv', 'output_samples': 10000, 'init_radius': 2.0, 'chain_id': 1, 'iter': 10000, 'adapt_engaged': True, 'init': 'random', 'eta': 1.0, 'grad_samples': 1, 'elbo_samples': 100, 'append_samples': False, 'enable_random_init': False, 'method': 'VARIATIONAL', 'adapt_iter': 50}), ('mean_pars', [])])\n"
     ]
    }
   ],
   "source": [
    "fullrank_results = stan_model.vb(data=stan_data, iter=10000,\n",
    "                                 output_samples=10000, algorithm='fullrank')\n",
    "print fullrank_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fullrank_samples = pd.read_csv(\n",
    "    fullrank_results['args']['sample_file'],\n",
    "    skiprows=7,\n",
    "    names=['alpha','beta','p']\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeEAAAHwCAYAAACRwNcwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3XucZGV17/9PV3dXd9elp2e6GxRBUNElRkQBA0EYMBlC\nPAEdE5ITThITchBFRPxhPBENJsZozA+DJiLeRonmePITxgDqKCDeGRMSkeSMkawZGK7DDNO36ltV\nX6p3/f7Y1T3VNX2b7q7au3u+79eLF7P3s3fVqurevfbledbTUCqVEBERkfpLRB2AiIjI0UpJWERE\nJCJKwiIiIhFREhYREYmIkrCIiEhElIRFREQi0rSUjczsPcDry9vf7O5fqmlUIjIvM2sAbgFOA8aA\nK9x9b0X7JcANwCRwq7tvM7NmYBvwYmACuNbd/6PuwYvILIteCZvZ+cAvufs5wGuBF9Y8KhFZyFag\npXxMXg/cNN1gZk3l5S3ABcCVZtYNvBkolPe5EvhCvYMWkcMt5Xb0RcDPzOxO4Gvl/0QkOucCdwO4\n+wPAmRVtpwB73H3I3SeBHwHnA78AfKu8z27geWbWXteoReQwS0nCXcAZwKXAVcD/qWlEIrKYdmCw\nYrloZol52kbK6x4CLgYws7MJj+t07UMVkYUs5ZlwH/CwuxeB3WY2ZmZd7t4718alUqnU0NCwqkGK\nrFPLPVCGgGzFcsLdg4q2yivcLJAD7gJeZmY/BH4M7Ab6F3oTHcsiS7bsA2UpSfh+4B3Ax8zsOCBF\nmJjnjqShgZ6e4eXGU3Pd3dlYxweKcTXEPT4IY1ymnYRXtdvLV7W7KtoeBk42sw4gD5wH3Aj8IvBd\nd7/OzM4AftHdxxd6k7gfyxD/n3Pc4wPFuBpWcCwvnoTdfYeZnWdm/0qY7d/m7pr1QSQ6dwAXmtnO\n8vLlZnYZkC73hL4OuJfweP28u+83s3HgK2b2XqBA2FFLRCK2pCFK7v6eWgciIktTPgm+qmr17or2\nHcCOqn36gQtrH52IHAkV6xAREYmIkrCIiEhElIRFREQioiQsIiISkSV1zIpKEATkcgOz1nV0bCSR\n0LmDiBxd5vp7uFIdHRtX9fXkyMU6CedyA2y/bxepTFh7ID8yxKVbTmXTps6IIxMRqa/qv4crNf33\n9NhjN6zK68nyxDoJA6Qy7WSyHVGHISISuSj+Hj700INcf/27+Id/uI3u7mMA+PSnb+bEE0/i05/+\nBHfddc/Mtg888M985zv3cuGFv8aXvhTOEfKzn/1fTj31NADe/vZ3kk5n+Nu//SjF4hT5/CinnfYq\nrrrqmrp+pjiJfRIWEZFoNTcn+fCHP8DHPvbJqpa5qzW++tVn8epXnwXAG97wa/zd3316pu3977+e\nSy/9HX7xF88G4H3vezc/+tH3Oe+8C2oReuzp4aqIiCzo9NPPpL29na9+9bYVv9amTZ1885tfZ9eu\n/6BYLPIXf/GRozYBg5KwiIgsoqGhgXe963puv/0f2bfv6UW3Xcjb3/5OfuEXTuUzn/kkr3/9RXz4\nwx9gZGRkNcNdU5SERURkUe3t7VxzzXX85V/+GaVSOH1A9UiVQiFPS0vLgq/z05/+hN/6rd/h5ps/\nyz/90w7a2tr44hc/X7O4407PhEVE1oj8yFCkr/Wa15zHD3/4Pb75za/ztre9g+c+9zh++tOfcPrp\nZwJhx6xXvOKVVXvNnu/nk5/8W1paWjjttFfR2trKCSc8n8HBQY5WSsIiImtAR8dGLt1y6qq/5pG6\n9tp38dOf/gSAP/mTP+WjH/0rPvvZWwiCgF/4hVO56KL/VrXH7NvTH/zgX/Gxj93IyMgwTU1NHHfc\n8fzxH1+/3I+w5ikJi8iiVDgneolEIpIaCa961Rm86lVnzCynUmluv/1rM8uf+MRnFtz/rrvunrX8\n/OefNEcv66OXkrCILCqXG+DOXd8g3Z4BYHRohK2nXqzCOSIrtKaTsM7O5WhkZg3ALcBpwBhwhbvv\nrWi/BLgBmARudfdt5X22AQZMAW92992HvfgC0u0ZMhuyq/QpRATWeBJWWUs5Sm0FWtz9HDM7C7ip\nvA4zayovnwEUgJ1mdhdwOpB293PNbAvwYeDS5QYQBAEDAzoBFlmpWCfhIAgYHTnUa250ZJAgCGZt\no7KWchQ6F7gbwN0fMLMzK9pOAfa4+xCAmf0I2Az0AhvKV8QbgImVBFAYyXNP33fp7A5PeHV7WmR5\nYp2EBwcH2Tv2INm2sAff8NgAg4Mn0NXVHXFkIpFqByrHdBTNLOHuwRxtI4RJ906gDfgvoBO4eKVB\npDIp3Z6uI82itD7FOgkDtKRTtGXDziATE+MRRyMSC0NAZfabTsDTbZXT7GSBHPC/gJ3u/j4zex7w\nPTN7ubsveEXc3R2+TSIxQVtPklQ6LMTQ2tZMU7JpZnlqcpyuriydnfVPytMxxtVqxdfX18c9j947\n0zlupUaHRvjds34TiP93CGsjxuWIfRIWkcPsJLyS3W5mZwO7KtoeBk42sw4gD5wH3Ej4jHj6CjlH\neOw3LvZGPT3DAPT3D1PIT9DYHJ4IjxUmSRQD8qPhciE/QW/vMEGQXPmnOwLd3dmZGONoNePr7x8m\n0ZSksXnhilRLlWgKf2adnZ3zxnjgwH7+4A9+B7NTKJVKNDQ0cPrpZ/KHf3jFnNt/+MMfYMuWi+jr\n6+WJJx7nrW99+6rEOjExxDXXXMtnPnPrgtt9+ctf5Lbb/pHt279Oc3PzTEzu/8WGDRsoFot0dGzk\nHe+4jvb2DfzO77yR2267i9bW1pnX+KM/+l0++MG/5iMf+SDvfvd7ef7zT1w0vpWcICgJi6w9dwAX\nmtnO8vLlZnYZYcerbWZ2HXAvYZWEz7v7fjO7Ebi1/Iy4Cbje3QuRRC9rygte8KJZsyAt1WI1pGvx\nevfeezdbtlzEfffdw+ted+iJy9VXXzsza9N//Me/c8MNf8LnPvclXvOazXzve/fNbOv+X2SzG3je\n845f1dgXoiQsssa4ewm4qmr17or2HcCOqn1ywBtrH52sN9N1ois99NCD3HnnV/nABz4MwBvecNGs\neYXnc801b2Hjxk0MDw/xl3/51/z1X3+IkZER+vp6eOMbf4utW3+Ta655Cy9+8UvYu/dR8vk8H/zg\nR9i4MQWEz8U/9KE/54UvfBG/+7t/cFhMxx9/PFu3/iZ/8Rd/OisJVzrttFfS1NTMvn1Pc8klb+Az\nn/nkzLY7dtzFG97wG0f0/ayUxhOIiMi8Hn98L+94x1u55pq38I53vJXe3l6g+sp06Ve9F154ER/7\n2CfZt+9ptmy5iJtu+gR/8zc385WvfHlmm5e97OV8/OO3cOaZv8h994XJvVgs8oEP/CmnnvqKwxIw\nwDe+cScXX7yVE054Ps3NSR5++D/njWHjxk0MDuZ42ctezvDwED09B5mcnOTBB/+NzZsvWPJnWQ26\nEhYRkXnNdTv6qaeeqNrq8KtlgO9//zt89au30dDQwNVXvxMIy1ZCOK/wbbf9Iz/4wXdJpdIUi1Mz\n+73kJQbAMcccy8BAPwCPPLKbdDpDPn/4U5Th4WH++Z9/zMBAju3bv8Lo6Chf/ept/OmffmDOuA4c\n2M8xxxwLwK//+hu4++5vctxxx/Ga12ymqam+aVFJWERkjRgdWr15d5f6WnPdjk4mW+jrC6+IDxzY\nz9DQ3DMyXXDBr3DBBb8ya930FfQ//uP/5uUvfwVbt/4mP/3pT/iXf9lZudVhr/XSl76MG2/8OG9+\n85s466xf4kUvOnmm7Z57dnDxxW/gbW97BwDj42P89m+/gVwud9hn+Ld/+xfa2tpmhrr+6q++juuu\neztdXV28/e3/z2Jfx6pbUhI2swc51LPyMXf/n7ULaX6lIGBwMEd/fx8AAwMDhxXvEBFZjzo6NrL1\n1BUP7z7sNRczV4eol770FDKZDG95y+WceOJJHHfc85b0fpWv9ZrXnMfHP34jP/zh93nBC15AKpVi\ncnJywQ5YyWSSd73rPXzoQ3/GZz/7xZmr1h07vsYNN/zFzHYtLa2cf/4v8/Wv3wnApz71Cb785S/S\n0JAgnU7zgQ/81cy22WyWE088if7+/lkdsla7Y9l8GuY6y6lkZi3Aj939jAU3PKS0Wl3yH330Eb74\n7/eyoTOswvPsE0/y3E0tHHd8+APv6+mjMfdijn1u2IV8ZDjHfzv7xAWr9sR9SAMoxtUQ9/gAuruz\n9TnKl690aIhSH99+4vszxTmefWo/ieZGup9zDAAjg8NceOIFda+YFfefc9zjA8W4GlZyLC/lSvg0\nIG1m9xCOK3yfuz+w3DdcqbaKKj2FQoHxXFSRiIiIrMxSekfngRvd/SLCYRFfNjP1qhYREVmhpVwJ\n7wYeAXD3PWbWBzwX2DffDqtVXiyXy9DclKC5OQyzqSlBW+uh0nmpVJJEW5J0ebk42UQiMUkicagS\n38aNh8/sshbKnynGlYt7fCIiS0nClwOvAK42s+MIa9HuX2iH5d67ry5Q/vjjTzMxOcXkZBGAYjGg\nMDYxUyovn59gvDDBaHm5r7ePL33tGTZ1hc+p5praMO7PFkAxroa4xwc6SRCRpSXhzwNfMLMfEg4G\n+6OKYvGrKpcb4M5d35gpUP7UY09SLB7Za6TSmtpQRETWhkWTsLsXgTfVIRYA0u2ZmY5Xbek2yKm8\nrYiIrE/qYCUiIhIRJWEREZGIrKmylaVSiYmxcQqF8BZ1vpBnbGSQkeFwsHB+ZIh0tr7FAkRERJZr\nTSXh4tQkj+0fZLQx7Lj1zKP7SCQOkJ8Ky1j2lZ7lBfnTowxRpObMrAG4hbCQzhhwhbvvrWi/BLgB\nmAS+4O6fN7M/AP6QsHNlW3nf57j73EV/RaQu1lQSBmhubiaZbAWgKZmkOZmkLRsm5dbRIRiPMjqR\nutgKtLj7OWZ2FnBTeR1m1lRePgMoADvN7Gvu/kXgi+Vtbga2KQGLRE/PhEXWnnOBuwHKJWTPrGg7\nBdjj7kPuPgncD2yebjSzM4GXufvn6xiviMxDSVhk7Wnn0KxmAMWKUrLVbcPAhorl64G5J1kVkbpb\nc7ejRYQhwsp10xIVBXSGCBPxtCyQAzCzDcBL3P0HS32j6apeicQEbT2HSsa2tjXTlGyaWZ6aHKer\nK0tnZ/2rgMW98ljc4wPFGCUlYZG1ZydwMbDdzM4GdlW0PQycbGYdhJOvbAZuLLdtBr5zJG90aCrD\nYQr5CRqbw04XY4VJEsVgpoRsIT9Bb+8wQZBc9odajriXJ417fKAYV8NKThCUhEXWnjuAC81sZ3n5\ncjO7DEi7+zYzuw64F2gg7IA1XevdgL2Hv5yIREVJWGSNcfcS4bSilXZXtO8Adsyx30drHJqIHKF1\nnYSDIGBgYGDWus7OdETRiIiIzLauk3AhP8yOnb2zpjZ8a1cWqO9zKxERkbms6yQMmtpQRETiS+OE\nRUREIqIkLCIiEhElYRERkYgoCYuIiERESVhERCQiSsIiIiIRidUQpSAoUSgUaEyGYY1PjEOpFHFU\nIrKYuQrjAHR0bCSR0Lm+yHxilYQHB3P4kz20b5oC4Jl9/bRlMhFHJSKLKYzkuafvu3R2d86sGx0a\nYeupF7NpU+cCe4oc3WKVhAGam1tIJlsBaEoeWWWrUhCQHx1mZDgHhBWy0ln9ARCph1QmRWbD+pxu\nTqRWYpeEV2K8MMb+xocpTg0B0Fd6lhfkT484KhERkbmtqyQM0JJqoy0b3sJuHR2C8UNtQRDQ399P\nEDTPrNMzKxERicq6S8ILKeSHuf3bP6ctHdaSzo8McemWU/XMSkREIrGkJGxmxwA/Aba4++7Fto+z\nVCZLOqMJHWTtMrMG4BbgNGAMuMLd91a0XwLcAEwCt7r7tvL69wCvJzzub3b3L9U7dhGZbdH7sGbW\nBHwayNc+HBFZgq1Ai7ufA1wP3DTdUD5ebwK2ABcAV5pZt5mdD/xSeZ/XAi+se9QicpilPAz9KPAp\n4JkaxyIiS3MucDeAuz8AnFnRdgqwx92H3H0S+BFwPnAR8DMzuxP4Wvk/EYnYgknYzP4QOOju3wYa\n6hKRiCymHRisWC6aWWKetpHyui7gDOBS4Crg/9QhThFZxGLPhC8HAjO7EHgl8CUze727H1xop+7u\n5Y0VzOUyNDclaG4Ow2pKNNLYvILlxkZaU02k0y0AtLWF446nl4OpJF1dWTo74ze2cbnfYT3FPca4\nx7cCQ0Dlh0u4e1DR1l7RlgVyQB/wsLsXgd1mNmZmXe7eu9AbTX+HicQEbT1JUuVjp7WtmaZk07zL\nAFOT43U5vuL+c457fKAYo7RgEnb386f/bWbfA96yWAIG6OkZXlYwfX0jTBYDJieLABSDKRomG5e9\nPDk5SX9ugFT62fD1e3rpOvY5jI6G45by+Ql6e4cJgiMrClJr3d3ZZX+H9RL3GOMeH6zoj8pO4GJg\nu5mdDeyqaHsYONnMOgj7cZwH3Eg4WO8dwMfM7DggRZiYFzT9Hfb3D1PIT9DYHB47Y4VJEsWA/Ojc\nywCFOhxfcf85xz0+UIyrYSUnCEcyRGnNFXGeq3hHOp8m294dcWQiK3IHcKGZ7SwvX25mlwFpd99m\nZtcB9xI+Qvq8u+8HdpjZeWb2r+X1b3P3NXdMi6w3S07C7v7LtQykVg4r3iGyxpWT51VVq3dXtO8A\ndsyx33tqHJqIHCGVihIREYlIpBWzgiAglzs0/dngYI6Spi4UEZGjRKRJOJcb4M5d3yDdHt4ufuqx\nJykWo4xIRESkfiKvHZ1uz8xMf9aWboNcIeKIRERE6kPPhEVERCKiJCwiIhIRJWEREZGIKAmLiIhE\nRElYREQkIkrCIiIiEVESFhERiYiSsIiISESUhEVERCKiJCwiIhKRyMtWisiRMbMG4BbgNGAMuMLd\n91a0XwLcAEwCt7r7tvL6B4HB8maPufv/rGvgInIYJWGRtWcr0OLu55jZWcBN5XWYWVN5+QygAOw0\ns7uAIajvvOBBEDAwMDBrXUfHRhIJ3YATmaYkLLL2nAvcDeDuD5jZmRVtpwB73H0IwMzuBzYDTwFp\nM7sHaATe5+4P1DLIwkiee/q+S2d3JwCjQyNsPfViNm3qrOXbiqwpOiUVWXvaOXRbGaBoZol52oaB\nDcAocKO7XwRcBXy5Yp+aSWVSZDZkyWzIzkxZKiKHKAmLrD1DQLZiOeHuQUVbe0VbFsgBe4AvA7j7\nHqAPeG7tQxWRheh2tMjasxO4GNhuZmcDuyraHgZONrMOIA+cB9wIXA68ArjazI4jTM77F3uj7u4w\n1ycSE7T1JEmlWwBobWumKdk07/Jc66Ymx+nqytLZmWU1TccYV3GPDxRjlJSERdaeO4ALzWxnefly\nM7sMSLv7NjO7DrgXaAA+7+77zezzwBfM7IdACfijiqvnefX0DAPQ3z9MIT9BY/M4AGOFSRLFgPzo\n3MtzrSvkJ+jtHSYIkqvxHQDhH+bpGOMo7vGBYlwNKzlBOKqScCkIyBeGGRnOATA6MkgQLPp3SCRW\n3L1E+Fy30u6K9h3Ajqp9isCbah+diByJoyoJjxfGeJL/ZGSqH4DhsQEGB0+gq6s74shERORodFQl\nYYCWdBtt2bCX5sTE+CJbi4iI1I56R4uIiERESVhERCQii96OLg/o/xxgQAC81d1/XuvARCS+SqUS\nY+NjFAoFAAqFAqmmdMRRiaw9S7kSvgQoufu5hEXhP1zbkEQk7sbGxnj0qT727MuxZ1+OR/b1Ma4+\nFiJHbNEk7O53AVeWF08CBubf+sgEQYlCoTDz3/jEOJRKq/XyIlJDTckkyWQryWQrzU2rN/ZX5Giy\npN7R7h6Y2a3AG4FLV+vNBwdz+JM9tG+aAuCZff20ZVRfVkREjg5LHqLk7peb2Z8A/2pmp7h7Yb5t\nl1o9JJfLkGprI50OE29raxuNzQmam8OwmhKNq74MzCw3NyXo7MzEshxaHGOqFvcY4x7felIqlZgY\nG595Rgx6TiyyFEvpmPX7wPHu/leEE4hPEXbQmtdSy4v19Y0wWQyYnCwCUAymaJhsrOlyE80zy5PF\ngL6+ETo64lUOLe4l2iD+McY9PlhfJwnFqUke2z/IaOOhO1kH9/VhLzzqShGIHJGlHCHbgb83sx+U\nt7/W3dUDQ0RmaW5uJplsPbSs58Qii1o0CZdvO//3OsQiIiJyVFGxDhERkYjogY2I1ER1Z61CoUAQ\naAiiSCUlYRGpierOWkP9/Zx3bI6urq6IIxOJDyVhkTXGzBqAW4DTCEcsXOHueyvaLyGsbjcJ3Oru\n2yrajgF+Amxx993UWGVnraamJIODOfr7+2baOzo2kkjoqZgcvZSERdaerUCLu59jZmcBN5XXYWZN\n5eUzgAKw08zucveectungXwUQReLE9z7wGMcfyBMuvmRIS7dciqbNnVGEY5ILBzVp6ClIJg5M6/8\nLwgWHAYtErVzgbsB3P0B4MyKtlOAPe4+5O6TwP3A5nLbR4FPAc/UMdZZUqksmWwHmWwHqUx7VGGI\nxMZRfSU8Nprn23u/x7Fj/zWzLj+S57dP/w09t5I4awcGK5aLZpZw92COtmFgg5n9AXDQ3b9tZu+t\nY6wisoCjOgkXpyY5MNRA0D41s26of5TBQXUekVgbAirLbU0n4Om2ykvMLJAD3gGUzOxC4JXAl8zs\n9e5+cKE3mq7qlUhM0NaTJJVuAaAtlaRpeGLeErFzrWtqbKQ11US6/BrBVJKuriydnSurHBb3ymNx\njw8UY5SO6iQMc1T5aW6JMBqRJdkJXAxsN7OzgV0VbQ8DJ5tZB+Gz383Aje7+T9MbmNn3gLcsloDh\nUAna/v5hCvkJGpvDYnmF/ATFqal5S8TOta44NcXYeJHR0fA18vkJenuHCYLlV9aKe3nSuMcHinE1\nrOQE4ahPwiJr0B3AhWa2s7x8uZldBqTdfZuZXQfcCzQA29x9f9X+GqwrEhNKwiJrjLuXgKuqVu+u\naN8B7Fhg/1+uUWgicoSUhEWkLkpBQH50mJHhHACjI4MaiSBHPSVhEamL8cIY+xsfpjg1BMDw2ACD\ngyfQ1dUdcWQi0VESFpG6aUm10ZYNy1hOTGhGVBElYRFZVBCUKBQKNCbDPxnjE+NQUv8ukZVSEhaR\nRQ0O5vAne2jfFI6pf2ZfP22ZTMRRiax9SsJVSqWSisyLzKG5ueXQZAzJ5Y/tFZFDlISrqMi8iIjU\ni5LwHKaLzIuIiNSS7rGKiIhERFfCIhIJ9b8QURIWkYio/4WIkrCIREj9L+RopyQsInN69uCz7D/Q\nH/6751mKU5MRRySy/igJi8ic7n7oO+Sbw+IcvSO9jBQG6OQ5EUclsr4smITNrAn4AnASkAQ+5O5f\nr0NckdFMLxJ3ZtYA3AKcBowBV7j73or2S4AbgEng1vIcwwngc4ABAfBWd//5Qu+TTCYppRoAaE21\n0tDQUIuPI3JUW6wb4u8Bve6+GXgdcHPtQ4rW9Ewvj089xONTD7F37EEGBwejDkuk0lagxd3PAa4H\nbppuKJ843wRsAS4ArjSzbuASoOTu5xIm6A/XO2gROdxiSfg2wgN2etuj4qHQ9EwvbdkMLelU1OGI\nVDsXuBvA3R8AzqxoOwXY4+5D7j4J3A9sdve7gCvL25wEDNQvXBGZz4K3o909D2BmWeB24H31CEpE\nFtQOVN6eKZpZwt2DOdqGgQ0A7h6Y2a3AG4FL6xWsiMxv0VHxZnYC8F3gi+7+ldqHJCKLGAKyFcvT\nCXi6rb2iLQvkphfc/XLgJcA2M2urdaAisrDFOmYdC9wDXO3u31vqi3Z3Z+dcXywWeeyxx2aWh4d7\naWxsoLk5DKMp0Uhjc6Kmy8C87XOta25K0NmZmfcz1Uq932854h5j3ONbgZ3AxcB2Mzsb2FXR9jBw\nspl1AHngPOBGM/t94Hh3/yvCzlxThB20FpRKt4T/TyVpbFz6sbaUbZoaG2lNNZEuv0cwlaSrK0tn\nZ/hzC4KAgYHZd803bjy8olbcf85xjw8UY5QWG6J0PdAB3GBm7wdKwOvcfXyhnXp6hudc/+ijj/DJ\n799GpiMcnP/sE0+wobuLTMcmAIrBFA2TjUxOFmu23ETzvO1zrZssBvT1jdDRMfdnqoXu7uy832Fc\nxD3GuMcHK/qjcgdwoZntLC9fbmaXAelyT+jrgHuBBuDz7r7fzLYDf29mPyA87q9d7DgGyI+Gm+Tz\nE0xNBUs+1payTXFqirHxIqMV79HbO0wQhNMk9vf3sf2+XaQy4YX9XBW14v5zjnt8oBhXw0pOEBZ7\nJvxO4J3LfvU5ZDo62NAZHkTDOfUNETlS7l4CrqpavbuifQewo2qfAvDfax/d0lUPBxweGqCvLzPT\nPjAwQFtaFbVkfVOxDhGJxPRwwOLUEAB9o88yeF8/x5/4IgB6DzxNZkMX2faFXkVkbVMSFpHITA8H\nBGgdHSLVdOjKd3RE4/Nl/dOcYSIiIhFREhYREYmIbkcvohQEmnhcRERqQkl4EeP5Aj/Y92P2NDwB\nwOjQCFtPvVgTj4uIyIopCS9BWyZFZsP6HCguIiLR0T1VERGRiOhKeBGlUomJsXEKhQIAhUKBIChF\nHJWIiKwHSsKLKE5N8tj+QUYbw7GMQ/39nHdsjq6urogjExGRtU5JeAmam5tJJlvL/26JOBqR9am6\njGV+ZIh0Vh0gZX1TEhaRWDisjGXpWV6QPz3iqERqS0lYRGKjuowli87zJLK2qXe0iIhIRHQlLCJr\nQhAEDAzMnv60szMdUTQiq0NJWGSNMbMG4BbgNGAMuMLd91a0XwLcAEwCt7r7NjNrAr4AnAQkgQ+5\n+9frHftKFPLD7NjZy6auY4Cw49Zbu7KEH0dkbdLtaJG1ZyvQ4u7nANcDN003lJPtTcAW4ALgSjPr\nBn4P6HX3zcDrgJvrHfRqSKXbyWQ7yGQ7SGU00bCsfUrCImvPucDdAO7+AHBmRdspwB53H3L3SeB+\nYDNwG+HVMYTH/WT9whWR+eh2tMja0w5UznhfNLOEuwdztA0DG9w9D2BmWeB24H31ClZE5qckLLL2\nDAGVM4pMJ+Dptsr7tFkgB2BmJwD/BNzs7l9Zyhul0mFxmlQqSWNjgubm8E9GU6KRxub5l5eyzaLL\njY20pprg+lvtAAAgAElEQVRIl2Noa0vS2NQ8sxxMhc+Cu7vjPblK3OMDxRglJWGRtWcncDGw3czO\nBnZVtD0MnGxmHUCe8Fb0jWZ2LHAPcLW7f2+pb5QfDQfq5vMTTE0FTE4WASgGUzRMNs67vJRtFl2e\nmmJsvMhoOYZCYYLGxoaZ5Xx+AoCenuGlfpy66+7Oxjo+UIyrYSUnCErCR6gUBAwO5ujv75tZ19Gx\nkURCj9elbu4ALjSzneXly83sMiBd7gl9HXAv0ABsc/f9ZvZxoAO4wczeD5SA17m7ymGIREhJ+AiN\n5wv8YN+P2dPwBACjQyNsPfViNm1SjVupD3cvAVdVrd5d0b4D2FG1zzuBd9Y+OhE5EkrCy9CWSZHZ\nsD6fT4iISP0oCYvImhQEAf39/QRB88wyNJBINMxso0dFEndKwis0Vyk9HfgitVfID3P7t39OW7oD\ngN4DT5NoSs6qqHXpllP1qEhibUlJ2MzOAj7i7q+tcTxrTmEkzz1936WzOzzQ9YxYpH5SmSzpTJiE\nR0cGaWxsIZPtiDgqkaVbNAmb2buB3wdGah/O2pTSM2KRVVcKAvKjw4wM54Dwyjad1cmtrC9LuRJ+\nBHgj8A81jkVEYqS3b4DxQvjvgdwgU1NTdX3/8cIY+xsfpjg1BEBf6VlekD+9rjGI1NqiDy7d/Q6g\nuNh2R4tSqcTE2DiFQmHmv1KpFHVYIqvusX19HBwucXC4RN9oiYnJ+iZhgJZUG23ZDG3ZDK3ptrq/\nv0it1aRj1nzVQ3K5DM1NKyhjtwrLwIpK7VEKeKpnmImWsHrLswdyvHxDaqa839TkOF1dWTo7V3Z7\nei2UaIt7jHGPL+4SNMx0MFyLHQ3VaVLWgiNJwg2LbxKar7xYX98Ik8Wll76rxXITzSsutdfclKQh\nEQ6LaGhopDA2MVPer5CfoLd3mCBY/hyncS/RBvGPMe7xgU4Sam2u+YfVW1ri5kiS8BHfcw2CgFzu\n0Jno4GAO3bkVkeWYq6NWKp1acJ/p+YdF4mpJSdjdnwDOOdIXz+UG2H7frpnJt5981Jk6Ro+XReTI\nzdVRK51Pk23vjjgykeWrebGOVObQmWhbOsMIfYvsISIyt+mOWgCto0MRRyOycuqhICIiEhGVrRSR\no4J6S0scKQmLyJpUCgLyhaVX1FJvaYkjJeFVprNtqTUzawBuAU4DxoAr3H1vRfslwA3AJHCru2+r\naFs3deDHC2M8yX8yMtUPLK2iVmVv6epjVbMwSRSUhFeosoIWQH9PH3f3foeuY7oATeggNbEVaHH3\nc8pJ9abyOsysqbx8BlAAdprZXe7esx7rwLekqzpqjS993+orY83CJFHQKd4KFacmeWz/IHv25diz\nL8cj+/poTDaS2ZAlsyFLuj0TdYiy/pwL3A3g7g8AZ1a0nQLscfchd58E7gc2l9um68BL2fSVcSbb\nQVsmO2t5emilSC0pCa+C5uZmkslWkslWmpuWXylLZInagcGK5aKZJeZpGwY2gOrAi8SRbkfXmJ4R\nSw0MAZU1LxPuHlS0VV7CZYHcct9oumZ6c1OCxkTDkuu0L2WbVa8F39hIa6qJdLmOe1tbksam5mUv\nB1NJ1YGPkbUQ43IoCddYYSTPPX3fpbM7fK6kZ8SyCnYCFwPbzexsYFdF28PAyWbWAeQJb0XfWLX/\nkuvAT9dMnywGTAWlJddpX8o2q10LfnJykv7cAKn0swD09fSSznYyminXdS9M0NjYwOjo0pbzqgMf\nG3GPcSUnCErCdZDKpMhsWJ9ncRKJO4ALzWxneflyM7sMSLv7NjO7DriXMNluc/f9Vfuvywrumn9Y\n1iIl4VVW3Vu6UCiQakpHHJWsJ+5eAq6qWr27on0HsGOefZdVB36tqCxr2TKcI59b+jjianqUJPWg\nJLzKpntLjzaGfwgO7uvDXqivWaTeVnplXD2EaWQox4Wvfj4bN26c2UZJWVaqptkhCAJGRw511MyP\nDlNKrcs7YbNM95YG1FtaJEKHTfhwBOOIYXZxj9GRQXbsfETjiGVV1TQJDw4OsnfsQbJt4ZnjM6XH\nyUxq7J2IrE2an1hWW83vk7akU4ee0aRba/12sbeUUnmdnXqGLLIeBUFALqfnzHKIHlbWWfWQpYP7\nnqUx2TRrCNMVXZcBuo0tsppKQUB+dPkdtaotp+NWLjfA9vt2zVTj0i1tURKOQOWQpdGhERLNjRrC\nJFJj1R21eqcO8Jyel5BKp4AjT8rLnZUpldEtbTlESbjGNGRJJD4qO2o1HGTF44oXmpUJ9GhJFqck\nXGMasiQSX6s5rniuK+O3dmVZ6NGSxiKLskEdHMmQpSAI6O/vJwiaZ9bpoBSpvdWouFV9ZVx9LA8M\nDFAqHRqmudxb2lI79e48pyRcZ4vdni6M5Lnj/36LVDbsuKFa0yL1s9pXxrd/++e0pQ89/+098DSZ\nDV1kK0ZqathTvNS789yqJ+FisUixGBZUn5qaWu2XX/MWuz1dKpVoTDbSmAzXJZobCYL1X+BEJG5W\n5co4kyWdOZRgK4sXSX0s58q2np3nVj0Jf+62H1LIh2Vpnnzs55SOVwKpVnl7uqmxedaV8eDQIAdH\nJtjQFX5vg729vKL18VnjiHV7WqQ+VvPKWKIR92Fhq56EN3QeR1NrmIRTB59hjJ7Vfot1pfrK+Jl9\n/WQ2ts8k6WAy4Af7fsyehicAGM4N8drnn6f6tSJ1plma1q44DwvTM+EYmHVlnDy841Zb1bjiex49\nVOxDSVmkfha6Mh4dypFobKG1rXVmuVQqUio1zuy/2lfP1bdaqyvwzVWRT38f4mXRJGxmDcAtwGnA\nGHCFu++tdWAyv+piH5VJWR251r/FjkkzuwS4AZgEbi3PMazjeJVVXxk/k3+c5pZmRqaeN7Pc2tDC\nptRzZ/apLhBSnbiHhwbo68vMbL9YEq2+1dp74GkSTcmZ3tbVy3PNBHUkY5lXmvTnej5bvc2RWuul\nQJdyJbwVaHH3c8zsLOCm8jqJQKlUYmx8bHbv6mxaFbeOLvMek2bWVF4+AygAO83sLuDc+faR5Zt1\nZZxupTmZnLXckm6dWYbDC4RUJ+6+0WcZvK+f4098EQA9zzzJ+OQkGzu7ABgdHuSSzS+jszM8ye7r\n6yMoBTOvX6JEa1tm1sxPjY0tC84EVTmWebEkOzAwwH0/eYp0dgNw5Em/ev+5toEjS6LVJyJzvWfl\nsLAjtZR6/93dy//7u5QkfC5wN4C7P2BmZy773eSIVQ9pGhwa5JncGAOT4djD6t7Vi/3C6HbVurDQ\nMXkKsMfdhwDM7EfA+cAvLbCP1NGCiXs4N2u6xXxhhL7UUxTbyqU2Bw/wv782wHEnnATA04/tZrSr\nn02pchJc5Ep7+nb4dFIuFos8+uijM2OZBwb6+cG/7yPTHrb3HniaRGOSTd3h6/c9+wxdzzlxwSS/\nUNKfHqJV+Xx2sSkigyCgr6+P/v7hmX2q/0ZVPvOd7z2zy5zAr3osd/WJRn5kiPfZSct7cZaWhNuB\nyn71RTNLuHsw18YD/QfJ5ycAGBnJMUKOhobwD3x+cIimliSDfX2RLU9NTtDcPHd7HGMc6u/jPwf7\n2Z8Lv9ODTz1FazpNIT0KwMToGIN9A7S2hAdZ7/4evjLxKBs2hr+QAwf7SDQ3Lnl5LF/gtSedx4YN\nC3diyOUy9PWNLLhNlOIeH0B396uWu+tCx2R12wiwAcgusM+SjA+PLPn3eCnb1Pp4jsNyZXxL2Wfg\nwEF6pw7Q3/M4AD1DT5NKtDMxEWbmwvAQI0GOQqE3bC8+TWpsdvsjU/96aP9nn6YxmWRTIkwYQ4O9\nHDvwUkqlSSBM4v/y6F7ayleRffufIdmWYlNzuH3PcLj/QKG8f7GX8ScnZvbve/YZEk1JpqbGF1xu\naW0JP+/oMGPj4zPtc21TGBnm8ccfm7mYGBzM8YOHHoeG8EShkB/l1855yczfqMHBHL0HnpkZ/rXY\ne4avf3ilsmlLeb3qeFdiKUl4iPAAnrbggXv1ZZsPXVKxZdmBSfy9+MVRR7CwuMe3Agsdk0OEiXha\nFhhYZJ85ffTaP2lYqF2kXi66aGXtq/1+q2kp9xx3Av8NwMzOBnbVNCIRWcxCx+TDwMlm1mFmSeA8\n4J+BHy+wj4hEpGGxB9YVvSpfUV51ubvvrnVgIjK3uY5Jwo5Y6XJP6F8H/gxoAD7v7p/WcSwST4sm\nYREREakNdYEVERGJiJKwiIhIRJSERUREIqIkLCIiEpFlTeCwnNq1qxDrasd4GXBtOcZd7v62OMVX\nsd1ngD53f2894yu/92Lf4auBvykv7gPe5O6TMYvxjcB7gYDwd/HT9YyvIo6zgI+4+2ur1kd+rFSL\na53pcknOLwAnEdZZ/BDwc+DvCX++P3P3q6OKb5qZHQP8hLBQwhTxi+89wOsJ//7fTDjk7e+JQYzl\n371tgBF+d28mRt9h5XFsZi+aKy4zezNwJeEx/SF337HQay73Snimdi1wPWEd2ukgp2vXbgEuAK40\ns+5lvs9KLBRjK/AXwPnufh7QYWYXxyW+aWb2FuDldY6r0mIxfhb4Q3ffDHwHeEGd44PFY5z+XTwX\neJeZbaDOzOzdwOeAlqr1cTlWqi36uxmR3wN6y79vv0aYQG4C3uvu5wMJM3tDlAGWf6afBvLlVXGL\n73zgl8o/29cCL4pZjL9KONTuXOCDwIfjEt8cx/FhcZnZscA1hGVifw34KzNrXuh1l5uEZ9WuBeas\nXVu+Krof2LzM91mJhWIcB85x9+naaU2EZ/z1tFB8mNkvAa8GPlPnuCrNG6OZvQToA64zs+8DHRGN\nO13wewQmgI1AW3k5ijF5jwBvnGN9XI6Vaot9p1G5jfCuAUAjUAROd/cfldd9i+jL9H0U+BTwDOE4\n7bjFdxHwMzO7E/ha+b84xTgGbChfEW8gvJqMS3zVx/EZVXFdCPwicL+7F8v12/dwaGz+nJabhOes\nXTtP2zDhl1lv88bo7iV37wEws2sIz7zui0t8ZvYcwmILbyc8kKOy0M+5i/Bs7+8ID4otZnZBfcMD\nFo4RwtvlDxJWiPrG9MQG9eTudxAmjGpxOVaqLfadRsLd8+4+amZZ4Hbgfcw+PiL9/szsD4GD7v5t\nDsVV+b3F4efbRVjY5VLgKuDLxCvG+wlPmP+L8ALk74jJz3iO47g6rnYOr9E+Xbt9Xss9sI60dm1u\nme+zEgvWyjWzBjO7EfgV4DfqHRwLx/dbQCfwTeA9wP8wszfVOT5YOMY+4BF33+3uRcIrpyiumOaN\n0cxOILw1dCLhc8Rjzew36x7h/OJyrFQ74jrT9VL+mX4X+KK7/3+Ez+OmRf39XQ5caGbfI3ye/iWg\n8vFC1PFBeNzeU75S2035yrOiPeoY/xew092NQ99hsqI96vgqzfW7d8TH9HKT8JHUrt1MWLu23har\nef1ZwudeWytuS9fTvPG5+yfc/dXu/svAR4D/4+5filOMwF4gY2YvLC+fB/xnfcMDFo6xlfDMddzd\nS8BBwlvTUam+qxGXY6VaLOvFl5+33QP8L3f/Ynn1Q2Y2fQv/dcCP5ty5Dtz9fHd/bbnz3b8Dvw98\nKy7xld1P+KwSMzsOSAPfKT8rhuhjzHDoSjJH+KjwoRjFV+mnc/xs/w0418yS5f4nLwV+ttCLLKt3\nNHAH4RnfzvLy5eXextO1a68D7iX8o7PN3fcv831WYt4YCW9PXg78qHzWWgL+1t3vikN8ceghW7bY\nz/l/Av9oZgA/dvdvxTDGLwE/NrMC8Chhb8aolGCmZ36cjpVqh32nUQZT4XqgA7jBzN5P+H1eC3yi\n3PnlYWB7hPHN5Y+Bz8UlPnffYWbnmdm/Ev7OXQU8DmyLSYw3AreW58FuIrwT+GCM4qt02M/W3Utm\n9neEJzsNhB23JhZ6EdWOFhERiUjknS1ERESOVkrCIiIiEVESFhERiYiSsIiISESUhEVERCKiJCwi\nIhIRJWERkRgxsxPN7LFFtvmz8lhpWeOUhEVE4kcFHI4Sy62YJWtAudTbnxJWbjkeeIBwbti6zvkr\nInMzs0bCWZdeDhwDOPCuivZbCWcCO52wDvEH3f3L5eazylXNjgP+3t0/UJ7c4vPA88rrf+juf1Cv\nzyNHTlfC69/ZwJXu/lLC2Ukin1RcRGacQ1jb/BzgxUCKct3uCs8jnCLvV4C/MbNjyuuPAc4nnDjl\n3WaWBn4deMjdXwO8BDjHzF5V+48hy6UkvP7d5+57y//+B+CXowxGRA4pz0f7KTN7G/C3wMmEkxhU\n+lx5+tV9hDWJzy2v/1Z5NqQ+oAfYVJ5Z6j4zuxb4BLBpjteTGFESXv+mKv6dYO55bUUkAmb2esI5\nfUeALxDOxPNE1WaVx2xjxXLl+hLQUJ4f/f8FniWci/dhop2TXBahJLz+XWBmx5YnZX8TEMVMRyIy\nt18BvlKeqvQg4XSWjVXbXAZhr2nC29JzTeU3nWi3AJ8pXxE3AK+c4/UkRpSE179nCM+0fwY8BcRl\nmkQRgc8B/6M8teCngbuA1zK7d3TGzH4CfB14s7sPzPE609t/HPhzM/tn4IbyPi+oVfCycprKcB0r\n947+E3ev7ughImtAuXf0t9z9tqhjkdrQlbCISHzpKmmd05WwiIhIRHQlLCIiEhElYRERkYgoCYuI\niERESVhERCQimsBBJKbMrAG4BTgNGCOcfGNvRfslhGNBJ4Fb3X2bmTURVl46CUgCH3L3r5vZK4Fv\nALvLu3/K3W+v24cRkTkpCYvE11agxd3PMbOzgJvK6ygn25uAM4ACsNPM7iIs4N/r7m8ys43AvxMW\nbDgD+Bt3/1gEn0NE5qEkLBJf5wJ3A7j7A2Z2ZkXbKcAedx8CMLP7CUse3gZMX+EmCK+SIUzCLzGz\nrcAe4Fp3H639RxCRheiZsEh8tQODFcvFcg3wudqGgQ3unnf30fK8srcD7yu3PwC8293PB/YCf17T\nyEVkSXQlLBJfQ4QTuU9LuHtQ0dZe0ZYFcgBmdgLwT8DN7v6Vcvud7j6dtO8gnGFnQaVSqdTQoAl4\nRJZg2QeKkrBIfO0ELga2m9nZwK6KtoeBk82sA8gT3oq+0cyOBe4Brnb371Vsf7eZXePuPyGcuefB\nxd68oaGBnp7hVfootdHdnY11jHGPDxTjaujuzi6+0TyUhEXi6w7gQjPbWV6+3MwuA9LlntDXAfcS\nnoVvc/f9ZvZxoAO4wczeT1h7+HXAW4BbzGwCOABcWe8PIyKHU+1oEZlPKc5XH7A2rpDiHB8oxtXQ\n3Z1d9u1odcwSERGJiJKwiIhIRJSERUREIqIkLCIiEhElYRERkYgoCYuIiERESVhERCQiSsIiIiIR\nURIWERGJiJKwiIhIRJSERUREIqIkLCIiEhElYRERkYgoCYuIiERESVhERCQiSsIiIiIRURIWERGJ\nSFPUAYiIzCUIAnK5gVnrOjo2kkjo2kHWDyVhEYmlXG6AO3d9g3R7BoDRoRG2nnoxmzZ1RhyZyOpR\nEhaR2Eq3Z8hsyEYdhkjN6L6OiIhIRJSERUREIqIkLCIiEhElYRERkYgoCYuIiERESVhERCQiSsIi\nIiIRURIWERGJiJKwiIhIRJSERUREIqIkLCIiEhElYRERkYhoAgeRmDKzBuAW4DRgDLjC3fdWtF8C\n3ABMAre6+zYzawK+AJwEJIEPufvXzexFwN8DAfAzd7+6np9FROamK2GR+NoKtLj7OcD1wE3TDeVk\nexOwBbgAuNLMuoHfA3rdfTPwOuDm8i43Ae919/OBhJm9oW6fQkTmpSQsEl/nAncDuPsDwJkVbacA\ne9x9yN0ngfuBzcBthFfHEB7fk+V/n+HuPyr/+1uEyVtEIqbb0SLx1Q4MViwXzSzh7sEcbcPABnfP\nA5hZFrgdeF+5vaF625pFLSJLpiQsEl9DQOWM9tMJeLqtvaItC+QAzOwE4J+Am939K+X2qbm2XUx3\nd3bxjWokkZigrSdJKt0CwNTkOF1dWTo7Z8cUZYxLEff4QDFGSUlYJL52AhcD283sbGBXRdvDwMlm\n1gHkCW9F32hmxwL3AFe7+/cqtn/IzDa7+w8JnxV/dykB9PQMr8LHWJ7+/mEK+Qkam8cBKOQn6O0d\nJgiSM9t0d2cjjXExcY8PFONqWMkJgpKwSHzdAVxoZjvLy5eb2WVAutwT+jrgXsJbzdvcfb+ZfRzo\nAG4ws/cDJcKk+8fA58ysmTCBb6/3hxGRwykJi8SUu5eAq6pW765o3wHsqNrnncA753i5PYS9qEUk\nRtQ7WkREJCJKwiIiIhFREhYREYmIngmLyJoUBAF9fX309x/qNdvRsZFEYu5riyAIyOUGZq1baHuR\nelASFpE1KZcb4J5H7yXRFA5ZGh0aYeupF7NpU+e829+56xuk2zNL2l6kHpSERWTNSrdnaGxuOaLt\nMxvWZ9EHWZt0H0ZERCQiSsIiIiIR0e1oEVkXgiBgYEAdr2RtURIWkXWhMJLnnr7v0tkddrRSxytZ\nC5SERWTdSGVS6ngla4ru04iIiERESVhERCQiuh0tImtCdcergYEBSqXSqm0vEgUlYRFZE6o7Xh3c\n9yzHHt9FU7J1ydtnN7WTpb1uMYssRklYRNaMyo5Xo0Mjq769SL3pmbCIiEhEdCUsIpGI+6xGcY9P\n1gclYRGJRPWsRsO5IV77/PPYuHEjEH1HKs26JPWgJCwikamc1Wh0aIR7Ho2uI1X1le/AwACpbFrF\nP6SmlIRFJDai7EhVfeWr3tRSD0rCIiJl1VfmUdNz6fVPSVhEJKZyuQG237eLVCa8Gs+PDHHpllP1\nXHodURIWEYmxVKadTLYj6jCkRnRPQ0REJCJKwiIiIhFREhYREYmIkrCIiEhElIRFREQiot7RIlIX\nc1Wk0vy+crRTEhaRulBFKpHDKQmLSN3ErSKVSNSUhEVEliAIAgYGBmYtQwOJRMPMOpWUlCOlJCwS\nU2bWANwCnAaMAVe4+96K9kuAG4BJ4FZ331bRdhbwEXd/bXn5lcA3gN3lTT7l7rfX5YOsEUFQolAo\n0JgM/ywWCgVSTemZ9sJInnv6Zs/y1JhsmlnWVIeyHErCIvG1FWhx93PKSfWm8jrMrKm8fAZQAHaa\n2V3u3mNm7wZ+H6i833sG8Dfu/rG6foIYq76yffLJx/EnDtLeOQXAwX192Atn/4msnuUp0dyoqQ5l\nRZSEReLrXOBuAHd/wMzOrGg7Bdjj7kMAZnY/sBn4KvAI8EbgHyq2PwN4iZltBfYA17r7aO0/QnRK\npRJj42MUCgVg8Svbpx57EhoaSCZbAWhuSh7x+1UmdYDOzvQ8W4uE9PBCJL7agcGK5aKZJeZpGwY2\nALj7HUCx6rUeAN7t7ucDe4E/r0XAcTI2NsajT/WxZ1+OPftyPLKvj/GJ8VnbTF/ZZjZkaUu3rfj9\nvv6Dn/PNf3mCb/7LE2y/b9dhSVmkmq6EReJrCKi815lw96CirXJsTxbILfBad7r7dNK+A/i7pQTQ\n3b16t1oTiQnaepKk0i0AtLY105RsWtEyMG97WypJa1sb6XQ4JKqttZW21vnfvy2VpGl4gubm8M9i\nU1Niwe2rl1OpJInuTo59zrEADA8lV/wdJhITpFJJ0uX3CKaSdHVl6exc3Vvgq/lzrpW1EONyKAmL\nxNdO4GJgu5mdDeyqaHsYONnMOoA84a3oG6v2b6j4991mdo27/wT4FeDBpQTQ0zO83NgP098/TCE/\nQWNzeDU6VpgkUQzIjy5/OZNsmre9kJ+gODXF5GR4U6BYDCiMTaza9tXL+fwE44UJRiuWV/od9vcP\nk89PkGg89Jq9vcMEwZHdKl9Id3d2VX/OtRD3GFdygqAkLBJfdwAXmtnO8vLlZnYZkHb3bWZ2HXAv\nYbLd5u77q/avLEf1FuAWM5sADgBX1jh2EVkCJWGRmHL3EnBV1erdFe07gB3z7PsEcE7F8v8l7Ogl\nIjGiJCwiModSqcTE2Pi8vavXgup63aCCInGjJCwiMofi1CSP7R9ktHG61vXh44bjLpcbYPt9u0hl\nwj58+ZEhLt1yqgqKxMja+o0SEamj5ubmZY8bjotUpp1MtiPqMGQeuichIiISESVhERGRiOh2tIjI\nMlSXxRwbH4NSaZG9RGZTEhYRWYaxsTEePzjIwGRYuav/QB8nlI6POCpZa5SERaQmqofHDAwMUFpn\nV4pNyeRMx62mZBLGF9lBpIqSsIjURC43wJ27vkG6fXqIz7NkN7WTnVXyev0oBQH50WFGhsMS3qMj\ngwRBsMhecrRTEhaRmkm3Z2bNv7uejRfG2N/4MMWpIQCGxwbI5U6ho+M5EUcmcaYkLCJHhXpUwGpJ\ntdGWDa/8x8fHGBgYIAiaZ21TWbGq3hWtgiDQnMcxoyQsIkeFelfAKhYn+Nr3nc5jCzPrqitW1bui\nVSE/zI6dvWzqOmbm/d7alQXWZiGS9UBJWETWheohQ+MT44cNGap3Bay2TGbRalX1rmiVSquCVpwo\nCYvIulA9ZOiZff20ZTKRxVMKAvKFQx21QJ215HBKwiKybhw2ZChC44UxnuQ/GZnqn1k3NNbPk09m\nZ575Vg/bqn5mux6HdclsSsIiIjXSkj7UUQtgdHSIex94jOMPhEm498DTZDZ0kS2P2qp+ZlvdLuuP\nkrCISJ2UggAqLmxLlA67PV35zHZ0ZHDB11tp7+ogCOjv75/Vg1vzDdeXkrCISJ1UjyXuKz3LC/Kn\nL3n/uW5X3/eTp0hnNwBH3ru6kB/m9m//nLZ0x7L2l5VTEhYRqaPKscSto0NHVOpyvtvVK+ntnMpk\nSWfUWzoqSsIismJz3RaNe6ei6uId/3979x4k2VnWcfzb17n1zGST3QiWKErwkX9A2SghhiwpCYoQ\nCFWWVVRJaTTESiFlFQLFxQAWIigSq8AKsbKYoGJZsFaishBiAHWzSBBYcBPWJxuSzbK7WfYy0zPT\n090z0xf/OD0zZ3u657J9Oadnf5+qre2333POPN3TM8+857zneVvd0hRH652u1sSuwaMkLCIda64T\nDehGgLAAABJiSURBVPGvFd1cvCPqW5q6QRO7Bo+SsIh0RbhONAxGrehw8Y6Nbmnqx8i5VquxUJql\nMBLEVCzMMja+teuzW5nYJdFTEhYR2YR+jJzLxQInOMJsdRew9YlbMniUhEVENmkrI+eLNTQ2etET\nt2Tw6GYwERGRiGgkLCISkXqtRnF+tb50cX6O+qhmM19KlIRFZCA0r5JUKpUYmxiOOKrONBfvOFU/\nRm5JU5kvJUrCIjIQmldJOnPyPLnJUXKTEQfWoXDxjqGxwf6jQrZOSVhEBkZ4laR+rAd8qWku9gGq\nJd1rSsIiIgKsLfahWtK9pyQsIiIrwsU+pPd0jkFERCQiGgmLxJSZJYC7gJcAZeBWd38q1H8TcAew\nBNzr7ntDfS8DPuruNzTaLwDuA2rAY+7+1n69DumdbpS57PTrd7KesWgkLBJnNwND7n4t8B7gzuUO\nM0s32q8CXgncZma7Gn3vBO4BhkLHuhN4r7vvAZJm9oa+vIIeCt+yVCqVBmYVpG4qFws8U/0ux6qH\nOFY9xPH645SL/avZnc9Ps+/hw3zxG8/wxW88w76HD69JyrI+JWGR+LoOeBDA3R8Frg71vQg46u6z\n7r4EPAJc3+h7Enhj07F2u/uBxuMvESTvgVapLvHk8SmOnsxz9GSep09OUa1Uow6rq8LFPApzeYqF\nWWq12gXbLJe5HBnPMTw20vcYR3PBNeTc+GWM5nSP81bpdLRIfE0A4WVwKmaWdPdai745YBLA3e83\ns59a57gr2w66dLb3tZyj1FzMY9AWdNjM6epOT2kP+ilxJWGR+JoFxkPt5QS83BcedowD+XWOFR4+\nbbStxEi4mMegLeiwfLp6eYTc6panzWzT6deIMyVhkfg6CLwO2Gdm1wCHQ31HgKvM7DKgSHAq+mNN\n+ydCjw+Z2fXu/l/Aa4CvbiaAXbvGN94ISCYXGTmbZXRs9TL08EiGdDa98txW20PDaSq1KolEcIo5\nkaiRSqXIZIJfW+lkCuCCdiqTjFU7HF9XjplKMTyaZmzlPcuSTqfa9o+MZEmlM+u2gbb9tWqWnTvH\nueKK1p+DZHKR0dFs2+2TyUV2XrmT8YkdAMzNrj3eZraB9p/Fze7fTqsCJTt29G8krSQsEl/3Azea\n2cFG+xYzexMw5u57zeztwEMEyXavuz/btH94ltI7gHvMLEOQwPdtJoCzZ+c2FejU1Byl4iKpzOow\nrVxaIlmpUZxfuKh2fnqOY2dmuHwueBmnnvoRI7kcS0sVACq1KmkyF7QTS6lYtcPxdeWY1SrlhQrz\nK+/ZIpVktW1/qbRIKpVYt50bz7btLxYXOXdujlqt9an+qak5isVFkqnW22/Uv9ltdu0ab/tZ3Mz+\n65maOt/xSHqzf6y2oiQsElPuXgdub3r6iVD/fmB/m32fAa4NtY8SzKIeKOEyldvxmq/Ew/Lksigo\nCYtIJFqtijSaHos4KulE86nd6elp6h3eNlar1Th//jxTU6sj4UGaeLURJWERiUSrVZHsZ/QraT1x\nX3+4ufb0udMnyE3uZLyDO5fy+Wn2f91JJFcLkgzSxKuN6BMvIpHRqkhbMwjrD4drT88XZjbYenPG\nchMkU6NdOVbcKAmLiAwQrT+8vWyPk+oiIiIDSElYREQkIkrCIiIiEdE1YRGRbWLN7Ok+L20oW6ck\nLCKxUK/XWSwvrNw3fCkuTdipbi/40KqkY9T36PbiXuSNvl4vF4hQEhaRWKhUl3j62RnmU8HM31Mn\npxjJ5SKOavBsZcGHWq3GfGGGej2oc908cm6+77cwm+fGX/xJduwI6jT3OgG20ot7kdfT6wUilIRF\nJDYyme29NGHclIsFTqefYHz4cqD1yLn5vt/9B5/sWwJspxf3Iq/79XpY1lJJWETkEjY0NrqlpRL7\nnQA7Fff1hpWERURk24r7esNKwiIi25RmSweiXCVpI0rCIiLbVLdnS0v3KQmLiGxjW5ktLf2nJCwi\nIoBOX0dBSVhE5BLRcj3iidX7fHX6uv+UhEVELhGt1iOerFw4YWnQT193WlGr3xW5lIRFRC4h2309\n4k4ravW7IpeSsIhsWXMBhCjKF4q0s15Bkc2MdPtZkERJWES2LJ+f5oHDX2BsIhhRnTn5I8Yvn2Cc\nPtcvlFir1WoslGYpjAQj7vnCDLVaLdKY+j3S3YiSsIhclLGJHLnJcQDmZwsRRyNxVC4WOMERZqu7\nAJgrTzMz8zx27twVaVxxKr2pJCwiIj0Trk29uDhgs7z6IB4VrEVERC5BGgmLiEhLGxXvaL7mu1Fx\nj3q9zsxMnqmp8yvPNU+Maj1xqmsvKXaUhEVEpKXm+4rPVU/znLM/y+jYKABTZ5/l/OgPma0Gk5w2\nKu5RqSzy0KNP8xOnV0/Cnj11nFRmmEQiERzjzEm+cO4MV1z5nOBrnj7Bruc+l7HcaE9eY9SUhEVE\npK3wfcWJM6wp9pHLTmxtPeLR8QtWNDqb/CHPVL+7MnnrfP1H/HTipbGZONVrSsIiIrJpvSj2EZ68\nNYhVujqhiVkiIiIRURIWERGJiE5Hi4hIXzTPtobGSk6j23j68waUhEWkL+r1OuWFMqVSCYCFxQW2\n9b0nskbzbGtoTO5auviakVu9TSpulIRFYsrMEsBdwEuAMnCruz8V6r8JuANYAu51973t9jGznwe+\nADzR2P1T7v75/r0aKJfLHDszw/RSBoBTJ6cYyeX6GYLEQHhiF3Q+uau5NOagrYGsJCwSXzcDQ+5+\nrZm9DLiz8Rxmlm60dwMl4KCZ/QtwXZt9dgMfd/e/iuB1rEhns2SzwyuPRbphkGdXa2KWSHxdBzwI\n4O6PAleH+l4EHHX3WXdfAg4Ae1rss7ux/W7gtWb2n2a218zG+vQa5BISvuZbmMsH13u3eMlhzTEK\ns5GvvNRLSsIi8TUBhCsVVMws2aavAEwC403PVxv7PAq80933AE8BH+xV0MvC14BLpZKuAV8Clq/5\nHqse4lj1ECfrTnVpsaNjHK8/Tqk416OIo6fT0SLxNUuQVJcl3b0W6gvPZhkHptvtY2YPuPtycr4f\n+MRmAti1a7zl88nkIiNns4yODQEwPJIhnU2vtAESyRrPnJymUA9OP588k2c4lyOTCX7tpJMpUplk\nR22gq8frdjscX1xi6vV7mJ4cZuLyoNrVzLmxdbffzDEWyvMAjDU+WyMjWVLpzEp7eCRLOp1a3T+V\nYng03Xb7Ttu1apadO8e54orWPxtbpSQsEl8HgdcB+8zsGuBwqO8IcJWZXQYUgVcAH2v0tdrnQTN7\nm7t/C/gV4NubCeDs2dYjkKmpOUrFRVKZ4OJbubREslKjOL96Ma5UXIREmkQymIhFIk11qcbSUgWA\nSq1KYinVUTtNpqvH63Y7HF9cYorTe7ipfapVSMF847NVKi2SSiVW2uXSIpVk9YLtywuVttt32i4U\nyhw9epxz51Z/Nsyez8VSEhaJr/uBG83sYKN9i5m9CRhrzIR+O/AQkAA+7e7PmtmafRr//z5wl5kt\nAqeB2/r3MkS2j1Jxjv0Hz3H5zmDRimJhlvcpCYtsP+5eB25vevqJUP9+YP8m9sHd/5dg0pbIQKnX\nahRL7ZdTjMLo2MQFi1B0QklYRERia6FU5jiPU6hOAWuXUxz0iltKwiIiEmtDYxssp9hBxa2oKQmL\niMhA6eZyilGXvVQSFhGRbWujJBt12UslYRER2bY2k2SjLHupJCwiIttaOMkOzeUp5kOzrSOe2KUk\nLCIil4zm5RSjntilJCwiIpeU9SZ2hReQgN5P1FISFhERaWgeKfd6opaSsIiIbBtrRrIXcc33gpFy\n8zXkLo+MlYRFRGTb6PY1316PjJWERWRDtVqNfH56pT01dZ5isUgqG/wKKRaLpIcylEqllW20frBE\npZvFPNYcr2lkPF+YWW/XDSkJi8iG8vlp9j18mNFcMKI4/oMj5HecZseVwfLGp46dJj00xHQls7LP\nqZNTjORykcQr0ivNI+O58jTw+os+npKwiGzKaG515ZiRsRyFzBDZbDDKSGezZDKZlfbycyLbUXhk\nvLjYWWWPZDcCEhERka1TEhYREYmITkeLyIZqtdoFE1CiLvUnsl0oCYvIhmZmZniq/G3GR3YA0Zf6\nE9kulIRFZFMuKILfhds+RETXhEVERCKjJCwiIhIRJWEREZGIKAmLiIhERElYREQkIkrCIiIiEVES\nFhERiYiSsIiISESUhEVERCKiJCwiIhIRJWEREZGIKAmLiIhERAs4iEhLX3z4IOfyCwCcfvYExUqJ\nyYhjEtlulIRFpKWpuQoLySsAWEwWqFRrEUcksv0oCYvElJklgLuAlwBl4FZ3fyrUfxNwB7AE3Ovu\ne9vtY2YvAO4DasBj7v7Wvr4YEWlJ14RF4utmYMjdrwXeA9y53GFm6Ub7VcArgdvMbNc6+9wJvNfd\n9wBJM3tD316FiLSlJCwSX9cBDwK4+6PA1aG+FwFH3X3W3ZeAA8CeFvvsbmy/290PNB5/iSB5r+t7\nT36Tx08c4PETB/jBme9QKs924zWJSIhOR4vE1wQwE2pXzCzp7rUWfQVgEhhver5qZikgEXpurrHt\nusoLsyxU5wEoLeQplWaZOX8egOLMLOmhbNv2ZrbpRru6tEgm07vjdzO+uMQUp/dwEGLcqF3I5+mE\nkrBIfM0SJNVlywl4uW8i1DcOTLfZp2pmtaZtN/zN8ck//lBio21EpDM6HS0SXweBXwcws2uAw6G+\nI8BVZnaZmWWBVwD/DXy9zT7fMbPrG49fQ3D6WkQilqjX61HHICIthGY6v7jx1C0E13jHGjOhXwt8\ngOBU86fd/e5W+7j7E2b2QuAeIEOQwN/i7vrhF4mYkrCIiEhEdDpaREQkIkrCIiIiEVESFhERiYhu\nURKRC2xULjMqjSphfws8H8gCHwa+T8zKcZrZlcC3CAqiVIlffO8GXk/w+/+vCWbh30cMYmx89vYC\nRvDevYUYvYdm9jLgo+5+Q7tSsGb2FuA2gnKyH3b3/esdUyNhEWnWtlxmxH4LOOfu1wO/RpBAYlWO\ns/GHwt1AsfFU3OLbA7y88b29AXhBzGJ8NcHs/+uADwF/Fpf4zOydBHcYDDWeWhOXmf0Y8Dbg5QSf\n0Y+YWWa94yoJi0iz9cplRulzBAtWAKSACvDSrZbj7LG/BD4FnCK4dSxu8f0q8JiZPQD8a+NfnGIs\nA5ONEfEkwWgyLvE9Cbwx1G4uBXsj8EvAI+5ecfdZ4Cirtwu2pCQsIs1alsuMKphl7l5093kzGwc+\nD7yPiyjH2Stm9jvAGXf/d1bjCr9vkcbXsJPgXvPfAG4HPku8YnwEGAH+D/gb4BPE5Hvs7vcT/OG3\nrDmuCdaWjV0uJ9tW5D9YIhI765XLjJSZPQ/4KvAZd/8ngutxyzZVjrOHbgFuNLOvEVxP/ztgV6g/\n6vgAzgNfbozUnqAx8gz1Rx3ju4CD7m6svofZUH/U8YW1+uy1Kie7brxKwiLSbL1ymZFpXG/7MvAu\nd/9M4+lDcSnH6e573P0Gd78B+C7wZuBLcYmv4RGCa5WY2Y8DY8BXGteKIfoYc6yOJPMEk8cOxSi+\nsFalYP8HuM7MsmY2Cfwc8Nh6B9HsaBFpdj/BiO5go31LlMGEvAe4DLjDzN4P1IE/BD7ZmPxyBNgX\nYXytvAO4Jy7xuft+M3uFmX2T4HTq7cAxYG9MYvwYcK+ZHSDIT+8Gvh2j+MLWfG/dvW5mnyD4YydB\nMHFrcb2DqGyliIhIRHQ6WkREJCJKwiIiIhFREhYREYmIkrCIiEhElIRFREQioiQsIiISESVhEZEY\nMLM9ZvZvW9j+g2b2y72MSXpPSVhEJD62UrhhD8FCFjLAVKxDRCQGGqUZPw5MAc8Bvg78AcGqQX9C\nUEHqaYK1al9LsObzswQr++wE/pRg8YMdBKU9/7nPL0EugkbCIiLxcRVwq7u/mGBhhfcCHwFe7e67\ngYeAP3f3vwe+Bfyeuz8OvLXx+GrgVuADkUQvW6ba0SIi8fEVdz/eePyPwH0Ep6i/1lhjN0mwEtKy\n5eX03gy8zsx+E7iGYGEGGQBKwiIi8dG8Xm0COODuNwOY2RAXLjO57BHgK8B/NP7/bG/DlG7R6WgR\nkfi4wcyea2ZJ4LeBO4GXm9kLG/13AH/ReFwB0ma2g+A09vvd/UHg1WjC1sBQEhYRiY/HgH8Avgcc\nBz4M/C7wOTP7HvALwB81tn0QuBswYC/wfTN7BCgAw2Y20ufY5SJodrSIiEhENBIWERGJiJKwiIhI\nRJSERUREIqIkLCIiEhElYRERkYgoCYuIiERESVhERCQiSsIiIiIR+X+rJEDK+ayHxAAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11dd6f550>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(8,8))\n",
    "plt.subplot(2,2,1)\n",
    "\n",
    "plt.hist(stan_nuts['p'], range=[0,1], bins=50, normed=True, label='NUTS', alpha=0.5)\n",
    "plt.hist(fullrank_samples['p'].values, range=[0,1], bins=50,\n",
    "         normed=True, label='Full-rank ADVI', alpha=0.5)\n",
    "plt.xlabel(\"p\")\n",
    "\n",
    "plt.subplot(2,2,2)\n",
    "plt.hist(stan_nuts['alpha'], range=[0,100], bins=50, normed=True, label='NUTS', alpha=0.5)\n",
    "plt.hist(fullrank_samples['alpha'], range=[0,100], bins=50,\n",
    "         normed=True, label='Full-rank ADVI', alpha=0.5)\n",
    "plt.xlabel(\"alpha\")\n",
    "plt.legend()\n",
    "\n",
    "plt.subplot(2,2,4)\n",
    "plt.hist(stan_nuts['beta'], range=[0,100], bins=50, normed=True, label='NUTS', alpha=0.5)\n",
    "plt.hist(fullrank_samples['beta'].values, range=[0,100],\n",
    "         bins=50, normed=True, label='Full-rank ADVI', alpha=0.5)\n",
    "plt.xlabel(\"beta\");"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}