Skip to content

Instantly share code, notes, and snippets.

@gngdb
Last active July 31, 2017 10:47
Show Gist options
  • Save gngdb/a122ae86c0b897cffebdb9481e321c9d to your computer and use it in GitHub Desktop.
Save gngdb/a122ae86c0b897cffebdb9481e321c9d to your computer and use it in GitHub Desktop.
Convolution with SeLUs
Display the source blob
Display the rendered blob
Raw
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Self-Normalizing Neural Networks\n",
"\n",
"[Self-normalizing neural networks](https://arxiv.org/abs/1706.02515) (SNNs) are carefully constructed to have zero mean, unit variance activations without an explicit normalization scheme such as batch norm, weight norm, layer norm.\n",
"\n",
"SNN theory is based around analyzing the mapping from the mean and variance of one layer to the next. Let each component of $n$ component vector $x$ be iid distributed from some distribution with mean $\\mu$ and variance $\\nu$. SNNs exist for any $\\mu$ and $\\nu$, but to be concrete let $\\mu=0$ and $\\nu=1$. Let $z = w^T x$ where $\\sum_i w_i = \\omega$ and $\\sum_i w_i^2 = \\tau$. As $n$ grows, the central limit theorem states that $z \\sim \\mathcal{N}(\\mu \\omega, \\sqrt{\\nu \\tau})$. If the weight vector $w$ is normalized such that $\\omega = 0$ and $\\tau = 1$, the distribution of $z$ has identical mean and variance to the distribution of each component of $x$, which is a good first step towards normalized activations. For the moment, let's assume $w$ is meets these conditions.\n",
"\n",
"## Scaled Exponential Linear Unit (SELU)\n",
"\n",
"Neural nets need non-linear activation functions to be powerful function approximators. If there existed a non-linear function $s$ that could map $z \\sim \\mathcal{N}(\\mu, \\sqrt{\\nu})$ to $h$ such that $\\mathbf{E}[h] = \\mu$, $\\text{Var}[h]=\\nu$, then it would be possible to non-linearly map $x$ to $z$ to $h$ while preserving the first and second moments.\n",
"\n",
"Conveniently, such a function exists! The authors name it the scaled exponential unit (SELU):\n",
"\n",
"$\\begin{align*}\n",
"s(x) &= \\lambda \n",
"\\begin{cases} \n",
" x & x\\geq 0 \\\\\n",
" \\alpha e^x - \\alpha & x<0 \\\\\n",
" \\end{cases} \\\\\n",
"s'(x) &= \\lambda\n",
"\\begin{cases} \n",
" 1 & x\\geq 0 \\\\\n",
" \\alpha e^x & x<0 \\\\\n",
" \\end{cases}\n",
"\\end{align*}$\n",
"\n",
"Exponential linear units (ELUs) are a special case of SELU when $\\lambda = \\alpha = 1$. The authors compute that for $\\lambda\\approx 1.0507$ and $\\alpha \\approx 1.6733$, the $s$ preserves mean 0 and variance 1. Let's investigate this."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"%matplotlib inline\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"scale = 1.0507009873554804934193349852946\n",
"alpha = 1.6732632423543772848170429916717\n",
"def selu(x):\n",
" return scale * np.where(x >= 0, x, alpha * np.exp(x) - alpha)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XlclWX+//HXxSLIoiiQG6iYSq6goohZmllqqX1LDS0X\n0sRl6pdTTY1my5RtY5Yz02haKm65l6mlTVraooFgmCumuYAbgqLs27l+f5wjae6cA/c5h8/z8TgP\nzsZ9v0V4c3Pf133dSmuNEEII5+FidAAhhBC2JcUuhBBORopdCCGcjBS7EEI4GSl2IYRwMlLsQgjh\nZKTYhRDCyUixCyGEk5FiF0IIJ+NmxEoDAgJ048aNjVi1ENeUkpICQGhoqMFJhLi6pKSkDK114I3e\nZ0ixN27cmMTERCNWLcQ1de/eHYDNmzcbmkOIa1FKHb2Z98muGCGEcDKGbLELYY8mT55sdAQhbEKK\nXQiLnj17Gh1BCJuwm2IvLi4mLS2NgoICo6M4DU9PT4KCgnB3dzc6ikNITk4GIDw83OAkQljHboo9\nLS0NX19fGjdujFLK6DgOT2tNZmYmaWlphISEGB3HIUyYMAGQg6fC8dnNwdOCggL8/f2l1G1EKYW/\nv7/8BSREFWQ3xQ5IqduYfD2FqJrsqtiFEMJZ5RaW8NqaPVwoKK7wdUmx/8mbb75Jq1ataNu2LeHh\n4cTHx9O9e3dCQ0MJDw8nPDycgQMHAvDaa6/x3nvvXfb5R44coXXr1pc9d7X3CSGqjnO5RTz2STwL\nfz5K4pGzFb4+uzl4ag+2bdvGunXr2LFjBx4eHmRkZFBUVATA4sWLiYiIMDihqEhvvfWW0RGEEzqR\nlc/wuQkcO5vHzMfb0+OOOhW+Tin2S5w8eZKAgAA8PDwACAgIMDiRqExdunQxOoJwMgfTcxg+J54L\nBSUsGNmJzk38K2W9Nit2pZQrkAgc11r3tWZZ/1i7h70nLtgmmEXL+jV4tV+r677n/vvv5/XXX6d5\n8+b07NmT6OhounXrBsDjjz9O9erVAbjvvvuYOnWqTfMJ423duhWQghe28WtaFjHztuOiYGlsZ1o3\nqFlp67blFvszwD6ghg2XWal8fHxISkrihx9+4LvvviM6Opp33nkHuPldMdcaiSIjVOzfpEmTABnH\nLqy39WAGoxck4udVjUVPRhIS4F2p67dJsSulgoAHgTeBZ61d3o22rCuSq6sr3bt3p3v37rRp04b5\n8+ff0uf7+/tz7ty5y547e/asnCQkRBWxYfdJ/t+SZBoHeLFgZCR1a3pWegZbjYqZDrwAmGy0PEOk\npKTw22+/lT1OTk6mUaNGt7QMHx8f6tWrx7fffguYS33Dhg107drVplmFEPZnacIxxi/eQasGNVg+\nJsqQUgcbbLErpfoC6VrrJKVU9+u8LxaIBWjYsKG1q60QOTk5PP3002RlZeHm5kbTpk2ZPXs2AwcO\nvGwfe0BAABs3bgRgypQpTJ8+vWwZaWlpLFiwgL/85S88+6z5j5dXX32V22+/vfL/QUKISqG15qMt\nv/Puhv10ax7IzKHt8apm3NgUpbW2bgFKvQ0MA0oAT8z72D/TWg+91udEREToP19oY9++fbRo0cKq\nLOJK8nW9eXKhDVEeWmve+mofH/9wmH5h9Zk2KIxqbhVzipBSKklrfcODfVb/StFaTwQmWlbaHXj+\neqUuhL269C8vIW5GSamJv3+2i5VJaQyPasRr/Vrh4mL8QAkZxy6EhUzXK25FQXEpT336Cxv3neaZ\ne5sxoWczuxn9ZtNi11pvBjbbcplCVJaLx03kghviRi4UFDN6fiLxh8/yj/6tGNGlsdGRLiNb7EJY\nTJkyBZBiF9d3JruQEXMTOHA6m38NDueh8AZGR7qCFLsQQtyk1LN5DJsTz6kLBXw8IoJ7Qm8zOtJV\nSbELIcRNSDmVzfC58eQXlbL4yUg6NKptdKRrkml7r+FWp9pds2ZN2fQDt2r16tXs3bu37PErr7xS\ntr9XCGG8pKPneHTWNrSGFWO72HWpg2yx20RJSQn9+/enf//+5fr81atX07dvX1q2bAnA66+/bst4\nQggrbE5JZ9yiHdSp4cHCUZEE1/YyOtINyRb7Jd58802aN29O165dSUlJAeDQoUP07t2bDh06cNdd\nd7F//34AYmJiGDt2LJGRkbzwwgvExcXx1FNPcf78eRo1aoTJZJ5dITc3l+DgYIqLi/n444/p2LEj\nYWFhDBgwgLy8PLZu3cqaNWv429/+Rnh4OIcOHSImJoaVK1eyYcMGBg0aVJZv8+bN9O1rnjjzf//7\nH1FRUbRv355BgwaRk5NTyV8t5zNr1ixmzZpldAxhR75IPs6T8xMJCfBmxdguDlHqYK9b7Ov/Dqd2\n2XaZddtAn2vvKklKSmLp0qUkJydTUlJC+/bt6dChA7GxsXz00Uc0a9aM+Ph4xo8fXzYPTFpaGlu3\nbsXV1ZW4uDgAatasSXh4OFu2bOGee+5h3bp19OrVC3d3dx555BFGjx4NwOTJk5kzZw5PP/00/fv3\np2/fvmVXZrqoZ8+exMbGkpubi7e3N8uWLWPw4MFkZGQwZcoUNm7ciLe3N++++y7vv/8+r7zyim2/\nZlVMaGio0RGEHVmw7QivrtlDx8a1+WREBDU83Y2OdNPss9gN8MMPP/Dwww/j5WX+jdy/f38KCgrY\nunXrZVvNhYWFZfcHDRqEq6vrFcuKjo5m2bJl3HPPPSxdupTx48cDsHv3biZPnkxWVhY5OTn06tXr\nupnc3Nzo3bs3a9euZeDAgXz55Zf885//ZMuWLezdu5c777wTgKKiIqKioqz+GlR1a9euBaBfv34G\nJxFG0lrzr02/MX3jb/RsUYcPH2uHp/uVP+f2zD6L/Tpb1pXJZDLh5+dHcnLyVV/39r76HMv9+/dn\n0qRJnD17lqSkJHr06AGYd9+sXr2asLAw4uLibmpOksGDB/Phhx9Su3ZtIiIi8PX1RWvNfffdx5Il\nS8r9bxNXmjZtGiDFXpWZTJp/rN3D/G1HGdA+iHcHtMHN1fH2WDte4gpy9913s3r1avLz88nOzmbt\n2rV4eXkREhLCihUrAPNv8p07d95wWT4+PnTs2JFnnnmGvn37lm3VZ2dnU69ePYqLi1m8eHHZ+319\nfcnOzr7qsrp168aOHTv4+OOPGTx4MACdO3fmp59+4uDBg4B5P/6BAwes+vcLUdUVlZiYsCyZ+duO\nMvquEKYObOuQpQ5S7GXat29PdHQ0YWFh9OnTh44dOwLmKyfNmTOHsLAwWrVqxRdffHFTy4uOjmbR\nokVER0eXPffGG28QGRnJnXfeyR133FH2/ODBg5k6dSrt2rXj0KFDly3H1dWVvn37sn79+rIDp4GB\ngcTFxTFkyBDatm1LVFRU2UFdIcStyysqYfSCRNbsPMGLve9g0gMt7GIyr/Kyetre8pBpeyuPfF1v\nnkzbWzVl5RUxMm47yalZvPlwG4Z0ss/rRUAlTtsrhBCO6vSFAobPSeBwRi7/faw9fdrUMzqSTUix\nC2GxcOFCoyOISnQkI5ehc+I5l1tE3BMd6dI0wOhINmNXxa61tpv5jJ2BEbvZHFlwcLDREUQl2X38\nPDHzEjBpWBLbmbZBfkZHsim7OXjq6elJZmamlJGNaK3JzMzE09OYi+k6omXLlrFs2TKjY4gKFv97\nJkNm/0w1VxeWj4lyulIHO9piDwoKIi0tjTNnzhgdxWl4enoSFBRkdAyHMXPmTIDLRjIJ5/LN3tM8\n9ekOgmpVZ+GoSOr7VTc6UoWwm2J3d3cnJCTE6BhCCCe1MimNF1f9Suv6NZj3RCdqe1czOlKFsZti\nF0KIivLJD78z5ct9dG0awEfDOuDj4dzV59z/OiFElaa1ZurXKczYfIgH2tTlg+hwPNwca96X8pBi\nF0I4pVKTZvLqXSxJSOWxyIa88VBrXB34bNJbIcUuhMXKlSuNjiBspLCklAlLk1m/+xRP3dOU5+5v\nXqWGUkuxC2EREOA8J6hUZTmFJYxZmMhPBzOZ/GALnryridGRKp0UuxAWFy+WEhMTY2gOUX5nc4uI\nmZfAnhMXmDYojAEdquZwXyl2ISyk2B3b8ax8hs2J5/i5fGYN7UDPlnWMjmQYq4tdKeUJfA94WJa3\nUmv9qrXLFUKIm3UwPZthcxLIKShh4ahIOoXUNjqSoWyxxV4I9NBa5yil3IEflVLrtdY/22DZQghx\nXTtTs4iZl4CriwtLx3SmVf2aRkcynNXFrs2Tu+RYHrpbbjLhixCiwv34WwaxCxPx96nGolGRNPK/\n+uUqqxqbTAKmlHJVSiUD6cA3Wuv4q7wnVimVqJRKlPlghBDW+mrXSUbGbadhbS9Wje0ipX4Jm15B\nSSnlB3wOPK213n2t913tCkpCGC0vLw8ALy8vg5OIG/k0/hgvrd5F+4a1mDuiIzW93I2OVCkMuYKS\n1jpLKfUd0Bu4ZrELYY+k0O2f1poZmw8x9esUuocGMvPxDlSv5vxTBNwqq3fFKKUCLVvqKKWqA/cB\ncmVl4XBmzJjBjBkzjI4hrsFk0kz5ch9Tv07h/8Lr8/HwCCn1a7DFFns9YL5SyhXzL4rlWut1Nliu\nEJVq+fLlAIwfP97gJOLPiktNvLjqVz7bcZyYLo15pW9LXKrIvC/lYYtRMb8C7WyQRQghrlBQXMpT\nn+5g4750nr2vOU/3aFql5n0pDznzVAhhty4UFPNkXCLbj57ljf9rzbDOjYyO5BCk2IUQdik9u4AR\nc7dzMD2bfw9uR7+w+kZHchhS7EIIu5N6No+hc+JJv1DIJyM60q15oNGRHIoUuxAWmzdvNjqCAPaf\nusDwOQkUlphYPDqS9g1rGR3J4UixCyHsRtLRszwxbzvVq7myYmwUzev4Gh3JIUmxC2Hx3nvvAfD8\n888bnKRq+m5/OuMWJ1GvZnUWjOxEcG05Yay8bDJXjBDOYN26daxbJ6dgGOGL5OOMXpDI7YE+rBgb\nJaVuJdliF0IYKu6nw7y2di+RIbX5ZEQEvp5VY96XiiTFLoQwhNaaDzb+xr83/cb9Levw7yHt8HSX\nKQJsQYpdCFHpTCbNq2v2sPDnozwaEcRbD7fBzVX2DNuKFLsQFtWrVzc6QpVQVGLiuRU7WbvzBGO6\nNeHvve+QKQJsTIpdCIv169cbHcHp5RWVMHbRDr4/cIaJfe5gTLfbjY7klKTYhRCVIiuviCfitrMz\nNYt/DmjLox2DjY7ktKTYhbB44403AHj55ZcNTuJ8Tp0vYPjceI5k5jFzaAd6taprdCSnJkcrhLDY\ntGkTmzZtMjqG0zmckcuAmVs5kVVA3BMdpdQrgWyxCyEqzO7j5xkxNwGApbGdad2gpsGJqgYpdiFE\nhdh2KJPRCxKpWd2dhaM60STQx+hIVYYUuxDC5r7ec4qnl/xCo9peLBjViXo1ZShpZZJiF8LC39/f\n6AhOYXliKn9f9Sttg/yYF9ORWt7VjI5U5UixC2GxatUqoyM4vFlbDvH2+v3c1SyAj4Z2wNtDKsYI\n8lUXQlhNa807G/Yza8vv9G1bj/cfDaeamwy6M4oUuxAWEydOBODtt982OIljKSk18dLnu1mWmMrQ\nzg35R//WuLrIFAFGkmIXwmLbtm1GR3A4BcWlPLP0F77ec5r/d28z/tqzmcz7Ygek2IUQ5ZJdUEzs\ngiS2/Z7Jq/1a8sSdIUZHEhZW7wRTSgUrpb5TSu1VSu1RSj1ji2BCCPuVkVPIkI9/ZvuRs0yPDpdS\ntzO22GIvAZ7TWu9QSvkCSUqpb7TWe22wbCGEnUk7l8fwOQmcOJ/Px8MjuOeO24yOJP7E6mLXWp8E\nTlruZyul9gENACl24VCCgoKMjmD3fjudzbA5CeQVlbBoVCQRjWsbHUlchU33sSulGgPtgHhbLleI\nyrBo0SKjI9i1X46d44m47bi7urBsTBQt6tUwOpK4BpsNNFVK+QCrgAla6wtXeT1WKZWolEo8c+aM\nrVYrhKgE3x84w+OfxFPD051VY7tIqds5mxS7Usodc6kv1lp/drX3aK1na60jtNYRgYGBtlitEDY1\nYcIEJkyYYHQMu7Pu1xOMmr+dRv7erBwXRUN/L6MjiRuweleMMg9anQPs01q/b30kIYyRnJxsdAS7\ns/Dno7zyxW4iGtXikxEdqVnd3ehI4ibYYov9TmAY0EMplWy5PWCD5QohDKK15j+bfuPl1bvpEXob\nC0ZGSqk7EFuMivkRkFPNhHASJpPmjS/3Mu+nIzzSrgHvDmyLu6vM++JI5MxTIUSZ4lITL6z8lc9/\nOc7IO0OY/GALXGTeF4cjxS6ERfPmzY2OYKj8olL+8ukOvt2fzt96hTK+++0y74uDkmIXwmL2m8/D\n6d1GxzDE+fxiRsVtJ+nYOd58uDWPRzYyOpKwghS7EABFebAkGnLSofUjRqepVOkXChg+N4FDZ3L4\ncEh7Hmxbz+hIwkpS7EIAfPsGsfP3AIrZk4wOU3mOZuYybE4CGTmFzI3pyF3N5BwTZyDFLsTh7+Hn\nmRw4p8BUar65uBqdqsLtPXGB4XMTKDGZ+HR0Z8KD/YyOJGxExjCJqi3vLHw2Bvxvhxr1zc+VFhmb\nqRJsP3KW6NnbcHdVrBwbJaXuZKTYRdWlNax9BnLTYcAn4GL5A7ak0NhcFezb/acZ+kk8gb4erBzX\nhaa3+RodSdiYFLuoupIXw7410GMy1G8HyvLj4MRb7J/tSGP0giSa1/FlxZgoGvhVNzqSqACyj11U\nTZmH4KsXoPFd0MV80a/w0Ebw+xGnLfa5Px7m9XV76XK7P7OHR+DjIT/+zkr+Z0XVU5QHy4eDqzs8\n/BG4mLfUp784Ej6Pd7pdMVpr3v/mAP/59iC9W9Vl+uBwPN2d/+BwVSbFLqoWreHLZ+H0Hnh8BdS8\n5KpJrtXMH51oi73UpHn5i918Gn+MwR2DefPhNrjKFAFOT4pdVC2Jc2HnEuj2d2h232UvDZ30HziS\nz6JY59hiLywp5dllO/ly10nGdb+dF3qFyhQBVYQUu6g60pJgw9+haU/o9uKVL58+CxdMUFpsQDjb\nyi0sYeyiJH74LYOXHmjB6LubGB1JVCIpdlE1XDgJy4aCT1145OOy/eqXubg1W+rYW+zncouIidvO\n7uPnmTqwLYMigo2OJCqZFLtwfkW55nlgCi/AyA3gVfvq77s43NGBD56ePJ/PsDkJHDubx8zH23N/\nq7pGRxIGkGIXzs1kgs9i4dQuGLIU6ra5zpsdexz7oTM5DJ+TwIX8YhaM7ETnJv5GRxIGkWIXzm3T\na7B/HfR+B5r3uu5boyLC4ZddDlnsu9LOM2JeAi4KlsR2pnWDmkZHEgaSYhfO6+eP4Kd/QcQoiBx7\nw7e//eqL8OEKKHGsYt96KIPR8xOp5V2NhaMiCQnwNjqSMJgUu3BOO5fBhhfhjr7Q559/HBi9nrJx\n7I6zj33D7pP8vyXJhAR4s2BUJ+rU8DQ6krADMleMcD4pG2D1OAi5GwbMAdeb234ZEDOeAcvzHObg\n6bLtxxi/eAetG9Rg2ZjOUuqijGyxC+dy+AdYMQLqtYXBn4L7zZdd5rnzkKcdYhz7R1sO8c76/XRr\nHsjMoe3xqiY/yuIP8t0gnMfh7+HTaKjVGB5fBR63Oh2t/Y9j11rz9vr9zP7+d/qH1ee9QWFUc5M/\nvMXlpNiFczj0HSwZYi71EWvBuxxD/crGsdvnwdOSUhMTP9vFiqQ0RkQ14tV+rXCReV/EVdjkV71S\naq5SKl0pVTUv8S6MdXATLBkMtZtAzDrwKed1O+34zNOC4lLGLd7BiqQ0JvRsxmv9pdTFtdlqiz0O\n+BBYYKPlCXFzdq8yX9ouMBSGrynflrrFvffeC5u3Q1oibP/EhiGtk19cyuKfj1EnM5el7erTueZJ\nSPze6FiivJr1Ar+KneZBaa1tsyClGgPrtNatb/TeiIgInZiYaJP1iirs54/Mk3o17AxDlkD1WtYv\n88OOkHHA+uUIcS2Pr4JmPcv1qUqpJK11xI3eJ/vYhePRGjb9A378wDxOfcAn4G6jS7yN+cE8p4wd\nOJ6Vz9Of/sLp7Hz+OSCMO5sGGB1J2IJnxZ8VXGnFrpSKBWIBGjZsWFmrFc6mOB/WPA27VkCHJ+DB\naeBim6sB9enTB4D169fbZHnWOHA6m2ELfqWg2Iu5T3ajQyMb/DUiqoxKK3at9WxgNph3xVTWeoUT\nOX8clj4GJ3fCva9A12dv7ozSm5Sfn2+zZVkj6eg5RsZtx8PNheVjogite6vDNkVVJ7tihGNI3Q7L\nHjdfr3TIEgjtY3SiCrHlwBnGLkyiTg0PFo6KJLi2l9GRhAOy1XDHJcA2IFQplaaUGmWL5QqB1pDw\nMcQ9AO5e8ORGpy31NTtP8OT87YQEeLNibBcpdVFuNtli11oPscVyhLhMfhaseQr2rTUPEXv4o2tf\nJMPBLdx2hFfW7KFj49p8MiKCGp7uRkcSDkx2xQj7lJYIK5+ACyfg/inQ+S9Xv5ydDfXt27dCl381\nWmv+vekgH2w8QM8WdfjwsXZ4utvmYLCouqTYhX0pKYQt78KP06FGA3hiAwR3rJRVP//885WynotM\nJs3r6/YSt/UIA9oH8e6ANri5yrwvwnpS7MJ+nNwJn4+D9D0QPhR6vQnV/YxOVSGKS008v2InXySf\nYPRdIUzs00KmCBA2I8UujFecDz+8Dz++D14BMGQZhPau9Bjdu3cHYPPmzRW6nvyiUsYtTmJzyhle\n7H0HY7s1Qdlw2KYQUuzCWAe+hvUvwLkj0DbafG1SJz1ACnA+r5iR87fzy7FzvP1IG4Z0kpP1hO1J\nsQtjnDsKGyZCypcQ0Nw8gVeTbkanqlCnLxQwfE4ChzNy+e9j7enTpp7RkYSTkmIXlSvvLPwwDRJm\ng4sb9HzNPOLFrZrRySrUkYxchs2N52xOEXFPdKSLzPsiKpAUu6gcxQWQMMtc6gUXIPwxuGcS1Awy\nOlmF23PiPCPmbsekNUtiO9M2yDkPCAv7IcUuKlZxPuxYCD9NhwvHodn95q30Oq2MTnaFRx991ObL\njP89kyfnJ+Lr6caCUZ1pepuPzdchxJ9JsYuKUZQLiXNh638g5zQEdzafORpyt9HJrmn8+PE2Xd7G\nvaf5y6c7CKpVnYWjIqnvZ6OphYW4ASl2YVsXTkLiHNg+B/LPQkg3GDAHGne16UyMFSEvLw8ALy/r\n52hZlZTGC6t+pXX9Gsx7ohO1vZ37GIKwL1LswjbSkiB+Juz5HEyl0Lw33PUsBHcyOtlNe+CBBwDr\nx7F/8sPvTPlyH12bBvDRsA74eMiPmahc8h0nyq/gPOz+DH5ZCMeToJovdBwNkbHmC0tXMVprpn6d\nwozNh3igTV0+iA7Hw03mfRGVT4pd3BqTCY7+BL8sgr1fQEk+BLaA3u+aR7p41jA6oSFKTZrJq3ex\nJCGVxyIb8sZDrXGVKQKEQaTYxY1pbd4i3/O5uczPp4JHTQgfAu2GQv32dr//vCIVlpQyYWky63ef\n4ql7mvLc/c1ligBhKCl2cXUmE5zYAXtXw54v4PwxcHGHpvfCva9Ci762u4C0A8spLGHMwkR+OpjJ\n5Adb8ORdVW8XlLA/UuziD/nn4NC3cOB/cHAj5GWYy/z2HuaTiUL7OO1siwAxMTG39P6zuUXEzEtg\nz4kLTBsUxoAOzn+ylXAMUuxVWUmh+YIWR36E3zdDajzoUqheC5reZz6ZqFlP8+Mq4FaK/XhWPsPm\nxHP8XD6zhnagZ8s6FRdMiFskxV6VFOfD8R3mIj/yA6Rth5ICQEHdNtB1gvkSdEER4FL1RnNkZGQA\nEBBw/XlcDqZnM2xOAjkFJSwcFUmnEOedjVI4Jil2Z2UyQcYB80HP44nmj6f3gKmEsiKPGGU+cahR\nVJXZKr+egQMHAtcfx74zNYuYeQm4uriwdExnWtWvWUnphLh5UuzOoCgX0veZi/v0HkjfCyeSoSjb\n/LpHDajfDu58BhpESJGX04+/ZRC7MBF/n2osGhVJI39voyMJcVVS7I4k/xxkHoLMg+bbxTI/dwTQ\n5ve4e0OdltD2UfMulQYdwL9ZhV8I2tl9teskE5Ym0yTQmwUjO3FbDU+jIwlxTVLs9sRUCtmnzOPE\nz6eZCzvzEJy1lHle5h/vVS5QK8S8SyVsiHm2xDotwa+xlLiNfRp/jJdW76J9w1rMHdGRml7uRkcS\n4rqk2CtLcQHkpkNOunm2w5zTcP74HyWelQrZJyz7wC/hWx/8b4cW/aD27eDf1Hyr1QjcPIz5t1QR\nWmtmbD7E1K9TuCc0kBmPd6B6tap3UFk4Hin28tDavF87/6x598jFW57lcV6mpbwvKfGC81cuR7lC\njfpQM9i837tmkPl+zWDwCzY/rib7cSvLuHHjyu6bTJo3v9rHnB8P83C7BvxzYFvcXeUvIeEYbFLs\nSqnewL8AV+ATrfU7tliuzWkNpcVQlAOF2eZbUQ4U5pgPNBZmW+7nQOGFS+5nQ37W5SVuKr72ety9\nwbcO+NSB21pAk+7gc5v5sU8d831vy2NX+d1qL6KjowEoLjXx4qpf+WzHcWK6NOaVvi1xkXlfhAOx\nulWUUq7Af4H7gDRgu1JqjdZ6r7XLvsLZw5aJpwrMY7Kv+7HAPEHVnz9q082ty80TqvmAh4951sLq\nfhAYah5N4lXb/LHsduljPznV3kGlpqZSUFzKuz+cYeO+dJ67rzlP9Wgq874Ih6O01tYtQKko4DWt\ndS/L44kAWuu3r/U5vr6+ukOHDpc99+ijjzJ+/Hjy8vLK5sW+VExMDDFdg8mY/X8MXJ5vnnRKuZoP\nIioXxt1dl+jIBqRmw7A5e8qev/ie5wZG0e+utqScuMCY91ebT8BxcTW/7uLG5GfH0/O++0jef4QJ\nL758xaRWb731Fl26dGHr1q1MmjTpinzTp08nPDycjRs3MmXKlCtenzVrFqGhoaxdu5Zp06Zd8frC\nhQsJDg5m2bJlzJw584rXV65cSUBAAHFxccTFxV3x+ldffYWXlxczZsxg+fLlV7x+cWz2e++9x7p1\n6y57rXr16qxfvx6AN954g02bNl32ur+/P6tWrQJg4sSJbNu27bLXg4KCWLRoEQATJkwgOTn5steb\nN2/O7NmWswXmAAAPMElEQVSzAYiNjeXAgQOXvR4eHs706dMBGDp0KGlpaZe9HhUVxdtvm7+dBgwY\nQGZm5mWv33vvvbz88ssA9OnTh/z8/Mte79u3L88//zwA3bt3588ufu91vesuEpKScQloTEiAN3Us\nI19iYmKIiYkhIyOjbKz7pcaNG0d0dDSpqakMGzbsitefe+45+vXrR0pKCmPGjLni9cmTJ9OzZ0+S\nk5OZMGHCFa/L957zf+9dt/cu+d7bsmVLktY64oo3/okt9gM0AFIveZwGRP75TUqpWCAWwMOjnAf9\nQu6GCbshYTjwp62oXuMgOhpSU2HdlT9ctB8GPfpBSgrUSLjy9VqNwK8heJ6t0jMVVmUnsvIpMZlo\nEehDoK8cmBaOyxZb7AOB3lrrJy2PhwGRWuunrvU5EREROjEx0ar1CmFL8b9n0r17dwJ8PUjdI9+b\nwj4ppW5qi90Wh/mPA8GXPA6yPCeEQzifV8xflyXj4e5KYzmbVDgBWxT7dqCZUipEKVUNGAysscFy\nhahwWmsmfb6L9OxCmt7mI1c9Ek7B6n3sWusSpdRTwNeYhzvO1VrvsTqZEJVgRVIaX+46yQu9QwmO\netHoOELYhE0GUWutvwK+ssWyhKgshzNyeW3NHjo3qc2Yu2/H1aWp0ZGEsAk5lU5UScWlJiYs/QV3\nVxc+iA7H1UWRkpJCSkqK0dGEsJqc9iiqpPe/OcDOtPPMfLw99WqaTyi7OMb8evOxC+EIZItdVDkb\n955m5uZDDOkUTJ829YyOI4TNSbGLKuVIRi5/XZ5MmwY1ebVfK6PjCFEhpNhFlZFfVMrYRUm4uihm\nPN4eT3eZglc4J9nHLqqEi+PVU05nE/dEJ4JrexkdSYgKI8UuqoRZ3//O578c5689m9OteeBV3zN5\n8uRKTiVExZBiF05vw+6TvLN+P33b1uPpHtceq96zZ89KTCVExZF97MKp7UzNYsKyZNo39OO9QWHX\nvWBGcnLyFdO+CuGIZItdOK20c3mMmp9IgI8Hs4dH3PBg6cW50GUcu3B0UuzCKWXkFDJ8TgKFJaUs\nGR1JgI/Mry6qDtkVI5zOhYJiRsxN4MT5fObGdKRZHV+jIwlRqaTYhVMpKC7lybhEUk5lM3NoBzo2\nrm10JCEqneyKEU6joNh8AtL2o2f51+B23BN6m9GRhDCEFLtwCgXFpcQuTOL7A2d4+5E29A+rf8vL\neOuttyogmRCVT4pdOLz8olJGL0jkp0MZvDugDdEdG5ZrOV26dLFxMiGMIcUuHFpOYQmj5yfy8+FM\npg4MY2CHoHIva+vWrYAUvHB8UuzCYZ3JLuSJuAT2nczm/UfDeLhd+UsdYNKkSYCMYxeOT4pdOKQj\nGbkMn5tAenYBHw/vQI876hgdSQi7IcUuHM6vaVk8MW87Jq1ZMroz7RrWMjqSEHZFil04lLU7T/C3\nlTsJ8PFg/shO3B7oY3QkIeyOFLtwCCaTZto3Kfz3u0NENKrFzKEdCPSVaQKEuBopdmH3sguK+euy\nZDbuS2dwx2D+8VArPNxsf/Wj6dOn23yZQhhBil3YtV1p53lqyQ7SzuXz+kOtGNa5EUpde+pda4SH\nh1fIcoWobFbNFaOUGqSU2qOUMimlImwVSgitNfN+OswjM3+iqMTE0tjODI9qXGGlDrBx40Y2btxY\nYcsXorJYu8W+G3gEmGWDLEIA5il3J362i2/2nqZni9uYOjCMWt7VKny9U6ZMAeRKSsLxWVXsWut9\nQIVuRYmq5ctfT/LyF7vJKShh8oMtGNU1RL6/hLhFso9d2IWzuUW8/MVuvvz1JG0a1GTao2E0l3nU\nhSiXGxa7UmojUPcqL72ktf7iZleklIoFYgEaNizfJE3C+WitWbXjOG9/tY8LBcX8rVcoY+5ugpur\nXCpAiPK6YbFrrW2yw1FrPRuYDRAREaFtsUzh2PadvMArX+xm+5FztGvox9uPtOGOujWMjiWEw5Nd\nMaLSncst4t/f/saCbUep4enGuwPaMKhDMC4uxu5LnzVLxgAI52BVsSulHgb+AwQCXyqlkrXWvWyS\nTDidguJS5v10hBmbD5JTWMLgjg15oVdopYx4uRmhoaFGRxDCJqwdFfM58LmNsggnVVJq4rNfjvPB\nNwc4eb6AHnfcxgu9Q+1ut8vatWsB6Nevn8FJhLCO7IoRFaaoxMTnv6QxY/MhjmbmERbsxwfR4XRu\n4m90tKuaNm0aIMUuHJ8Uu7C5guJSlm1PZdaWQ5w4X0CbBjWZNawD97esI2PShagEUuzCZk5k5bPw\n56MsSThGVl4xHRrV4q1H2tCteaAUuhCVSIpdWEVrTdLRc8zbeoQNu0+htaZnizqM7BpCZEhtKXQh\nDCDFLsolPbuAz3ccZ3liKofO5OLr6cbIOxszPKoxwbW9jI4nRJUmxS5uWkFxKZtT0lmZlMZ3KWco\nNWk6NKrFO480oV9Yfbw9HPvbaeHChUZHEMImHPsnUVS4guJSthw4w1e7TrJx72lyi0q5zdeD2Lub\nMLBDkFNdmi44ONjoCELYhBS7uEJWXhFbDpxh0750Nu0zl3ktL3f6h9fngTb1iGri75RzuSxbtgyA\n6Ohog5MIYR0pdoHWmn0ns/kuJZ3v9qez49g5TBpqebnTL6w+D7atR+cm/rg7YZlfaubMmYAUu3B8\nUuxVkNaaQ2dy2Pb7WX7+PZP43zPJyCkCoHWDGjx1T1O633EbYUF+uBo8f4sQ4tZJsVcBhSWl7D1x\ngZ2pWWw/co74w38Ued0antzVLJCo2/3p3jyQ22p4GpxWCGEtKXYnU2rSHM7IZWdqFjvTstiZmsXe\nkxcoLjXPlFyvpid3NwukcxN/IpvUpmFtLxlrLoSTkWJ3YGeyC0k5lc3+UxfYfyqblFPZHDidTWGJ\nCQDvaq60DfJjVNcmhAf7ER7sR92askUuhLOTYrdzBcWlHMnM5fCZXA5f/JhhvmXmFpW9L9DXgzvq\n+jI8qhGhdWsQFlSTJoE+so/8FqxcudLoCELYhBS7wfKLSjmelc+JrPwrPh7LzOPE+YLL3h/o60FI\ngDc9W9SheV1fWtT1JbSuL/4+Hgb9C5xHQECA0RGEsAkp9gqSW1jCmexCzuQUmj9eesspJD27gBNZ\nBZy9ZKsbwEWZD2jW96tOZBN/QgK8y26NA7zxcfCzO+1ZXFwcADExMYbmEMJa0hLXobWmsMREdkEJ\n5/OLOZ9fRFZesfmWX0xWXtFl98/nF3Mur4jMnCLyikqvWJ6ri8LfuxqBvh4E+nrQNsiPBn7VaeBX\nnfp+1anv50ndGp5OefKPI5BiF87C4Ytda01RqYmCYhOFxaUUFJsoKCml4OL9Ysv9kj/u5xSWkFtY\nQm6h+X5OQQm5RSVXPJ9bWEKJ6drX3XZRULO6O35e1fDzcsffuxpNArwJ8PEoK++ym48HtbyqGX5d\nTyGE83OoYv/Xxt9YnXz8j7K2lLi+dvdek4sCHw83fDzc8LbcfDzcqOPrabnvWva8r6dbWYHX8nLH\nr3o1anq54+vhJkUthLA7DlXsdWp40Kp+DTzdXfF0d8HTzfWP++6ueLi74unmYnnuj+fN7/vjeR8P\nNzzdXWT8thDCKTlUsQ/u1JDBnRoaHUMIIeyaQxW7EBXpq6++MjqCEDYhxS6EhZeXXPlJOAcZVyeE\nxYwZM5gxY4bRMYSwmlXFrpSaqpTar5T6VSn1uVLKz1bBhKhsy5cvZ/ny5UbHEMJq1m6xfwO01lq3\nBQ4AE62PJIQQwhpWFbvW+n9a6xLLw5+BIOsjCSGEsIYt97GPBNbbcHlCCCHK4YajYpRSG4G6V3np\nJa31F5b3vASUAIuvs5xYIBagYUMZiy6EEBVF6fKcj3/pApSKAcYA92qt827yc84AR8u5ygAgo5yf\nW5Ek162RXLdGct0ae80F1mVrpLUOvNGbrCp2pVRv4H2gm9b6TLkXdGvrTNRaR1TGum6F5Lo1kuvW\nSK5bY6+5oHKyWbuP/UPAF/hGKZWslPrIBpmEEEJYwaozT7XWTW0VRAghhG044pmns40OcA2S69ZI\nrlsjuW6NveaCSshm9cFTIYQQ9sURt9iFEEJch0MXu1LqOaWUVkrZxeXllVJvWObNSVZK/U8pVd/o\nTGC/c/oopQYppfYopUxKKcNHMCileiulUpRSB5VSfzc6D4BSaq5SKl0ptdvoLJdSSgUrpb5TSu21\n/B8+Y3QmAKWUp1IqQSm105LrH0ZnupRSylUp9YtSal1Frsdhi10pFQzcDxwzOsslpmqt22qtw4F1\nwCtGB7Kw1zl9dgOPAN8bHUQp5Qr8F+gDtASGKKVaGpsKgDigt9EhrqIEeE5r3RLoDPzFTr5ehUAP\nrXUYEA70Vkp1NjjTpZ4B9lX0Shy22IEPgBcAuzlIoLW+cMlDb+wkm73O6aO13qe1TjE6h0Un4KDW\n+netdRGwFHjI4Exorb8Hzhqd48+01ie11jss97Mxl1UDY1OBNsuxPHS33Ozi51ApFQQ8CHxS0ety\nyGJXSj0EHNda7zQ6y58ppd5USqUCj2M/W+yXkjl9rq4BkHrJ4zTsoKgcgVKqMdAOiDc2iZlld0cy\nkA58o7W2i1zAdMwbo6aKXpHdXkHpenPUAJMw74apdDeaO0dr/RLwklJqIvAU8Ko95LK854Zz+hiR\nSzgupZQPsAqY8Ke/WA2jtS4Fwi3Hkj5XSrXWWht6jEIp1RdI11onKaW6V/T67LbYtdY9r/a8UqoN\nEALsVEqBebfCDqVUJ631KaNyXcVi4CsqqdhvlMsyp09fzHP6VNqfprfw9TLacSD4ksdBlufENSil\n3DGX+mKt9WdG5/kzrXWWUuo7zMcojD74fCfQXyn1AOAJ1FBKLdJaD62IlTncrhit9S6t9W1a68Za\n68aY/2RuXxmlfiNKqWaXPHwI2G9UlktZ5vR5Aeh/sxO1VUHbgWZKqRClVDVgMLDG4Ex2S5m3quYA\n+7TW7xud5yKlVODFUV9KqerAfdjBz6HWeqLWOsjSWYOBbyuq1MEBi93OvaOU2q2U+hXzriK7GAKG\nnc7po5R6WCmVBkQBXyqlvjYqi+Xg8lPA15gPBC7XWu8xKs9FSqklwDYgVCmVppQaZXQmizuBYUAP\ny/dUsmVr1Gj1gO8sP4PbMe9jr9ChhfZIzjwVQggnI1vsQgjhZKTYhRDCyUixCyGEk5FiF0IIJyPF\nLoQQTkaKXQghnIwUuxBCOBkpdiGEcDL/HyoENFe9lZ/CAAAAAElFTkSuQmCC\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x7f92d507b3d0>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"x = np.linspace(-4, 4, 1000)\n",
"plt.plot(x, selu(x), label='SELU')\n",
"plt.plot(x[1:], np.diff(selu(x)) / np.diff(x), label='derivative')\n",
"plt.axhline(0, color='black', linestyle='--'); plt.axvline(0, color='black', linestyle='--')\n",
"plt.legend(loc='upper left')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$s$ is continuous, with a minimum value of $-\\lambda \\alpha$ and no maximum value. $s$ maps positive numbers to positive and negative to negative. $s'$ is not continuous, with maximum value of $\\lambda \\alpha$ at $x=0$ and a value of $\\lambda$ for $x\\ge 0$.\n",
"\n",
"Let's visualize how $s$ transforms a distribution."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEICAYAAABRSj9aAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGbxJREFUeJzt3X2UXXV97/H3h/BkASWYMQ1JICihtwnW0DuNXpXbVKog\n0uZyexc3oBRaudELWlzFJQFtefDGlXWroF1WbVQElTSNFUoKPgUuqUWBMLGBkoRghEQSEzIKSFBK\nV+L3/vH7TdgZ5jzMnHNyZn7zea111uzz2w/nu/ec893f89sPRxGBmZmV66BuB2BmZp3lRG9mVjgn\nejOzwjnRm5kVzonezKxwTvRmZoVzoi+QpKslfbXbcdQjKSSd2O04zMYDJ/oRkLRF0i5JR1TaLpK0\nuoth2Rgh6TBJN0h6VtJOSX/eYPrzJG2V9AtJ/yjpmMq4cyR9X9Iv673/JP1x3rle1MZVaTtJh0r6\nh/wZC0nzGkx/jKRb87bZKum8QeNPk/RI3j53Szp+0PjflvRdSc9JelLSpR1Yra5zoh+5CUDLbwol\n/j+ML1cDM4Hjgd8DPiTpjKEmlDQb+FvgfGAy8EvgM5VJngI+CSyp9WKSJgJXAuvbEPuwSTp4mLPc\nA7wL2NnEtH8D/Adp27wT+GzeZkiaBNwC/AVwDNAH/H0lrknAt0jb95XAicB3hhnr2BARfgzzAWwB\nFpE+ZEfntouA1ZVp3gg8APw8/31jZdxqYDHwPeB50htsNfB/gO8DzwH/RHrz3Qw8m5cxo7KMTwFP\n5HFrgVMr464Gvloj9knA7cAzOf5/AQ7K4xYBPwJ2AxuAsyvzXZjjvT7P+1hexwtzHLuACyrT3wh8\nDliVl/fPwPGV8QGcmIcPAz4O/Bh4Ms/3shrxDzeOmssGJuZt0Q88nYenDfo/fTS/3m5SEpjUhvfP\nT4C3VZ5fCyyvMe3HgGWV568hJbajBk233/tv0LjPARfn9bmoyRgvB/5hUNungL/Ow38CbMzb5THg\nPZXp5gHb8jJ2Al8Z4XbaBsyrM/6IvC1OqrR9GViShxcC3x80/fPAf6ps2xHFNtYeriRHro/0wfng\n4BH5q/UdwF+TkvV1wB2SXlmZ7HzSG/EoYGtuW5Dbp5I+0PcCXyJVIxuBqyrzPwDMyeOWAV+TdHgT\ncV9G+gD1kKqgK0lJF1KSPxV4BXAN8FVJUyrzvh54KK/TMmA58DukHdW7gE9LOrIy/TtJiXISsI60\n0xrKEuCkvD4n5vX/yzrrMJw46i37INL2PR44jpQEPj3otc4jJbVXAYdS+X9LeqbOY9FQgefqegrw\nYKX5QWB2jXWdXZ02In4EvJDXqSFJc4FeUrIfjuXAmZKOysuZAJxD2t6QdqhnAS8nbZ/rJf12Zf5f\nJ703jwcWSjquwfbar8ulSScBeyLi0UpbdVsO3na/ADZXxr8BeCp3fe2S9E+SjhtBHKNft/c0Y/FB\nquh/HziZVLH3UKmoSMl6zaB57gUuzMOrgWsHjV8NfLjy/BPANyvP/wBYVyemp4HX5eGrqV3RXwvc\nRq6mG6znOmB+Hr4Q+GFl3GtJO4jJlbafAXPy8I1UqlTgSGAvMD0/D1LiFfAL4DWVaf8L8HiNmJqO\nYwTLngM8Peh/8pHK84uBb7X43pme4z280vZWYEuN6e8C3juobTuDKl2GqOhJ3Yt9wBsq69NURZ+n\nvwf440qMP6oz7T8Cl+bheaRK+/BmX6vGMhtV9KcCOwe1/S9e/Bx+kVzdV8Z/jxc/h4+SvhX+DnA4\nqTD7Xisxj9aHK/oWRMTDpK/7g6u3Y3mxSh+wlVRNDnhiiEU+WRl+fojn+6plSR+UtFHSzyU9Q6rC\nJzUR9l+RqprvSHqsWnnmA3brBqos0o6suszB8RARNWOsrmNEPEfqKjp2UDw9wK8Bayuv+63cXkuz\ncdRdtqRfk/S3+SDes8B3gaNz9Tqg2k/8y0Hr15Ckz+UDfc9JupLULQepEh7wClIXyFCeGzRto+mr\nLgYeioj7hhNzxTLg3Dx8Hi9W80h6u6T7JD2Vt+uZ7P9e6Y+Ifx/h6zar0bZpNP554NaIeCDHeg3w\nRkmv6FC8XeNE37qrSFVENYn/hPSVteo4UiU2YMS3DZV0KvAh0lfpiRFxNOmbhRrNGxG7I+KyiHg1\n8IfAn+czE44HPg+8D3hlXubDzSyzjumVmI8kfZX/yaBpfkr6wM2OiKPz4xURMayEWkOjZV8G/Abw\n+oh4OfBfB8JtZuGVBD7U40qAiHhvRByZHx+LiKeBHcDrKot6HbUPlK6vTivpNaQupEdrTF91GnB2\nPrNnJ+lYxickDe6equVrwDxJ04CzyYle0mHA10nHPibn98o32H+77ff+zl039bbXO5uMqepR4GBJ\nMytt1W05eNsdQeoSHRj/0KA4i72VrxN9iyJiM+lI/p9Vmr8BnKR0WtzBkv4nMItU/bfDUcAe0kHE\ngyX9JS+tXIYk6SxJJ0oSaeewF/gV6UBV5GUi6U9IFX0rzpT0ZkmHkvrq74uI/b7JRMSvSDuY6yW9\nKr/2VEmnt/jazSz7KNKO4Jl8XOWqoZdUc/lH1nl8rM6sXwY+ImmipN8kFQo31pj2ZuAPJJ2aE9VH\ngVsiYndenwn52MzBwEGSDpd0SJ73QuA3SV1Sc0jdONcAH87zzpNUM7lFRD+pu+dLpO6ujXnUoaSD\n3P3AHklvB95WZ32JiB832F77jt8onX46cLzp0LxOL9n5RupzvwW4VtIRkt5MKl6+kie5FThZ0h/l\n5V0FPBgRj+TxXyLtCOfkbfYXwD0R8fN66zIWOdG3x7WkRAlARPyMdKDqMlJ/8YeAsyLip216vW+T\nuiAeJXUJ/TtDdwUNZSZwJ+lr7b3AZyLi7ojYQDoucC+pa+S1pP7MViwjfbieAv4z6UDpUC4ndSfd\nl7tQ7iRV2u1Qb9mfBF5GqvzvI23TA+Eq0oHvraRE+n8jYt9r5wr3VICIWA+8l5Twd5HeZxdXlnU+\naWf1WVKf9fOknRsR8UxE7Bx4kPrNn60ksumks7zqWUY6HrWv2ybvZP4MWEE6NnQesHJ4m6CuTXk9\nppLe68+TvyFLulLSNyvTXkz6H+7KMf7vvM0GdlR/RDrD7WlgLumEh4H1+H+kkxHuyPOfmNelOMoH\nJczaStKNwLaI+Ei3Y7GhSfoC8LWI+Ha3Y7HOGu6FDGZWiIgY1VfJWvu468bMrHDuujEzK5wrejOz\nwo2KPvpJkybFjBkzuh2GmdmYsnbt2p9GRL2LC4FRkuhnzJhBX19ft8MwMxtTJA2+An9I7roxMyuc\nE72ZWeGc6M3MCudEb2ZWOCd6M7PCOdGbmRXOid7MrHBO9GZmhWuY6PNN/9dIelDSeknX5ParJW3P\nPz23TtKZlXmukLRZ0qZ2/ICEmZmNXDNXxr4AvCUinsu/wnJP5cb/10fEx6sTS5pFurn/bNLvg94p\n6aSI2NvOwM26bcaiO/YNb1nyji5GYlZfw0Qf6faWAz9ofEh+1Lvl5XxgeUS8ADwuaTPpl13ubTFW\nszHNOwbrlqb66PPvUq4j/dzWqoi4P496v6SHJN0gaWJum8r+P2u3jf1/OHtgmQsl9Unq6+/vb2EV\nzMysnqYSfUTsjYg5wDRgrqSTSb9R+WrSjw7vIP3eaNMiYmlE9EZEb09Pw5uvmZnZCA3r7pUR8Yyk\nu4Ezqn3zkj4P3J6fbif96PCAabnNzBpw9451QjNn3fRIOjoPvwx4K/CIpCmVyc4GHs7DK4EFkg6T\ndAIwE1jT3rDNzKxZzVT0U4CbJE0g7RhWRMTtkr4iaQ7pwOwW4D0AEbFe0gpgA7AHuMRn3Nh4Va3Q\na7W7crdOa+asm4eAU4ZoP7/OPIuBxa2FZjY+1NoZmLWLr4w1MyucE72ZWeGc6M3MCudEb2ZWOCd6\nM7PCOdGbmRVuWFfGmtmB43PtrV1c0ZuZFc6J3syscE70ZmaFc6I3MyucE72ZWeF81o1ZAz77xcY6\nV/RmZoVzRW/WZr7tsI02rujNzArnit6sDVzF22jmit7MrHBO9GZmhXOiNzMrXMNEL+lwSWskPShp\nvaRrcvsxklZJ+mH+O7EyzxWSNkvaJOn0Tq6AmZnV18zB2BeAt0TEc5IOAe6R9E3gvwN3RcQSSYuA\nRcDlkmYBC4DZwLHAnZJOioi9HVoHswPGB11tLGqY6CMigOfy00PyI4D5wLzcfhOwGrg8ty+PiBeA\nxyVtBuYC97YzcLPxylfq2nA11UcvaYKkdcAuYFVE3A9MjogdeZKdwOQ8PBV4ojL7ttw2eJkLJfVJ\n6uvv7x/xCpiZWX1NJfqI2BsRc4BpwFxJJw8aH6Qqv2kRsTQieiOit6enZzizmpnZMAzrrJuIeAa4\nGzgDeFLSFID8d1eebDswvTLbtNxmZmZd0MxZNz2Sjs7DLwPeCjwCrAQuyJNdANyWh1cCCyQdJukE\nYCawpt2Bm5lZc5o562YKcJOkCaQdw4qIuF3SvcAKSe8GtgLnAETEekkrgA3AHuASn3FjZtY9zZx1\n8xBwyhDtPwNOqzHPYmBxy9GZmVnLfFMzsyH4fHkriW+BYGZWOCd6M7PCOdGbmRXOffRmY4CPGVgr\nnOjNxjDf98aa4a4bM7PCOdGbmRXOid7MrHBO9GZmhfPBWLNC+MCs1eKK3syscE70ZmaFc6I3Myuc\nE72ZWeF8MNasQD4wa1Wu6M3MCudEb2ZWOCd6M7PCOdGbmRWuYaKXNF3S3ZI2SFov6dLcfrWk7ZLW\n5ceZlXmukLRZ0iZJp3dyBczMrL5mzrrZA1wWET+QdBSwVtKqPO76iPh4dWJJs4AFwGzgWOBOSSdF\nxN52Bm5mZs1pWNFHxI6I+EEe3g1sBKbWmWU+sDwiXoiIx4HNwNx2BGtmZsM3rD56STOAU4D7c9P7\nJT0k6QZJE3PbVOCJymzbqL9jMDOzDmo60Us6Evg68IGIeBb4LPBqYA6wA/jEcF5Y0kJJfZL6+vv7\nhzOrmZkNQ1NXxko6hJTkb46IWwAi4snK+M8Dt+en24Hpldmn5bb9RMRSYClAb29vjCR4s5HwVaM2\n3jRM9JIEfBHYGBHXVdqnRMSO/PRs4OE8vBJYJuk60sHYmcCatkZt1gHVHYBZSZqp6N8EnA/8m6R1\nue1K4FxJc4AAtgDvAYiI9ZJWABtIZ+xc4jNuzMy6p2Gij4h7AA0x6ht15lkMLG4hLjMzaxPfvdLG\ntfHQXeNjEuZbIJiZFc6J3syscE70ZmaFc6I3MyucE72ZWeF81o3ZODL4LCOfhTM+uKI3MyucE72Z\nWeGc6M3MCudEb2ZWOCd6M7PCOdGbmRXOp1fauDAebl5mVosrejOzwjnRm5kVzonezKxwTvRmZoVz\nojczK5wTvZlZ4ZzozcwK1zDRS5ou6W5JGyStl3Rpbj9G0ipJP8x/J1bmuULSZkmbJJ3eyRUwM7P6\nmqno9wCXRcQs4A3AJZJmAYuAuyJiJnBXfk4etwCYDZwBfEbShE4Eb2ZmjTW8MjYidgA78vBuSRuB\nqcB8YF6e7CZgNXB5bl8eES8Aj0vaDMwF7m138GbWmuoVw/4RknINq49e0gzgFOB+YHLeCQDsBCbn\n4anAE5XZtuW2wctaKKlPUl9/f/8wwzYzs2Y1neglHQl8HfhARDxbHRcRAcRwXjgilkZEb0T09vT0\nDGdWMzMbhqZuaibpEFKSvzkibsnNT0qaEhE7JE0BduX27cD0yuzTcptZR/nGZWZDa+asGwFfBDZG\nxHWVUSuBC/LwBcBtlfYFkg6TdAIwE1jTvpDNzGw4mqno3wScD/ybpHW57UpgCbBC0ruBrcA5ABGx\nXtIKYAPpjJ1LImJv2yM3M7OmNHPWzT2Aaow+rcY8i4HFLcRlZmZt4itjzcwK50RvZlY4J3ozs8I5\n0ZuZFc6J3syscE70ZmaFc6I3MytcU7dAMButfNsDs8Zc0ZuZFc6J3syscE70ZmaFc6I3MyucE72Z\nWeGc6M3MCudEb2ZWOCd6M7PCOdGbmRXOV8aaGbD/VcZblryji5FYu7miNzMrnBO9mVnhGiZ6STdI\n2iXp4Urb1ZK2S1qXH2dWxl0habOkTZJO71TgZnbgzVh0x76HjR3N9NHfCHwa+PKg9usj4uPVBkmz\ngAXAbOBY4E5JJ0XE3jbEauOYE4vZyDWs6CPiu8BTTS5vPrA8Il6IiMeBzcDcFuIzM7MWtdJH/35J\nD+WunYm5bSrwRGWabbntJSQtlNQnqa+/v7+FMMzMrJ6Rnl75WeCjQOS/nwD+dDgLiIilwFKA3t7e\nGGEcZtZh7jYb+0ZU0UfEkxGxNyJ+BXyeF7tntgPTK5NOy21mZtYlI0r0kqZUnp4NDJyRsxJYIOkw\nSScAM4E1rYVoZmataNh1I+nvgHnAJEnbgKuAeZLmkLputgDvAYiI9ZJWABuAPcAlPuPGbOxxd01Z\nGib6iDh3iOYv1pl+MbC4laDMzKx9fGWsmVnhnOjNzArnRG9mVjgnejOzwjnRm5kVzonezKxwTvRm\nZoXzTwnaqOWLdszawxW9mVnhnOjNzArnRG9mVjgnejOzwjnRm5kVzonezKxwTvRmZoVzojczK5wT\nvZlZ4ZzozcwK50RvZlY4J3ozs8I1TPSSbpC0S9LDlbZjJK2S9MP8d2Jl3BWSNkvaJOn0TgVuZt01\nY9Ed+x42ujVT0d8InDGobRFwV0TMBO7Kz5E0C1gAzM7zfEbShLZFa2ajkpP+6NYw0UfEd4GnBjXP\nB27KwzcB/63SvjwiXoiIx4HNwNw2xWpmZiMw0j76yRGxIw/vBCbn4anAE5XptuW2l5C0UFKfpL7+\n/v4RhmFmZo20fDA2IgKIEcy3NCJ6I6K3p6en1TDMzKyGkf7C1JOSpkTEDklTgF25fTswvTLdtNxm\n1hT38Zq130gT/UrgAmBJ/ntbpX2ZpOuAY4GZwJpWgzSzsaO6s96y5B1djMQGNEz0kv4OmAdMkrQN\nuIqU4FdIejewFTgHICLWS1oBbAD2AJdExN4OxW5mZk1omOgj4twao06rMf1iYHErQZmZWfv4ylgz\ns8I50ZuZFc6J3syscCM968asbXxKpVlnuaI3MyucE72ZWeGc6M3MCuc+ejPrGF8lOzq4ojczK5wT\nvZlZ4ZzozcwK5z56O2B8vrxZd7iiNzMrnBO9mVnhnOjNzArnPnrrKPfLm3WfK3ozs8I50ZuZFc6J\n3syscE70ZmaFa+lgrKQtwG5gL7AnInolHQP8PTAD2AKcExFPtxammZmNVDsq+t+LiDkR0ZufLwLu\nioiZwF35uZmZdUknTq+cD8zLwzcBq4HLO/A6ZjaG1DrV1rcv7rxWK/oA7pS0VtLC3DY5Inbk4Z3A\n5BZfw8zMWtBqRf/miNgu6VXAKkmPVEdGREiKoWbMO4aFAMcdd1yLYZiZWS0tVfQRsT3/3QXcCswF\nnpQ0BSD/3VVj3qUR0RsRvT09Pa2EYWZmdYy4opd0BHBQROzOw28DrgVWAhcAS/Lf29oRqI0dvu2B\ntYN/hrB9Wum6mQzcKmlgOcsi4luSHgBWSHo3sBU4p/UwzcxspEac6CPiMeB1Q7T/DDitlaDMzKx9\nfGWsmVnhfJtiM+sq98V3nit6M7PCuaK3tvCZNmajlxO9DYu/Zls3+H3XGid6Mxs1/M2wM9xHb2ZW\nOFf01lCtKsvVl9nY4IrezKxwTvRmZoVzojczK5z76G1I7n83K4cTvZmNKT6nfvic6G0fV/E2lnkH\nUJsTvZkVx0l/fz4Ya2ZWOFf045C7aMzGF1f0ZmaFc0VvZkVzf70rejOz4ikiuh0Dvb290dfX1+0w\niuO+eLPmjNVKX9LaiOhtNF3Hum4knQF8CpgAfCEilnTqtUrVzFdOJ3Mza6QjiV7SBOBvgLcC24AH\nJK2MiA2deD0zs1Y0WzCN1cq/UxX9XGBzRDwGIGk5MB/oSKLv1sGWWq873PZay2ym3cwOnFY+h8PN\nBe3UkT56Sf8DOCMiLsrPzwdeHxHvq0yzEFiYn/4GsKntgXTeJOCn3Q5ilPK2GZq3S23eNrXV2jbH\nR0RPo5m7dnplRCwFlnbr9dtBUl8zB0LGI2+boXm71OZtU1ur26ZTp1duB6ZXnk/LbWZmdoB1KtE/\nAMyUdIKkQ4EFwMoOvZaZmdXRka6biNgj6X3At0mnV94QEes78VpdNqa7njrM22Zo3i61edvU1tK2\nGRUXTJmZWef4FghmZoVzojczK5wTfZtIukxSSJrU7VhGA0l/JekRSQ9JulXS0d2OqdsknSFpk6TN\nkhZ1O57RQtJ0SXdL2iBpvaRLux3TaCJpgqR/lXT7SJfhRN8GkqYDbwN+3O1YRpFVwMkR8VvAo8AV\nXY6nqyq3BXk7MAs4V9Ks7kY1auwBLouIWcAbgEu8bfZzKbCxlQU40bfH9cCHAB/ZziLiOxGxJz+9\nj3QtxXi277YgEfEfwMBtQca9iNgRET/Iw7tJSW1qd6MaHSRNA94BfKGV5TjRt0jSfGB7RDzY7VhG\nsT8FvtntILpsKvBE5fk2nMxeQtIM4BTg/u5GMmp8klRE/qqVhfgXppog6U7g14cY9WHgSlK3zbhT\nb7tExG15mg+TvprffCBjs7FH0pHA14EPRMSz3Y6n2ySdBeyKiLWS5rWyLCf6JkTE7w/VLum1wAnA\ng5IgdU/8QNLciNh5AEPsilrbZYCkC4GzgNPCF2z4tiB1SDqElORvjohbuh3PKPEm4A8lnQkcDrxc\n0lcj4l3DXZAvmGojSVuA3ogY93fgyz88cx3wuxHR3+14uk3SwaSD0qeREvwDwHmFXjE+LEpV0k3A\nUxHxgW7HMxrliv6DEXHWSOZ3H711yqeBo4BVktZJ+ly3A+qmfGB64LYgG4EVTvL7vAk4H3hLfq+s\ny1WstYkrejOzwrmiNzMrnBO9mVnhnOjNzArnRG9mVjgnejOzwjnRm5kVzonezKxw/x+CspPOcpTH\ndwAAAABJRU5ErkJggg==\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x7f92d5070f90>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"z = np.random.randn(10000)\n",
"plt.hist(z, bins=100)\n",
"plt.title('Normal sample mean=%.3f, var=%.3f' % (z.mean(), z.var()))\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEICAYAAABRSj9aAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFqhJREFUeJzt3XuUZWV95vHvIyIooMLQdhBom3HwAk7ArA6QyEwweEEh\nMhknDMQLJjgdEx3NClFBxyW6gsOacYwZr9NRBgwqsoIsWYIiEpXxAthM0NBcHOQi3QKNCnIxwTT8\n5o+9C49FXU5VneJUvfX9rFWrz9n73fu8+/Q5z3n3b791KlWFJKldjxl3ByRJi8ugl6TGGfSS1DiD\nXpIaZ9BLUuMMeklqnEGvRZPklCRnjbsf0kpn0I9YkkOTfDPJT5P8JMk3kvx6v+41SR5Mct+kn6f2\n629O8oIp9vmaJF+fYvmU7bU0JDk8yXVJfpbkK0meNkPb3ZKcl+T+JLck+f2BdY9L8rf9/3clOWzS\ntm9OcnWSe5PclOTNi3hYI5FkhySnJ7knye1J/myGtkny9iQ/6NufneSJA+v3TPK5/v22OcnrJm1f\n/fM68X772GIe21Jk0I9Q/+L7PPABYDdgT+BdwAMDzb5VVTtP+vnhGLqrRZRkd+CzwDvoXgsbgc/M\nsMmHgJ8Dq4FXAB9Jsv/A+q8DrwRun+rhgFcDuwJHAG9IcuxCj2Eu+jCeS56cAuwLPA14PvCWJEdM\n0/bVwKuA5wFPBR5P9x6bcBZwE91zdyTwniTPn7SPAwbeb6+dQz/bUFX+jOgHWAfcPcP61wBfn2H9\nzcALht1uuvYD29wI3Ev3JnhFv/zpwN8BPwZ+BHwSePKkfb4Z+C5wP/BxujfQF/p9fRnYtW+7Fihg\nPfBD4Dbgzwf2dQpw1sD9Q4BvAncD3wEOm+W5GKofs+0b+APg2n67G4E/Glh3GLAZOBHY2h/DH4zg\ntbAe+ObA/Z2AfwSeNUXbnehC/hkDyz4BnDZF280zPW99m/8JfGCIPu7QP1/PGVi2qu/nU+g+OD4P\n3Anc1d/ea6DtV4FTgW/02/yrOTw/PwReNHD/3cDZ07T9W+AtA/d/E/gn4AnAzv1r8CkD6zcAfzNw\nv+bStxZ/HNGP1veAB5OcmeQlSXYdRyeS7ET3Zn9JVe1C98a4amI18F/pRkbPBvamC+RBLwdeCDwD\n+B26cH0bXQg8BnjjpPbPpxudvQh46zTlpz2BC4C/oBvh/jlwbpJVMxzKUP0YYt9bgaOAJ9KF/l8m\n+bWBx/kV4El0Z2AnAB+a+L9LclKSu6f7maHv+9N94ABQVfcDN/TLJ3sGsK2qvjew7DvTtJ1RkgD/\nBtg0W9uqeoDurOO4gcXHAF+rqq10z/H/pht1r6EL8w9O2s2r6D7UdgFuSfLhGZ6v7/Z93BXYg4Hn\nZ47HG7oPqX3721Otf86kZZf2JaLPJlk75OM0w6Afoaq6BziUbgTx18CdSc5Psnqg2SGTXvzfX6Tu\nPAQ8J8njq+q2qtrU9/GGqrq4qh6oqjuB9wG/NWnbD1TVHVW1Bfg/wOVV9fdV9U/AecBzJ7V/V1Xd\nX1X/QBcMx/FIrwQurKoLq+qhqrqYrpzx0hmOYdh+zLjvqrqgqr5fna8BX6ILwwn/DLy7qv65qi4E\n7gOe2W97WlU9ebqfGfq+M/DTScvuoQvEqdreM2Tb2ZzCLwJ6GJ8CBss8v98vo6p+XFXnVtXPqupe\nutH75NfKGVW1qaq29c/fn8zwfP1qv83O/b+Dz89Mx/tF4LVJ1iZ5EvDWfvkT+n59A3hHkh37D/CX\n0432J/wW3dnns+jOJD6f5LHDPDmtMOhHrKqurarXVNVedKOKpwLvH2hy2aQX/9OH2O02YPsplm9P\nF1KT+3A/8B+B1wG3JbkgybMAkqzuL2ZtSXIPXX1z90m7uGPg9j9OcX/nX27OrQO3b6E75smeBvze\npNHwoXQju+kM248Z992fXV3WX6y7m+4DYPCYf1xV2wbu/2yKY5xWkjUDF/ru6xffR3cGMehJdOWj\nyebSdqZ+vIGunn1kP1ofxleAJyQ5uB/pHkj3IUqSJyT5X/3F4XuAS4EnJ9luYPtbJ+9wCBPP0eAx\nz3S8pwOfpisVber7DF0ZC7prGvv0ffkI3Wt6Yh1VdWlV/byq7gbeRBf6z55Hv5ctg34RVdV1wBk8\n8jRyrn4ArOlPy4HuTUhXR71lmse+qKpeSBd219GdYQC8h+6M419X1RPpRsNTnf7Oxd4Dt9fQjZom\nu5Wubjr4IbdTVZ22wMeecd9JdgDOBd4LrO5H4Rcy5DEneVseOUvql0K9qn5QAxfX+003AQcM7Gcn\nuusjU5VUvgc8Nsm+A8sOmKbtdP38Q+Ak4PCq2jxb+wlV9SBwDt1Z2HHA5/tRMnTXLZ4JHNy/Vv7t\nxMMN7mJSPz46w/M1cVZ5F921kAMGNp32ePuztHdW1dp+ALUJ2NL/UFW3VNVRVbWqqg6m+xC/YpZD\nX+hrflkx6EcoybOSnJhkr/7+3nRvnsvmsJvt+1PQiZ/HApfTXXw6qV+2E3AaXXniEUHfj9qP7ts9\nQDeCeqhfvUt//6d9bXsUU/He0Y/+9qergU81u+Qs4HeSvDjJdv1xHDbxXC3QTPt+HF09905gW5KX\n0F1LGEpVvaceOUtqcqhP5Ty60tnLk+wIvBP4Tv/hP/kx7qerlb87yU5JDgVeBvzNRJt00xF37O8+\nrj/G9OteQfcB/sKqunHy/pN8NckpM/T1U3RngK/ob0/Yhe7M6e4ku/XHMKOqet0Mz9dgDf4TwH9J\nsmuSZwP/iW5Q9Ajppp4+PZ396MqN766qh/r1z06yS7ppqK+k+/99X79u/yQH9q+LnfvlW+guzq8Y\nBv1o3QscDFye5H66gL+abmQ04TemGOn8+sD6C+neXBM/p/Sn4UfyixkiN9KVR46pqqn+oMBjgD+j\nG1n/hK5G+cf9uncBv0ZXH72ALmAW6mt0FxovAd5bVV+a3KCqbgWOpruYeifdKPzNjOA1ONO++9Hp\nG+lGrXfR1aDPX+hjDtGnO+lqxaf2j3sQA7Xw/kzhCwOb/AndtMGtdGH7xxPXVXrX070e9gQu6m9P\nzMv/C+BfAN8eeE19dGDbvenq2NP19XK6mU1PpbvgPeH9fZ9+RPda/uJQBz+cdwLfpxuofBX4b1X1\n8P77Y5i4jrI73fvi/r5/p1fVhoF9vZjuPXEXXbnyiP75h26m1mforgHcSPecHVVVjyh5tixT54Q0\nu76mexOw/aQat5aI/qzmnKr6zXH3ReNj0GveDHppebB0I0mNc0QvSY1zRC9JjVsSvx22++6719q1\na8fdDUlaVq688sofVdVMXyMCLJGgX7t2LRs3bhx3NyRpWUky5S9MTmbpRpIaZ9BLUuMMeklqnEEv\nSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGrckfjNWy9Paky54+PbNpx05xp5ImokjeklqnEEvSY0z\n6CWpcdbo58i6tKTlxqDXWPiBKT16DPppGESSWmGNXpIaZ9BLUuMMeklqnDX6BbCOL2k5cEQvSY0z\n6CWpcZZuFpnlHUnjZtAPYTCsJWm5sXQjSY1zRK9HjWdG0ngY9Bo5r0tIS4tBr5FwtC4tXdboJalx\nBr0kNc6gl6TGGfSS1Dgvxo6IM00kLVUGvRaVs3Gk8Zs16JPsDXwCWA0UsKGq/irJbsBngLXAzcAx\nVXVXv83JwAnAg8Abq+qiRen9EmW4SVpKhqnRbwNOrKr9gEOA1yfZDzgJuKSq9gUu6e/TrzsW2B84\nAvhwku0Wo/OSpNnNOqKvqtuA2/rb9ya5FtgTOBo4rG92JvBV4K398rOr6gHgpiQ3AAcB3xp151tk\nrV/SqM2pRp9kLfBc4HJgdf8hAHA7XWkHug+BywY229wvm7yv9cB6gDVr1sylG1qB/ACU5m/ooE+y\nM3Au8KdVdU+Sh9dVVSWpuTxwVW0ANgCsW7duTtuuFIabpFEYah59ku3pQv6TVfXZfvEdSfbo1+8B\nbO2XbwH2Hth8r36ZJGkMZg36dEP3jwPXVtX7BladDxzf3z4e+NzA8mOT7JBkH2Bf4IrRdVmSNBfD\nlG6eB7wK+IckV/XL3gacBpyT5ATgFuAYgKralOQc4Bq6GTuvr6oHR97zhsx1OmZrJZ3WjkdaaoaZ\ndfN1INOsPnyabU4FTl1AvyRJI+J33UhS4wx6SWqcQS9JjTPoJalxBr0kNc6vKR4Tv+FyNJyaKc3O\nEb0kNc4R/TI3+czAUa2kyRzRS1LjHNFr2fH6hjQ3juglqXGO6B9FjkQljYNBryXF6ZLS6Fm6kaTG\nGfSS1DiDXpIaZ9BLUuO8GDvAWTGSWuSIXpIa54heK4LTNrWSGfRasiylSaNh0DfGkaukyazRS1Lj\nDHpJapxBL0mNM+glqXFejF0mnIEiab4c0UtS41bkiN4piJJWkhUZ9JqdH4ZSOyzdSFLjVvyI3ouc\nklrniF6SGrfiR/Rqk2dq0i8Y9CvESri4Omy4r4TnQhpk0GtOHClLy8+sNfokpyfZmuTqgWWnJNmS\n5Kr+56UD605OckOS65O8eLE6LkkazjAj+jOADwKfmLT8L6vqvYMLkuwHHAvsDzwV+HKSZ1TVgyPo\nqzRylnG0Eswa9FV1aZK1Q+7vaODsqnoAuCnJDcBBwLfm3UPNm2UWSbCwGv1/TvJqYCNwYlXdBewJ\nXDbQZnO/7BGSrAfWA6xZs2YB3dBcOYqVVpb5zqP/CPAvgQOB24D/MdcdVNWGqlpXVetWrVo1z25I\nkmYzr6Cvqjuq6sGqegj4a7ryDMAWYO+Bpnv1yyRJYzKvoE+yx8Dd3wUmZuScDxybZIck+wD7Alcs\nrIuSpIWYtUaf5NPAYcDuSTYD7wQOS3IgUMDNwB8BVNWmJOcA1wDbgNc740aSxmuYWTfHTbH44zO0\nPxU4dSGdkiSNjl9qJkmNM+glqXEGvSQ1zqCXpMb57ZXSFPztYbXEoJd6fjeQWmXpRpIa54heD3NE\nK7XJEb0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnNMrNSunXUrLmyN6SWqcI/oVztG61D5H9JLU\nOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z\n6CWpcQa9JDXOoJekxvmHR6RZDP5xlptPO3KMPZHmxxG9JDVu1qBPcnqSrUmuHli2W5KLk/y//t9d\nB9adnOSGJNcnefFidVySNJxhRvRnAEdMWnYScElV7Qtc0t8nyX7AscD+/TYfTrLdyHorSZqzWWv0\nVXVpkrWTFh8NHNbfPhP4KvDWfvnZVfUAcFOSG4CDgG+NprvS8mWtX+My34uxq6vqtv727cDq/vae\nwGUD7Tb3y8Zu8E0mSSvJgmfdVFUlqblul2Q9sB5gzZo1C+2G9KiYbsDgCF1L2Xxn3dyRZA+A/t+t\n/fItwN4D7fbqlz1CVW2oqnVVtW7VqlXz7IYkaTbzDfrzgeP728cDnxtYfmySHZLsA+wLXLGwLkqS\nFmLW0k2ST9NdeN09yWbgncBpwDlJTgBuAY4BqKpNSc4BrgG2Aa+vqgcXqe+SpCEMM+vmuGlWHT5N\n+1OBUxfSKUnS6PibsZLUOINekhpn0EtS45r+9kp/SUqPFl9rWsoc0UtS4wx6SWqcQS9JjTPoJalx\nBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktS4pr+PXlqq\nBr+//ubTjhxjT7QSOKKXpMYZ9JLUOINekhpn0EtS4wx6SWqcs26kJcTZOFoMjuglqXEGvSQ1zqCX\npMZZo5fGbLAuLy0Gg15aoqb7APAirebK0o0kNc6gl6TGGfSS1Dhr9NIy5i9YaRiO6CWpcQsa0Se5\nGbgXeBDYVlXrkuwGfAZYC9wMHFNVdy2sm5Kk+RrFiP75VXVgVa3r758EXFJV+wKX9PclSWOyGKWb\no4Ez+9tnAv9uER5DkjSkhQZ9AV9OcmWS9f2y1VV1W3/7dmD1VBsmWZ9kY5KNd9555wK7IUmazkJn\n3RxaVVuSPAW4OMl1gyurqpLUVBtW1QZgA8C6deumbCNJWrgFjeirakv/71bgPOAg4I4kewD0/25d\naCclSfM376BPslOSXSZuAy8CrgbOB47vmx0PfG6hnZQkzd9CSjergfOSTOznU1X1xSTfBs5JcgJw\nC3DMwrspSZqveQd9Vd0IHDDF8h8Dhy+kU5Kk0fE3YyWpcQa9JDXOoJekxhn0ktS45r6m2L+/KUm/\nzBG9JDXOoJekxjVXupFaZ3lSc2XQSw3yTwxqkKUbSWqcI3pJM/LsYPlzRC9JjTPoJalxBr0kNc4a\nvbSCTJ6aac19ZTDopcY5714GvdSI+QT6dDNq/HBoizV6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS\n1DiDXpIa5zx6SYBz51tm0EtaNH7F8dJg0EsamsG9PFmjl6TGGfSS1DhLN5LmZbqLt5Z0lp4mgt7Z\nAtLSZ31/fCzdSFLjDHpJalwTpRtJy8sw5dbp/hCKZZ+5c0QvSY0z6CWpcamqxdlxcgTwV8B2wMeq\n6rTp2q5bt642btw478dy1o20Mk1XxlkppZ4kV1bVutnaLUqNPsl2wIeAFwKbgW8nOb+qrlmMx5O0\nMs010FfKB8Bki3Ux9iDghqq6ESDJ2cDRgEEvaVFMd2Y/zBn/fKoCC7lY/Gh/4CxK6SbJfwCOqKrX\n9vdfBRxcVW8YaLMeWN/ffSZwPbA78KORd2i8WjwmaPO4Wjwm8LiWk7ke09OqatVsjcY2vbKqNgAb\nBpcl2ThMvWk5afGYoM3javGYwONaThbrmBZr1s0WYO+B+3v1yyRJj7LFCvpvA/sm2SfJ44BjgfMX\n6bEkSTNYlNJNVW1L8gbgIrrpladX1aYhNt0we5Nlp8VjgjaPq8VjAo9rOVmUY1q0efSSpKXB34yV\npMYZ9JLUuCUX9En+e5Lrknw3yXlJnjzuPi1Ukt9LsinJQ0mW9XSwJEckuT7JDUlOGnd/RiHJ6Um2\nJrl63H0ZpSR7J/lKkmv619+bxt2nhUqyY5IrknynP6Z3jbtPo5JkuyR/n+Tzo973kgt64GLgOVX1\nq8D3gJPH3J9RuBr498Cl4+7IQgx8tcVLgP2A45LsN95ejcQZwBHj7sQi2AacWFX7AYcAr2/g/+sB\n4Ler6gDgQOCIJIeMuU+j8ibg2sXY8ZIL+qr6UlVt6+9eRjcHf1mrqmur6vpx92MEHv5qi6r6OTDx\n1RbLWlVdCvxk3P0Ytaq6rar+b3/7XroQ2XO8vVqY6tzX392+/1n2M0qS7AUcCXxsMfa/5IJ+kj8E\nvjDuTuhhewK3DtzfzDIPjpUiyVrgucDl4+3JwvUljquArcDFVbXsjwl4P/AW4KHF2PlYvgIhyZeB\nX5li1dur6nN9m7fTnXp+8tHs23wNc0zSOCTZGTgX+NOqumfc/VmoqnoQOLC/fndekudU1bK9vpLk\nKGBrVV2Z5LDFeIyxBH1VvWCm9UleAxwFHF7LZKL/bMfUCL/aYplJsj1dyH+yqj477v6MUlXdneQr\ndNdXlm3QA88DXpbkpcCOwBOTnFVVrxzVAyy50k3/B0veArysqn427v7ol/jVFstIkgAfB66tqveN\nuz+jkGTVxEy8JI+n+5sX1423VwtTVSdX1V5VtZbuPfV3owx5WIJBD3wQ2AW4OMlVST467g4tVJLf\nTbIZ+A3ggiQXjbtP89FfJJ/4aotrgXOG/GqLJS3Jp4FvAc9MsjnJCePu04g8D3gV8Nv9e+mqftS4\nnO0BfCXJd+kGHhdX1cinI7bGr0CQpMYtxRG9JGmEDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLU\nuP8PYKFjFJlvfCMAAAAASUVORK5CYII=\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x7f92d1933990>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"h = selu(z)\n",
"plt.hist(h, bins=100)\n",
"plt.title('SELU sample mean=%.3f, var=%.3f' % (h.mean(), h.var()))\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$h=s(z)$ is clearly not normally distributed despite the normal distribution of $z$. However, SNNs don't claim normally distributed activations, but only that the activations will have 0 mean and unit variance.\n",
"\n",
"Recall the special property of the activation function is to map $z \\sim \\mathcal{N}(\\mu \\omega, \\sqrt{\\nu \\tau})$ to $h$ with $\\mathbf{E}[h] = 0$ and $\\text{Var}[h] = 0$ when $\\mu =0,\\ \\omega=0,\\ \\nu=1,\\ \\tau=1$. In practice, none of these equalities will be exact because of issues such as sampling as well as the difficulty of maintaining appropriately normalized weight vectors. The power of SELU comes from the fact that mean and variance in the neighborhood of 1, $\\mathbf{E}[h]$ is closer to 0 than $\\mathbf{E}{z}$ and $\\text{Var}[h]$ is closer to 1 than $\\text{Var}[z]$. On the domain of near standard normal distributions, SELU maps to a (non-normal) distribution with closer to zero mean and unit variance.\n",
"\n",
"Stacking layers of nearly normalized matrices followed by SELU activations will lead to activations with moments approaching (0, 1). Each SELU layer is a [contraction](https://en.wikipedia.org/wiki/Banach_fixed-point_theorem) acting on the first two moments of the distribution.\n",
"\n",
"For SELU to move mean and variance closer to 0, 1, it is critical that it is applied to a normal distribution and not just any distribution with appropriate mean and variance. As an example, consider applying the SELU function repeatedly to a normal distribution. If each application contracted the mean and variance, after 10 iterations the mean and variance should be very close to 0, 1. Observe below that is not the case. Thankfully, multiplying by a weight matrix restores the activations to a normal distribution due to central limit theorem."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAEICAYAAAC55kg0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGrtJREFUeJzt3Xu4VdV97vHvKxAhKipxhxJQt+nBpEAf8UiQ1EutGsXa\nBJsnISSngg2RtmprTtKTA0lbtSfk8TR5cvEkmlprwGqknFwq1RJLUGpsY3DTaBTxQrwB5bKjImIi\nOejv/DHG1slirb3Xhr332nS8n+eZz55rzDHnGnOuyzvnmHPNrYjAzMzKdFCrG2BmZq3jEDAzK5hD\nwMysYA4BM7OCOQTMzArmEDAzK5hD4AAl6SJJ97a6HQcCSaskfazV7TAbjBwCmaSnJZ2dx/v9C7a/\nv5gkzZX0qKSXJG2V9E+SDsvTFkn6paSdleHBPK1dUkgaWmeZiyR9tqasYX0beJJGSfqupJclPSPp\nIz3U/++StkjaIelGSQdXpl0mqUPSLkmL6sw7U9K6/B57RNIF/bBKfUbS+ZLulbQ9r/MNXZ+JBvXb\nJd0t6ef5s3R2ZZokfUbSs3nbLZE0sjL9C5KeyNvmUUmz+3v99pVDoB+0+gtR0m8CnwM+HBGHAb8G\n/H1Ntb+KiEMrwwkD3lDrD18DfgmMBv4bcJ2kifUqSjoXmA+cBRwLvB24qlLlP4DPAjfWmXcscDPw\nCWAk8D+Ab0p6a5+tSRN6+Vk7nLQ+byN9JsYCn++m/q3Aj4G3AJ8BviWpLU+bDVwInJKXNwL4P5V5\nXwbem59zDvAVSb/Ri7YOnIjwkH41/TRwNunN8QrwKrAT2J6nHwx8AXgW2Ap8HRiRp50BbAT+J7AF\n+DvgSOB2oBN4IY+Py/UX5uW/kp/jq7n8ncAK4HngMWBmpX1vAZYBO4DVwP8C7m2wLn8K/EM367oI\n+GyDae1AAEObma+H+gK+BGzL7X4ImJSnnU/6gO0ANgBX1lnm7+dpLwB/CLwL+AmwvWub5foXAf8K\nfBV4EXgUOKsyfRXwscrjjwLr8nLvBI7tYVs01Y6elg18JS9nB7AGOK0y7UpgKXAT8BKwFpjSy/fw\nIaQAOL5SdhNwdYP63wQ+V3l8JrClTr3PAotqyk4GttWUdQLvbqKd1wFfqCm7DfhEHp8P/DRvh0eA\n363zWn8JeK72/djL7fV+4KEG044HdgGHVcruAf4wj38L+FRl2m+QPs9vbrC8ZcAn97Wt/Tn4SKBG\nRKwjfdB/GGkP+Yg86WrSG2My8F9IexF/UZn1V4BRpD2qeaSjrG/kx8cAvyB9SRERnwF+AFyWn+My\nSYeQAuCbwFuBWcC1kibk5X+N9CYbQ/qi+Wg3q/Ej4FxJV0k6pXqIP8DOAU4nbbfDgZmkDy6kPaXZ\nwBGkQPijOt0JJwPjgQ8BXybtjZ0NTARm5iOeat2fAkcBVwDfkTSqtkGSZgCfJn0BtJFeh1t7WI+m\n2tHEsu8nvX9GkV7n/ytpeGX6+4AleZssI79f8rJvz90Y9Ybbc7Xjgd0R8XhlmQ/mdtYzMU+v1h0t\n6S09bA+ADmCdpPdKGpJfu12kcOzJrcCHJCmv25Gk98qSPP2nwGmk98xVwM2SxlTmPxl4knS0s1DS\nqd1sm+2STm3QjtNJYVvPRODJiHipUtbdthRpR3H8XhOkEaQdh0bP1VqtTqHBMpCPBPL4RVT2skkv\n8MvAr1bK3g08lcfPIO2BDe9m+ZOBFyqPV7Hn3umHgB/UzPPXpC+0IcD/A95ZmfY5GhwJ5OnnAf9I\n2lvdCXwRGJKnLSIFyvbKsDhPa6fvjgTOBB4HpgEH9bD9vwx8qWaZYyvTnwM+VHn8beDjldfrPwBV\npq8GLqzd1sByYG6l3kHAz6lzNLAP7Wh62Xn6C8AJefxK4PuVaROAX/TyPXwaNXvywMXAqgb1fwpM\nrzwelte3vabeXkcCuXxufm/tzut5fpPtFOmI+vRKG+/qpv4DwIzKa/1sb7ZLg2W+J2//4xtMvxC4\nr6ZsYdd2AD6W39vtpLBalrfdXkdCwGLge9X352AafCTQnDbgzcCarr0L0ovaVqnTGRGvdD2Q9GZJ\nf51Pzu0gHUoeIWlIg+c4Fji5ugdD6tP9lfw8Q0ldCV2e6a7BEbE8It5L2uucQfrwVE9EfyEijqgM\nc3rcCunDPqymbBjwWh5q23AXaW/2a8A2Sdd3nTyTdHI+6dYp6UXS0ddRNYvYWhn/RZ3Hh1Yeb4r8\nicueIfXV1jqW1D/btY2fJ30pja23wr1sR7fLlvSn+UTqi3n64ey5zlsq4z8Hhveyz3snqX++6nBS\nt0oz9Q/PfxvVf10+SfpXpB2gNwG/CdwgaXJP8+bXaQnw4Vz0EeCWyrJnS3qgsh0nsed2qn4Oek3S\nNNKR2Adiz6Omqp625Y2kI5pVpD38u3P5xprn+jyp/TNr3p+DhkOgvtoX62ekD/vEypfm4RFxaDfz\nfBJ4B3ByRIwkHXpC+lKoV38D8C81X8yHRsQfkfpadwNHV+of09SKRLwWESuBu0hvxv3xLGnPp+o4\nYENE7BUC+fmviYiTSHu2x5NOIEL6EC4Djo6Iw0nnWFRvGU0a29W9kB1DOjqotQH4g5rtPCIi/m0/\nnrvHZUs6DfgUqUvsyEjdjC/S5DpLWq49r+aqDstztceBoZKqXRIn0LgbYm2eXq27NSKea1C/ajJw\nT0R05PfY/aRuyLN7mK/LrcAHJB1L6t75dl7PY4G/AS4D3pK308PsuZ32+OxIOq2bbbMzb/uuuieS\n3ncfzZ+LRtYCb6+5euj1bZnX+YqIaI+Icbl8Ux66nusq0hH5ORGxo8ntMuAcAvVtBcZJehOkF5z0\nxvxS19UPksbmqysaOYwUHNtz3/QVdZ7j7ZXHtwPHS7pQ0rA8vEvSr0XEq8B3gCvzEcYE0hUHdUma\nIWmWpCOVTCXtqd3Xi21wsKThleEg0gf1fEnn5H7gtwF/xht9ubXteFfe4x9G6k57hTeOGA4Dno+I\nV3L7ur2UsQlvBf4kb7cPkk7w/1Odel8HFihfMSPp8Fy/L3S37MNIQd5J+qL+C/be02woIs6LPa/m\nqg7n5Tovk94nfynpkNwX/j7ShQr13ATMlTQh98v/OanLj9z+oUrnLIYAQ/L7oOvI5H7g1K49//zl\nehr5nIDSZdZPd7M+PybtXN0A3BkR2/OkQ0hf8p15Ob9PDzsvEfGDbrbNoRHxg7ysSaQj+D+OiH/s\nYZmPk7qhrsjr/X7g13kjrEZJ+tX8+ZpA6m79y66dIUkLSO/ps5sM1dZpdX/UYBnY85zAm4A7SIfz\nP8tlw0n98E+Sru5YB/xJnnYGsLFmeW8jHSruJO2h/QGVvnPSOYXHSf2S1+Syd+Tn7ST1Pd8FTM7T\n2khB0czVQacDK0kfspfy81SvZFhEOoexszJ0rWd7bmft0LVt3ku6suVFUpfL58lXSdVpx1mkL4Wd\nuS23AIfmaR/I87+U1+urwM01bRhaWdZG4IzK45uBP8vjF7Hn1UGPk/a+uuquYs/zLxeSrlTqujLp\nxgbt71U7uls26Yv0xly+mXRU8HRlu17Ztf6NnrvJ9/Eo4B9Iofss8JHKtGPya3FMpewTpB2SHaQL\nGQ6uTLuyzvvgysr0y4D1+TV8ksrVL6RAuaWHtv55XuYHa8oXkj97pC/Xf+GNczoX0c25sB6e7xuk\nnZDq+35tZfrXga/XvAarSDtzj3W9Vnna8bns56T38SdqnitIJ8qrz/Xpgfgu6+2g3GCzA5aki0hf\nEo2uArEBJumfgcsjXW1ng5h/5WlmfS4izml1G6w5PidgZlYwdweZmRXMRwJmZgUb9OcEjjrqqGhv\nb291M8zMDihr1qz5WUS09VSvqRDI1/u+RLrp2e6ImJKvff970mVUT5N+EfdCrr+A9JPyV0mXUd6Z\ny08iXZ44gnQN9+XRQ39Ue3s7HR0dzTTTzMwySd3eVaBLb7qDfisiJkfElPx4PrAyIsaTrkmfn594\nAunmZxOB6aSboHXdKuE60n1Cxudhei+e38zM+tj+nBOYQboxEvnvBZXyJRGxKyKeIv2YZKrSXQBH\nRsR9ee//pso8ZmbWAs2GQADfl7RG0rxcNjoiNufxLaTbukK6WVb1Bk8bc9lY9ry5Ule5mZm1SLMn\nhk+NiE35vjkrJD1anRgRIanPrjXNQTMP4JhjmrpPmpmZ7YOmjgQiYlP+uw34LjAV2Jq7eMh/t+Xq\nm9jzbpfjeOPueuPqlNd7vusjYkpETGlr6/HktpmZ7aMeQyDfjbDrH5QfQvoPQA+TbsfadSfLOaR/\nD0cunyXpYEnHkU4Ar85dRzskTcu3/J1dmcfMzFqgme6g0cB3863ahwLfjIjvSbofWCppLukuejMB\nImKtpKWk/w26G7g00q2QAS7hjUtEl+fBzMxaZNDfNmLKlCnh3wmYmfWOpDWVS/ob8m0jzMwKNuhv\nG7E/2uff8fr401ef38KWmJkNTj4SMDMrmEPAzKxgDgEzs4I5BMzMCuYQMDMrmEPAzKxgDgEzs4I5\nBMzMCuYQMDMrmEPAzKxgDgEzs4I5BMzMCuYQMDMrmEPAzKxgDgEzs4I5BMzMCuYQMDMrmEPAzKxg\nDgEzs4I5BMzMCuYQMDMrmEPAzKxgDgEzs4I5BMzMCuYQMDMrmEPAzKxgDgEzs4I5BMzMCuYQMDMr\nmEPAzKxgDgEzs4I5BMzMCtZ0CEgaIunHkm7Pj0dJWiHpifz3yErdBZLWS3pM0rmV8pMkPZSnXSNJ\nfbs6ZmbWG705ErgcWFd5PB9YGRHjgZX5MZImALOAicB04FpJQ/I81wEXA+PzMH2/Wm9mZvulqRCQ\nNA44H7ihUjwDWJzHFwMXVMqXRMSuiHgKWA9MlTQGGBkR90VEADdV5jEzsxZo9kjgy8CngNcqZaMj\nYnMe3wKMzuNjgQ2Vehtz2dg8Xlu+F0nzJHVI6ujs7GyyiWZm1ls9hoCk3wG2RcSaRnXynn30VaMi\n4vqImBIRU9ra2vpqsWZmVmNoE3VOAd4n6beB4cBISTcDWyWNiYjNuatnW66/CTi6Mv+4XLYpj9eW\nm5lZi/R4JBARCyJiXES0k0743hURvwcsA+bkanOA2/L4MmCWpIMlHUc6Abw6dx3tkDQtXxU0uzKP\nmZm1QDNHAo1cDSyVNBd4BpgJEBFrJS0FHgF2A5dGxKt5nkuARcAIYHkezMysRXoVAhGxCliVx58D\nzmpQbyGwsE55BzCpt400M7P+4V8Mm5kVzCFgZlYwh4CZWcEcAmZmBXMImJkVzCFgZlYwh4CZWcEc\nAmZmBXMImJkVzCFgZlYwh4CZWcEcAmZmBXMImJkVzCFgZlYwh4CZWcEcAmZmBXMImJkVzCFgZlYw\nh4CZWcEcAmZmBXMImJkVzCFgZlYwh4CZWcEcAmZmBXMImJkVzCFgZlYwh4CZWcEcAmZmBXMImJkV\nzCFgZlYwh4CZWcEcAmZmBXMImJkVzCFgZlawHkNA0nBJqyU9KGmtpKty+ShJKyQ9kf8eWZlngaT1\nkh6TdG6l/CRJD+Vp10hS/6yWmZk1o5kjgV3AmRFxAjAZmC5pGjAfWBkR44GV+TGSJgCzgInAdOBa\nSUPysq4DLgbG52F6H66LmZn1Uo8hEMnO/HBYHgKYASzO5YuBC/L4DGBJROyKiKeA9cBUSWOAkRFx\nX0QEcFNlHjMza4GmzglIGiLpAWAbsCIifgSMjojNucoWYHQeHwtsqMy+MZeNzeO15fWeb56kDkkd\nnZ2dTa+MmZn1TlMhEBGvRsRkYBxpr35SzfQgHR30iYi4PiKmRMSUtra2vlqsmZnV6NXVQRGxHbib\n1Je/NXfxkP9uy9U2AUdXZhuXyzbl8dpyMzNrkWauDmqTdEQeHwG8B3gUWAbMydXmALfl8WXALEkH\nSzqOdAJ4de462iFpWr4qaHZlHjMza4GhTdQZAyzOV/gcBCyNiNsl/RBYKmku8AwwEyAi1kpaCjwC\n7AYujYhX87IuARYBI4DleTAzsxbpMQQi4ifAiXXKnwPOajDPQmBhnfIOYNLec5iZWSv4F8NmZgVz\nCJiZFcwhYGZWMIeAmVnBHAJmZgVzCJiZFcwhYGZWMIeAmVnBHAJmZgVzCJiZFcwhYGZWMIeAmVnB\nHAJmZgVzCJiZFcwhYGZWMIeAmVnBHAJmZgVzCJiZFcwhYGZWMIeAmVnBHAJmZgVzCJiZFcwhYGZW\nMIeAmVnBHAJmZgVzCJiZFcwhYGZWMIeAmVnBHAJmZgVzCJiZFcwhYGZWMIeAmVnBHAJmZgXrMQQk\nHS3pbkmPSFor6fJcPkrSCklP5L9HVuZZIGm9pMcknVspP0nSQ3naNZLUP6tlZmbNaOZIYDfwyYiY\nAEwDLpU0AZgPrIyI8cDK/Jg8bRYwEZgOXCtpSF7WdcDFwPg8TO/DdTEzs17qMQQiYnNE/HsefwlY\nB4wFZgCLc7XFwAV5fAawJCJ2RcRTwHpgqqQxwMiIuC8iAripMo+ZmbVAr84JSGoHTgR+BIyOiM15\n0hZgdB4fC2yozLYxl43N47XlZmbWIk2HgKRDgW8DH4+IHdVpec8++qpRkuZJ6pDU0dnZ2VeLNTOz\nGk2FgKRhpAC4JSK+k4u35i4e8t9tuXwTcHRl9nG5bFMery3fS0RcHxFTImJKW1tbs+tiZma91MzV\nQQL+FlgXEV+sTFoGzMnjc4DbKuWzJB0s6TjSCeDVuetoh6RpeZmzK/OYmVkLDG2izinAhcBDkh7I\nZZ8GrgaWSpoLPAPMBIiItZKWAo+Qriy6NCJezfNdAiwCRgDL82BmZi3SYwhExL1Ao+v5z2owz0Jg\nYZ3yDmBSbxpoZmb9x78YNjMrmEPAzKxgDgEzs4I5BMzMCuYQMDMrmEPAzKxgDgEzs4I5BMzMCuYQ\nMDMrmEPAzKxgDgEzs4I5BMzMCuYQMDMrmEPAzKxgDgEzs4I5BMzMCuYQMDMrmEPAzKxgDgEzs4I5\nBMzMCuYQMDMrmEPAzKxgDgEzs4I5BMzMCuYQMDMrmEPAzKxgDgEzs4I5BMzMCuYQMDMrmEPAzKxg\nDgEzs4I5BMzMCuYQMDMrmEPAzKxgPYaApBslbZP0cKVslKQVkp7If4+sTFsgab2kxySdWyk/SdJD\nedo1ktT3q2NmZr3RzJHAImB6Tdl8YGVEjAdW5sdImgDMAibmea6VNCTPcx1wMTA+D7XLNDOzAdZj\nCETEPcDzNcUzgMV5fDFwQaV8SUTsioingPXAVEljgJERcV9EBHBTZR4zM2uRfT0nMDoiNufxLcDo\nPD4W2FCptzGXjc3jteV1SZonqUNSR2dn5z420czMerLfJ4bznn30QVuqy7w+IqZExJS2tra+XLSZ\nmVXsawhszV085L/bcvkm4OhKvXG5bFMery03M7MW2tcQWAbMyeNzgNsq5bMkHSzpONIJ4NW562iH\npGn5qqDZlXnMzKxFhvZUQdKtwBnAUZI2AlcAVwNLJc0FngFmAkTEWklLgUeA3cClEfFqXtQlpCuN\nRgDL82BmZi3UYwhExIcbTDqrQf2FwMI65R3ApF61zszM+pV/MWxmVjCHgJlZwRwCZmYFcwiYmRXM\nIWBmVjCHgJlZwRwCZmYFcwiYmRXMIWBmVjCHgJlZwRwCZmYFcwiYmRXMIWBmVjCHgJlZwRwCZmYF\ncwiYmRXMIWBmVjCHgJlZwRwCZmYFcwiYmRXMIWBmVjCHgJlZwRwCZmYFcwiYmRXMIWBmVjCHgJlZ\nwRwCZmYFcwiYmRXMIWBmVrChrW6ANa99/h39uvynrz6/X5dvZoOPQ8DqqgZONRyaCSKHidmBwyFg\nr2v0Bd/bI5D+OmJxuJj1PYeAHTAahYvDwWzfFRMCjbo37MDXTNdVo9d8f+Y1+89gwENA0nTgK8AQ\n4IaIuHqg22D/ee1Pl9b+zNvbkDEbLAY0BCQNAb4GvAfYCNwvaVlEPDKQ7TDra/sTMlUOChtoA30k\nMBVYHxFPAkhaAswABjQEenvi0h9MGyj9fRnw/urtlWKN5rXBY6BDYCywofJ4I3BybSVJ84B5+eFO\nSY8NQNuOAn5Wb4L+9wA8e3MatnEQcRv7xqBsY81noVdtbNHnaFBuxxr91cZjm6k0KE8MR8T1wPUD\n+ZySOiJiykA+Z2+5jX3DbewbbmPfaHUbB/q2EZuAoyuPx+UyMzNrgYEOgfuB8ZKOk/QmYBawbIDb\nYGZm2YB2B0XEbkmXAXeSLhG9MSLWDmQbujGg3U/7yG3sG25j33Ab+0ZL26iIaOXzm5lZC/lW0mZm\nBXMImJkVzCGQSfqgpLWSXpM0qC4pkzRd0mOS1kua3+r21CPpRknbJD3c6rbUI+loSXdLeiS/zpe3\nuk21JA2XtFrSg7mNV7W6TY1IGiLpx5Jub3VbGpH0tKSHJD0gqaPV7alH0hGSviXpUUnrJL17oNvg\nEHjDw8D7gXta3ZCqyq02zgMmAB+WNKG1raprETC91Y3oxm7gkxExAZgGXDoIt+Mu4MyIOAGYDEyX\nNK3FbWrkcmBdqxvRhN+KiMmD+LcCXwG+FxHvBE6gBdvUIZBFxLqIGIhfJvfW67faiIhfAl232hhU\nIuIe4PlWt6ORiNgcEf+ex18ifdjGtrZVe4pkZ344LA+D7soNSeOA84EbWt2WA5mkw4HTgb8FiIhf\nRsT2gW6HQ2Dwq3erjUH15XWgkdQOnAj8qLUt2VvuZnkA2AasiIhB10bgy8CngNda3ZAeBPB9SWvy\nrWgGm+OATuAbuWvtBkmHDHQjigoBSd+X9HCdYdDtWVv/kHQo8G3g4xGxo9XtqRURr0bEZNKv6adK\nmtTqNlVJ+h1gW0SsaXVbmnBq3pbnkbr/Tm91g2oMBf4rcF1EnAi8DAz4Ob9Bee+g/hIRZ7e6DfvA\nt9roI5KGkQLgloj4Tqvb052I2C7pbtJ5lsF0sv0U4H2SfhsYDoyUdHNE/F6L27WXiNiU/26T9F1S\n1+pgOue3EdhYOdr7Fi0IgaKOBA5QvtVGH5AkUt/ruoj4YqvbU4+kNklH5PERpP+78WhrW7WniFgQ\nEeMiop30XrxrMAaApEMkHdY1DpzD4ApTImILsEHSO3LRWQzwbfXBIfA6Sb8raSPwbuAOSXe2uk2Q\nbrUBdN1qYx2wdBDdauN1km4Ffgi8Q9JGSXNb3aYapwAXAmfmSwYfyHuzg8kY4G5JPyGF/4qIGLSX\nYA5yo4F7JT0IrAbuiIjvtbhN9fwxcEt+zScDnxvoBvi2EWZmBfORgJlZwRwCZmYFcwiYmRXMIWBm\nVjCHgJlZwRwCZmYFcwiYmRXs/wPUmhfNGg0ykwAAAABJRU5ErkJggg==\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x7f92d1a55690>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"def iter_compose(f, n):\n",
" '''Returns f composed with itself n times.'''\n",
" def g(x):\n",
" for _ in range(n):\n",
" x = f(x)\n",
" return x\n",
" return g\n",
"\n",
"selu10 = iter_compose(selu, 10)\n",
"h10 = selu10(z)\n",
"plt.hist(h10, bins=100)\n",
"plt.title('Iterated SELU sample mean=%.3f, var=%.3f' % (h10.mean(), h10.var()))\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's observe the moment contraction in action."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from tqdm import tqdm_notebook as tqdm"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "fd726539a34848a49aeff2c235a04ab4"
}
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEKCAYAAADpfBXhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXeYFEX6x781YXMgLXFJknMOigQBCeoZTj3kMJ16yk8x\nnHd63J0oZs90ZhEVA0YEAyqKoCAgICxRcoZd0i4sm8Ok+v1RXd3VPd0zPbuzYbA+z7PPzkz3dNd0\n+NZb7/vW24RSColEIpGcXTjqugESiUQiiT5S3CUSieQsRIq7RCKRnIVIcZdIJJKzECnuEolEchYi\nxV0ikUjOQqS4SyQSyVmIFHeJRCI5C5HiLpFIJGchrrracZMmTWi7du3qavcSiUQSk2zYsOEUpTQj\n3Hp1Ju7t2rVDVlZWXe1eIpFIYhJCyGE760m3jEQikZyFSHGXSCSSsxAp7hKJRHIWIsVdIpFIzkKk\nuEskEslZiBR3iUQiOQuR4i6RSCRnIVLcJZI65mhBOZbtyq3rZkjOMqS4SyR1zCUvrcRf3l1f182Q\nnGVIcZdI6pgzZV4AgHxYvSSaSHGXSOoJHn+grpsgOYsIK+6EkDmEkFxCyDaL5VMIIVsJIb8RQlYT\nQvpEv5kSSWxAKUV+qadK3630SXGvLwQCsT+KsmO5vwtgQojlBwGMpJT2AvAogNlRaJdEEpMs2HgU\n/R9dgu3HCiP+bqVXint94bynfsIt78V2HCRsVUhK6QpCSLsQy1cLb9cCyKx+sySS2GT1vlMAgJ3H\ni9GjZXpE3630+WuiSZIIKa7w4kRRBU4UVdR1U6pFtH3uNwP4LsrblEhiBreT3VK+CPznTgcBAHik\nW6ZeUFThq+smRIWoiTsh5AIwcf9niHVuJYRkEUKy8vLyorVriaTe4HIyofZGIu6Efef15ftrpE2S\nyPCeJZ1sVMSdENIbwFsALqOUnrZaj1I6m1I6kFI6MCMj7INEJJKYg1vuHr/9gBy33D/bkIPcGHcF\nnA2IWUuxnJ5abXEnhLQB8DmA6yile6rfJEksQik9KzIMqkuci91SkVju3NoHgOwzZVFvkyQyRPeY\nN4JOur4RNqBKCPkYwCgATQghOQAeAuAGAErpLAAPAmgM4DXChpc+SunAmmqwpHbJLaqAn1K0SE8M\nud4fX1+NbUcLsffxi2qpZfUTN3fLRDC059Y+ABSWe6PeJklkiB2zxx9QO+xYw062zOQwy28BcEvU\nWiSpVwx+4kcAwKGnLg653qYjBbXRnHoPF+qdJ4psfydeEI9563MwumuzqLdLYh/RWvf6AkB8HTam\nGsRmlySR1FN4cHT3iWLb30l0O9XX328/EfU2xTKUUhw6VVqr+zRa7rGKFHeJJIp4lbhDSWXodLpy\njx+5xSx4Gi+Iu0TP3LWHMerZ5diSbW9keOhUabVTSsXvx3J6qhT3OuDnPXl4b/Whum6GjkqfH499\nswOnSyqrtZ1YvhmiAc9vD+c7v2rWagx+nLm8Etzabdi2cVLNNc6EwnIvSsN0RHXJmv0s+c5OoLmg\nzINRzy7HQwtNK6XYRrTWyzx+jH52OX7adbJa26wLpLjXATfMWYeHFm6v62bo2HD4DN5adRD3fLq5\nWtupz0JRG/AhfYU3gAqv9YzT7cc0nzx35QC1X4Kgz8M/4LynfqrVfUYCNxbinOGlitf04R1CVRHd\nModPl+LAqdJ6d7/aQYq7gtcfwF0fb8JXm4/WdVPqBH7ziL5iPpU+HMcLy9XX4dwRZztiMK7IZuaL\nT0ghrYsSBPU5Q4db0W4bGSv82LttdASht6OJe6nHF5Vt1gWx1+Ia4uiZcizccqzalmuswm8icUj6\n57d+Dfu9Cq8fH6w9rL6P9fooHl+gWq4lX0A/pLdDgFKM6pKBW85vX6OVIfecLMbk2WtRbrNd9QF+\nLvw28s35utUWd5+2r5JKdqzsjBzqG7HX4hqC34gxPCGtWnCrh98g6w/l2/renR9vwqvLtGnzFTFe\n2bDvIz9g8BNLq/x9nyBC3OoLBaUUPj+Fy+FAgttZo+I+c+F2rDlwGhsOn1H3bca2o4VYXE+ydrgV\nbee4lCnHu7p56ZWCgcNHX/ExmOseey2uIcq99t0Jvx44jQ2H7YlfXeEPUNz76WZsO2qv9CyfdMPF\n/epZa4K2Z8bPu/U1gkL5mWuLdtO/xX+/32W6bOfxIhRVWLshyjx+FJRV3U0humXsWO5eP4U/QOF0\nMAHxB2hERccigZ9DpdoBSi3ad8nLq3Db3A22t/vJuiMY8OgSNfsnHBVeP175aW9ESQV2RoRl3uhY\n2RXCceEuK+mWiWHKPfZvqEmz1+LK19eEXzEM+/NKqr0NK44VlOPzTUfx1/ezbK3PLSRfgJoKtJWr\nwmjR1LXlzgXMrAiXP0Ax8cWVuOU9e8ekKpRUah2DHXH3BQLwBQJwORyIV7JmKmrIeg8olrpDUfeS\nKFU/nP75bzhd6sHDX++wtf6vB/Px7A978NDC7dh7MvR8AJeDHZN7520JGygtU1wobhexXIdSivWH\n8kPWjBGv/zNKkNbOaGDHsSJsOnIm7Hq1ReyJ+46vgEebArnmlllVKbMxhAYQVX/lmOd+jtq2jPAb\n2Wo4++BX2zDjSy1lTPS1HysoD1rfStyNt0hd+9xD+ctPl7I0z801NJv2VEklFm/XUuZ+sOHa8Pop\nApQVD4t3sXz3mhr98MEXL1QmxgeiQYWNeyO3uALFwsgpXO0Wh6BQ//hsi+k6AcUg4W6wUJb70p25\nuHrWGny07ojlOuXC8T9ZzK4ZO+J+0UsrccVrq8OuV1vEnrjTAOCvRLCsVI8NNnvc2z+0P1ytDtuO\nFuKlH/dW+fvcaqy0EIr31xzGXCUQWunzY+Nh7febWd9Wot2learufbQs9605BVWqyBeqc8ktYjdq\nepI77HYCAYqrXl+NJTvs5zd/u/W47v2Hv1oLCMfr55Y7QVoiqwZiN8smUoxuGStXW01xorACgx//\nEdM+2qR+Fs4Y4B0RAEu3z8yvt6PrjO/VNNxQLpRTyjyOjYetO3hR3A8oo+vUhPDXjB0WbMjBtI82\nRmVb4Yg9cedEMfJZ4fXjjZ8PhF0vO78MP++Jbh369YfyEQhQPPz1duzL1Yaol76yCs8v2VPlG1AV\ndxtD/BlfbsN7a0JnvFhtR8zRBvRWZ2G5t0ojndX7T+HSV37B26sORvzdUJ0Lv7HziiuxNIxoF1V4\nkXX4TESdedNUfRGSpLjwM099fgq/n8LhIGiYFAcAOFNWtWew7j1ZjLxi60lovLPkQV8xBfO15fuC\n1j8ZYfnhH3flYsCjSyyXnzKZIFceZpTiEkx3buV/su4I5q3PVj/nnSiPldixsotDxF0qvH40SYmH\ngwA5Z9goNtnGubTD3z/bgm8MRkBNEYPibu1Pqyp2c7OHP70M0TZ2rp61BvvySvDOL4dw18daGibf\nTySlY0W4yPrCNLjC68eaA3pfptmTaES3TWG5F0OeWIrb5mah2HDsKpSOIRCg6PPwD7hhzjrbbf52\n63E8v2SPamFvyanCc0hDWIJiB3VLmFhEUTn7XZGUfOW+7Et6t8CIzhno1Cw1zDfY+fVTCpeDoFEy\nE/f80qpZ7hf+bwVGP7scALPKjRPK/Iq4c4NBLNH89Pe7gwK519pIhQWAtASt/uDpEA8Hd5Dgezfc\npK2AiRE3/fPfcP+CrcJ22f8CpVM02w+HT3QKsQoqvAEkxTmRnqhZ6+HuIxE7qbS1kXgQg+LOiZ7K\nioElQuwV6Oc3IsAsen7DHCsox8cm/ryvtxzDO7+YW6J/fpPdROI0dE5VCxdxy90Rpi/sOuP7oIvx\njMkNKq6z/WghThYx//LO40Xo07oBnr6qNwDtZj2uWH3rDCmVc9ceNrUSAeCOjzbipR/3qnMNIrH6\nD+SVYPnuXJ2AG89jJB1lqIyaf33+G4YJszrfX3MIe04Wq8fonrGdkOByWPqgD5/WCmF5/QElW0aw\n3EMIJGdeVjbaTf82SCR4Z/vAl7+hx0OLdQLOfz6vf2MUrGMFekv9cL692vLG0ZLVcTYTauP1YcTO\nyJUoBt8ZxXIPFUtYd5DtL9QIr9zjR6LbiRSh04pkBF1QHv78VScjyy6xJ+6hutwqIlrulJoLqlEo\n+E2VnV+G4U8vw4tL2XNKbnxnHf71+W9BN+idH2/Cw1/vMK0WyIeraYnBfr2qTKjZdOQM3l7F3EzO\ncOqO4AvNLH1SFE2ji2ZI+0a4tE9LAJrlbuXrn/HlNjz9/W5stBHjsJueWuH1Y/RzP+PGd9bjQJ4m\nnNz65lgdy3d/OYhz/vWtbnmoWZsfrzuCo0rQuajCiwe/2o4b5qwTpso7kRjnVI+FkZHPLFdf+wIU\nvgCz3LkbJ5yrAgBmKdlAh0+bC/CXm46x5YJAc6FfpLgFjIJ1rFAfSKeUglKKnceLLA0enz8Ajz+A\nVEEIrdxKZq6915fvD2lM2bGYuSQUKOcs1GjroFJhsiDE+S31+JAY51QD3Gyb9u/DQhvCbacDqC6x\nJ+6cMNb11LkbMOLpZbY2VWxwQ1SYpEUae/oKL6vqx9OzVuxlU/X5sM9rYT28udLat28WtLEj7jMX\nbke76d+q7694bTXWHmAWijG4VOH1B5VQNd50bwm+bu5r9OjEXS8+DZPi1JRIfpzMbmTRyrSThhfK\ncvf4AqpYiQFI8bXxBrK6QR9ftBMBCuwVYh7Xva13SZwqqcQLS/foLGFKqZrK53ISHFIs8jiXAwku\np+nQ29gGrz+g+twT3PazZZqkMP/+st25oJQGXSetGrKHqxwRxV25Zz7NYv5qo3AaA7leP8Wk2Wsx\n8cWVulo4IjxtM0OINxwvqMB9n20J8rFbXctG156Izx/A0HMaYVz3ZiDE3O3G3TDcLePzB/DmigPI\nMhkV8O8XWnRA6w7mY+XeU/D6A7qRtM+kw6CU4tFvdmCPIZ0zVMfB2Z9b82WMY1Dcw1uivx44je+3\nn9Bd2KHglvvkwa0BwNTiEmcbnntOYwQoMPjxH1Xfn5pDrFxoVr7E+Rty0DDJjbtGdwxa1lhw9XDs\nWAzvhpgMkhyvfx7LVbNWY5Til7VDj1bpAIzirm9TgyQ3CCGIczlUi93sRhY7UTv1TKyGzpRSdH7g\nO0x8cSUOnipVh+MAdL5YYzutxIWLqhikMxqMd360CS8s3YsdxzWR+27bCaxUOvXuLdLw8k/M3RTn\nciDB7TBtvzHg6fVT1edu7CBDwXPin/puF9YdzA/qCF3KiK1cuG79BoPIbzBAiip8QX537sawOl98\nv2Iw+ZP1R/DZhhxMX7AVE15YgV+UGkWiMLdqoD3Zy0w4tTZSJLiduGZwa1AaPGkO0Cz3IsFyf3zR\nTlw1K3guCr8m+Oho7trD+FWIOb38E8tQO3iqVGe5m40gcosr8faqg7hhzjrdKCiUy6WBkqm14XDN\n58PHoLhzrC+IAxEW9+cTT7g1ZGY58YvqiSt6YXTXpkHL+cnlbhDxQjYOf91OB4Z2aBy0DbczuOMK\nZ7mLvmE+vO0qpCcan2u67ai5Bda+STLe+csg3WeTB7fG3y/sDED/e4zHh7sTElwOnCrxgFKqDn+t\n2lpgIyNkx/GiIDfW5uwCPKDk5+8+WYzb5mZZbuuxb3fqRjTiQ6sdQmyFPyxjp4nLjAvHtmPMVSWO\nvG7/cKP6O1PitVFXnMuBhDinqXvFeD7LPX74AhROhwMupwMuB7F054iIgrg3twRlggvrdEkllEde\notzrx+p9p7A5u0B3LZR5fDDaDYXlXkuXkNVogn+ekZqgflaqTCZaujMXu04U4/ONrBif2Nn2b9sQ\nfx7SRvktAWw7WhhkAQNQXVbtGicD0I9E+LHnBhVPBAiXCQOwDmDjkTOY8eU2TJq9Vl3OO6vkeJfe\ncjcZifP7urTSpzPCQmUsBQIUE3o0x4xLulmuEy1iT9wtfO6fZWVj1DPLUFjmDUrPCwd3EXDL2cxy\n+rsygaJhkts08GkUd3HGq7HGiNvpwHkdmuDgk/rnjZr5CsOlMt7xoZYzy62LBkIet92o/MxLe6Bj\nRorus7+N7azGAfKKK/HgV9tQ7vEHtYlbOEUVPizYmIOvNpsXYBNFOMdkopQZ419YgQUbctT3l7/6\niy5/fM/JEksf7wolbZWLuCisAQpkHT6D77cdV8VBDHRyKGXf5+f3q83HdMtPl7B9i51fnNOB1HgX\nPL5AUMaKMZ5TWumDXxEwgI0iwmWQGDOcss+U6WbDrt5/Wg2kbzh8Bn9+61dc/uovOsu9pNIXJFhF\nIcXdvE18/YwUzXI3zszlowzx+F83tC36ZLJRYVGFF5e8vAp/eWd90PZ9fhZsTopjI9DHvt2pLrtA\nGYHy38pFfbfFrFdKqXrt5pd68EdlwlHrRtoogv+eOTcM0lvuJvcmX7eowodsodNZvjvXdP8Au0db\nN0pUO9+aJPbEnWMYYs7Lysah02XYklOgpqSJlHl82Ha00LTeOPf5NUm1ttw5SfEu0yfncLcMF3fR\n+jL69LmFbjzBZtZBOLeMaNnydcs8fnRplooJPZrbLkTVPC0hKKAb53KoOcNPfb8L7685jK+3HAsS\nH2NnZzUFe9VeJjot0xNsP1kH0DpWK04WhX7ACD8G/PjcN74LAJaGOvWDjaDKKNBqO9fPWWdpMHC/\n8lGhs3I7CXplNgAA/LDjBL7fpuU1Gy33Ug8Td4cq7o6wlrsx/7yo3KuLYezNLVGvww/Wah1hdr7W\nxrJKf9CIsqDMo+Z1G7G6J7ilK/rcjbO9+a3Kz8OK+y7A4PaN1Bz2HcfZNXy0oDyoTb5AAC6nA4kW\neeaBAFXvI24cGTuij9cdQdahfHj91DRU17ohe0DKnR9vwvZjRbhuaFv0ykzXXddmSRbiMbnwfyvU\n18bAtIjXH6i1OjUxKO7mNxk/aWUef1D6H6UU3R9cjEteXqUO6UWe/n43AKBBIrfcQ4h7nFP10Yqo\nU7v5cFiwXozBQ/Hk/ueibhjcvhGapcWr1oF4gYdzy7iEH8uHqaWVPnRomozuLdNYNoZyYRpdNJw5\nNw5E52YpSDX45+NdTnUqN/cjxrsdQUGtRMPxaKsMoTm3vMdcJ/mllUhNcKNbi7So1n0/EWayzUUv\nrgTAbiwHAZqlJeiWixNlmqQExz1W7j1l2km2a5ykBtDFB4QTQlSf8t8+3YKpH2xUM5CMIsFdCfw8\nxlsEYkVOlehHKqWVft0xOFlYETYyVaa4g0TeW3NYtWaNWFn0quWeam25F1V4sftEsbout+RdipGz\nXcjOMj4JjI9qrCaEVfiC73eR4gov/vX5b7hq1hr1ujXWQ+IG39db2KhsUPtGynraPsMlCIhYuWUo\npfD6af0Rd0LIHEJILiHE9NlVhPESIWQfIWQrIaR/9JsZHn7AKrx+/fA7QHXWyAFDsS4xDSvBRuGm\npDinae0KLpzcAtNlhlTqfYDiyf3riHMw77Zz4XI4VMtDDF55/AEUlnnR75Ef1OCWiFe4QS9+aRUA\n1rEkxbnUSRhLd54EpTToBiUEGNutGUZ3bQZCCBwOgnvGdtIdDzHFjX2HoMIb0KVYmnV2Ikt3nsSH\nvx5BpY9lILidDl3NbH+Ahs3tDpUut+u4eRyBw2MwHh+zmoxCIf4WnmtuxCjKCW52voxC1iGDdWxp\nhuPGOzNjZ82DgE7Bcg/nlnnLkHFVWunDcWHk8GlWdtgJYGUen60a6RxjMS0uYPyaEgOqxuDut1uP\nY/wLK7Dp8Bk0So5TY1v8PhAzcYwi6lXcMm6nw/S+23i4QBdQF89tWoJLF/vh2zZmpRmrY7ZqwDp/\n0XIvrvAGXaNmrqpmafE4VeIxNaR4Z2oWW6sJ7HQh7wKYEGL5RACdlL9bAbxe/WbZQX/wuBVQ5vHr\nROyHHSdxh1DLoZfi5+OIFxP365UJVqU/QDHyGS2lMjnOZer25+4LpxDI4hiH+2ZPlXE7ieo2ECPp\nXn8Au04U4UyZNyg9DzAv1lTq8SM5zokeLdMAAFM/2IgvNx8NEqKDT16Mt24YqPvskt4t1deEEKQm\nuHW/t7TShzKPHwnCb+Dizi0iXqBLpLDciwqvH/EuJ9wuh/pb//v9LnT49yL0CzFtHQg9i3j9oTPo\n16aB+n7uzYNN1/P4A4gzEXexMxXjFQPbNrTcZ6emqfD4A0HxlD6tWTtSDOLOLXMu7vNuOxeAJu58\neUq8S21PmceHSW+s0VVODAQovtumL0hW6vGpIwi7lHn8qg/+27vOxxDFWrVCvE/6PboEgx5nNe/5\n9ddYGPFY1bE/dLoUnZulqB0Z/82iG8Mo7v4AhVsZWZm5Rv7+mT62071Fmvq6a/M03bnltX+MBovR\nVcszzETL/UBeqXqNev0BnC6pNE3VbdkgkRkrJnEgfs3XG8udUroCQKhpZJcBeJ8y1gJoQAhpEa0G\nBmHh++S9erlXL+5TP9iArYIVYzyw3Pc7eXAbrXCTEG0vqfTpJookxZtbqfzEqdkyQq/+y75TSE1w\noYsyHT3OpOd2OR2qz13MCPD4Amq6ndnQ0Mz/X+7xIzHOhfZNNPfIwbxSWzM+zazaNMHS+XLTUZwq\nqURjIYDGLZzV00cDgKnQFJR5UOkLIN7lgNtJ2Iik3GtamtcMoytCpMzjQ3PB1WIsZgZowbR4tyMo\nPVQUAHHK+Qe3DDENngNAcrwTJRW+IB8ud6kZXVX8cy7uCW4HEt1O9Vrj102PVunYnM2Kpq3Zfxq/\nHszHE4u0IKKZcHKDxrhPM7j4lSm+foC5pcwm0IlYXTv8XuOGEWDdEZ8sqtRZzfxeFCf9cNfJgg05\n+Cwr29TlNrxTE902RZqkxGPNv0ajU9MUgADXva2Vv+DPQW2Rrl0r5zRJDhb3OC7uwed+yY6T6PnQ\nYgx4bKnpueDuuDyTOjr83Lvqi7jboBWAbOF9jvJZzWK4q/iFUu7xhSw9ahRIngbVr3UD9cYWb3bj\nRZ0c5wq6oRPdTjX9yyygerSgHO0aJ6sWg1nP7XY6hICQ9t0j+WV4ZZn5dH0A6G0YiXh8bMZgcpxT\nVyIBhOCVZeGrTHLL9bK+LYM+A1gt7rziSt0wnIsKt+BPmwhxQRmz3BPcTsS7HPD4Ath+zH7tmFMl\nlZYpbgGqb6MoNJxKH+tM0hLdIQt6id9NcDux+cFxpuulxLtUYesjnAMeNzEGy7mrb76S+eN2sk6m\n0OCWadsoCSWVPpR6/KqLT/SeGIPzALM8K7wBy45IhF8TpZWaz90pTKCygo/GRCEurfSpghXvcuCd\nGweBEOs87xNFFTpDgY+2dW5IZXt//2wL7pvP5izwh3C8cd0AAKyY1+tTzL2/328/gRbpiWienmAZ\nrxrXvZn6ekTnDJR6/Dq3n2q5mxyTv76fpWqIWfCZTx7LFTqd4govvP4AHvyKdS615JWp3YAqIeRW\nQkgWISQrL6+q1RXNjwy/l8q9ftPCVwAL+lj5M8s8PiTHueAg+mnrxsh/otupBoQ4KQkutffnN2lp\npR8Xv7QS89Zn40RhBZqlJai+RrOe2+0kauBT7FCMhZgCAYrCMq+akWIM6vCbMDHOqRMYAmBeVg7C\nkRTnwuYHL8QLk/qqnxkDkNuPFaJpWjzuGt0RY7s1VYNp/GbllvvlQgdRUO4VLHfmlvl+m/1HuV09\naw0GPmb9+DuxjWIFv/6Ku6bc40dBmQcNEt06K9+IUfgT3E5cPSAzaD3R+r/+3Haq2FhlN/Hz9L1S\n4z3O5UBKvFO91riLglvQReVaSq/ov+Xi/uqfNXEr9/jVjjMcvNxxmdevTmJyOUiQP3t8j2a691zI\nbv9Iq5LZ46HF6u+KczlwQdemOreIEX+AqqNjtl+2T/0DwoOP333jWHbTyM4ZAIALuzezrP1zcW/m\nNIhzOkzdg38amKl2ljee1w4ZqfHwB6huv8nxejejFWYZX02S2b0gdsK9Zv6Am9/LwkIlYFtTD2Mx\nEg1xPwqgtfA+U/ksCErpbErpQErpwIyMjGru1pgyxd6XKTexGSnxLstCXE4HCyimJbp1loTop/5D\nn5ZwOAhGdsrQDYGv6NcKxYoVw8U9v7QS248V4f4FW3EgrxQZqXFolsZOvJlbRrTcxX0arWBvIIAp\nb6/FFa+tBqVUF0wCtGGq0fUgVsp75qreeGlyP9PjAAANkuJ0HUM/xY/cUBGGogofmqYm4N5xXfDW\nDYPUdblvlB+/poKI7jlZjIIyL+J5QNVPQ1YQNCNUWicX7IbKbFnOpEHs0izzssfnNUyKQ+OUeNw/\noYvpdswEMlNJlRNFXjy+yfFOtG7E1unRUj+S4hg7Yeb7Fy13duz46DE7v0wVfDFFlo9eRJ++N0BN\n3TIdm2rzFsZ2YxPvkpR1yip96ijD6SBBZXLfuG4g7hqjBddPFFYgEKD4ZZ++gigXMS6ExuvOiN4t\nE37S3vBOTdCmMTu2CW4ntj88HtMndEVTYdJUG+XYA8D9SpprSaVPl/rJGdOtmRrnSEtwIUVpryjU\n3Nceri7TPpMnqfHReaXPj+W7c9Wc9xVCqfCyKGaKhSIa4r4QwPVK1sxQAIWU0porWGzhc+dWr0cZ\nfhtZef8FiHM6gupndFJugEmD2Gy59ES3ahXc/O56XPLyKnVdfiE4HATTJ3YFAFzUq7maYdH5ge/U\nYOibK7X6LB5/AElxLlXszCrMpcS71Bu33OtXb+xck+nqfJapxx8ImnhzopD5KI0WqHjYruyfqRb6\nsgMfcXRvqVllGYba5QA7LoRoASqxnEJBmRc7jhchweWE2+mAxx+o1kMpruyfiYcv7aG+T4xzYs2/\nRmPJvSN16yUqbpZyjw8FZV7Vcr19VEcsnDYsaLspJuLEA48thSnz4noNk+LQs1U6vrnzfEwzKSsB\nAM/+sFu13MRtaD53/XYnzV6rdlJc23edKMIj3+wI2r/XH0CFNxDkRhDTOrkrkI8Myjx+3dwMM4ND\nvIbKPH4s2RlcA5/fa7xzMDt++t+sbdNsBGvswI0lOZLjXXA4CC7o2hTf3Hk+dj06AT/8bYS6nHfO\nv5pklgHQxTnSEt1qZ6S6Z4XAvHFiopEck86Dd7oV3gBufGc9bjSZmGX17NpoYycV8mMAawB0IYTk\nEEJuJoQnaRSRAAAgAElEQVRMJYRMVVZZBOAAgH0A3gRwe421VoSaW+4eXyCocM+kga3RulESdp8s\nxs978vQ+da8fV/RrpV6caQluVXR+3KWfaSbmd/OL2OenugvWCrE+tFkKVePkONWSrfD61ZtwheHh\nIF7h4j9ZWIkyjx8XdNFGQdwlYvQ788lO943vYjrJKxT8gk1LcKsXupm4A8x65wG1Ria1cuLdDsQ5\nCTy+gM4vyeF+VY4xswEAPrh5CJ65qreug0p0O9EiPVHtiBomudEsLV6zVD1+lHl8OvExszLbNErC\n8E5NcOuIc9TP1DRXoYdMFo4vPxY9W6VbCkF2fjnu+niTKpiZDRORHO8UUiEVcRR+74Nfsexj3rnM\nXnFATQ5IdDuxdeY4/HlIG/j8FJU+f5DPPd7lFGZds2s3Jd6FeJcDn2/KUYPULhPLnX1f+6zU4zOd\nyMWFkrt1wom7OLoQ52hwd87sFfoAeyhXU89W6Uhw6+edJChWt8viPCTFOdX5HJkNE4MewnF+Ry1Y\ny11sDS2e3GXmBeCuplDPFbDzbN1oYCdbZjKltAWl1E0pzaSUvk0pnUUpnaUsp5TSOyilHSilvSil\nNff0YQBWPnc+xKw0sdwfubyH7r3otuG515x0g1uGc/WATPz7Iq0eBLeym6UlmAZejCTGOVXBdpgc\n9YbJcWoebbnHr8vaEBF9ul9sYt6vqwe2xlN/7AVAK7dqvGi//Y0Npoz513ZIUCedONTtGp86xHE5\niRrzaGwyISjB5cR+pSyv2TRx8UZNjXeZBjTP79SEPblI6DyMndmv/x6LlfePVsW0tNKvBB217eut\nb3a8uzRPxdybh+jOtWbhiu3U3lh1dGakxLsweXBrEEKQHO9SDRG+7f5ttPTLvbls2M8tSD4q4/tP\nS3AjLcENjz+g1iB//Iqe6jqEaK4xPurq16YBKn0BZOeX45nFu5HZkHWIorh/f89wAPpUwIIyL7bm\nBPuYC8u9iHM61FFGOLeMePxFVxV3k60/pJ/dHOksfR4P47/BbP+3X9ARz/+pD8b3aB7UXrFz5poS\nySP2uL/ezIDjmTR/GdbO9vaqQwzOUOUYK9yx92fKPDiQV6rrbcWLFNAHO3juNSfO5cDGI/oiS12b\np+KZq/uoViEAXNa3Fcb3aIa7x3ay9XCPRLdTFUazJ8U0So5DqcePLdkFKPP6g0T4r8PbA9APW/+n\n1JBvkhKvpj3yjstquna4lDcz+EXudhD1JhZ9niKiVdIoWTte/PfEux0hZ5SKndr6B8baqkcPAC0a\n6NvDSyfw4zD5zbUoNwQdRbfDnBsHYdk/Rql56iLccnY4iBqgFV0K4axVEbENqQludQAqXoOD2unz\n63nnInbsfBt8fkR+mQdJcS5cKGSCEGhpg9ef2w7L/jEKo7roi96dk5EChzJJCGCZP12bpyn70MvD\nSz8FZ20Vlnt1HUO4Uax4Xeo6cuF6F++9UBUjzeCjDR4nAaAWKOP7T3A78cf+mSCEBN0nYiYbL9tt\n53GJnGEd2PH+YUdwsgAhLD7X2cYTuqJB7Im7RVfOTwTPCTdLGXvscmbViHm4PPeZw7NjxFlzxok+\nALO037huIJqkxAdV1zMTpKQ4p1qiwGzIyP32l736C1bsyUO6YaYkD9SZZWM0SYlTbzD+iDZukfDY\nAMfMzREOh2CVcfE1m6ZvROxgeTZLvMupyyt++NIe2Pzgher7RsLvtiPsopvDfLn+9yYaLPd7xnbC\n4ntGoF+bhrp5ASJcbJyEYO7NQ7DivgvU4/2ngZkRFYHiFjagPxeiyCQa2mzMkRd/h9vpAKVsks3A\ndg11HQ0hBPeP74KV91+AZmkJpr+Pt8HMLWPnWaSF5T6DuAcbD7/+e0xQuwHmVuGc0yQFfZWOVTRg\nIn2GMD8XCW4nhnVklVfFshpi8NXYHkB/zZ2nCPX5Ql69EZ4kAbDZ3g4l80gsR8E5VVJp676JFrEn\n7hyLySMcs/rLPCd81s/Mr8cfciBaTTee1w4Ae9CDgwDTLuioswLMMA7L1yiTeUQS41xq8FacBcox\n+vVyzuhr0XOLwqxyZJPUePU3cMudi97UkR1066ZV4Snul/RpgVvOb49/jOuCOTcOxJ2jO9pyRYhC\n01S5CeLdDtwyXPNnd8hIQQNB0Bsma+2zU93zi9uH4ck/9goanXGMVlei0JETQnDP2M6mk55EeH/q\nUFwpbRonqSMRO8ExLiBxLgd8Aaq2SfTbJ4kjCoPglCvxArGkQIIg7pzmaQlIinOpmVC3jjgHLqdD\nzeQxgwsf95mLQm1HV4vKvTrffLLBcv/PRd10cxCsJlrFux1q6qyYfuy1Ke5m7kZuIYsdvHGOifH6\nEA2vkZ0zsPORCbh2SFvL/Ypptfzes0qhrPAGdKP/miYGxd3CcjeI3rnnBNdL576z5bvzUOnTSteK\nw09+I1z39joEaPhcV4ClmfFniAL6FEBOktuJdk2SsfORCbjSJG/aaKWK+fhjuzVVbzqziRmp8S51\n9JFfFhxQ7Su4GsL5RM2IdznxwCXdkZ7kRsemqfj7uC62rFVxX9yNE+9y4irh95+j1GLhnUVKvAsL\npw3DtAs6BgV+P7l1KH6bqffBd2meismD28AK47DbTi64kfZKG3lKHqBZvGYjRCM8HZN3Vnz0I05R\nF8+Xsc0HTpViwUZ9djG/LsV0Qv69S/u0xKGnLsZQk3vAiNFyF0cNfMQiPljDSLBbRn99DTmnkS6H\nPsHCxRHvcmrlP4QO0/jwECuW3jsSX92hz37iLi9XiFlDxpGdcVSdGMfSXPc/cRF6tgrO4b9qoJYF\nzmelhorBSXGvAsaLwOiOAPQX6afrs9WAjmj1JRtOtp2hKSEEfxrYGgPaNtRZKSL8xrPyhbcyuBVe\nUybF9MlMx1s3DFJv4nyTHH5CtKf48NmBokUyorOWTVNbdS0AfcfIBc3YWXLL5+tp5+PDW4aAEILe\nmQ3wj/HBeegt0hMiCm4BJpZ7BP5TzrVD2mDebedifI/m6me8s2pikhFkZMYl3XHuOY3VGa3c5Sa6\nSfRumeA2zjBUM+Udn3g+I/ENc/jx5AIsjhr46HdI+0Z4W3BNup1EzSYqqfTpxJtn+7gcBPsen4je\nmQ10hoAxUYCPHJwOogr/MaEImp1nqALMoDLGS3KLWWznHAt3G2DilrG4P5wOEqQNAHBJrxY49NTF\nuHN0Rzx5BUtqSE+0NqCaRBB8ry6Rm3H1BvOAKsAq85lV94tzOfD0Vb1x//ytePCr7eqFJlruxtox\ndsSdM3/quUGf8eybcKLSo2U6erZKw7ajRRjeqQn6tG6ArTPHacNl5b+xlvcUJVjEO6hTxZVwO4le\nRIUnDvGqhTXFvRd2xvNLWKDX7KY2TubhItU8PQHN061njorbiATjzVsVy50QgsGGwlo9W6Xjpcn9\nMKqL9WS8m89vj8+ysuF0EF2OPP8d1w5tq9Y7EYXZ6JYJRXXFnT+1Kymeu5m0kQjPdmrXJFkXiN3z\n2EScKvFg9gpWnTJO55bRxFoMOn9++3ko9/iDhPbbu4Zjx3HmbuK/+9a52izYqsSIOPde2Bmp8W6M\n6dbMcp2ggGqIOM95HZoE5c/zzuzv4zRjpG3jZDUjzEhGLVrusSfuFu4AMQiTfabc8qIQrXfu+hAt\nd+OwMhJxN3NVtGuchC05hbZuvN6ZDbDtaJEqSKJ/nLtdTgrpcJf3bYnHFWuBLy+u9KFleoKuLZXK\nqObOMR1r/Akwd43phGsGt1bzt3u1SscFXZua1u2JlEitdiD4nFRHLIyEmwg245LumHFJdwD664i7\nZ5wOos5vEF0H4a4VMT6jc8u4I/9t3ZT88pGdM9AgyY2bhrVXl43qnIE5Nw7EiE4ZcDoIOjdLQW5x\npVItVNuXmVvGOEITUzxF2jROUt1dXGj5NTKuezM89Icept+zQ8emqfiv4i594OJuONfk0ZbG+zvU\n7TFtdEcMOacR+rdpiM4PfAfAfCTctrF1jKNlg9AGTDSJPXHnGNIPxUqQTkKCSq5yxNxrPgFDtNyN\nlp5VoC4cGanxyCuuVIfgdir28RvGzMrnvsFjgrg7hYR58WYyBju5n96sHnZN0DQ1QXVbfH3n+QCA\nzzeyujZVEffnru6DLzcftZ0WaeSbO89XZxpXJaAcDT5df0R9PfQcbRTw3k2D8eGvR9SaJEBwtozI\nvNvO1fl+RXGqiuXOA93pie6gOQWEEIzuqlm9X94xTA2yJridiFMKwInXFXddVKUjNsYv3rhuQNSM\nETGIb2T+1HPx3bYTeHvVQd2DW4w4HUSNY8y9eXBQTj6ntUUCRs9WabrkgZomBsXd/GSXeXwY170Z\nfthxEoSwHnXqyA66vF9AP4WczwhNNqSPidgJqJqx+J4RKKnw4Zkf2FOe7Ph6+ZDNLLeX37hfbtIC\na2LwR+yEjIW+KoXKfXUFt5qu7B8cTA7HlQMyTYPQdhFT7qri2okGVw9ojU+zWPFU0V3Rs1U6nlQm\noHH4ue7UNAUjOmfg7VWslMX0iV0xqF1D3TUqilFVxD0S48UYfExLcOFUiUcXQOQdsJgiaBej66s2\nnjMKAAPbNULf1g3QOzM9pAtHZHinDAzvZO6S69SMZcW5HEQXM4hkPkQ0iOGAqnbQAgGKCm9ADUpy\nAZ8+sSsGGB64IFpuvJZ4qINe1WF8o+Q4XcqcWRlaq32ZPUuVdw58hNKrVTruFp6aJGIUC16yIBIX\nU7RpkZ6IQ09drAZ3Z13bHx/eMqTW21FXlrvVuTKDn2uX04HOilB0b5GGqSM7BAmeWGXRKhPFCL/O\nnr26j+02mcFThEXLvVPTFNw5uiNetSjJG4pGyXFY/5+x1WpTVXE5Hbisb6uoCPD5HZtg6b0jcO1Q\nfQplpDn71SX2LHeTzpxbps3SEvDkH3vpivmb8cxVvdVa0UBwhgwhmtenKjM6RRomxYEQe26ZFums\nUxrXvXnQMmPn8NnUc4OCgw9c3A0llT7dgzQArQZGbWbKhGNCz5p7nksorNx1NU0kRoJ4rfBzbCUL\njQV3jt1A7KK7hmPXieKgUW2kdG2eis3ZBboRocNBdMHFSOG1cPpkmlfXjAUIIejYNDWo3vuu48Hl\nNmqS2BN3juBz59ZsotsZMueZM9KQ4WDsrRskutVSutW19CYPaYMuzVNt+YtHdM7Akr+NQCeT6cnB\nsYBgobbyK942ogPWH8wPmnr+e6J9k2T4A7TKfvvqYpZGZ7mukrFFKVXPu1WJiyapmg/X7hN+WjdK\nCjmxyS68wmY0R4QOB8HSe0fqnpYUq5R7WQyhbeMkHD5dZqsGVTSJQXEPvjlFcbeD0c9onFXXIClO\nFffqDtNaNUgMOQnEiJmwA8GTnCLxR3ZvmYbV/xoTfsWzmB/+NiLoCVq1CU/55FPiQ8EDcieLKsLG\nahrVYoDOCO+wzGolVQexDn0sw4uHTRrUGk9/v9tWDapoUn/G6RGjHShe/N6uz9Fo9RpdHu2EVCar\nSUl1AU+rk0SO2+mo05gDAGyacSHevmFQ2PX45KaJvVqorropQ82nwNfW8zjN4AFcsxiRRHuiGu+A\n/bUs7rFnuZtYCbyGuV0rxijuxvf/m9QX8zfk4KZh7SOufV6T3Hx+ezyqPKxBEns0tDGbFWCCvXXm\nOCS5nXA5Hdj92IQqp+TWJDzLzO4s0t8b/DnKPP0xUMvHKYYtdw1eo8Ps4RBmEEJ0WTRGAW+QFIdb\nhp9Tr4Rd8vsiLcGtWuX1UdgBzVUo7xJzuKHOR/+13QfGnuXOEYY4PF/drrgDwN1jOuH6Oeui3qya\nZtU/L4i4xrXk7OajW4aoVmJtwmchny0+8mgz+7oB+GxDjlocLyDdMuEwccsojwsTy8WGoyoTPuoD\n4coPS35/nNcxdOpvTTFpUGucKKzQPZJQotGpWSr+fVE39altMs/dNtqBenkZe0JMJMPXqlQHlEgk\nGqkJbjwgg/xh4XMVhtVyJxx74m4IqJ4orDCtcR4OOzNGJRKJpLq4nQ4svXeErvRJbWAroEoImUAI\n2U0I2UcImW6yPJ0Q8jUhZAshZDsh5C/Rb6oBxX/FhzyRYjcnXiKRSKpLx6aptW5QhhV3QogTwKsA\nJgLoDmAyIcQ4FrsDwA5KaR8AowA8RwipodkVesudVxkc3yOyqdTRLP0qkUgk9Q07lvtgAPsopQco\npR4AnwC4zLAOBZBK2LTJFAD5AMI/f6xaMMudl+39v1EdI/p2rAZUJRKJxA52xL0VgGzhfY7ymcgr\nALoBOAbgNwB3U0prZtoaMbfcIy3lWlvlRCUSiaQuiNYkpvEANgNoCaAvgFcIIUFPkyWE3EoIySKE\nZOXl5VVvj4rPvbCsauIukUgkZzN2HM9HAbQW3mcqn4n8BcBTlFXG2UcIOQigKwDdLCFK6WwAswFg\n4MCBVUz6NFruzPtTFR/62n+NgZyEKpFIzkbsWO7rAXQihLRXgqTXAFhoWOcIgDEAQAhpBqALgAPR\nbGgwiuVe7kVynLNKtcqbpyegaVrslxaVSCQSI2HNXUqpjxAyDcBiAE4Acyil2wkhU5XlswA8CuBd\nQshvYKb1Pymlp2qkxSY+d+mSkUgkEj22fBmU0kUAFhk+myW8PgZgnPF7tUFRhbfaT0uSSCSSs43Y\nrQpJNbeMtNwlEolETwyKu94tU1QuLXeJRCIxEoPizpGWu0QikVgRe+IuBFT9AYr8Uo8Ud4lEIjEQ\ne+LOoRRrD5xGpS+AHi2D5ktJJBLJ75oYFHfNci9QZqf2aJleV42RSCSSekkMijuHotzLHi0my/dK\nJBKJntgTd8HnXqGIe0Jc7P0MiUQiqUliVxUpVcVdWu4SiUSiJwbFXbPcyz2K5S7FXSKRSHTEoLhz\nmM/d5SBVKhomkUgkZzOxp4qCz73c65cuGYlEIjEhph8kWuH1I0E+Lk8iqdd4vV7k5OSgoqKirpsS\nUyQkJCAzMxNud9UmacauuFPmc5eWu0RSv8nJyUFqairatWsnH29pE0opTp8+jZycHLRv375K24g9\ntwykW0YiiSUqKirQuHFjKewRQAhB48aNqzXaiUFx51BUeAPSLSORxABS2COnuscs9sQ9KKAaez9B\nIpHEDhdddBEKCgrquhkRE8M+dzaJqXFyXF23RCKRnIVQSkEpxaJFi8KvXA+JQbNXP4lJTmCSSCSh\nmD59Ol599VX1/cyZM/HYY49hzJgx6N+/P3r16oWvvvoKAHDo0CF06dIF119/PXr27Ins7Gy0a9cO\np06xR0JffvnlGDBgAHr06IHZs2er20xJScF//vMf9OnTB0OHDsXJkycBACdPnsQVV1yBPn36oE+f\nPli9ejUA4IMPPsDgwYPRt29f3HbbbfD7/VH/3bFruSuTmGRAVSKJHR7+ejt2HCuK6ja7t0zDQ3/o\nYbl80qRJuOeee3DHHXcAAObNm4fFixfjrrvuQlpaGk6dOoWhQ4fi0ksvBQDs3bsX7733HoYOHRq0\nrTlz5qBRo0YoLy/HoEGDcOWVV6Jx48YoLS3F0KFD8fjjj+P+++/Hm2++iQceeAB33XUXRo4ciS++\n+AJ+vx8lJSXYuXMnPv30U/zyyy9wu924/fbb8eGHH+L666+P6nGJPXEXYgwV3gDipbhLJJIQ9OvX\nD7m5uTh27Bjy8vLQsGFDNG/eHH/729+wYsUKOBwOHD16VLW227ZtayrsAPDSSy/hiy++AABkZ2dj\n7969aNy4MeLi4nDJJZcAAAYMGIAlS5YAAH766Se8//77AACn04n09HTMnTsXGzZswKBBgwAA5eXl\naNq0adR/ty1xJ4RMAPAiACeAtyilT5msMwrACwDcAE5RSkdGsZ3BUAqPz494Vwx6liSS3ymhLOya\n5Oqrr8b8+fNx4sQJTJo0CR9++CHy8vKwYcMGuN1utGvXTk07TE5ONt3G8uXLsXTpUqxZswZJSUkY\nNWqU+h23261mtzidTvh8Psu2UEpxww034Mknn4zyr9QTVhkJIU4ArwKYCKA7gMmEkO6GdRoAeA3A\npZTSHgCuroG28r2przz+AOKkuEskkjBMmjQJn3zyCebPn4+rr74ahYWFaNq0KdxuN5YtW4bDhw+H\n3UZhYSEaNmyIpKQk7Nq1C2vXrg37nTFjxuD1118HAPj9fhQWFmLMmDGYP38+cnNzAQD5+fm29h8p\ndpRxMIB9lNIDlFIPgE8AXGZY588APqeUHgEASmludJtpjscXQJwsGiaRSMLQo0cPFBcXo1WrVmjR\nogWmTJmCrKws9OrVC++//z66du0adhsTJkyAz+dDt27dMH36dEvXjciLL76IZcuWoVevXhgwYAB2\n7NiB7t2747HHHsO4cePQu3dvXHjhhTh+/Hg0fqYOQikNvQIhVwGYQCm9RXl/HYAhlNJpwjrcHdMD\nQCqAFyml75ts61YAtwJAmzZtBlSptzq6EXjzAvgnfYQO7wH3XtgZd43pFPl2JBJJrbBz505069at\nrpsRk5gdO0LIBkrpwHDfjZbZ6wIwAMDFAMYDmEEI6WxciVI6m1I6kFI6MCMjo2p7UvxavgDrlKRb\nRiKRSIKxE1A9CqC18D5T+UwkB8BpSmkpgFJCyAoAfQDsiUorTfD6AgCc0i0jkUgkJthRxvUAOhFC\n2hNC4gBcA2ChYZ2vAJxPCHERQpIADAGwM7pN5XDLPQBAWu4SiURiRljLnVLqI4RMA7AYLBVyDqV0\nOyFkqrJ8FqV0JyHkewBbAQTA0iW31WTDmeUOablLJBKJCbby3CmliwAsMnw2y/D+GQDPRK9pFkif\nu0QikYQlZpXRH2C1GKS4SyQSSTAxqIyK5e5XLHfplpFIJJIgYlYZfX4ZUJVIJBIrYk8Zuc9dirtE\nIrHJoUOH0LVrV9x4443o3LkzpkyZgqVLl2LYsGHo1KkT1q1bh9LSUtx0000YPHgw+vXrpysDPHz4\ncPTv3x/9+/dXy/YuX74co0aNwlVXXYWuXbtiypQpCDcptDaJvaqQCl6ZCimRxB7fTQdO/BbdbTbv\nBUwMqmUYxL59+/DZZ59hzpw5GDRoED766COsWrUKCxcuxBNPPIHu3btj9OjRmDNnDgoKCjB48GCM\nHTsWTZs2xZIlS5CQkIC9e/di8uTJyMrKAgBs2rQJ27dvR8uWLTFs2DD88ssvOP/886P7+6pIDIq7\n9LlLJJLIad++PXr16gWA1ZoZM2YMCCHo1asXDh06hJycHCxcuBDPPvssAPZg7yNHjqBly5aYNm0a\nNm/eDKfTiT17tLmZgwcPRmZmJgCgb9++OHTokBT36sLdMrLkr0QSQ9iwsGuK+Ph49bXD4VDfOxwO\n+Hw+OJ1OLFiwAF26dNF9b+bMmWjWrBm2bNmCQCCAhIQE022GK/Vb28SsMkqfu0QiiSbjx4/Hyy+/\nrPrNN23aBICV+m3RogUcDgfmzp1bI4/EqwliTxkNk5jc0i0jkUiiwIwZM+D1etG7d2/06NEDM2bM\nAADcfvvteO+999CnTx/s2rXL8mEe9Y2wJX9rioEDB1IelIiIk9uB18/D8t7P4sZ1LZH1wFg0SYkP\n/z2JRFInyJK/Vac+lPytRWThMIlEIglHzCqjKu7SLSORSCRBxJ4yEpkKKZFIJOGIWWX0BwJwOwkc\nDhJ+ZYlEUqfUp5mbsUJ1j1kMirtWfkBa7RJJ/SchIQGnT5+WAh8BlFKcPn1al1MfKTE9iUkGUyWS\n+k9mZiZycnKQl5dX102JKRISEtTZr1Uh9sRdyHOX4i6R1H/cbjfat29f18343RGz6uiXlrtEIpFY\nErPq6AtQOTtVIpFILIhBdWRuGa8MqEokEoklttSREDKBELKbELKPEDI9xHqDCCE+QshV0WuiOYFA\nQFaElEgkEgvCqiMhxAngVQATAXQHMJkQ0t1ivf8C+CHajTTsCACbxCR97hKJRGKOHXUcDGAfpfQA\npdQD4BMAl5msdyeABQByo9g+S3wBGVCVSCQSK+yoYysA2cL7HOUzFUJIKwBXAHg9ek2zglnu/gCV\nPneJRCKxIFrq+AKAf1JKA6FWIoTcSgjJIoRkVXdCg5zEJJFIJNbYmcR0FEBr4X2m8pnIQACfEOYP\nbwLgIkKIj1L6pbgSpXQ2gNkAq+depRYrPndvIIAEt7NKm5BIJJKzHTvivh5AJ0JIezBRvwbAn8UV\nKKXq9DNCyLsAvjEKe7Tx+2S2jEQikVgRVtwppT5CyDQAiwE4AcyhlG4nhExVls+q4Taa4g0EEO+S\nlrtEIpGYYau2DKV0EYBFhs9MRZ1SemP1mxUenz+ANLe03CUSicSM2FNHoXCYtNwlEonEnNgTd4VA\ngEqfu0QikVgQs+pIQBEv3TISiURiSgyqo/ZYPZkKKZFIJObEoLgzCCDdMhKJRGJB7Kkj0Sx3GVCV\nSCQSc2JP3BUIkQFViUQisSIG1VH63CUSiSQcMSjuDAJpuUskEokVsaeOos9dpkJKJBKJKTGtjjKg\nKpFIJObEoLiLPvcYbL5EIpHUAjGrjsznLi13iUQiMSP2xF2X5x57zZdIJJLaIKbVUVruEolEYk7M\nijuBzJaRSCQSK2JQHaVbRiKRSMIRs+rodgBE8L9LJBKJRCP2xF0RdJdTCrtEIpFYEXviruB2SHGX\nSCQSK2JQ3Jmou50x2HSJRCKpJWwpJCFkAiFkNyFkHyFkusnyKYSQrYSQ3wghqwkhfaLfVD0yUUYi\nkUisCSuRhBAngFcBTATQHcBkQkh3w2oHAYyklPYC8CiA2dFuqNAgAIBLWu4SiURiiR2FHAxgH6X0\nAKXUA+ATAJeJK1BKV1NKzyhv1wLIjG4zg5Gl3CUSicQaO+LeCkC28D5H+cyKmwF8Z7aAEHIrISSL\nEJKVl5dnv5X6rQAA4qTlLpFIJJZEVSEJIReAifs/zZZTSmdTSgdSSgdmZGRUa19xMhVSIpFILHHZ\nWOcogNbC+0zlMx2EkN4A3gIwkVJ6OjrNs0b63CUSicQaOwq5HkAnQkh7QkgcgGsALBRXIIS0AfA5\ngOj75mQAACAASURBVOsopXui30zdzgBIt4xEIpGEIqzlTin1EUKmAVgMwAlgDqV0OyFkqrJ8FoAH\nATQG8JpSEsBHKR1Yc80G3NItI5FIJJbYccuAUroIwCLDZ7OE17cAuCW6TbOCW+5S3CUSicSKmPNt\nBCgFIC13iUQiCUXMiXulj4t7zDVdIpFIao2YU8gKnx+AdMtIJBJJKGJQ3AMApOUukUgkoYg5hSz3\ncHGXlrtEIpFYEXPiXumrQ3Hf9CGQk1X7+5VIJJIIsZUKWZ8o9zKfe508rOOr29n/mYW1v++6oiQP\nCPiAtBZ13RKJRBIBMWe5V3gVy91Vy+KupGD+7ni2I/B817puxdlNWT5wbHNdt0LCKcsHZqYDWz6p\n65ZUi9gTdyVbptYDqr6K2t2f5PfD2+OA2SPruhUSTv5B9n9dzT2WojaIOXFPVAq5J7hquOmBALDy\nOaC8gL2vKKrZ/Ul+v5zey/7/XkeH9Q3KDEiQ2H5oRMyJ+7kdmgAAGibF1eyODvwE/PgI8N397H1l\ncc3uTxI5JblA6ama3cf+ZUD2+uhsa9mTwHt/sF7uq4zOfiTVI+Bj/0nMyaOO2G19TVs5fuUElysP\nmPII4l6Yoy2PBvNuAPYuid72wpG7Ezjya2TfqY9W5bOdgGc61Ow+5l4OvD02Otv6+Sng4Arr5b7y\n6OzHisoSYPPH9fNc1icCiuXuiLl8Ex0xKO61FEjlJ5b34n6vtux/PYDF/67adgMB4Lf52gXk8wA7\nvgQ+vKrqbY2U14YCc8ZF9h1vWc20pSqUFwCL7ov8exveBY6sjV47ls5kgbdoiaXRcqcUWP9W9FyC\ni/4BfDlVpvN6yoBd32qGmxF+Hri7LEaJQXHn1LD14Qwh7gCwd3HVtrtuNrDgZi0SX6ncuPXdv+ex\nEPecLOCHB6K7r/wDTDQP/WK+fPlTVQt2fX03MGd89domsup/7H+03Cleg+W+6xvg278DP/83Otsv\nOML+Rys54KNrgNWv2F8//0Dwb6wL5l0PfPJn1jmbwQ2ZkpPMGOP4vcCuRTEz8ok9cSe1ZLlzsfV5\ngDkT2I0mcuZQ1bZ7Yiv7zzuNCiVn3p1Yte2FYuFdwDf32luXUmDfj9qIInud3vUUsHBDvTUGWP1y\nsJvKWwEc2xR5mwHg4Er2f8vH5ss9IeIfR9YC696s2n5DcXg1sOCvwPPdgzt6ozvlo2uAdy8BSk+z\n79iN1xg7iRPb2H+nEF8qOGJPXFY8C2ydp/8s2r7kPd8BP/xHe7/yeeDQquD1PGWApxR4qR+wwKQy\nOP89J3ewJAYAmHsF8MGVVWtXZTFwal/w5/t+ZPdbyQn23uo4iB2QeN3//DTwyWRg/49Va1ctE3vi\nzol27+n3Ad/8TUuD4iODyiLgyBpg7WvB36ksiXw/JSfZ/13fsA6CC5ErIfJtWfHLS+xC3vgekPW2\n9Xpl+cBnN7L/B38GPvgju7mObwXevhB4tLG2rpW4c3zlbDu7lWejL3kQmD0KWHQ/cODnyNrPO3Cr\nc+zzWH93znjmfoiE/APAxvdDr/POROC3eUDR0WA3ybo3NQvP72Oid2glsOJp9p1NH+jXX/YE8KvJ\nyMPYSZTns/8pTdn/I2uBF3ox95LVSIrz06PA53/Vf8bPYbTTegMBYMUzwI8PA+9ebFjmB55oAbw6\nlL3f+4N++dENwMMNWIf+9oUsicHvBfb/BOxbar6/TR+y2IEVc68AXhmg/6wkj13f828G0lqxz9xJ\n5t/3WYh7/n72v0w5L/NvAl427McKSvWjgFogBsXdwnI/vR/Y8VXw55TaC34e2wRkzQE+v5W95yfV\nU2r9ncIcNvnEeNL8Xs0iN8Ktgr0/AHMmAr++zt6L1p23HHhjBHB4jfk2Tu0Ntk53f8dcGWcOAUtm\nsAuZ89t88+38/DSw/Qtmeb9/Gfsse535bw4YrNUfH2GZJBxfJfC/nsDH17CLP3cH+3zdG8D7l5rv\nH2D7f/8yLeUU0CwqqhzX4pPasdi2ACjNM7TNb77tyhKhszbh9H4m1LNHAQvvZKONV4dowW2jhc4p\nOcmsTM6yx4Gtn7LXoiXLryFjYO7n/wLfmcQMzHzuADseJbnAnu/Z+2/uYYIZKbw91YmfZL0DLDe4\nifb/BPz0mPaen8vNHwGPNGKvCxWXEP9N+QfZdcYt/b2LtXvDL3TeG97V7snVrwCzL2Azxb+cKvyu\nALt/NrzH1slRspvE+96jGGKndmv7sXKniZb7k5nA1/cobVeuR359blsAnFZGCMUngJPbzbcHAK8P\nAx5pyEZUtSTyMSjuHINV9/ow5ksryQXWC9bqimeYBbp3KRO/ouPmm+M+dt5r+22I+2tD2OSTJTP0\nn396HfBUGzasFl0Tv80HDgt+5OJj2mt/pebqOb0fOL4F+OoO8/2+MZJZp+JF8stL7L/ZBSZ2eqLQ\nb5rL/q96XvusNA9wuoO3YRTQlc+xTBLOMx0Ar3KsPCVAcoZ52wN+1oF890/2ez+7ETiwnAkWRx0u\nUyYGb48F3pnArLz5NwEHlum36bEYQb1/KfBSX/ba2MFTCrzcH3j3Iq0jPrUHyNsFfDWNvbfyD79+\nLvsT4dvI26X/rQAbFZiNQrZ8om/Xqb3s3KttDGhtfbaT5uMXWf4UcwEBTGByNpi3GdD2VVW/Nz9P\ny58A8nZrn1caDJn/tmVGD+/wdCjH4aW+wEeTNPdnIKCN2ERx//putp3ik6zjPLYxeJPf3a+I8F36\nztUr3LsBIXedj1z8VuIudH7UD2x4h903orgbz+f/egKvn2e+PQDIVe7Lnx4FvrjNer0oEnvibuVz\n56L8yRTg23s1odzwHvu/9CH2/389gMXCBfDrG8CexaxTADRrzY7lztk4V/9+j+KamDWMWYWcBTeH\n3k7xCf2+8/ebr8cvWtGaPrKa/TcLzHKxLsvXt8FMFPMPmMcTXjtXCXKuCh9A9JQFu5m2f8n+r3mV\nDb9/ncWsZXG/KtwtE2A+bh4IfOtC8/1VlgA7FgJPttY++/Bq1hkALBdedDEBWqbEid+0z94Yzv7z\nm7sqIuhO1l5zq+7EVvMsnS9u0wLqALBwGutwts4DCo9CFULjqInjLQeWP8lcQD4P8FwX4K3RQJFg\nNHx+GzNsjqw1v6YDATZ6WfGsubvrl5fYKBLQ5/u/Olh7bXY9zB5p3qEFfJpL6dBKFssA2PmwSl4A\ngOc6B3/G2fCO+eei64rrQ9FRIPtX63YfWqXvYDnHN+sNnM0f6pfzc1RewEYs2eus2/vbPOtlUSR2\nEzmt/LGnFIvi3T8Ag28B4lPY+6Kjyvf8wJpXgBZ9gN5/0iYpcbjVwE+WndzjykLgwz8BFz8LNGgT\nvLz0dLArwYw544Er39Y6GoDdfA4HG3G4E4HEBvq2uuL12/CadEYOFxtFbP/Cxm8pMu+E+PF492Lg\njhAXLsCGyH6DUHx2A9CjkFnHavuFm/j4FhaQ6z1J8LkH9KMbauF+KToGzLtO/5no293/k37Zm6OB\nXleHaH8Ru5EjcV/wNov+7EMrtddWbjqzz42+civ30OaPtNefXqu9FsVp6yfsDwAylBpBxcro1VMK\nPN1Bu8YzBwLnjNLvg49Kc3dqrjYjfKRjxGpEteRB7fXub9n/LcJviST7qLzAOh7kKWXHruiY5vYU\nr0u+H0rZ72vUPjhmwFn9svZ6/l/0y8TR16fXauf9gVwWDK+tJBADtsSdEDIBwIsAnADeopQ+ZVhO\nlOUXASgDcCOl1GT8FA3CHCh+sxQe0V9ExpzWz//KxN2Iz8OsuXABRCN7FwMvWKRHvjJQC46Fwyis\nPz0CjJ3JinelNAP+YSKOosU1/6bgbW791GKIXEW4f96KyiLzDJFtnwNxgmXrcAFNumgd8o8Ps78B\nys1jN2geLhXTKJZHN2hWvRWVRZH7ppc9aZ1JYeUCqLSRw251LX4rZEKJqbnZFrn8fDvcWi7NMwQP\nlc6z6Di7Xpv10Ja9NtS6fVadbomFQROuINdvn4VezgkEQrs4fpzJxF/sZEV8lewa2/Elcw/2v97e\nfo2IBpW4r8eUQPh/TlRtu9UkrLgTQpwAXgVwIYAcAOsJIQsppWI3PhFAJ+VvCIDXlf+1Q5lN4TRi\nFmgtygFmnQ/0vqZ6bRKxK+xmrPofMFgJKJWc1HdSebtYJoZVVkFNUWwRt+B88EfzTIT5fwEcgj/f\n6qZTh9k2xd3KQqwOntLwGTQ6CJuBCjALWfS9A/rRmMjCu8Jv2spyt0IMborwLJ+83azzNVrI3nL2\nu3kV0OqWti61+M2hUlkB1sHb4ZGGoZfv/Dr0cn8lcw3y2FOks7Y5PEPMih8fqdp2q4kdn/tgAPso\npQcopR4AnwAwmm6XAXifMtYCaEAIqeEC4MKNP/eKqm3C6IcVEX2xdc3z3bTX/22nvV7/NvP98fTK\n+oSV1WvlPzbD7mxSswBwdVn+FIsL2EUMWjdsF7w820I4jtso9RvpKNIKPkoozWUJCMaYwqdTWGyF\nEyrl1A71vZKqz6MJO6CNICMlXIDULI26FiZC2RH3VgCyhfc5ymeRrhMdzPxXdm6QSMkNkdZUX9hm\nkeIo0ipEHm5aZvTaUhPwOEk4yk5Hf9+b5oZfR+SwOHnH5Bq1EvcgTL5rp0NMaRZ+HVFsCw6blxku\nOKy9DjdCi3XCjSBqdN82EjWqSa1myxBCbiWEZBFCsvLybAQYQ8F7vmhV7KtNbq/i8K8qdAtRhfCO\nWmxHKKpboIln09QXzPzrdttoNmuSx5FaDbT+nqcUOOeC8Nu3SlE1ozAHiE+3v36scXyL9bLOE2p2\n36Fy4qOEHXE/CkDIMUOm8lmk64BSOptSOpBSOjAjI4KLTIfBsolWxb7aJLEBcFOY2jRxqSG+H8bX\neJ7gxw1lufNMourwlzD+xk7jAJdJaYXzhDTIqSZT1u3irOHSz5yOFmmYZvi9wFSLujhNurD/HcZo\nn6W2DL097vvvapHJAbC4Q+OO4ds27O7w63COrA7OYa9PNOkMXPx8+PWqQndhDoczDhhwIzCiCsXq\nrPjp0ehtywI74r4eQCdCSHtCSByAawAsNKyzEMD1hDEUQCGltGbHdEsfYnnX0eLuLUDrEBkB0eKC\nB9gQ2pjCaCQhxG+LTwv9XTHqb9zOzEKgZX/gjxb1Vy5+Dmg/wnyZWafSNsTEjSvfBqZ8BrQeHLxM\nzAdPbx283Ein8cB1XwZ/Lqa2XfoycMVsoP8N4bcXCRc/DziE+QPthgMXPau0y6S6pq8SaN4TSGgQ\nvCxOCTSLluHFyrbCucnCderhrqnkpsCQ/wOG2yzPIAZmW/a39x0jI+4HRgvZTGkW3tq7t0a+bVc8\n0C3E7Ofq4BKMhn8eBv7wIjD879Xf7h3rWJuvfKv62wpDWHGnlPoATAOwGMBOAPMopdsJIVMJIXwO\n8CIABwDsA/AmgNtrqL01lzPasB1wbhSbbTWsG3mfvd8QyqpudE7o74puDjOBuXWZeRoowITxuq+A\nf+xlVnnLftp2mvfS1rv4Oc3iPu9OINUkft5TKfxETaZbizePVY0PkWY9gA4h3A7jHmOdWp9J7EYc\n97j5euOfBDK6BX/eeqj+9wHst0/LAgbdrP8NxAEM/ivw12XAJMNkFgBooySK/fOQ9llSE+CebVoe\neWOhDr0zHrhtJXBbmBo8cWFGWoQADdpaL5/yGZuJbVUwa/KnTMQvfk4/Ga5BW+Dqd7X3134euh0i\nrnjgPGG00NOiGJidDj5o24lASob5KCndZL6JCB8tWe2XH6Pul2kdsjNM5xmuIFujDkBGF2DSXCC1\neeh1o4AtnzuldBGltDOltAOl9HHls1mU0lnKa0opvUNZ3otSGlsFo6crsWCHRdYF92XyLIhzDZM2\nRj/ALOL79rOhIgBMeFK/zm0rgemC3zVcvZskJZPHGa+5Na75mAmpKMz37WdCPEkoTiVmjySaiLvI\n1F+AKQuAgUp+vMPFJk2lNGVW+WWvss9d8foIf8N2mhiOewz4uyH1D9A6se4mefHisXbYuAzNJra0\nETI7xNEGIcB505iQi1z6MuvA+W8SSc8MvnkH3gw06cReG8UdAFr113dSADtXYx7S2pHSnLlg7t8P\nNGjNRm73bNOLuysOaNEbSG6izxlve35wO0ORfxC4fQ1wiVKmQHRZXfs50LKvvv1GukxgHf+gW4Ck\nRtrnSY2BeMFN2HFM8HeN8M6BOPTXo5W4m14DxPBfgY+w+Uilec/gr94cxu2ZphgjVveHWq5AaJfD\nod3fZvARdYfR2me3LmcjPcC6fnwNEXvlB6rysI4bvtEHMa9doM3WA4AE5aQYre3pR5ho91dmPzbr\nCfz1JyZmDdtr63FfXHIT5ku/dgHQoB3QUYgHJDbQu0jEm9uMpt2Z3+8PL2oimdaS7ZsLTeNObJ8p\nTYG2w5QvEr1whvLdA+zG6DQWuOg54D8ng0cVfBRgHPKHs2JEBt0CTDZMXLFKX5z4tPnnFUoxqkte\nYP/bDgOmCNlC8Sa/0zi5hrurzMorl+cHt6mfMOvTTNzN6H6Zfjt3b2HXA8fpYiIvjlasKoKOms7O\nMScpjFvG4WKTxLj7Rr0moBkLgN7FZIXYmQZ8oWMborHzh5eAq94Bhv6fti/xmmrZF5hxKnTHxe+b\nnkrxO3Ekeu3nmtWrmydhuG7FwHHHscBtK/QjHz7SzBxsPhJRi7YZjpXx3IsxDK4j4vFo2Q+44Wug\n7xRgcohKljVADIp7CIgz2KoGgPbDgaaCmHccC1z0TPB6ovXQsr8mxo0UIU5rxQKUhDCRNyOpEdu+\nw2G4qeOC1xMniVxvCGM4nMCf3gP6TtYuKC4CHS9kfvs/vSesr9wATrdeXMTfNEwozmXE4QDcJiLD\nRc0oQGY3+z+EJ9fc+K32mhCgy0TgijeC29tXEFCAWctm8OEzF8UWffSuK7M4BG97gzbA5a9rn5uJ\n+4T/6i2uNufpRcmOuJ93JzDOEChzJ5i74cQ2WD2opWl37Zw07cFGAOffy4Sp9VAmGv+3Wlufxx/4\n9sTrQGyzHbfgYGFWb8Af3LmLhso5FwB/ep9lZg24gYky94V3NEl4cLq1MsbjHgdmKKmsd28F7t2l\nHesk9rxkNGrPfvstP7FRg1nq612bgOuE8hpON7seh97OYia83AjADLuJ/wVu+oGNsDuOYeefM+pf\nwRUgVf6/vfMPsqquAvjn7C7Luov8WBZQWJSlBAUVxAVRSUJhFNDAZBxEDVKnsslMplFMRm2aGjMj\nKaeaApz8kRbqFFrOqNlMk04mWhqBJkQpqImlaY4lo6c/vt/Lve/uffv2vX27b9/3nc/Mnb3v++57\n+z333Xvu+Z5zvucbGVo+PrIgMUFpzhXub9q6F4Gl34XD+iGml6D6asvkCxqt3ecU4t6nXe2YLI46\nK04rK5QSlrxQxk53T/ekG6C5FU671imBnpDP5RMxca57MEV9TyrTA8rdy35wqgwBxMqyeWTX1MLe\nzDSMLLj0eU+7IyC+YQEmZFhm05bHEz6ioWpknUXUJ/p+0mVwyDQnb3Tujz7HpRVGlmFElj86ukGn\nng3TV8TtaR//scth1CQXaGxpc3X90/IdvcxVRYSuD7aLH3YKZ/LCrn3IR7IPyaDwube56qYALSPj\nuXozL3JKYv51bjsgY8JVFuWxR9dLXR7lftSS/LNYI+atdcbML1Y7yz19Tc1bG5ct/u+bcMyyXPfb\nYSd0f92duc7Vspn92fhhM8LHC9omu3pASdfkhQnrOromj18Vt7V2uC3JkNG57tGF33AB3sglMywR\nwI7iWB9e4EZMUd2etHKP+rr8TjjkWLf/yQddLaPOi9xW7IziPqL6lLuIu5GTU86bhsc3Yzp4l0yV\nSvqlI6sgH+kskyw/YzHR83xuiMXfdBZa+pimhCUaXVDdLejR2Axn3ACTTi9vemCksKee7cq4RhTj\nlsli9JG5N/+lj8eV9I480y1msuArXa3M+gYXlI6Qeud+yXro57O+0pZ7dJPX1cU3bDroOuNCV2zs\nketiCy0iKxuoEEnXSHIEMmVJ7nk5YMXmmU2dPD/7vXKP5Bs8xPl8H/1qrjU5ahLOAlVY9UtX9rhL\n/+pyXYfp3yH5urvMrojF6+LgPDjX0Yl5SlrPv96NotqOcOWFT03VDoomAEVxop5S3xAr9i6kZoxG\nMnUJfHq56+rjUfHhJ+VmjQ2QhbUHRi+KZfUOuCER5Y78sQCjj3J+5pY2eGN3dqYGFL4gy52Vk0/h\nzrwk+5gcN4N0/x0RkTWbb/GKUhg6Fq7c7W7Gd9+AHd59lG8E1TYJ2mfm/76V92cvoDFmalyoatkm\nZwn35De49HFX4zvr2Klnu2Je01bktqeVezL7qL0TVmzuWh0RnItkYZnWMwU32nzx8a5ZOkmiB1N3\naZKXPwPrp8UGyMR5zl0wY6WL9VyQMZN59XZXkykrGBkRWch5FSLOMMlyvaSZWaDcdZJBTTDJp5he\n92bX3zYq1FUoe6gUovM9eREs/X7XkWWhVcKiY45f1Xdpmj2kOpV701CXyvT2y3DMuXDc+bnvfWmP\nswI3Lsj1oybJciv0JT2pfzLrU/FiyMmc5vGz3LCvp30+YBWW6QEVZU40t8aWcr4HzecKzBjuOCV/\nHn1Ew2D3UOkJo4/MjackaZ0I177etT35W8xb68oMJ4kUS1/T0Jj9EEmy8Aa3KHN7N7NTR0yAL+6M\nrfu6usKTlYaOjc/xoBaYurTrMdGkqFkZtVMiJTj+hL4taZv13dPOc3Xss0ourNhcWn8mznMB97lX\nxf93+nnFf0/EWetL/2yZqE7lDi7Vac+T+VOrxs/qfVW7cvDxH7oFD3qSodDS5qzkx9bn+i+X3eoq\n+WVlhOTj9K9BR0btkN7SMNgVBSs0YaZamFvGWYd9wYQ52fGLNENKnfENXPNydvvID8G1b2SnKaaX\nQuxP5l7lXGNZ12CpD+aGRpcqW4goVtcXxerKTPUq9+GHZS+MMdA49tz8E4ayaG6FBamSp4OHQHsP\nF+KNyOfP7C31g2A/A8avaPQx+eYgJBdU6W9EKmdcnLPBLXozKs9ocQARVipkOVh0U98XDapmFq9z\nw+FiRhFGGDQNc2UgIHbLhTKC6yktbS5NtEKrKxWDaD/UFc6is7NTt26t8ETWf+5yw8t0CpURPjvu\nd/ni3fmyjfy8v98Vv5qzuvAsaKOsiMhTqlrwwq3tsXWhWaJGuHRXCtkoTP2g3Ak8xoDD3DKGYRgB\nYsrdMAwjQEy5G4ZhBIgpd8MwjAAx5W4YhhEgptwNwzACxJS7YRhGgJhyNwzDCJCKzVAVkX3A30v8\neBuQUe4vaEzm2sBkrg16I/PhqlqwUlzFlHtvEJGtPZl+GxImc21gMtcG/SGzuWUMwzACxJS7YRhG\ngFSrcv9BpTtQAUzm2sBkrg36XOaq9LkbhmEY3VOtlrthGIbRDVWn3EXkDBF5XkR2isiaSvenHIjI\neBH5tYhsF5E/i8jlvr1VRB4WkRf83xGJz1ztz8HzInJ65XrfO0SkXkT+ICIP+NdByywiw0XkHhF5\nTkR2iMiJNSDzFf663iYid4lIU2gyi8gmEXlNRLYl2oqWUUSOF5E/+fe+LdKLJZ9UtWo2oB7YBUwE\nGoFngCmV7lcZ5DoUmOH3Dwb+AkwBbgTW+PY1wNf9/hQv+2Cgw5+T+krLUaLsq4EfAw/410HLDPwI\nuMTvNwLDQ5YZGAfsBg7yr38KrApNZuAUYAawLdFWtIzA74HZgAAPAgtL7VO1We6zgJ2q+ldVfQ+4\nG1hS4T71GlV9RVWf9vtvAztwN8USnDLA/13q95cAd6vq/1R1N7ATd26qChFpBxYDGxLNwcosIsNw\nSmAjgKq+p6pvErDMngbgIBFpAJqBlwlMZlX9DfCvVHNRMorIocBQVf2dOk1/W+IzRVNtyn0c8FLi\n9R7fFgwiMgE4DngCGKOqr/i3XgXG+P1QzsPNwJXAB4m2kGXuAPYBt3pX1AYRaSFgmVV1L3AT8CLw\nCvBvVX2IgGVOUKyM4/x+ur0kqk25B42IDAHuBb6gqm8l3/NP8mBSm0TkTOA1VX0q3zGhyYyzYGcA\n31PV44B3cMP1A4Qms/czL8E92MYCLSJyQfKY0GTOohIyVpty3wuMT7xu921Vj4gMwin2O1X1Pt/8\nDz9Uw/99zbeHcB5OBj4mIn/DuddOFZE7CFvmPcAeVX3Cv74Hp+xDlnk+sFtV96nqfuA+4CTCljmi\nWBn3+v10e0lUm3J/EjhCRDpEpBFYDmypcJ96jY+IbwR2qOq6xFtbgJV+fyXw80T7chEZLCIdwBG4\nQEzVoKpXq2q7qk7A/Y6PquoFhC3zq8BLIjLZN50GbCdgmXHumNki0uyv89NwMaWQZY4oSkbvwnlL\nRGb7c/WJxGeKp9JR5hKi0otw2SS7gGsq3Z8yyTQHN2R7Fvij3xYBI4FfAS8AjwCtic9c48/B8/Qi\noj4QNuCjxNkyQcsMTAe2+t/6Z8CIGpD5y8BzwDbgdlyWSFAyA3fhYgr7cSO0i0uREej052kXcAt+\nomkpm81QNQzDCJBqc8sYhmEYPcCUu2EYRoCYcjcMwwgQU+6GYRgBYsrdMAwjQEy5G8EjIv+pdB8M\no78x5W4YZcIXxjKMAYEpd6MmEZGzROQJX8DrEREZIyJ1vvb2KH9Mna+rPcpv94rIk3472R9zvYjc\nLiKP4SboGMaAwJS7Uav8FpitroDX3cCVqvoBcAdwvj9mPvCMqu4D1gPfUtWZwDnklimeAsxX1fP6\nrfeGUQAbRhq1SjvwE1/QqRG3oATAJlw9j5uBi4Bbfft8YEpiYZyhvoonwBZVfbdfem0YPcQsd6NW\n+Q5wi6oeA3waaAJQ1Zdw1fxOxS0S8aA/vg5n6U/32zhVjQK17/Rz3w2jIKbcjVplGHE51ZWp9zbg\n3DObVfV93/YQcFl0gIhM7/MeGkYvMOVu1ALNIrInsa0Grgc2i8hTwOup47cAQ4hdMgCfBzpFzT6n\nWwAAAGNJREFU5FkR2Q58pj86bhilYlUhDSOFiHTigqcfqXRfDKNULKBqGAlEZA1wKXHGjGFUJWa5\nG4ZhBIj53A3DMALElLthGEaAmHI3DMMIEFPuhmEYAWLK3TAMI0BMuRuGYQTI/wGsgK4IfrfjUAAA\nAABJRU5ErkJggg==\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x7f92c9e67dd0>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"n_dims = 1000\n",
"input_mean = 1.1\n",
"input_var = .1\n",
"\n",
"x = np.random.randn(n_dims)\n",
"x = (x - x.mean()) / x.std()\n",
"x = np.sqrt(input_var) * x + input_mean\n",
"\n",
"assert np.allclose(x.mean(), input_mean)\n",
"assert np.allclose(x.var(), input_var)\n",
"\n",
"activations = [x]\n",
"for _ in tqdm(range(1000)):\n",
" W = np.random.randn(n_dims, n_dims)\n",
" W -= W.mean(axis=1, keepdims=True)\n",
" W /= np.sqrt(np.sum(W ** 2, axis=1, keepdims=True))\n",
" h = selu(np.dot(W, activations[-1]))\n",
" activations.append(h)\n",
"\n",
"plt.plot([a.var() for a in activations], label='variance')\n",
"plt.plot([a.mean() for a in activations], label='mean')\n",
"plt.xlabel('Layer')\n",
"plt.legend(loc='center right')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Even with initialization far away from the fixed point, the random SNN normalizes the activations over 1000 layers. This is the key property and benefit of SNNs. Re-running this experiment with different activation functions such as as sigmoid, RELU, or ELU gives dramatically different results.\n",
"\n",
"## Dropout for SNNs\n",
"\n",
"The paper introduces a new type of dropout (called alpha dropout) for use with SNNs. Rather than dropping out to 0 value, they drop out to the minimum SELU value $\\alpha' = -\\lambda \\alpha \\approx -1.7580$. To keep the self-normalizing property, the dropout scheme must not alter the mean or variance of the activation. One can solve for an affine function $am(h) + b$ that preserves the moments of $h$ after setting units to $\\alpha'$ with probability $1-q$ through mask function $m$."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYEAAAEICAYAAAC55kg0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xm8HFWd9/HPlwQS9gSSiSxJLoMBBYZFM8CDooyIBlAB\nnzGAigFBBsUBHGYgoA6gwuDo+Mg8ihiRAWQzDlsGRAQ0g4oIiYNIWDQKkWBCwhJI2DTwmz/OuVDp\ndN/b9/Z2+9b3/Xr1q6vqnKo6tXT96pxT3a2IwMzMymmdThfAzMw6x0HAzKzEHATMzErMQcDMrMQc\nBMzMSsxBwMysxBwEOkTSxZK+0OlymFUjaW9JD3W6HNZ6DgI1SHpE0jJJGxamHSNpbgeL1XKSzpR0\nWafLMZRIGiXpIknPSloq6R/6yf9BSYskPSfpOkmbFdKmS7pD0vOV55Kk7SRdL2m5pKck3Sxp+xZt\nVp8i4icR0ZF1N5ukWZIekvSKpCP7yVvzWPd3fCTtlKc9IalrvoDlINC3EcCJ7V6ppJHtXqf16Uxg\nCjAZ+BvgFEnTqmWUtCPwTeAIYALwPHB+IctTwFeBc6vMPgaYA2yf570LuL4pWzAAw/D8+xXwCeCX\ndeQ9k9rHur/j82dgNnB0U0rdLhHhV5UX8Agwk/ShHZOnHQPMLeTZC7gbeCa/79XH8nYjnYQrge8C\nVwFfyGn7AIuBU4GlwHfy9I8BC3MZ5gBbFpYXwAnA74EngC8B6+S0dYDPAIuAZcClwKbFdVXZ1ncC\n04A/kU7mVcCv6thPPbksRwGPAk8DxwF/DdwLrAC+VjHPR4EHct6bgcmFtPPycp4F5gN7F9LOJH3I\nLs37cQEwtQ3nwh+BdxXGPwdcVSPvOcAVhfFt8z7duCLfGudSjWVtlvft5nWU8VTgPyumnQf8ex4+\nKu/zlfmc+btCvrXOv8rzhPRZ+F2e/37gkELakcBPgS/nY/owsH/FdvxH3o9PA9cV0t4D3JPPkzuA\nnVt4HH8KHNnEY131+ACvB6LV52XT9kunCzBUX4UL4zW8drF+9YObT4CnSXd8I4HD8/haH1hgPdIF\n+VPAusDfki60xSCwGvgiMApYH3gH6eL+pjzt/wO3F5YZwI9zOSYBvwGOyWkfJQWPvwQ2ytvwncK6\nqgaBPHwmcNkA9lNPLssFwGjgXcCLwHXAXwBbkQLR23P+g3LZ3pj322eAOwrL+zCweU47mXRRGl0o\n24vAAaRa2r8Ad/ZRtt4gVO11fp3bNzZv34TCtP8L/LpG/uuBUyumrQTeXDGtniBwMLCkznJOJtU6\nNs7jI4AlwJ55/EBSQBLw9pz3TX2cf2ucJ8AHgC1JNxiHAs8BW+S0I0nn88fyej9Oupgqp99IuvEZ\nSzr/e8+F3fK5sUeeb0Y+F0e14njSTxAYxLGuenxwEBgeL14LAjuR7vTHs2YQOAK4q2Ken1c7yYC3\nFT8UedodrBkE/kS+2OVp3wb+tTC+Uf6g9eTxAKYV0j8B3JaHbwM+UUjbPs87svLDXdzWPHwmgwsC\nWxWmPQkcWhi/GjgpD98EHF1IW4d0QZpcY/lPA7sUynZrIW0H4IUWnwcT8/YVj81+wCM18t8GHFcx\n7TFgn4ppfQYBYOs83+EDKOtPgY8Uyvi7PvJeB5zYx/m31nlSMf89wEF5+EhgYSFtg7zPXgdsAbwC\njK2yjG8An6+Y9hA5SLTgWPYXBOo+1n0dH7osCLhPoB8RcR9wA6k6XLQl6e6+aBHpzrfSlsBjkc+Q\nQt6i5RHxYq3lR8Qq0sW1uPxHK5a3ZY2yLSIFgAlVytYsjxeGX6gyvlEengycJ2mFpBWkpi6Rt0vS\nP0p6QNIzOX1TYFxhWUsLw88Do5vZhi3pAkmr8ut0UrMYwCaFbJuS7u6rWVWRt7/81cowHvgh6e72\nynrnA64g1UgBPpjHe5e5v6Q7c4fmClJtqrhfK8+/yjJ9RNI9heO2EzWOS0Q8nwc3Il1Yn4qIp6ss\ndjJwcu8y83In8tp53G51HesGjs+Q5CBQnzNIVd3iBfiPpJO4aBLp7qDSEmArSarIWxQV42ssPz+l\ntHnF8idWLO+PNco2iVTdf5xUjd+gsNwRpFpOrXI026Ok9ugxhdf6EXGHpL2BU4DppDvHMaRamPpa\nYC2SFhQu6JWvC6rNExHHRcRG+XVOvngtAXYpZNuF1B9RzYJiXknbkpoDf1NnmceSLjBzIuLseuYp\n+B6wj6StgUPIQUDSKFJt7Mukpo4xwPdZc7/WPO6SJgPfAj5Jau4cA9xHfcflUWAzSWNqpJ1dcS5s\nUOvCOpjjORD1HOsGj8+Q5CBQh4hYSGrTPKEw+fvAdvlxwJGSDiU1T9xQZRE/J12ET5C0rqT3A7v3\ns9orgaMk7Zo/xOcAv4iIRwp5/knSWEkTSU8xfbcw76ckbSNpozzvdyNiNeliNFrSgZLWJbXJjyos\n83GgR9Kr50Z+bHRuP+Wt1wXAafkpGiRtKukDOW1j0n5aDoyU9M+sfVddt4jYsXBBr3wdN4BFXQp8\nJu/rN5JuCC6ukfdy4L1Kz9lvCHweuCYiVkIKupJGk2pm60ganY8DkjYhdZT/LCIqa55I2qevRw8j\nYjkwl9QJ+3BEPJCT1iMd4+XAakn7k/pu6rUhKUgsz+U4ilQT6FdELCE1AZ6f99+6kt6Wk78FHCdp\nDyUb5vNy4xrLGtTxlLRe3ucC1s37vNa1r+axruP4KK9nvTw+On92hzQHgfp9jvRhACAiniQ92XAy\nqZnmFOA9EfFE5YwR8Sfg/aS206dIHWvX9LWyiLgV+CzpDm4JqVPvsIps15OeoLmH1Pn27Tz9ItIT\nHreTntR4Efj7vNxnSP0HF5JqFc+Rngzp9b38/qSk3kfqJgI/66u89YqIa0kdkFdJepZ0R7l/Tr4Z\n+AEpUC3K5X602nLa7AzSkzGLSBfZf42IH/Qm5jvRvQEiYgHp6ajLSZ2eG5L2d68jSM1j3wD2zsPf\nymmHkJ6qOqriLre31jiR1JfUlytIfVmvNgXlAHQC6cmqp0lNRXPq3fiIuB/4N9LNzOPAXzGw8+EI\nUp/Ug6R9clJe7jzSRfZruVwLSZ+RZvshaT/vBczKw28DkPQhScVaXV/Hur/jMzkvu3d5L5D6OIa0\n3t576zL5jnBKrqW0el33APvmwGcdIulC4HsRcXOny2LDh4NAl2pnEDCz4cvNQWZmJeaagJlZibkm\nYGZWYkP+h6LGjRsXPT09nS6GmVlXmT9//hMRMb6/fEM+CPT09DBv3rxOF8PMrKtIqvxVgqrcHGRm\nVmIOAmZmJeYgYGZWYkO+T2Ao6Jl546vDj5x7YAdLYmbWXK4JmJmVmGsCLeZahJkNZa4JmJmVmGsC\nLVC8+zczG8ocBDqkMlC4qcjMOsFBwDrO/SZmneMg0CRuAhoY7y+zoWHQQSD/r+2lwATS/4/Oiojz\nJG1G+q/bHuARYHr+A2cknQYcDbwMnOB/SBq8brl77pZympVVI08HrQZOjogdgD2B4yXtAMwEbouI\nKcBteZycdhiwIzCN9MfTIxopvJmZNWbQNYGIWEL6A3QiYqWkB4CtgIOAfXK2S0h/1nxqnn5VRLwE\nPCxpIbA76c+ru5KbNKrzfjHrHk3pE5DUA+wG/AKYkAMEwFJScxGkAHFnYbbFeVq15R0LHAswadKk\nZhSxa7k55TXeF2bN13AQkLQRcDVwUkQ8K+nVtIiI/IfoAxIRs4BZAFOnTh1S/3/pu1wzG04a+saw\npHVJAeDyiLgmT35c0hY5fQtgWZ7+GDCxMPvWeZqZmXVII08HCfg28EBEfKWQNAeYAZyb368vTL9C\n0leALYEpwF2DXf9w46aOpN4v0Xl/mTVHI81BbwGOAH4t6Z487XTSxX+2pKOBRcB0gIhYIGk2cD/p\nyaLjI+LlBtbfdcrelFT27Tcbihp5OuingGok71tjnrOBswe7TjMzay7/iqiZWYk5CJiZlZiDgJlZ\nifkH5Gropk7Msj8pU/btN2uEg4A1RTcFTTN7jSKG1Bdy1zJ16tSYN29e29c7HC5qxbviVt8tD8X9\n5VqBlZmk+RExtb987hMwMysxBwEzsxJzn8Aw1kgTjTtbzcrBQaBgKLZrt4Iv8GbWy81BZmYl5iBg\nZlZibg6yYaueZi83jVnZOQiUnC+Cr/G+sDJyELBS8AXerDr3CZiZlZhrAlY6ZXkU2KwepQ8CviCY\nWZmVPgjYaxwQzcrHfQJmZiXmmoBZFbVqRe38eW6zdnBNwMysxFwTsAHx3a/Z8OKagJlZibkmYIPm\np4nMup+DgFkTuJnMupWbg8zMSsxBwMysxNwcZP1y27/Z8OUgYDYADog23Lg5yMysxEpZE/DdnJlZ\n4pqAmVmJNRQEJF0kaZmk+wrTNpN0i6Tf5vexhbTTJC2U9JCkdzeybrNu0DPzxldfZkNRozWBi4Fp\nFdNmArdFxBTgtjyOpB2Aw4Ad8zznSxrR4PrNzKwBDQWBiLgdeKpi8kHAJXn4EuDgwvSrIuKliHgY\nWAjs3sj6zcysMa3oGJ4QEUvy8FJgQh7eCrizkG9xnrYWSccCxwJMmjSpBUU0ax03/Vg3aWnHcEQE\nEIOYb1ZETI2IqePHj29ByczMDFpTE3hc0hYRsUTSFsCyPP0xYGIh39Z5mlkp+EfmbChqRU1gDjAj\nD88Ari9MP0zSKEnbAFOAu1qwfjMzq1NDNQFJVwL7AOMkLQbOAM4FZks6GlgETAeIiAWSZgP3A6uB\n4yPi5UbWb2ZmjWkoCETE4TWS9q2R/2zg7EbWaWZmzeNvDJuZlZiDgJlZiZXyB+TMOs1PCtlQ4ZqA\nmVmJOQiYmZWYg4CZWYk5CJiZlZg7hs2GqL46j92xbM3imoCZWYk5CJiZlZibg8w6rJ6mHf9HgbWK\ng4DZEOKLvbVbaYKAP1w2XLmT2BpRmiBgVjYODlYPBwGzYcQ1XhsoPx1kZlZiDgJmZiXm5iCzknFf\ngRW5JmBmVmIOAmZmJebmILMS8FNDVotrAmZmJeaagFmJDbST2J3Kw4+DgJmtxRf78nAQMDOgsX4D\nB43uNayDgDvDzBrnz9Hw5o5hM7MScxAwMysxBwEzsxIb1n0CZtY6regrcAdz+zkImFlT1bqQ1zPd\n2s9BwMxaptYF3hf+oaPtQUDSNOA8YARwYUSc2+4ymNnQVytQuJmoudoaBCSNAL4O7AcsBu6WNCci\n7m9nOcxseKgMFA4QA9fumsDuwMKI+D2ApKuAgwAHATOrS19NSfV0LA+0Kaqefo1u1u4gsBXwaGF8\nMbBHZSZJxwLHAuOAVZIeak/x2moc8ESnC9EC3q7uMmy3S19sznbpiwOb3mIDOV6T68k0JDuGI2IW\nMEvSvIjo6XR5WiFv29ROl6PZvF3dxdvVXVqxXe3+sthjwMTC+NZ5mpmZdUC7g8DdwBRJ20haDzgM\nmNPmMpiZWdbW5qCIWC3pk8DNpEdEL4qIBX3MMqs9JeuI4bpt3q7u4u3qLk3fLkVEs5c5LEi6GFgc\nEZ9pZt5BlONI4JiIeGuzl23WDJJWATv3PvVn3aX0PyAnaa6kpyWN6nRZuomkfSQt7nQ5hhpJn5K0\nVNKzki7q67yStKuk+ZKez++7FtJ2knSzpCckrXWnls/bFyWtyq+OPUEXERt1YwCQNF3SHXn/z60j\n/wclLZL0nKTrJG1WSPuypN9KWinpQUkfKaRtJ+l6ScslPZWP6/Yt2qwBK3UQkNQD7A0E8L6OFmYQ\nJA3Jp7vKStK7gZnAvqTH8/4SOKtG3vWA64HLgLHAJcD1eTrAn4HZwNF9rPKT+QK8UUS0/aIyDM6/\np4CvAv3+aoGkHYFvAkcAE4DngfMLWZ4D3gtsCswAzpO0V04bQ+r73D7Pexfp2A8NEVHaF/DPwM+A\nrwA3VKRdDHwhD+9D+k7D6aRndB8BPlSR9+vAjcBK4BfAtoX080jfj3gWmA/s3UeZNiedMM+STpbP\nAz8tpAdwPPBb4OE8bS9Sp/sz+X2vQv65wL/kZT1LOvk2K6S/D1gArMh531ixrtdX7hNgQ+AF4BVg\nVX5tWcf+fgT4J+Be0ofm26QPxU15v90KjC3k3xO4I5ftV8A+hbSjgAfyfL8H/q6Q1nu8TgaWAUuA\no9pwPl0BnFMYfwewtEbed5GejFNh2h+AaRX5Xp8+pmvNP5fUTDjQMu4BLAVGFKYdAtybh3cHfp73\n+RLga8B6/Zx/r54nwIHA/+Rz7VHgzMK8PTnvjLytTwCfLqSPIH3GfpeP63xgYk57A3AL6cL9EDC9\nicftGGBuP3nOAa4ojG8L/AnYuEb+OcDJNdI2y/th81afk3Vtf6cL0NGNh4XAJ4A3k+68JhTSLmbN\nILCaFCxGAW8nXcS2L+R9Mn+ARgKXA1cVlvVh0sV9ZL4wLQVG1yjTVaQ7wA2BnfKFojII3JJPpPXz\n+9OkO5SRwOF5fPOcf25exk55mVcDl+W07fJ27AesC5yS98l6hXWtFQQK+2TxAPf3I8CdpAv/VqQL\n9C+B3YDRwI+AM3LerfI+PYBUY90vj4/P6QfmD6Ly8XgeeFPF8fpc3q4DcvrYGuU6n3TRq/a6dwDb\n9yvg0ML45tT4sAOfAm6qmPZfVFw46DsILCddSH9GIUDWUc7fAfsVxr8HzMzDbyYF35Gki/YDwEm1\nzr/K8yTv+7/Kx2xn4HHg4JzWk/N+i3Tu7gK8RL7xIN0g/Jp0x6ycvjnpvH2UFPhH5vPlCWCHPN/M\nPo7fijr2Rz1B4Hrg1IppK4E3V8m7PimATquxrIOBJY1cu5r56ngBOrbh8FbShX9cHn8Q+FQh/WLW\nDgIbFtJnA58t5L2wkHYA8GAf634a2KXK9BG5TG8oTDuHtYPAOwrjRwB3VSzn58CReXgucG4hbQfS\nHcwI4LPA7ELaOqSAsU9hXc0OAsUa1NXANwrjfw9cl4dPBb5TMf/NwIway74OOLFQtheAkYX0ZcCe\nLT6nflf84JMCUAA9VfJ+lsKNQp52OYU75zytVhDYA9iYdFMyI1+Qtq2znF8gPZlHXsZzwOQaeU8C\nrq11/lU7TyrSvgr8vzzck/NuXUi/CzgsDz8EHFRlGYcCP6mY9k3yDUMTjls9QeA24LiKaa9+Viqm\nXwL8gEItr5DW+92ow1t5Lg7kVeY+gRnADyOi9yvYV+RptTwdEc8VxhcBWxbGlxaGnwc26h2R9I+S\nHpD0jKQVpHbDcVXWMZ50p1P8aY1FVfIV07eskmcR6U66Wv5FpIvTuMp5I+KVnLc4b7M9Xhh+ocp4\n736bDHxA0oreFylwbwEgaX9Jd+aOthWkwFvcp09GxOrC+BrHpFGSPlTolL0pT14FbFLItml+X1ll\nEZV5e/NXy7uWiPhFRKyMiJci4hJSbeCAOot/BfD+3Gn9fuCXEbEIXu3EvKG3c5t0E1J5rj5KDZL2\nkPTj3An6DHBclflrfVYmkgJppcnAHhXnwoeA19W1tc1R1/GS9CVSrXt65Kt+IW088EPg/Ii4soVl\nHZBSBgFJ6wPTgbfnk30pqXq+i6Rdasw2VtKGhfFJwB/rWNfepGaW6aTmiDGktntVyb6cVOMofqt6\nUpV8xZPrj6z9GyGTWPOb2JXL+zOpOr3GvJKU8/bO+zywQWHe4odujRO8BR4l1QTGFF4bRsS5+eJ1\nNfBlUhPeGOD7VN+n/ZJ0QeGCXvmq+j2WiLg8XuuU3T9PXkBqwui1C/B4RDxZZRELgJ3zPu+1c54+\nGEGd2x/pV3sXAfsDHyQFhV7fINWKp0TEJqQ2+srl9nXsryC1h0+MiE2BC+otF+mYb1tj+n9XnAsb\nRcTHASSd3sfxW1XnuvuzxrGVtC2wHvCbwrSzSPv0XRHxbHFmSWNJAWBORJzdpDI1RSmDAKlN7mVS\n08iu+fVG4CfAR/qY7yxJ6+UL+3tIban92Zh0YV8OjJT0z6x9RwFARLwMXAOcKWkDSTvQd+0E0sVv\nu/z42khJh+btuqGQ58OSdpC0Aamd/D/zumYDB0raV9K6pP6Kl0idsQD3AB+UNELpfyDeXljm48Dm\nknrvdnsfG21WcLgMeK+kd+f1j87L35r04RtFDpqS9id1tA5KRBxXuKBXvnYcwKIuBY7O+3osqcnn\n4hp555LOwRMkjZJ0Auni+iNIAVnSaNK2krd/VB4ek/fL6HzMPwS8jdQEQc4Tkvbpo6xXACfm+Yrn\n8cakTt1Vkt4AfHwA2987/1MR8aKk3UlBpl4XAp+XNCVv/86SNiedy9tJOkLSuvn115LeCBAR5/Rx\n/GrW/nrPK1Lte528P9etkf1y0vm4d74Z/DxwTUSszMs6LW/rOyuDvqRNSE2ZP4uImQPYH+3R6fao\nTrxIH5Z/qzJ9OqmqOpLqTwd9mnQH/QfgiMJ8r+Yt5s/DI4CLSB+sJaRawSOkk6Va2caTTvq+ng56\nfcU8byU9SfFMfn9rIW0uaz4d9F/kfpCcfgjpp7yfAf4b2LGQNpV0B7QS+A5wZcV2XkTqrF1Balo6\ngnSi19rva2w36UJ/ZmH8GODWwvgeuUxPkS74NwKTctrxpEC0Ipftqsrj1de6W3hu/UMu17PAfwCj\nCmk3AacXxnfLx+sFcgd5Ia0nH+vi65HCOXJ3Pi4rSJ3txY7eiXn9NZ8+IdUIXwFurJj+NlJNYBXp\npuhzdZx/xY7hvyXVMlaSzuOv8dqDCL3bVOyrmUt+yon0WfkM8HCe/25y/wGps/jGfB48SQqWuzZ4\nrI6sso8vLqSvovAkH+ki/wdSH0rlU3ZBuoFaVXidntNm5PTnKtIntfp8rOflbwzXId9RXRYRW3e6\nLAOl9CWYyyLiwjas60LgexFxc6vXZbVJ+jApmJ/W6bLY0NftX/awISQijul0GQwi4rJOl8G6R1n7\nBMzMDP+AnJlZqbkmYGZWYkO+T2DcuHHR09PT6WKYmXWV+fPnPxER4/vLN+SDQE9PD/Pmzet0MczM\nuoqkar82sBY3B5mZlZiDgJlZiTkImJmVWL9BQOkv8pZJuq8wbTNJtyj9ndot+XdSetNOk7RQ0kNK\n/7TUO/3Nkn6d0/694oezhhSdpVdfZmbDWT01gYuBaRXTZgK3RcQU0u9szwTIP3h2GLBjnud8SSPy\nPN8APgZMya/KZZqZWZv1GwQi4nbSD3gVHUT64wTy+8GF6VdF+o3zh0n/UrW7pC2ATSLizkjfTru0\nMI+ZmXXIYPsEJkTEkjy8lPR3gZD+jKT4hxOL87St8nDldDMz66CGO4bznX1Tf3tC0rGS5kmat3z5\n8mYu2szMCgYbBB7PTTzk92V5+mOs+S9Wvf+n+VgerpxeVUTMioipETF1/Ph+v/BmZmaDNNggMIfX\n/vFqBukPFnqnH5b/KWkbUgfwXbnp6FlJe+angj5SmMfMzDqk35+NkHQl6Z+axklaDJwBnAvMlnQ0\n6V+EpgNExAJJs0n/VLUaOD7S3xgCfIL0pNH6pH9YugkzM+uofoNARBxeI2nfGvnPBtb6I+WImAfs\nNKDSmZlZS/kbw2ZmJeYgYGZWYg4CZmYl5iBgZlZiDgJmZiXmIGBmVmIOAmZmJeYgYGZWYg4CZmYl\n5iBgZlZiDgJmZiXmIGBmVmIOAmZmJeYgYGZWYg4CZmYl5iBgZlZiDgJmZiXmIGBmVmIOAmZmJeYg\nYGZWYg4CZmYl5iBgZlZiDgJmZiXmIGBmVmIOAmZmJeYgYGZWYg4CZmYl5iBgZlZiDgJmZiXmIGBm\nVmIOAmZmJeYgYGZWYg4CZmYl5iBgZlZiDgJmZiXmIGBmVmIOAmZmJdZQEJD0iKRfS7pH0rw8bTNJ\nt0j6bX4fW8h/mqSFkh6S9O5GC29mZo1pRk3gbyJi14iYmsdnArdFxBTgtjyOpB2Aw4AdgWnA+ZJG\nNGH9ZmY2SK1oDjoIuCQPXwIcXJh+VUS8FBEPAwuB3VuwfjMzq1OjQSCAWyXNl3RsnjYhIpbk4aXA\nhDy8FfBoYd7FedpaJB0raZ6kecuXL2+wiGZmVsvIBud/a0Q8JukvgFskPVhMjIiQFANdaETMAmYB\nTJ06dcDzm5lZfRqqCUTEY/l9GXAtqXnncUlbAOT3ZTn7Y8DEwuxb52lmZtYhgw4CkjaUtHHvMPAu\n4D5gDjAjZ5sBXJ+H5wCHSRolaRtgCnDXYNdvZmaNa6Q5aAJwraTe5VwRET+QdDcwW9LRwCJgOkBE\nLJA0G7gfWA0cHxEvN1R6MzNryKCDQET8HtilyvQngX1rzHM2cPZg12lmZs3lbwybmZWYg4CZWYk5\nCJiZlZiDgJlZiTkImJmVmIOAmVmJOQiYmZWYg4CZWYk5CJiZlZiDgJlZiTkImJmVmIOAmVmJOQiY\nmZWYg4CZWYk5CJiZlZiDgJlZiTkImJmVmIOAmVmJOQiYmZWYg4CZWYkN+o/mrX46SwDEGVF33nrz\nm5k1wjUBM7MSc01gCCje/fc3faC1g4HUQtqhVk2n1j4o6m8bXIsyGzgHgTaq50LXjGVUuwAOxQvk\nQPdHtYBWTwAdKttrNhQ5CLRIMy747Vh3N14sO7lvzYYbB4Em6vaLU63yd0vNopbBNqt10zaaDZaD\nQIO68cLfjGaY4cQXeyszBwGr23Bqfx9IZ3yt9Fp9E92yD8zAQcDapBtrTEXDvZ/FystBYBC6/YLW\nSmXaN2XaVhu+HATM2sw1BRtKHATMWmiwfQxm7eKfjTAzKzHXBMyGuP5qC65NWCMcBOrkTkBrtXrO\nMTcvWbM5CJgNU/0FDAcJgw4EAUnTgPOAEcCFEXFuu8swEK4B2HDlL74ZtDkISBoBfB3YD1gM3C1p\nTkTc385ymNmaBvsN6iIHjO7U7prA7sDCiPg9gKSrgIMABwGzLtfKWnO7O8TLVBtqdxDYCni0ML4Y\n2KMyk6RjgWOBccAqSQ+1p3htNQ54otOFaAFvV3fpiu3Smf10iK+d3rTt6m/dbTaQ7ZpcT6Yh2TEc\nEbOAWZLmRURPp8vTCnnbpna6HM3m7eou3q7u0ortaveXxR4DJhbGt87TzMysA9odBO4GpkjaRtJ6\nwGHAnDaXwczMsrY2B0XEakmfBG4mPSJ6UUQs6GOWWe0pWUcM123zdnUXb1d3afp2KWJ493ybmVlt\n/gE5M7NjrgurAAACqElEQVQScxAwMyuxIR8EJH1J0oOS7pV0raQxnS5TM0j6gKQFkl6R1PWPskma\nJukhSQslzex0eZpF0kWSlkm6r9NlaSZJEyX9WNL9+Tw8sdNlagZJoyXdJelXebvO6nSZmkXSCEn/\nI+mGZi53yAcB4BZgp4jYGfgNcFqHy9Ms9wHvB27vdEEaVfg5kP2BHYDDJe3Q2VI1zcXAtE4XogVW\nAydHxA7AnsDxw+SYvQS8IyJ2AXYFpknas8NlapYTgQeavdAhHwQi4ocRsTqP3kn6bkHXi4gHImK4\nfBP61Z8DiYg/Ab0/B9L1IuJ24KlOl6PZImJJRPwyD68kXVy26mypGhfJqjy6bn51/dMvkrYGDgQu\nbPayh3wQqPBR4KZOF8LWUu3nQLr+glIWknqA3YBfdLYkzZGbTe4BlgG3RMRw2K6vAqcArzR7wUPi\nZyMk3Qq8rkrSpyPi+pzn06Qq7OXtLFsj6tkus06StBFwNXBSRDzb6fI0Q0S8DOya+w+vlbRTRHRt\nn46k9wDLImK+pH2avfwhEQQi4p19pUs6EngPsG900Rcb+tuuYcQ/B9KFJK1LCgCXR8Q1nS5Ps0XE\nCkk/JvXpdG0QAN4CvE/SAcBoYBNJl0XEh5ux8CHfHJT/hOYU4H0R8Xyny2NV+edAuowkAd8GHoiI\nr3S6PM0iaXzvE4SS1if9d8mDnS1VYyLitIjYOv+Y5mHAj5oVAKALggDwNWBj4BZJ90i6oNMFagZJ\nh0haDPwf4EZJN3e6TIOVO+57fw7kAWB2Pz8H0jUkXQn8HNhe0mJJR3e6TE3yFuAI4B35c3VPvtPs\ndlsAP5Z0L+nm5JaIaOojlcONfzbCzKzEuqEmYGZmLeIgYGZWYg4CZmYl5iBgZlZiDgJmZiXmIGBm\nVmIOAmZmJfa/x3FS8M6YvrUAAAAASUVORK5CYII=\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x7f92d02843d0>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"alpha_p = -scale * alpha\n",
"def alpha_dropout(h, q):\n",
" '''h is activation, q is keep probability'''\n",
" mask = np.random.binomial(1, q, size=h.shape)\n",
" dropped = mask * h + (1 - mask) * alpha_p\n",
" a = 1. / np.sqrt(q + alpha_p ** 2 * q * (1 - q))\n",
" b = -a * (1 - q) * alpha_p\n",
" return a * dropped + b\n",
" \n",
"keep_prob = .9\n",
"x = np.random.randn(10000)\n",
"h = selu(x)\n",
"hdrop = alpha_dropout(h, keep_prob)\n",
"\n",
"ax1 = plt.subplot(2, 1, 1)\n",
"plt.title('No dropout, mean = %.3f, variance = %.3f' % (h.mean(), h.var()))\n",
"plt.hist(h, bins=100)\n",
"plt.setp(ax1.get_xticklabels(), visible=False)\n",
"\n",
"plt.subplot(2, 1, 2, sharex=ax1)\n",
"plt.title('Alpha dropout, mean = %.3f, variance= %.3f' % (hdrop.mean(), hdrop.var()))\n",
"plt.hist(hdrop, bins=100, color='green')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The affine linear combination keeps the activation mean and variance steady."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Convolution for SNNs\n",
"\n",
"To repeat the above experiment and iterate for many layers, but apply a convolution operator rather than a fully connected layer. It's not clear to me if the mean and variance will also tend to 0,1 in this case. [This implementation][snnconv] suggests the following way to initialise weights for a convolutional layer:\n",
"\n",
"[snnconv]: https://github.com/bioinf-jku/SNNs/blob/master/SelfNormalizingNetworks_CNN_CIFAR10.ipynb"
]
},
{
"cell_type": "code",
"execution_count": 112,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def conv_weights(shape):\n",
" # Determine number of input features from shape\n",
" f_in = np.prod(shape[:-1]) if len(shape) == 4 else shape[0]\n",
" sdev = np.sqrt(1. / f_in)\n",
" W = sdev*np.random.randn(*shape)\n",
" W -= W.mean(axis=(1,2,3), keepdims=True)\n",
" W /= np.sqrt(np.sum(W ** 2, axis=(1,2,3), keepdims=True))\n",
" return W"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And using the numpy implementation of convolution from [here](http://wiseodd.github.io/techblog/2016/07/16/convnet-conv-layer/):"
]
},
{
"cell_type": "code",
"execution_count": 92,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def get_im2col_indices(x_shape, field_height, field_width, padding=1, stride=1):\n",
" # First figure out what the size of the output should be\n",
" N, C, H, W = x_shape\n",
" assert (H + 2 * padding - field_height) % stride == 0\n",
" assert (W + 2 * padding - field_height) % stride == 0\n",
" out_height = (H + 2 * padding - field_height) / stride + 1\n",
" out_width = (W + 2 * padding - field_width) / stride + 1\n",
"\n",
" i0 = np.repeat(np.arange(field_height), field_width)\n",
" i0 = np.tile(i0, C)\n",
" i1 = stride * np.repeat(np.arange(out_height), out_width)\n",
" j0 = np.tile(np.arange(field_width), field_height * C)\n",
" j1 = stride * np.tile(np.arange(out_width), out_height)\n",
" i = i0.reshape(-1, 1) + i1.reshape(1, -1)\n",
" j = j0.reshape(-1, 1) + j1.reshape(1, -1)\n",
"\n",
" k = np.repeat(np.arange(C), field_height * field_width).reshape(-1, 1)\n",
"\n",
" return (k, i, j)\n",
"\n",
"\n",
"def im2col_indices(x, field_height, field_width, padding=1, stride=1):\n",
" \"\"\" An implementation of im2col based on some fancy indexing \"\"\"\n",
" # Zero-pad the input\n",
" p = padding\n",
" x_padded = np.pad(x, ((0, 0), (0, 0), (p, p), (p, p)), mode='constant')\n",
"\n",
" k, i, j = get_im2col_indices(x.shape, field_height, field_width, padding,\n",
" stride)\n",
"\n",
" cols = x_padded[:, k, i, j]\n",
" C = x.shape[1]\n",
" cols = cols.transpose(1, 2, 0).reshape(field_height * field_width * C, -1)\n",
" return cols"
]
},
{
"cell_type": "code",
"execution_count": 124,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def conv_forward(X, W, stride=1, padding=1):\n",
" n_filters, d_filter, h_filter, w_filter = W.shape\n",
" n_x, d_x, h_x, w_x = X.shape\n",
" h_out = (h_x - h_filter + 2 * padding) / stride + 1\n",
" w_out = (w_x - w_filter + 2 * padding) / stride + 1\n",
"\n",
" #if not h_out.is_integer() or not w_out.is_integer():\n",
" # raise Exception('Invalid output dimension!')\n",
"\n",
" h_out, w_out = int(h_out), int(w_out)\n",
"\n",
" X_col = im2col_indices(X, h_filter, w_filter, padding=padding, stride=stride)\n",
" W_col = W.reshape(n_filters, -1)\n",
"\n",
" out = np.dot(W_col, X_col)\n",
" out = out.reshape(n_filters, h_out, w_out, n_x)\n",
" out = out.transpose(3, 0, 1, 2)\n",
"\n",
" return out"
]
},
{
"cell_type": "code",
"execution_count": 125,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"ksize, f_in, f_out = 3, 3, 3"
]
},
{
"cell_type": "code",
"execution_count": 126,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "bcb805138c5a4202930dad6f60cc7019"
}
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEKCAYAAADpfBXhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsXXec3MTZfmbL7fmabexztzkMLti4gbFNM2DTayAQCCSE\nkEAoCSGdj0ASQnOAkNB7CYQWerPpxuAG7t3G7Wyf6/ns621Xmu8PaaSRNFppd3W3e8c8v599u1qV\nkTTzzDPv+847hFIKCQkJCYnOhVC2CyAhISEhETwkuUtISEh0Qkhyl5CQkOiEkOQuISEh0QkhyV1C\nQkKiE0KSu4SEhEQnhCR3CQkJiU4ISe4SEhISnRCS3CUkJCQ6ISLZunDPnj1pWVlZti4vISEh0SGx\naNGivZTSUq/9skbuZWVlWLhwYbYuLyEhIdEhQQjZ4mc/aZaRkJCQ6ISQ5C4hISHRCSHJXUJCQqIT\nQpK7hISERCeEJHcJCQmJTghJ7hISEhKdEJLcJSQkJDohJLlLSEhkBcsrqrGioibbxei0yNokJgkJ\nie82znloDgCgfNqZWS5J54RU7hISEhKdEJLcJSQkJDohJLlLSEhIdEJIcpeQkJDohJDkLiEhIdEJ\nIcldQkJCohNCkruEhIREJ4QkdwkJCYlOCEnuEhISEp0QktwlJCQkOiEkuUtISEh0Qkhyl5CQkOiE\n8CR3QshAQshMQshqQsgqQsivBfucQAipIYQs1f/9pW2KKyEhISHhB36yQiYA/I5SupgQUgxgESHk\nE0rpatt+X1FKzwq+iBISEhISqcJTuVNKd1JKF+uf6wCsAdC/rQsmISEhIZE+UrK5E0LKAIwD8LXg\n56MJIcsJITMIISNdjr+KELKQELKwsrIy5cJKSEhISPiDb3InhBQBeAPADZTSWtvPiwEMopSOBvAg\ngLdF56CUPkEpHU8pHV9aWppumSUkJCQkPOCL3AkhUWjE/iKl9E3775TSWkppvf55OoAoIaRnoCWV\nkJCQkPANP9EyBMDTANZQSu9z2aePvh8IIRP081YFWVAJCQkJCf/wEy1zDIAfA1hBCFmqb7sJwCAA\noJQ+BuACANcQQhIAmgBcTCmlbVBeCQkJCQkf8CR3SulsAMRjn4cAPBRUoSQkJCQkMoOcoSohISHR\nCSHJXUJCQqITQpK7hISERCeEJHcJCQmJTghJ7hISEhKdEJLcJSQkJDohJLnnCLbta0TZjR9gxoqd\n2S6KhIREJ4Ak9xzBqh1aup63lmzPckkkJCQ6AyS55xjktF4JCYkgIMk9R0CSzgGWkJCQSA2S3HMM\nMiOPhIREEJDkniOQwl1CQiJISHLPOUjpLiEhkTkkuecIiDS6S0hIBAhJ7jkGaXOXkJAIApLccwRS\nt0tISAQJSe4SEhISnRCS3HMM0iojISERBCS55wikP1VCQiJISHLPMch1xSUkJIKAJHcJCQmJTghJ\n7hISEhKdEJLccwTS5i4hIREkJLnnGKTFvX2hqhTTZqzFjuqmbBdFQiJQSHLPERA5jSkrWLG9Bo/N\n2ohfv7Ik20WRkAgUnuROCBlICJlJCFlNCFlFCPm1YB9CCHmAELKBELKcEHJ42xS380MGy7QvFP2B\nxxX54CU6FyI+9kkA+B2ldDEhpBjAIkLIJ5TS1dw+pwMYov+bCOBR/a+EX0jhLiEhESA8lTuldCel\ndLH+uQ7AGgD9bbudC+B5qmE+gG6EkL6Bl7YzQwrHrICNlKRDW6KzISWbOyGkDMA4AF/bfuoPYBv3\nvQLODkAiCajO7pLj2xvaE5fcLtHZ4JvcCSFFAN4AcAOltDadixFCriKELCSELKysrEznFJ0W0tae\nHcjnLtFZ4YvcCSFRaMT+IqX0TcEu2wEM5L4P0LdZQCl9glI6nlI6vrS0NJ3ydlqoOsnI9APZgVws\nRaKzwU+0DAHwNIA1lNL7XHZ7F8BletTMJAA1lNKdAZaz00OSenYgn7pEZ4Uf5X4MgB8DmEIIWar/\nO4MQcjUh5Gp9n+kANgHYAOBJANe2TXE7B1SV4uf/WYDZ6/ea2yTLZAWGQzW7xZCQCByeoZCU0tnw\nqPtUk53XBVWozo6WhIpP1+zBl+v34tvbT9e3SnbPJqRVRqKzQc5QzQJYZExrQjW2mTb3bJRIQkKi\ns0GSexYgMsFIUs8OmK9Dpn+Q6GyQ5J4FKAJ2V3OY3ZvjSqd1+Bp3JbldopNBknsWICJKRu40x2zv\ne+qaMfyWD/HsnPJsF0VCQiIFSHLPAkTKPVexo7oZAPDOUse0hU6BTjogyXl01pFgLkGSexYg4nZD\nuedYnQ/rYSRKrhUMQEJRMyYJKtMPZAU5WJ06HSS5ZwEiQsrVyh7Sa0gix1Li1rckcMifZ+DhmRsy\nOo9MHJYd5FZt6pyQ5J4FiFRwrlpqIjq755rDt7YpDgD47/ytGZ0nnftqSShoaElkdN3vOqRZpu0h\nyT0LEIdC5mZlD+mKNtf8BJGwVrCEqnrsmRzstlIJhTzv4bkY+dePMrrudx25VZs6JyS5ZwGqgCgZ\nt+cox+fsyIKfCJYO2LtIxSyzemdaSVElOORqPfeLP7y2DGU3fpDtYiSFJPcsQGQKyLUQSAZG6pkq\n5KDBHmGmy+OxdyFt7u2LXK3vfvHaoopsF8ETktyzAPEkJu1vrlV6Rn45xu1GuYIyy0i0Lzq6cu8I\nkOSeBSQLhcw1sI4o12zurDyZKvdcu69cxNOzN2ONNEV1OEhyzwKEZpkc5RhWrlyLcw+qOJnmllle\nUY0d1U3BFCZHcdv7q3H6/V8FGiGUY9WpU0KSexYgJvfcnMTEypprCjeokY4RLZOmzf2ch+bg6Gmf\nB1KWXMeSrdWBnSvXzI+dEZLcswARUeZqVVdylNyDKk+ujUhyGUE6nTviY1dVipXbaxzbchWS3LMA\nUcVmlSTXqgrNUXIPqji5Or8gV8A/n0DJPbhTtRuenr0ZZz04G19vqjK2JXKsXfCQ5J4FZKrct+1r\nxJ7a5uAKlASsqLlH7gEpdyPOXcZCisC/9lCAz6gjdqpsfsN2zseSa+2ChyT3LEBETEYd8VFXjrt7\nJibc+VmwhXJBrg47g7a5p3Vslp5NU6uCGSvc159/Yf4WlN34AVoSSsbX4p9zkN1fbtaq5BB1SLk2\n/4OHJPcsIJlDNdeg5Ogkn6DalDGJKY1js2Wv/9u7q3DNi4uxbJvYwXnfx+sAAPXNmUe38HU1FApQ\nuecuJ6YEqdwlLEi2zF6uRREYWROzWwwHeNKprGtJ/zwpph+obY4bn7PVsLfuawQA19BEVqwgzCjU\nYpbJ+HQGOqIjW2S6kzZ3CQuEuWVyjNQZzOn5uUXvPLkfk0Eooppi5zV3w17jc7bI3agrLoUOckKc\nxSwTYB3wa86Yt7EKL32dWebPtkSumi0BIJLtAnwXkSzlb64Jmlytu3y5WpX0x/ipEuG2faYzLVdV\nGyOcIEiev8Ugu3e/ZrUfPjkfAHDJxEEBXj09iG3uuVkHAKncswJRm1NSNA+0FwyzRZbLYUdQqjlV\nAqysN01A2VJtXkVmxQqiePxzzoZyz3VIm7uEBcLEYQaJ5haNqlk0un+wfCe26fZlO4JyQJs2d383\nGOdGCdlSbeyqbnWFjQyDeEb8OYK0ufPc3p7BBHtqm3HSfbNQsV9cr5Kh09ncCSHPEEL2EEJWuvx+\nAiGkhhCyVP/3l+CL2bkgUotKjnouU7VJB4nrXlqMcx6aLfwtqDaV6v3xHXN5VUMwhUgVHikTjIln\nAZtlgoxz55V7e3D73R+uxc//sxCvL67Ahj31eGH+lpTPIeqElBwegfixuT8H4CEAzyfZ5ytK6VmB\nlOg7AGGce44qgCCdc698sxX9unXB5KGlnvuyhrS/MS78PVtmGV6p/eiprwMpQ6rwWtQ7SLNMW2Ur\n5c/bHjX/kS82AgAOP7BbWsfXNMbR2KrNG+D7uFxW7p7kTin9khBS1vZF+e5A1NkHOZQOEqmaLZLh\nxjdXAADKp53pua8XeQdmlkkxjl/hUgy3ZLgKVKZweyfs2QUhGCwknObp4vE4Kioq0NxszqqOKyqe\nPKcvAGDd2jWu98L2WbNmTXoXt52na5dGPHlOXxTnx1M6Z8X+Jlx4SAgXHtIXB4T2G+drqdyGNfvb\nxrqdn5+PAQMGIBqNpnV8UNEyRxNClgPYDuD3lNJVAZ23U0I0XGam3Bzj9oyzJqYLL0UUVJx0qvzH\nlysvEsp4mb9M4PVOglDd/CnSPV9FRQWKi4tRVlZmkHhTawLYUw8AOLR/V1dyj1doE7WG9isBQBBO\n0/DPztOnaz521TSjtDiGvl27pHw8AAzsXoCobrM/pFcRCvKCDzqklKKqqgoVFRU46KCD0jpHEF3O\nYgCDKKWjATwI4G23HQkhVxFCFhJCFlZWVgZw6Y6JZLa7XFu0I5MZnJnAS7kHNRo2r+PvDnkba0Fe\nOJhCpAi/VSRos0y6p2tubkaPHj0sBM6fy895V+2oxdocXDCkrZorIQQ9evSwjHZSRcbkTimtpZTW\n65+nA4gSQnq67PsEpXQ8pXR8aam33bWzQhFECrBtuWbCy1Zn46XcgyrX52v3APA/MuHLVVoUC6QM\nqcJvdxR0nHu6pjBKKXbXtljNRGmcKmdmtbaT0snUFJoxuRNC+hC9FISQCfo5q5If9d0G3+hM51eO\n2tyzNEPVK+4/CHtyU6uCRVv2p3SMolIc2KMAADD10N4ZlyEdGAu7eOwXiM1dzVy517co2FPXbJkj\nwJ+rrlnsNM8VnHHGGaitqRH+lmPN1QI/oZAvA5gHYBghpIIQ8jNCyNWEkKv1XS4AsJIQsgzAAwAu\nprnGUDkGntzta5TmnHLXRxQijm2OK23WGbFQObfwuyCeE5810W/XlVApCvMiiEVCWeuI2VW9yLu8\nKvVYbjuCcKiKnhO/ZUsA5WwLUEqhqiqmT5+Okq5dxfuk0eXtqG5CbVPbd2ie5E4p/SGltC+lNEop\nHUApfZpS+hil9DH994copSMppWMopZMopXPbvNQdHFblbo1J9hpKt3fIpFs0SUJRMfyWD3HTWyuS\nHr9oyz6U3fiBYwUbL7DOzs1/FoTJgY928Xs2RaWIhAlChGRvEpPPUMcrn1+Y8bUCMcvofy2vsh07\nxhtvvBGvPPek8f3R+6bh3/dMw9SpU3H44Ydj1KhReOeddwAA5eXlGDZsGC677DIcdthh2LZtG8rK\nyrB/n2aMuOFnl+LEYybh/KlH4fUXnzPOWVRUhD//+c8YM2YMJk2ahN27dwMAdu/ejfPOOw9jxozB\nmDFjMHfuXFQ1tOL5F17AhAkTMHbsWPziF7+AomSentkOmVsmC7Da3LW/qk/lbrc7/t+by5EXDuHW\ncw8LsogG3NogI8aXv9mGu84f7Xr87PVao/ggSf5xERJKcnNQEJ1cc9xsUH7PF1dUhENa1Ea2p563\nhz8k0Hh0Atz63iqs3lELRaWW518YE1ORPfOlaL8R/Urw17NHul72oosuwpXX/BIXX34lAODj99/G\nq2+9hz//8XcoKSnB3r17MWnSJJxzzjkAgPXr1+M///kPJk2a5DjXrfc+hJGD+2Pjzn344Zkn4opL\nL0bxgD5oaGjApEmTcMcdd+CPf/wjnnzySdx88824/vrrcfzxx+Ott96Coiior6/HF4vX4t0338Cc\nOXMQjUZx7bXX4sUXX8Rll13m/QxTgCT3LMBilrEtY+eljuyE8vI32wCgzcjd7EysJJtQ/DX1Pl01\np+P2/U0ee1qR8FTuKZ1OCF65+3XWKSpFJERASPbyoxhmmXYgdxqIWUb7m63UGuPGjcO+qr3Ys2sn\nqrbWoaRrV/Tq3Rs33XQTvvzyS4RCIWzfvt1Q2wceeKCQ2AHgpWcfx1efTEdcUbF753Zs2LgBZQP6\nIC8vD2eddZZ+vcPx/oyPoFKKzz//HM8/r83/DIfDKCkpwfw5s7B82WIceeSRAICmpib06tUr8PuW\n5J4FUJFZxmcmv/ZWi25mGb+ZGIti2gSMNSmGsbGQw7DbRB3bc4orKqLh1OIDeOXo97kmVIpwSDPL\nZJCMMjMYJry2v1QQZhnWHRECQ2HXNsUt6RtGDxDPHF1eYV2QxG0/L5x85rn4ZPq7aK6twilnn483\n//cKKisrsWjRIkSjUZSVlRlhh4WFhcJzLJg3G/Nnf4EZn89CTWsIP73gTDQ3aaIlGo0ao8yGuIr6\nphbsb2gVnodS4Ac//BEeuO+etO7FL2TisCzAYpbRPys+G2x7h4O55V7xq1pZedfrE1b8wlTuYnK3\nE01TPHWbZXPcvIe4T6bWlHsIIZK9vCLtqdyDMMuIxn7tbdA69ezz8NG7b+D9t9/CKWeei7raWvTq\n1QvRaBQzZ87Eli3euWbqa2tR0rUbCroUYNOGb7F8idinwad/mDp1Kh599FEAgKIoqK6uwcRjJuP9\nd97Cnj1aGO6+fft8XT9VSHLPApKZZXLOoepyvXjCrxnDSYB+7sG0ubuUy07uremQe/rKPRzKjkN1\nzc5aLK/QnNPtEa3Dv760OxMmEPiX2c4i5ZBhh6Khvh59+vVDae8+OO8HF2HhwoUYNWoUnn/+eQwf\nPtzzHMecMBVKIoGjjhiDf9/1N4weN96zk7r//vsxc+ZMjBo1CkcccQRWrV6Ng4cOx//d8leccsop\nGD16NE4++WTs3JmaT8oPpFlGgKr6Foy/41M8/ZPxmDI8+FhmUbSMGeee/NhcMcvEfapWkW2+VVGR\nH0o+u9OIlnExutuFdqbkHvfpQ1BUVbe5Z8ehevr9Xxmf22Lg8NGqXXhv2Q48dMnh2jX4CpkZt3tu\n83WuDFaGeuPTuUb6gR49emLevHnC/VautCbALS8vN8xDj7zwOvp164LdNc1QKMWA7tqch/p6c2R6\n1rnnYdLUM0AI0Lt3byMSB9Dqz6odtfje93+AKy//cUrlTxWdQrlvrKzHA5+tD0zJzFi5C5QCby7e\nHsj57OCVa+7b3LW/dmdYKmYMO/wk3PIyyyRs10/HLMMrb9/KXWE29+xnBGwLE90vXliE95ebKjJI\ns0xQxpjKuhas2F6TUVuobUpgn4tN3As7qpuS3lOyUhnO5XbwLXcKcr/0ya9x3yffotolPWyqaGzV\nwq/6lOQHcr4F5fssi07wIpGpLzP9gAe5t/Nwlropd59mGREB8pOH3OAV5x63nbcxDeXOP+uUbO5h\ngjAhlgyR6YBSmlHisaDNMtbIGKcPKH1/qnNUmi4vUwBVOimn5fPQr9uSUNJasIMh6Ug7yYMSx561\nDToFuTfrZBFUVWeEFA4H8woufGwejrt7pvFdFC1j/PWor35DEIOCkQbAtp03y+yodg9zFCr3uB/l\nru3jNvS2K/fmNJQ7X7TUbO4hkAAmMb38zTYMvXkGtid5fskQ9MBh7kYza4hoJJnuIu7U9hdARsTK\nqkQQfVumTvFkRcj2ujsdntz31DYHptgZmCKLhtrm8SgCswwjCi81lsli0OnATPlrM8twivPoaZ+D\nUoqEouKu6Wuwl8shIlbu3vfgpdztnVw6RMs/a7/HtyZU5IVDCIUyJ4Z3l2lmvy1701vRKehoGb6D\nFNXHjDuTIIpLTRNhOqezH5NxcxIUQpplAsJR0z4P/JxsiB7EmpHC9VIFZhm/M1Q/W7Pb+Mw3vDMf\n+ArPztmcfkFd4EYgvMoDtPv8bO0ePP7lJoy//VPM1LMt2hU24M8s42Vztzt00yFa/tb8hna2Kiry\nIgGnH0izngWt3PlnzdpApnHulFIj5DSo4rJiphM5Zje/pTsaMY9P94i2Z/cOT+48eQZlg2TKMgj7\ntsjRJ8wt48Oh2hxXcOf0tcb35+aWG59X7ajFre+tzrS4Drg90/s/W2/5nlCpZar4vR+vM7bb4cuh\nqng5VLXfjxrcA4D/aBce/LP2Y+5qalVQWdeCvHBIs7l7kMv8TVV48stNrr9nWr0CD4Xkl49TmJkw\nM3fosgozpxAj0kzKTWGKrnRGLnYnasbvINlTEVRdqdzTRFDORkY+QSizRltuDMAlWsbHJCb77c36\nNvUFT1SVYv4m/xmZmdDxqoxarhCTtOuaE8Z2O/jRh/v5mM1d/DsbEfz1nBGu1/GCVbl7H/+715YC\n0DpsQryvefET83HHdPel3EwNl15LDzpyii+FSLmnw+7WZGHW3/Kj6S12wp5XELfv5xRJOyOhQ9X9\nN+lQTRNBCRlme8w0GgIAGgRRHIpAuTOySlaR7J1XOisBvTB/Cy5+Yj4+XrXL1/5+1VFCpZZRCrsf\nEWk+PHOjr/MBcF1WLa7neInofhH+OndOX+Nr8WqrcvceTSzZqsU6tybUYMwyGaq4oM0yvF+FRSNZ\nImjSYHeewO2cl959U9MsE0SDz/AcKXK7AUnuKSIoJROkcrdntQNsNnf9c72+X7IKa7+/WCR1ct9Y\nqU22SBbhwsNYGMJWrMlDrStp2bP8MXJws4W7xRg/M3szyvc2cA5V90WgI2GCiE7+7DpLtu7HE19u\nwuwNe5PdFoDUlTvraAgJZoYqI8t0G3rQDtWQxSyjPU9+gY10/Md8nW1oSWDDnnpz0XWPOxfZ1NMd\nSGzfthXnnjABt/zmWpw9eTz+71dXYv5XX2DK8ZMxZMgQfPPNN2hoaMAVV1yBCRMmYNy4cZY0wJef\nfzouOv14XHT68Vi6UBMOC+bNxnlnnIILLrgAw4cPx6WXXmoVZyLl3o52mU41QzU4clcCO58oflpk\nlmFmjOSeduuveSkmytLOof31O7vP7REQAIf1L8HK7VpCsISqosWncgeA3bXNOKAwz7KtsTWBv7+/\nGk98uQl/OHWYXk7x9eOKimgoZBAus7mf94i5nICipwpwA69E/djc2bOjlIIQ4tuhp6pUONOWnc9t\nFq53eYI2y1gdqo/P2oi7Zpg+nnSuxndA9S0J9J13K0jdOgxuVaxpkwWpfFVVxWBb2Gw4FkbfuApF\npYhFQ0AoBPQZBZw+zbMs28o34d5Hn8Wtwx7CJWdNwfS3X8dj//sAa7+eiTvvvBMjRozAlClT8Mwz\nz6C6uhoTJkzASSedhNJevfD4S28hlp+PLZs34sbrfo6Xp88EAKxcsQz/e2UV+vXrh2OOOQZz5szB\ngSMOd31e7WmW6VTkHlRdbw1QufMdRE1jHF0Loo6VmDbsqcfOGi0jXTLCsHc20UjqVcQtnYDrNQ3l\nrv2tbmzFmQ/MRl1zHEN6F1vK1qo4ydLNtCUiU0bQDS0JT+WeUHTlHmbK3Xm+qoYW9Cp2n4iWTpw7\noE1CS2WGqle6hfSVe5oH+sDGygYLsQPpdSbJRhftnRO//8ADMeRQLSvlwUOHY+Kxk0EIwYDBw1Be\nXo6Kigq8++67uPfeewFoC3tv3boVffr0xa1/+jXWrVqBcDiMLZtMs+LYw8djwIAB2uexY1FeXo5B\nBrln1+jeqcg9KIdqwsOkkM65AGDM3z/GvP+b4pj1d98n64zvtc0JrN1Vi+F9Shznst+ffQatH8JO\ntW6Zk6y073M2VBmTbiKc4kwo1FKZWaiiGwGK4vX5fDJe+dwTqopIOCS0uTM0tihAsWOzAcsMVR/v\n2pi1qY8I4i4Tpz5YvhPXvbTY+N6qqELnYaa1NWizDF+/RJ1vOlezk/fOo/4KpTiGPXUt6Ne1C+Kq\nir31rRjV37mMXaMtLTAAjOhbgh1VjWhoTWBA9y44oNB7kXLWjqN55kgxRELIy9OODYVDSCQSCIfD\neOONNzBs2DDL8X/961/Ro2cvvPbxbKiqigmH9DF+y+POGQ6HkUiYZljR65EO1TQRVGVnFTto5Q4A\nVfWtjlDIPiVdLPuc9u+vIIKdf7rkWftmN5XLI1WzjGGK0KtlK7ccGJ8/PaFSS+s3lLvLMxQ5MNm2\nSIgYDVJ0T/+dvwVzN1YhGuJs7qLOwqM+WMyj1Dtumv2qqDTpDFUWBsoQdwn9THUU5Tje5frMXFeY\nosOdFzMsBQePoJT7njrdjk+090sp9Vxnld9mOCx9FCehqCjfm3w2LDNHnXrqqXjwwQeNsixZsgQA\nUF1Tg569eiMUCuH9N171XhIvSbkkuaeJoNLhMjt5EENGOwG0JBRLORWVon93jdzPHtMv6bnsDcUx\nIcNX40uNUOyTq/hcKPz1FVW11Gl2324EKIpLZ2qeV+6iTujmt1diS1UjIuGQkSJCdB1PsrY/Tw/1\nznZPqFTP5+4yKrGRudus4ky1iOvt6Y/sgKI8lx3E4Ispmp+RTnmTvQKC5KkEMvUpKKqK1Ttr0SDo\nqCzl0Mtwyy23IB6PY/To0Rg5ciRuueUWAMA111yD915/GReeciw2b/wWXQrEi3kY5U4Wzy8dqukh\nKPOdFzGlArtpp75FMRIfAVqZGUn2LnYfYj48cwPu+ciqCO3JsnxRO6tbPrWDaXPXvvN29a837zM+\nuz0r/v5DxHxHIiJlhM9PEErma+SjZRKqU/2JlPtt76/Gxsp6PPfTCY764tWZDy4txK7aZkw9tBde\nXbANW/eJFaF9Bq5bcrBUFCjDqP5dUb63AXUtCdeRqtEhq9rnBz/fgIuOHIg+XZMnwrMqdwG5+y8m\nd87kR5kx6xQhW510fy7+SsJPlus/cBDe/MxM8Xvbvx4xPg8cdKCR5vfxxx93nOeQQ4bg9U/mAADy\nIiH85qZbAQBHHnUsTpk6xdjvoYceAgCU6+kk3EYegFTuYsSbgNodgOLMJ+NWkZrjCnbXNvu+hGFz\nDyDO3W67XLOzFm8tMVMJU0qNYXuySR12YgfEE6S8kKpw4CNEAHeienjmRqFS4UmfN+OITBWsk+PD\nDHfVNKPsxg/w1XrnhK08zuauqNQxGhDVh6dnb8YX6yot92ReP/n7Hti9AIV5YVx85EAs3Vbtvni4\nLcLDNeOkj4lrdigqxcj+mj/G7V3wC8As2VaNf336LY6e9pmPc5ufheSejnLXb84eGQVodZB13mIi\nFJhqKG8qTA6/AsbeFirrWpLyBW8qTEbg0uaeKtbNAO47FKja4PjJTcn86KmvMfFO78rN4BXGlwrs\nBDPNFoGgkZIKQoAxA7X1IfMiyV/LMYdoU+5FE6S8YA4Z/e1vpDbVv7tl83tv2Q5hEib+/vPsNnob\nGFnxURQOXr2ZAAAgAElEQVRslPPu0h2O/bsX5JnKXaEOEvXykaaq3BMqRbeCPE9/RYutHG7pFngb\nvl+olKI4PwpCzLkRlnNSarxbhVLUNsX147zTGvMjnSbOlBENs0Rd6djctb/dC/JwUE+7OYMYz1Jo\nc/e6nMfv6Vo+dtY0JSV3S6ZMYblzw+je8cg9iZHOjdwXbtmf0iVMZ2Cw0TIiqFQzdUTDIZw8Qlv1\nif11Q4HuSLU7vVJRVn4TZdlTEi/eal2w+LbvHWZ8Fj0vi3LnOi0R0fDK/fFZ1lmsvfXIIL7hdC+M\nIhQiIES7H/soydOhqv/942ladITXLFVFVY3Qy2Sw2/q9cranYltOqBTRMEFRLGLMjbCW0Uo8tc3m\nCFc0oc56rNgsw0ZH6Wgd/h1EBDY2M0+M81jx9aix3ZP7fZbXS+EnO00ys3oyJ7FU7kKwx+J8cEHF\nzCY8wvjSOdeYAc5QL0A3yyiqoWqH9S72JJkifcJHQ4tTuXutLsPqm/9l5azH1TVbzWEDu5uRPqLn\nxZu2+Mb961eWOvblwx9rbcTFhvWLuI66W0Gecd6ESh12fK/oKdb42LP3et9xj0lRxnntx7k8a1a8\nVKqZqlKECEFJftRC3Aw8mSoqtTxHkdK3HMs7VHlyD7uray9UN7Zqil+f+GVHUuXussqR6dPw6rz9\nlTcVhe/nERij42Qn8BXZlhn/dDxyT6rckx/KK6r7PvkWS7dVC/cz49yDs7k/dMnhQnOLQrXVeNhv\nkTDxnC3JcsqIwtXW7qpNeiw7s59cKgAX200pVu+oxaZKa9wxr0pFzyuhUuPevIiR2eEjgjz6zDRw\nwWOmU6y3PkEpEgoZ5i0eXtEyjPxZ+byeu6JQofq0w94o3R2q1lGRHyRUrQzF+WLlzvdvikotIaIN\nLVqk1uOzNqKmSeSzEiv3aBozoRl+/cpSbKmOo7p6H0R0R5Iod9FjsYxMPK7tX7l7ncj8aH/9ondn\njCwysLlTSlFVVYX8/PRXg/OMliGEPAPgLAB7KKWHCX4nAO4HcAaARgCXU0oX2/cLDu7K3c8SdSFo\ncbUPfLYeD3y2HuXTzjR+TyiqFr/cBnHu4RBBQV7Y0dCZLZSRVyQc8rwuU+4ip9e6XXU4+uCersfy\n4Xx+YDxT6ozfBqz2ZKFyV1XEIiG0JlTPCs2HQjogUDrM99AUV/DEl5tw9mhrKKlX58xuLWoo9+Qd\nnkas2r5/OHWY0MkNOGsmPzegurEVs76txCkj+mBvnTbKSoXcFT2VQVEsgnqRWcZiD7Yu6VjfksDc\njVW4a8ZarNpRiwd+OM52bvNzY5w3yzB17buYFjz49X5MPOgA7K+qwu4a05bdujeKSIigsr4V6r48\nxGwBBbVNcccITt0fQ6UeJ9/cJYKq/KjldyZ4CvIiaEkoqKzzXifVfp7d+7VJemvqtFGpolLsrmlG\nfjSErl2i2F1r5tupjoTQstca5ba7thlxhaIuL4z63c4UG/sa4iDVMUQ8Os38/Hxj9ms68BMK+RyA\nhwA87/L76QCG6P8mAnhU/9s2SKLcPR1iCkU07L7fsFs+xJBeRW0S566F7TlfpkopWhXVIBjNxOAk\nGd7JyKJqmgTk7rUqFVOL/tcMNctZJMj/kUy5U0qRUClikRDq4D1xygiFFNR50RC1Z5G1UX1Tvs/y\n3cvmrjrI3UMccDb36048xJXc7Wjl1ps956E52LqvEf27dcEu3WnHirmzpglvLKrAdSceYnlWqkqx\nrKIa4wZ11xKmhQjyo2HhyI0PIVUotYxeGrjwyf2NTtLjlbvVoao9n3QXtqhtURHp2gc9i/Nw1h1m\nYMO080dhSO8iXPnSPPznigk43paM7q4Za/D4rG2WbS/8bAKufPEbAMDvTxmKX04ZYvm97MYPAADl\n087EnA17ceWL3tlB/3DqMFx34iHG99O5cwDAtn2NOOuFmbj7gtE47rCBxjUAYPyB3fH6NWMt57vm\nnpkor9ICD166cqJFbL2+qAK/f3cZvvzDiRjUo8CzbJnAc7xFKf0SwL4ku5wL4HmqYT6AboSQvkEV\n0In0lbuXLV1RKdbuqjMaSJDKPRIKCYf093y4DnGFGnbfSIgIbbS8SYORu31yTIh4L4TBRiV+yd3I\nCgmglIvDP+aQHvjohsk4uFeRsY0P8dSuQbUcMHqnRggwblA312sZDlVBJyDqaEu6WFWbPY2xd7SM\ndk42avIyyyRsNnc3M1NXW7nYe9pS1WDExvPrprJyXPviYtz78bfYsKfecvw/P1mH8x6Zi5Xba9Cc\nUJAfDSMWCQnfNXuGBXkRJFRq6eAaWxVL6Kgd/LYF5aZvg3VomcQXhENOcaNSIGyUR3AvgkXY+TUD\nvJqn3zrOv3fRKmHsEYrqpX2hdu265rYb31hh+W3ldm3xkqDWZ06GIGzu/QHw3WuFvq1twCl3u03V\nq/L5JW1TuQcXLRMOEYu97gfjteHW6p21aGpVDLtvNBwS2sOj3MExfd8W2yzCQn0oyt/H5c9+Y4k8\nYfe2eIvY32AHHy3DyrjpzjPw4s8nYVifYowd2E0Q4qYhoaqOzIyvX32067XiScwyIjIqzreOJPhJ\nVYD/XEN5fs0yCrWsq+tmf7d3Emx0c/w9Xwj3Z/fGzCz2W317yQ59O0Vjq4IueWHEomJyZ9cqyAsj\noaiO9XrZu7C3gb+/txqzN4gXcTHMMsJfrbj82W8wmVsMniEcgiPSSKXUEspqR1xRHc+Yr9+eYs5v\n0IB+nqXbqjHs5g8dv7PriJZUFrVVXnTZ11xgq6f58d1kinZ1qBJCriKELCSELKysTH0VIf0s+l/q\nUK5eL5v1qF6Tk4wZqmlMYnrp660ou/EDRwdhf5m8c3VXbZMRDRIJEyzeWm1JPKVtDzmOtd9/LBq2\nKJtZ6yrxxbpKI7vfpsp6zFipqdt5m6qwfned5/3wzqGmVgUl+REH+dqVKsNna/YgoapcHnStkzv/\n8P4Y0L2LY39GTKJ8MqJ3y8wF9144BgBwZFl36zFeDlXV5lD1jHNXLR2V6yQmnzNUjXKYbg0ATvcC\nH6vemlDRJRpGLBIWqkxG+IWxiCO2nR952EMmn5mzGZ+6rJBlmGV8dJZfrKvE1n2Njn0JIY42QGES\nvujZxwUJ1/gJYl7v12+4r6Kq+PHTX+N7D88R/87IXaTcmVjauh9v6yPXeBJyZ/CTBypTBEHu2wEM\n5L4P0Lc5QCl9glI6nlI6vrS0VLSLNzjl3mxTrl5KjVVoPmROUbVoFb4HZo7KdGzud+rLqrFz8Mqd\nPxsfgbCjutmwH7MG8MHynZbzRoTK3Vp599a34OVvthrl5smfUorzH51r2X+Xj1m7fPqBJl012uFG\nXr96eQkUlaKXbs65+MhBxr24RdawsjrKkaSdMnNRqpOSHDZ3rjN/b9kO3PfJt5b961sUFHJ+B1F9\nE82U9TIP8BFJgDMig12HxakX5OlmmXhy5c5/18qmCk2TbrlvGIxQyKR7AQs5n8feeqtNP6wvcGIB\nNU12bpPa7OTebFHuyctjfw/5UTHdJVSKr9a7L+7CnqFo/QRW7vMfmYsbXtXCe/nZ14UCPxXQcZT7\nuwAuIxomAaihlO70Oih9mMrdPiz1UhYiR+nBN03H0JtnCGPG0yF3e0Nmo4RIiNhmtpn77GtoNcjd\n7ZL8kJZVeP7++end23S7Ll/+hEodzlZRtI0dps2dojGuGBOoeIgUJH/dbgVRbLzzDFx7wsEANDur\nm1ID3Ca0aBvZBK9vbppq/MZsoY5n7zmJidncnWaZX728BA/YFgGvb4lbTEGsTDxpiZ6FX+Xuxp6G\n2UYn9y5JbO7sWoX6e7JEMynUOBdvcvTy0zACFjnwebz09VazHIpqeR8imzsF7++wlmFTZT3W7a5z\nKF+LcvfpY2M4tK8zjTYA1DYlj/9n9yEKZRaNHvhOpYtLSpF0F2hJBZ7kTgh5GcA8AMMIIRWEkJ8R\nQq4mhFyt7zIdwCYAGwA8CeDaNiutViDtL3UqVy//iaHcBTtO+9C5kHE6DlWDoGz2/XCIWAjd3nHY\n7cd28A1DZJbhyZ39zs9IFDXgldtrUC2ImuDB2odKtQgKUWVNRg7M5h4OmVPN3ZQ7IybRU2f7xyIh\nDO5ZiF5cLnvWThxhpr6Vuz+Han1zwhIxNLKfRhbF+REs2bof1720WNhheinjBeX7sGFPPZeOwF5O\nq3LvkhdBLGqaZc55aDbKbvwAC8v3GWGXBTGRcqdmmC93r822sMeozTbOvv/13VVJ78OeO6iRE0wh\nm88JMPPi28sDAFP+OQurdtSiIC+Mx398BG7XZ0Lzdc3eedsXXrc7ZN3U8oY9YvPkg3rnzp6hKN7f\nzhGUWifTuZllckK5U0p/SCntSymNUkoHUEqfppQ+Ril9TP+dUkqvo5QeTCkdRSld2LZF5pW7zSxj\ne9A/eGyeYQcDktvSX/5mm2NbOsqdHcKyFN6vVxBCrGYZu+oQqYKmVgV79TUswwKzDA+edFm5q7mJ\nKiL1+ODnG3DsPzTn1+a9DRblZS8n1Z15osqaTJnyseEM4RBxKDUtJlm712X65DL+PhXu3dkdc0wF\n2dPUeudz1xZbZv6MZO+bUor6lgSKuE74Pz+dgINLNWfyVS8swgfLdxqjJh5eyv25ueU46b5ZxijJ\nrjhZuX77v2UArMqdUorlFVoExgWPzcP1L2umAabceeJOcBO9+HvlRVI0HHKQmCiEVwR+ZbC4omJH\njRkRFCLOUFgK7zDUSJjg1JF9DNMefz/21/v07M2W7/YZy273YY9OYvinbpZrTaLc7XVGUamlXAUu\nZhk/M50zRYeeodpsU+68WaY5ruCb8n2GHQxIPa2Al0Pm7SXbcc9Ha4W/qZQ68rDw5bOTOyMy/pV/\n/9G5GH/7pwDsNncnwfK2cNaAa1zIvX8305nJhvo/eeYb3PTWCkfsNL8SU1NcbHM3Fl/Q0ZPLIy5a\nx5SlC+Bx7X8X4/EvN1m28SpZ5YhP1FkAwJYqK7F6kTWlmmOLPdtktvGWhIq4Qi0jrB5FMRw/tBcS\nXCiraNaol3I3ysSVew/nD7HfRiwSQiwSAqVOuzILsSyMOU13ikqN585v5z/zaZQZmgUi6p2l2/HF\nuj2W7RblrlBLRycKI6QUXPSO+BmxNXpZx8CX1T4ys9cpuxkp6pKQb7/H3JBUlLvDzs+1Vb79S3IX\nIoly5x6eaJKGOfPUvbGdwy2Y4aXcb3h1KR6eaYYZ2lWSvXzJzDIGuXPvfPVOM5UAr1ZjAsdQl2gY\nQ/SYc1bheHLnc49TSh1Db1aBd1Rbnax8MWsa40KzTF8uT/jCm0/C384ZaXzXyNh6rXDYSe6frbUS\nBWB1RvFhrA7l7hJ5kMwmq6hUyyFOzOOT7c+Iwn7/0QjRcgPp70+U74XPdpkM7PIfrNiJCXd+hlnf\nVuK3rzpz8EQjIRTrMyrdzGrMN7JyR42xLcGRO2+y4+ttOEQcJHbsIdYZz1P++QV+/cpSXP7sAsv2\nZ+eUG5/jioqrXlhkfBcuEA4YoaVeJjF2ON+m7CMznuwppQ4/Wl6aseWMsO0j5oN6FkJVKe7jZm4n\ni+Djq7yoswsaHY/cDW53OlR5wtzf4GxkfkIcrzj2IADA5KGlKdvcKzkFq6rUEXfPn80u5swhn/il\nh3mbezhkdAIl+REcN6Qn/nL2CPzuFJbdUCd3TpH84HEzJ0tcpY4ohJ7FmtreyQ2lAWvlrKhuEppl\nXr/maFwycRB+c9JQ9CyK4azR/XDSob0xuLQQiuJU7lE9F4wXRJEpCUF+FzfOTCaYFaplFyQgXHig\n+/5siG+fMp4XDlnSRzAVyBMBGxG4OdcYmIP38VnaCOaZ2Zvx5hJn4Fk0RDBYNwdttOX6YWDKnc8F\ntKe22TCH1bv4Y5rjiqPzvHjCIMt3+whJBOcEO+s5TxvZBxeOH8CFQiYf3bDj+Q7E3hfz7TWuUMcK\nTHazymVHHZj0mgxGtIx+/PA+xThhWCmO1znigc/N9OP2uSeKpUzmPeaEQzX3YCp3pjjG6nnQedIW\nKZpVuopJRtpjB3ZD+bQzcUhpUcqLdfDONE0luffiFx050PKbyNTCgx/SEWJW7KJYBC/8bCIOLi3i\nViVymmXs4Mn90Fs+NIa/docgXzlbE6pj3VZAM/Pced4o/Pokcyp4QV4YlIqVNkul4EXwRTGzjKaD\nWnUQrJsiTuZQVVXt+Wix69q2ZDZ6Vrei9o4qHIJKTfKp1rNy8kTCyKFVUY18OG5lsnx3KU8kHMKg\nA7Sp62759YtizrkHj3+5Ca/oviWe0PlMn81xMxXGeeP6Y8MdpwtDAL1gTy9sj2R77MdHoCQ/6jqp\nygHbK9ZW9bIpd+57S0JxmMjsI5I/njYcP/FB8IyU2fEf3jAZz/10giUlCIOj/XBlCiprrV90PHLn\nbO6sgjJS4XvG5dtrHIf++S1tKS23macnDDNj7yMC04EX+CG5iLwe/9EROHlEb2y68wyMP9A64SbZ\nAh2q7Vw8mfFOKvuEkJqmuFDV3v69wyyjDN4R6bBjKtRiKnLz/tvBE7idfNno4P5PzThykZPYapYx\ny+NU7mJyT0bWCqV4+ZutFmIWdQb2CCs7Qdi/i5Q7I/e4onqqdx5usdfRMEG3LtpIy60DP6BQPLHM\nnn8HgMO8wu4pHCKIcKNEAPjlS+KcgHbytptE3N6FaI6BCPZ3XJgXcbQv/lk0tiqODsbeSUVCxBgR\nXzpxEKZff5zw2nblzhAW5IFi5H7vhWPQuyRmEYjpTIrMBB1wDVWnzb1YJwHWABeW78M3m/ehrEeB\nkcAHAE7RY6RFuVtikRAevfQI47uoV3bDV+sr8dmaPZg81LRNqpQ6OoeJg3tg4mBTufGORZHNnaGy\nvgXrOY8+TxC8bzFis19WN7WiV3G+ZbLS3d8fjVNH9nG9F77Ie+qaMW9TlZaBkJtA4weEmA48uwO0\nQs+6N2djFX6rbxPFbfftajp+mSqLqxQFPpV7svcn6ixFobJxRUU4FDbqjH0UYvddsBEjTyStehoA\nSpMvpeg3O2Q0HEJxfgSEuJM7y3WfDKpKheYB1nmyv/we7y8XT2GxtykWNnn50WUojIVRWiReH5hd\n3k1IDe9T7CgDoM3GZod8u7sOn6zebTEX1bckLOQeDhGHQ5UXCQf1LMSIfuI4eLaylr1zCIeIY7TV\nFNeuGdUTBfKdmtcC7EGjYyt3PVqGhafFFYqV22twwWPz8PnaPRjau9hyaK8SrYKJGv3IfiWWSBC3\n7Iwi/Pjpb/Dc3HIjhStgnTDihhV/O9X4nCeIlmGwh9fxinZ3janAI7YJITWNcZT1tGaeY7Z1N/CV\n8S9vaw2Ut8/6IQ1AcxhRKlbuTAnx6laUcXIIl5TMiHQS5BtxU+57ki2Vxr0bdjy7d344by6W7le5\na3UgalPufuzufsVEJEwQCmkLdjz51SbhPl5LNQLaiI05ivmRZIRT7oB3Nk/AGWnEQngP7VuMP5w6\n3PUchGhx9W7rC9x5/igAznccDpmjhUue/NqRobOhJYE6rt52iYaF5JwMAw/QxAWbcSpS/naOqNdH\nLHnhEKobW/H6ogpj/V9plvGE0+ZexCn3lZw5pldJDN0KzOEpqz+ikDf7iw6HCFTqbrcVnWNjpamu\nVUo945v5ziSZzX29LQ6XJ0LeccVIb9v+Riwo34e6loSjg2MzYXuXiJUUf7+iiKPuBeLhvh3hEIFC\nqTBapkVA7vykJIbD+nfFWaO1BKPsNmua4o5cNm4m4Qc+34ANe+pQ3diKF+ZvsaxSxYf3sXfP7p23\nQRvr6XIzjXnYyX2fbpaxK3czW6P7e7aHlLqBXbM4P+IIBzbL7U0kja0Kqhq0a/blwmNZVEkqi3S4\n1XUvXxJgLrbCwNdB9hztPBwi5shaNCuYKXeWciA/GnZ0eHyHYx80FeaFjRGnUV+jAuVuO27eRi35\nWlnPQmON4+d0J7DfLJVBoeORu8DmXqw7j+KKajFBRMMhS5wpqzSiimhXFqwRu9kKFwvWZV3HJeIS\nhUImQzKzzP+9ucKx729PHurYjymuP72xAhc+Ng+UwnC8MfTQyf1Ppw0XloNvZEzB8mXq5pPcQyEC\nRdVIxt5xskgOPpWBaIZut4IoHrrkcPQuiUHVJ4VV1rVYUg8DyZMwnXTflxj7909wy9srcelTZm5v\nXumZjmh35e5uc7dee7+LQ/W/87UJYvk+zVrJwMIHmXkLgNEJAlr92O+x3CKgdWLXvbQEgDWc1a7c\n/azaxZ7PL44fbNnuZwQRCRGLSY4XLMbzFpA7q6oiU1dDi2ZzL9FDRrvkhRzvKhnyo2HjnphPyu4X\nYuGMvOCp2N+ISIhYUh3YzX63f8+x5lGboOORO6fcX12oef4ZWcQVanEO5kVCFjsmI+pmAbnblQGr\nVG6KpDHuJO5vd5nkrqjOUM1kYKYWr8V6AeirRTnPLZrSbG9cjETdFNWyimqccf9XaGhJGOYYvlL7\nNcuwaAa2LByP+y/WVgBi4XyARg7FsYilUbARSphoo4D6lgRaEqplkhTgPrxmUVQMa7h5A3zueWZ3\nZnWl1qLcGbnr0TL2UEjb8zXMMpYJPSr+8aE22S0Vh6obRIt0H6PHop8yojfW3X46zh7Tz+G0Z2CT\n2BaU7zNmA/fmRk52m7tXh0QpNUwgRbZoKpGj3I6BBxRg/iYz3bBlNKrfq70DD3FmGdEzZfWX5fzP\nj4SRF3buZ2YzsU0+iobNkUFcQX405BCALCf7hIMOMLbVNSccE/1YXWB1yL4OQVuh45G7/oDrm+NG\nDG8krPXKcUVFMxeKlKeHqTEw5W7PJgnAkRCLKdxKl6GyKCPfDm4JMUWlwn3cILI5J4NokQBRo7c7\nM9lI5tgh4qX4np+3Bat31mJ5RY1RuXkTQzefFTNMiKF87KGLvUvykRcJWZxwCYVi9MCuligP1hGF\nQgSqaiY+Y5EiDG7KPZnPhCcdpsAUgXKPKyoWbdlnzBNwOlSteX7YsbxK5AWCX4f0XbqtWQSRucQ0\nX2jX7ZIXxt/PFStEloeIn5nZhyN3trSdQUb5Ufz+FOdIkeGfH3+Lqf+cBQDo3dVqXhMpdz4PEgCM\nGdjNkkWSd4Ta7wsAVvztFM0sk4Tc63RyZ3WoS17Ykh6BgZknDyi0jgZj0RB21jTjqucXYndts/Aa\nrPPjRVxNU9wxkrAr9/bIKwN04GgZLZzOfEhRfTJJfYuV3KlIuQvIvcRmFmCVfVdtM8p6FuLZOZsR\nCRH8+KgyAO6ZEDXFypS7f7MMG30k81299POJGFyqz0LVyeTG003ziih3hlt0R9cuUZRPO9OyZJj9\nPozP3JfufpV7iCSdmckm/wBa4qaFW/bj+KGlln1ZfpRwiKCyvsWIDLErSb58xbEIrp86BF9vrrLM\nyrWDt1Wz41ftqMX8TVUWcr/iuQUWk5EzqZb2zPnJQuEQMcraqzhmIS6eJPLCIZw7th9eW1RhOWfP\nopirT0RUhnsvHGM4cPnnJ+rsAZNc+Rj5Pl2164WIOcKZvmIn/nL2CADAyP5dXcvDRtCA9vz5yCd7\nR7T676c6OuP8aMjSVvgsjewe+EOK86MI82YZQYdZ1xxHc1w1xEy+wKEKAFcccxBKi2M4d4x1fSF2\n3MertWRk/bo6fUIsjJIXcfsbWx0dASNzltrDLfVw0Oiwyt0+E4wtT8dPXopyZpnifDMuVkTuxbaF\ndvvoL3OXrsZvfW81bnlnlUGqbiYXpsAV1Zn7JhkKBZOD7OhRFDPKFRc4+ESKIBom+Pg3k43vfiIf\nAK0jFKVQ9jukDBFiqFm3cj09ezOOv2cmfvy0tiZmfUvCUNTXTx1ikG5+JIyv1u811HO+i+0T0BTb\nlZMHIxYJOxKJAcAhXASOcbx+ndcXVeDiJ+ZbHKrr99Rj6TYzR5C9AxXZcWORkDHiK+tRaFlSj1d1\nt593mDBXz4h+JcJJSEYZbCQ1ZXgv413xr9dNITJy387Z7HvoypU31+3j2pJbKCNgNYdHwyGLAGCh\njAwFeRGHss2Phi0EyT9/ZqZkt5LH+QNYFIuoD2M+B5VT96JRRCQcwnnjBjhCQu3OU5Fdn12XNyPt\nrWtxkDu77h0faJln+5Q4F6ppC3Q8ctdfdqvecK8+XssRnhcJoTmuYO5G03aXFw7hF/rv3QvyjBct\nIt3RA6zKhJHobe+vxrvLdhjb2ZCVJ3fepMJ+V2hqyp1VrmTcyxPJlZMH4/BB3XDeOFNxiJRaOBRy\nRMz4QVOrYlgheY73m/CIT3EsOoYpui1VjYZSppQa5M6bCVj4JpsgYidEnm8vOEJbvjASJmhqdb7n\nsh4FDoesPc+HPb83bwFzM8vwyI+GcZk+wuvXzar4+LJHw8ShYq+fOgT/vmhs0hTQ9g6FzQYGbMqd\nezBf/fFETBneC4DTLPPxbyYbZMYT4HOXH2l8dltty468SMhQpvdfPNaXjyY/EjbmAgBWnwd7PEyU\nlHTRnktBLGL4vZq49nzR+IGWezMXlbdmuzx1ZG/PMlm+C8jdHPHVGO+koVVxjCQY+S/UgzD4xH1t\niY5H7vpLbtWJ84IjNHKLhkN4ZcE2i1qLRkK4+viDUT7tTHSJhjF9xS4c+4/PLft0L4jizWuPNkiB\ngRF2VUMrrn95ibGdjQxe54aiEzmHCoOiqr4cqj+aNMi3F5+vnP27dcGb1x5j+AYAcQW0T5f3i4ZW\nxSAMSilOG9kHP9fz7vgBT1pi5e7MvaKo1DBn8HbZnjbVaL9P/lrTdFt1JBRCU6szQ2OhbjawHG9r\nBXWC5F8Mjjw5AnKPRUL4zclDsfmuMxxl5VVdOBRy2OCvmjwYBxTmJR3JRW0FjkXMUEL+WfCLMA88\noMCo0+zZsuX7BnTvYihkntyP5hKGJZt8xaN7QZ7xjEQLu4jAOhZmxmOd/Zmj+xqpftmdsOiXgmgY\nTa0JfLu7Dmt21qJPST6e/emR+McFo9G7JOaIFirIixjvatABBXj8x+OTlskuIEQjLDbyaY6rRrkA\noE8HMkwAACAASURBVIt+P8y09ubi7ZYOi3VQbY2OR+5gTgym4rQHJVKtMa7hsV62Yn+TxSyTHw3j\n8EHdfZsrGPksqzDj6fkFotmwUVHFTlc7bv/eKKy/4wzje7JoGa/IA5FT1j6E94tGzqlFqZYL5Oaz\nRvg+nufAsMgByN0Lv6AJI3I+ksauGu2qiinviD5dHtDUbYNg4QwRuduVe1VDq6uD2z6tXtQxs4ZO\nCHGYAvjv0RAxyIuBla0gJibT2793mFGX7794LE4YVgrCORf5W7F37KwDYP4lVpej4ZBBOFcdN9hY\n7YpHMjsxf83eJTFDNR/Yo8DlCNu59Xtm7ZK1m5vOONRol2z0VKzXhcJYGA0tCk7515cAgOOHluLE\nYb303yJG1NLZY/rhhxMG4uYzDzWevZ/JRHYfnOj++ZXNeHMl68C/+uMUY9u/P9HWdbjlrBG+uSZT\ndDxytyl39iDtDR6wNiSeX5rjqtGILrQpdi+IJvbw9vqnfqIpAkWljjzYfnDdiYe4Dtu8JpXwpGUP\nZ0sVfpbgS4awD18AA2ts0XAIfzl7BJ69/EiLKckR8WNraCKTlpszsSjmtPna1fhbS7ajZ1GeMEJk\nmM2GzL+TqbrZgydme0fCd1TRcAjDbUu/sWfVsyiG164+CqtuPRWLbznZ+P34oWb+o3PH9sdzP51g\nuU4x1ynZ74tFD7EZ3VW6uo2ECAryIiifdiaunDwYT142HpvvOsNyrP2ZjR3YDYUCNdujKIYzR2kx\n92U9Ch2/ixDTz83aCzNj8J0TC8tlpFuQF7GsPcAr6+JYxCDeolgEd50/Gj2KYsbkLD9pHuzmJFG0\nDD9T3ELuell4/nlmzmYAVnNjW6PjkTvE5C4KMeMbO6/OmuMKepXEsOrWU3HDSe4hXqMEEQL8xBGG\nLnnmY2TxwqmGQjKM6FeCOTdOwfTrj8PvTh5qUcBeHT6vCFgFYyT3r4vG4I7znKFxdocXg6JSc3Hs\nVG5Ah8U8ICB30QhFy5kSxYk6SZrHW/dzI2f+nG6r7hQJlLtISZVXNVrMEvzxPPgGzMIA+bpoV+68\niakgL4xJg61ZIvmyHFl2AApjEUvddeu0zhzVFzecNAR/4Can2UdtzO/XJWq9B9H927fZhUVRLGKE\n4/LPPRwi+OvZI7DklpN9TWACzM6atRfRhDFmQjLMMnlhiwDhn3lhLGI4g/k0EKw8fsjdHhUmMkv9\n/lSTOwrzwka6Al5oPnrp4ZZj+giibtoKHY/cmXLXh3CsYohsYrz5hfeGN8cVdImGURiLJM2r/PwV\nExzbNu915s/mIwzyuCEmMx29fOUk9/txwYh+JfjV1CG4Rl9U+keTBvmeQASY0TeM5M4bNwCXTnSm\nN33z2qOFI4W4qhpNduqhvRy/e8HT5i6IOXYjLvs7cjhUDacbd36XcxXalLuoA2cQjQbt4AmIzVQs\n5iJd7BNnenATsPzOVuX7KbdFHiLhEG44aail87E/d5YNNdM4675d8zG8T7Ex4mITgNikqUg4hO6F\n/usqe85MuRvplTlinjj4AMQiIVw1WZsBW5AXsaSTsJN7tZEGwrzXKGcy9ULXLnazjPNdFeRFDHEU\nDhEM662Nwvj3evqovpYZrMlCXINGh41zb4rHURyLGAqDd94cNbgHCmNhnDfONLnYlbsfB5Hd1luc\nHzEiKfiskbxCYTbUyvoWVNa3YPLQUhx1sHsOby/8/pRh+MH4gTjQ5xCXocCm3N33i6Brl6glXA+A\nkar0+KGluPuC0SldG7CqbVHGP9EM2ZvOOFR4LjsZ2R2sDDzvuXXaRbGwodyvO/Fg/OFUcRoGwF88\nMt+JMOXJEzirG90KohjVv6vlvv3OVuVHPl19pn/gjzMm0bCImjRXJGKYe+MU3P/ZeigqxRG3fWKa\nd9I8L3Oosig2UQhtr+J8rLv9dON7Wc8CS73i1xngOzi+800lV46dH9xyTLFrRULEEB3291qUH8H+\nxji+N7YfBnT354cIAh1QuWt/GlsUi52L77kvO+pAPPWTIy2kyzf22uaEL1VmJ4jSohhaEgreWbrd\n4pTJC4fwzwvH4O3rjkFhLILiWAS7appRvrcRB/l0KrmBEJIysQOmuvUTuiiqtnGVolVRUdajwFfy\nJzv4ZyfKc2Iv1n9/NtE1ZJPvmO0hq4Cm/AZ074J/fN/shNwSZxXGIka9SNbYn738SOFo0A5+YszZ\nY/qhMC+MnxxdZmxjRb/wiAF44WcTLccyEjhF4MDkwY+CUnkXsUgIk4eW4qnLND/Q0brIsOcbShWE\nEEs0GYPfhbTtYG2RzV1xy+PD4yibOcuq3PlwU6dZptDFWc3DHufullqZRfbMXFdpOIYd5K6P5Ozm\nxrZGh1Xuza1xSyyw17RuniAWbdmPyZxjyi/y9Jl3v35FW9eyT4mWK31waSHGDTLzePTtlo/1e+pQ\n35JAn67tE9PK8OENx2HWukpMX7kLgDPbnQiiyUqKqqI1ofq2m9rBE1KVgNy/f/gALCg3k68li6Pm\nOwpRLv5QiGD2n6ZYtrll4OvbNd/o8JKRx+DSQkeaAxF4u/aQ3kVY9ffTLL/v1hPZ2ae3A2YH/PCl\nh2PIn2e4XoOVk19Mxg8IIRbT4lXHDcY5Y/qhX5px1scN6YkRugO4JN/5vtJd9NlQ7gkzA2c4RJKe\nzz4ZzW6WYeDfMeOAZGGmH1x/LCr2NzkiaqpdyJ0PsGAjF7soYE7uVFOMZIqOR+6EmWUSNnI3P4v4\nzF5R7LMc/SAWDVti16+aPBiTh/bEIb2sinNo72J8ri/47EclBInhfUowvE8JPlrFyD29HNIJlWZE\n7nxnOlWgWC6eMAhNcQW3vrcaQPLYXz7U1E+GQsA9DHVkv65Gx5MstLQwFnE00h9NGuTYjzfLiEaD\n7FojBQtBsKG/l7kgHCL4+qapruYovwiFSNrEDsAy8hDNVBZlKvUDNhpp5pS7l1+AEIL1d5yOiXd+\nhn0N1in/BZzDOI/z7TAHbDKSHdmvK0b264pP9bQDvUti2F3b4qrcRe3DbtJhHCDzuXtCV+5xxRKC\nOGagOVwX8ZmXU84Nn/72eONzLBLCgs3mMmX50bCD2AFt2GvMpgwgC2A6MHKU+1Luzm2tCRUJlQoz\n6fkB4/Yu0bBl9SkefMPokYS4zhnTD9dP1ZZS9Lv0oZsTOD8aNgg3GanaCWDVrafiNkEiLv4cIjv/\nb04ein9fNBbHCRK18XXj9u8dhnd/eYxreXqX5KetjNsC9qJMHlqKMbYsnH7ByJAJp1ZF9bVuazQc\nwtDemoLn27N1FG8WdJBuIrWvXywCixQborfva/SZ7nYwW/wd5x1mmQ3LY+xAbVTfXtkgGTqsck8k\nVIvyOuZgs/GIVKDd1+PH5g5Yh3+xSMgyu9VN+XUVxLy2N5ij2Y9aEIWGsRV60lXuS7Zq+VhE+V0Y\n+JmWophpBkIIThhWigc+W+97wYNTbEsJHnFgd/zt7JEA4MsswwjnxGGl2FHdbBnqW+7Bg4SKYhF8\nb1x/4W+86v/RJGckUy5jkM2XZJ/0kwpYO/r3J9/inDH9MHv9XssqSsnAfCt8Z8CTKx8J1r9bF2y8\n8wxfnSTraFiCPTew5nXCsF74Yp224pI9jPKXUw7BkWXdHWGvbQ1fLZcQchohZB0hZAMh5EbB7ycQ\nQmoIIUv1f38JvqjG1QBoNmG+YbF8Id0LojhWEJ9sf6HJcne4we7McjtHVxdHb3viWj2E8tC+3nll\nRLG3jRmS+yiB49MOt4W+RWB20nRXszlhaKlRJnYle3SHSF0//ZMjMePX4oWTgfTszEWxCHoWxdpt\npmIypBuaN7yP1cyUiSplHemmvQ1IKKpj5bFkYLH2vO+DSZVLJg5y1G2/74uJjYEezudLJ2qmuu4F\nUcN006PQud6AaM5EW8OT4QghYQAPAzgZQAWABYSQdymlq227fkUpPasNymgvEABAtZE7IQSb7jzD\nNQTOnqDJngUyGWb/6USoKvCPj9ZatrsRmHUqcnYGRycM65VUcfB44OJx+GztHvz+tWUAtFBDNvsv\nL83wtiuPG4xdNc2GA04ERq5+VB/rJNNdQV40mYVXzhvv1GZkHnzTdMtxyeZBpIuFN58U+DlTxUc3\nTMaOmiZjyn6mSEcsMfCRKRsrnfNIkkEUuy9S86liyvBe+NdFY3DmqH5J9/vllENwzQkHIxIO4Y7v\nHYZHv9iIw10WSWlv+HkjEwBsoJRuAgBCyCsAzgVgJ/d2gm6WUVWLswRI3hB31lgXS/a70jwAIzaV\n7yB+eeIh6OsSCdMtB5R7KuhemIcLjhhgkHssEspYuYdDBH87Z2TSfVjonJ8JL+w5pqvc+cZ/0ZED\nUdeSwEmHmiGITNGdO7Zfymp8zo1TLAtMeMFvEq62xLA+xY5UCplAFD3jF7zvgV8tyw+OGNQdK7fX\noieXoycRwGQtQohlnkyy/ZhIGdK7GPddNDbtawYNPy23P4Bt3PcKfZsdRxNClhNCZhBCkrfqTKAT\nLLUpdy+s0BfOZjPKGgUZA71Qz2V2SzYxiV92qyOQux3hEMmY3P2ANQo/M2+ZzXv0AP9OO37qN9/Q\nxw3qjocvOVw4erv/4nG47wepNdD+3bqklVa5M8HLfJEM0XAIj/3oCADADa8uTenYP585Ah/dMNli\nWz9cD0t2W23su4KgWu5iAIMopaMBPAjgbdFOhJCrCCELCSELKysr07yU6ShMhdwZ/qSvXFTWM/WJ\nQXy45bhB7iTD2/9SMf/kCiIhYixQkW60jB+wfCF9fSRTyo+G8da1R+PRHx3uuS/D6aP64ocTNJuo\nKDOlRGZ45zozumdsCp2uCJMGW9Nm33me+zKDPPIiIccIZHzZAVjxt1NwQkAmp44KPzV+OwA+dmiA\nvs0ApbSWUlqvf54OIEoIcXSblNInKKXjKaXjS0tTn0QEwFDudoeqFx744ThcedxBOHFYL7x93TG4\nZIIzZtkLVxxbBgCY9YcTfOeqzsQWmQ10K4haHI32hYODxFEH98Bh/Uvwf2e4pwDgMW5Q95Q7y4Qg\nw6BEMOBDH/0u5uEGu6kq0/w3HVFUBQ0/zLMAwBBCyEHQSP1iAJfwOxBC+gDYTSmlhJAJ0DqNKseZ\nAoFJ7qk4+84Z0w/njNGcI2PTjMc94sADfDspGTqSWebz3x2P7gV5uOSpr41tu2y+iiAxoHsB3v+V\neyRKEFC4XPESbYeiDEWMPaw4WQithD94vhFKaYIQ8ksAHwEIA3iGUrqKEHK1/vtjAC4AcA0hJAGg\nCcDFNN2pkV7QlTuh6Zll2hu5EO7mF2zxbT6K5FRbvHhHA0vDm+nsTonkyLTztLcTt7kBEv7hq7vV\nTS3Tbdse4z4/BOChYIvmBmL8n+4qQ+2B4ljE90SMXANrqEcf3CMjR1ku4IaThmBE35KU87JIZA9H\nlnXP2Mwj0YFnqALU99qj2cAXfzjByBjX0cCGyJmEt+UKYpEwzh6TPFZZIrfw/BUTvXeS8ETHI3dD\nudM2DdPLFD2KYknzpeQymGpqr4V8JTou5tw4xbLebhDIVsqOzoaO13oJZ5ZJM3+0RHKwVLedQblL\ntC3c1vuVyD46HrnrIKAdKhKlI4FNGGrvLHYS3228ctUk2aYDRMcjd2KaZdwy9UlkBuanzl2PhkRn\nRHtnTezs6IB2DdMs094LYXxXcJi+aHS8nRcXkJCQCA4dT/oy5U5ouy9b9V3B2aP7obFVkVEmEhId\nGB2QHaVZpq0RChEjJ4uEhETHRMczy3Az2aRyl5CQkBCj45G7VO4SEhISnuh45K4r9xCAghxY9EBC\nQkIiF9HxyF1X7rEIaZMl0CQkJCQ6AzoeuevKPT8iiV1CQkLCDR2P3A3l3gGLLiEhIdFO6HgMaSj3\njld0CQkJifZCh2NINmeyV7H3osoSEhI5jF0rgF0rs12KTosOF0vYGFdRCKB3ScdMpyshIaHjsWO1\nv3+ryW45Oik6nHJvbNXWVozl8CpMEhISASPeDCRas12KDoUOx5CNcc0wI+cvSWQNi18AZtyY7VJ8\nt3BHb+DBw7NdCkBVgc/+DlRvy3ZJPNHxyF1X7nk5vMSeLzw5FXh4kvk90Qp88Dtg/5bslem7iIYq\n4G9dgaUv+T/m3V8CXz/admXKNSgJYMeSbJcCqMkBQt21DPjqn8BbV2e7JJ7ocOTe1KoCAGIdndy3\nLwQq15jfK9cCC54Cnj8ne2XqaFDiWgf5/m/TP8e+TdrfBU8FU6bOiNt6AE+cAOxene2SAF/em93r\nM9OQkvsmog5H7g1xTblHO9skptYG7W/j/uyWw46Gvakr2/ZAogWYNkjrIBc+nfn5mnLsuSfDplnA\nbaVA4z7x7zUVwXVWqmJ+bqwK5px2rHhdGx34wee3aZ36jD8BdbvapjxuqK/URBgA1G4HNn7evtdP\nER2O3BuZcu+o66fe1gt44Tzn9pY67W84x5a2Y8r268ezWw476ncD8cbMzxPXO9V9m4CVb6R+/Oe3\nA2unZ16OVPDlPZpy3LVC/PtLF2kmviBMKfwzDmcQfrxzufU51e40P7/xM2D5K/7PteFT4OvHgOl/\nSL886eDRo4H3rtc+124Xt+McQodjyLEDuwMAuhZ0II/q57cDaz/QPist4h6/lZG7jwa0+l1NvQHa\nULlyXTDlFKG5VvvLK7hkqN4G3DsMqNqY2XUp1f65wU21xps0O7pfsE4VADZ/5b0/6+wArXxf3gO8\n8kP/1wsCqq5yQy6J82oqtL9PnJD5tVo5cidp0gWlwOPHWZ/TF3c69/EL1uGQNEfv8ab0jmvY49zm\nFsHjdyTShuhw5N67q7bausXmHm9qW4LLFF/eA7xyifg3VqkZWflR7v/7sWmbf/Qo4OEJmZfRDfW7\ntb9UQO4bPtM6rk/+am5b9SZQvwtY+Ixz/y3zgMpv/V33zauAW7sBe9aIf29yIff/nAPcM9jfNQCr\nmgz5EAzLXjU/Z6vOsY7Wze7rtyMWYcXrwOLnze9sZAMAajy9cy7+j2Db89bvbqJm3iPAUydbtynx\n5McIz/OwVqe2fg3c0UdziP6jzDSHesGt8xGZ85a+DNw1QButiPD1E1o52hgdjtyN3po97HgTcPfB\nGsG11Kd2rrrd4qH43Ac1O3PtjszKClh7dtH5Ei3aMHP677XvqVTYioXm5wVPA8v/J95vyzztfpii\nSwWs8iaardtrdwL/PV/ruOb821R40QLtL2s0TdWafXT3KuDZ04CHj0xejoXPAPvLgRX6vTx9avJy\n2VHxTfLfeVSuA5ZxvgQvcqcUqN5qfn/6ZPd9k52DHy3U7U6djJlyd6vvSov7savech/17FqhmUje\n/ZX5PnnySyQ5bzLw5qOVb4rPE3ch2S/uMt+pUQ69LoZTmMj40U3A8leBZ0/Xvi97Wasj/EgsGW7v\nLd4uEhlr3wcSTcDqt52/KQngwxuB9R/7u24G8EXuhJDTCCHrCCEbCCGOAF+i4QH99+WEkDYMSGWK\nXSf3t681K0ZzdWqnmvFH4PUrrOqwcR/w8c3a54/+nFFJAVhf/nNnOX9vqbNOwY6kUGHf+7X5+YPf\nAm9eKVYYTDml4wBiRMQTyfZFwH3Drfs1VmnXZuWv14ewK17T7KOPHm3u+6+RYkJrrgHe/w3w3+9z\n168Bts537uvV8f6jzGrXFWHnMut3NzMHw6y7rZ1BS614v9odGimufgf44PfW3+Y/oqm6ut3aM/rn\nUGDWP7TfKr/V3qnXkJ6Ru0h1UmpV9Eao58uac/y1yzWbvLDc3PPaq49KeLOMkqZy5010r/9ULELc\nTCWiZ9ykt/MIJ4Ra6oBP/6ZNdkoG+whU5JTds8bpr3DrMLfpCnz1O8Dnd+jX0PyCqNrg3L9hj1aG\nkrZfn9iT3AkhYQAPAzgdwAgAPySEjLDtdjqAIfq/qwC0XRCwXbnzvaObInEDewmPcPHmj082P8eK\nUy+fHbxK3SewQ8/4I9BQaX7fvVI8ZKtYBMy5H5h5l3VfO/aXm59b6oEXLzQrcEu9RjwvXew/OoSR\ne+NebQIHpcCTU5z7bZkDPHiEpvoAoHy29pc9YztEZW/Yq/+ttG5/5lSzQQMaWbEOmEEVXKde0HC3\nL9JGSoCTOLziqOc/kvx3QBup/XuUVub/XQYseFLrtADtHuY+qH3evdL0m7C/r/0EWPScmBQYqrcB\nu/ThPj8CADQitxN31Xrt74InzXde8Y1mUuPx1jXAppnm9/3lmhnjpQvNbemE/9VXWs8LaPME7GhN\nwTneqNcTXrnPuR+Y/S+bSalJq6vLX3M/Fy8SKNVGw49MsvorknVqzPTyv8uAL+/WPrO6uvoda73l\nr1fS3/2cAcGPcp8AYAOldBOltBXAKwDOte1zLoDnqYb5ALoRQvoGXFYdnHLfvdpKHm522PWfOu1f\ni18w1SWghfrNusfawO2Nxw2qYjoeHdf+JPmxq950NuZnTtHs2ErCVDRPTQE++Qswa1ry8z0wVqtA\njfu0od/6j83Gtfh54L8XAN/O0JQtf38Ln9EieeorgfmPmp0nI0CqAhULNDs4jyG62eStX1g7r5Ya\njSDcwtVEk7VYqF1I4HfwUur3Hers3O0NC9AaOxsZ2N/Z6nfcbfyAe0fFo7FKU9a8KYKd89kzgDpd\nHc9/BHjz59rnolK9vDr5Lv2vNoJheOc64P6x2mfeFFTLCYf/nK35GtZ/ZG6L5GvOdwAgYWuH/iE3\nAG/cp41I+M7rtcs1M0Yzl/clmbkH0EyDr1yqtTdAq0PTf5/8mLLjtL92s0xTtdWXAwCDT9T+1mzX\n/ka7mNdhvpCdS4HPbtM6+/1btM6cPWcR3rvefD8VC4Cnppq/MWHGRIcIzCfFoKrW57x/s/X3ffr3\nbm2/AL0fcu8PgJc0Ffq2VPcBIeQqQshCQsjCyspK+8/+wCv3R4+y/rboOfPz0peAO/pqDerF72ve\neqbuGvdp6mEbN9x/+xpg5u3W81V7zBYtn6MNd+8+CJg2UCOLZ88AXvyBVr66Xd5kDADffggceg7M\njguaHfvZ0zXnTyqRBIBGdHcf5By2V64B9qwyv//vMvPzZ7dpjfepKVrD3zpPs40uecHc53lbn37e\nE8DFSeLf7x+jNS4R9pdrpglWxm0LtPA9wFRmlrLr8cUf3yI+X/0u7Z55fPVP7V0nWpzTxbcvEhM5\nM5FsX2xGOAFaOd3MMACwZS7w6o+cjR3QRES82frs2egB0MwWdbvMjnzug1pnO+tujTCX/FcjiS3z\nzM4B0NRq5TpNmW7+0nrNXiM12/T8h7XvFd9YO4a932oktOY94NFj3O8LAA7WR2pN1ZrgmHO/5kjn\nQalmGlz7vtbeqrcBL18stjsz/OQ94PL3gfxuzo72H2VaG2A4+npgwlXa51Vvan/XzTD/1ui+kKUv\nAl/dC+xcIo5uEeGRSdpo0O5DYTPI+fdmR/0ezY/B8PfuWjsrPVT7/tY12t+5D2rv6p1rte89DvZX\ntgzQrvGElNInADwBAOPHj0+RsRh0AhQNoXl197b+ULfMMbc9cwpw2j+SN1KGLt2BHUu1Rrh/i9ao\nmmuB0+7SbLM7lwHPnWE9ZtpAa1kWPWt+v7kSuL3U/XpDTgbWvGvdxhxJdrXshqGna6qcgW8cImz8\nHJj+R41IWCQEcxg27deULI+EzS466gJvO3XFAvPzrxab+UE+uUX7l1ek2em9Jsi8/tP/b+/Mo7So\nrgT+u93YILvIpjRKGwk7gjRKRAkqI7IFcQHZjctEHHACnhDFxNFMnGMYNxLPcXQUVDQg7kwSxoUw\n42hUlgQdNgWFERVlERRwAek7f9wqqr6vv96+brrp6vs7p05VvXpV9W4t9913361XNh1GoNMQaHEK\nvHFfat7GbU3Zb/mf4go/JO5ayu8blXPtc2atxvsn2vSAzwJLfNCt0GM03JPmmQw76jK5FxZNLF22\n7evgrk7QoFlq+rLb085xYfF9S4qUataudKUE8N6L1uIqi7CT/48zbArpcwU0y7dWSvi+hdzbvfRj\nNmwJBYELtHUXs7jBKjjJJRrcGzjnBjj/ltTObDAl+uGbxcMqAVbMtZZHaeQ1iUKQM3Fgryn9kii8\nCt5eaK2cdFp3tvLtWG8hti/9InIlnvp3Fetby5LyWO4fAzGtRX6QVtE8VUO9+tCoVWqo3VUvQ++J\n1rlxW4uSb8hHK8wynX9R5u2jAyu142Bb1kPWhP/jDIsKWf6A+eQXTU71zWfinq62z+Fy58Gg22z5\nB1NtmNNRD0bbu19q6dnQazy07QHjFsKJvaP00ny3LYJwweUPZA5xWzjOOmjrN4WhaZ98t+0JZ02L\nFHu9BjZvnCGiIIxJrt/UrJVbv4C+10TbD+zL7svHIb+BsQtg8O1R0x7guAK4YUPFjjXh2dT19I7n\nz2Iulg7nmOKclFbxHd53aeb0kPNjrob8vnbfQkIXSN+r4bIM4YMATYKOuNAyjNO+H4eNnxFzrLIv\njbhiD/3XA2ZGz2HDljYvKYronm4w/2Ibujd0Q3Ut4d0KOaaRzU8dFKWder61pBZNtpZq3AiSXOh5\nuS03PwkuTQuxnTs488dcqx+PnusBM21+wmnR9qF3wg3rYWTMFdWiFGu63rFpCQI9R5cc5dP+zGj5\n0VggRbs+MLYCH2xVgvJY7iuAjiJSgCnsy4H0oO3FwFQRWQicCXyhqmWEKmSJiL0U7wbxyVNXQsuO\nZmVD5njs8tCoFXQZAWMet5citx6MfdJCw3LqwbmzrDn62ZoMnYHCYUujYEDxJnL4wjVoavPwJT5t\nDHQ429LzGpqi2r3FmrZxcurBtFXRA/byL1OtkotiD+iEZ03p3hH49DoOjvywvSeam6XXBBh+t0UD\nffe1NftLYsQcaHh8tD5jAzRunWqxT11hFVn/n5rle1wHs8bCCJlJL8ApA6P8w+60Dr5M9Bxj4alF\nJUSM5ObBuEWR1Qcw+rHIQp+2yp6RzsNTr2OXEXDeLeaq+P0Y2PuJWZ4DZ0X3JU6P0VE4ZpwTayH3\ncgAAC3pJREFUT4/mTU5IdZMcvvdiL/DKh80l9H+v2/XIzYMzroGlQSU/6DZ7lpbMjI7RbRQMu8uW\nO++ycV3AKtQBPzMXSW6ePRMHv7LvIuJW4Ld77ZyNWlpl/86iIBw1+JqzaX6qrx7gF9th50Zz/XQZ\nYW6ON+6z5VXzMofnNm0XfIKfVpl1vyRyxfzD8tSWRYdzzA3z6ZrUaJHOw83NE3fhnNgbrl5avGXY\n/RKb0g24yxdYuO2uTVB4pbXSAWZts3frvJvN8n/wXBhxr8kG0Hs8nDbWZG/W3sZ82rnRKvB41Nb0\nNfCvMeX/8y3W0ir4IWz+b2tZLP2VbRvzBHQaaq3f0M0H0O1iGDLbdEs1UOZZVPU7EZkKvAjkAnNV\nda2IXBts/zfgT8BQYBPwFfDjko5XJXS8wJT7uEWm2MFqxJBuo0zJDJltvuO4Twzggl+nRltMfA6O\nP9WUQnjTATpdCLNiDZDeEy0+tvlJpiTrNzHFOOZxeHK85ek60qzUULFM+6u5HiCybOMfKjVL65oo\nvDJVKV37OrTqnPpAXPygTf85C7qkhVc2bGHzK1+Ep6+CobNhTqDc8wut3Dk5phCGBRb5gJ/B354w\nxXpf7DpedD90vziqOAGaZugnb34S/CiIAhkQ60C7Zpl91Xhir+L7XDrXwlDjnDoILrgd+l1n/u4w\n+uCyR8wvO/8iq9i/d27qfsceB2f8xMofKoMRc6BNN9v2/QvNfRBe99GP2dgrQ+8q/qK1/L5dm/7X\nR8p94vN27jbd7dqBVQg3bIiUzPinzYe/+VU4e7o9O50yuFEAzpwCxzaHDv2Luxrig3Pl1rOWyO7N\ncPUrxZvy9RsXP3b9JqlRXj1H2zxU7uMX2TMbdp4W/NCO27a7TWDurlmfmD9+1Tyr3KeuSn02pq+N\n3IUn94dRD5iya9AUpq+zii8nx1oYO9abERZ2IrZNc9m07JRaGZ89w1wxZbn8wFrYHyyzZ6dzzE06\n6kFzS+U1jNKanwQzM0Ss5eREPvCTz7LpwH4zglYHhk+jlibz63Og1zi7f2DPxle7rFP8zCnw4V+i\nVsnAm1KV+/C77XmsLlS1RqY+ffpo1hw6pLpna2rawW9U/6mp6pxetv75FtWiItWtKy09nBZfb9vD\n9TB/NhQVqe7YqPr1nuh4K+fZts2vqa5eULzcr82x/GWxZ6sdv7IUFUVl+2av6lM/Vv1yW+a83+5P\nvVZbV1j6dwdV/7m16pKbKl+eOEtusvP8x3TVP/1c9esvUrfHy11UpPrG/ao7N1VtGdLPdeg7u0/7\ndtr6v7S37Vv+orpvR8n7FRXZfX3uOtX9u8p/3g9ejY4xd4jqpqWp2/dsVX3v5ezlCpk7RPW/Zkfr\na19QXXZH6fsc/Fb1xZtVv/rc1lcvVF23WHX3h7b+zN9HspfE7g9V33ygfGU8dMiuf3kIr9mR5tdt\nK3eez9alPiNVALBSy6FjRSsaiVFFFBYW6sqVK8vOWBE2v2p+s3Rr+Mtt0Uc3v9xlFlFocd2yO7LG\nsuW7A5Gf8LJHrOVwNLH8361l064c35YtuTEaq/y6t6xjCCzcU3KyH88jE/u225eDw+4q3pkIcHc3\ncyFUx2/YwuchPJeqWV1dRlgLoCQWTbKO52zLqGqRId/sqX2/mys6ZLHvx6T7o6uB9Pt1pNi/y9xf\nzduXnbck3v+zRcr0m1J23nIgIqtUtbCsfLVo9K1yUFBCJ2fclRA2wy971JRLVYwuGXezlNWhVBOc\ncU3ZeUKG3BEp93jzvjxN5IrSuDVcUsrQtFNey/6T94py5pTUyk8EBpbjb0uXPpJ9P094npmbK3eM\nmiInF3JqQLFXJ42OB44vM1upfO+8KJy0GkmWci+Ntj1Se9W7VaESjluzVWnZ1hRh6F8mn251Up3+\nySHl+B4hEzk5VHqIpqo4Rl0j/4zivnsnhbqj3K96pVb8PeWoYNLzNnxAJleJ4xwNXF3Gl99OHVLu\nxzSw6UgxYo5FtSSBRi2rtmXjOE61U3eU+5GmzxU1XQLHcZzDuKPPcRwngbhydxzHSSCu3B3HcRKI\nK3fHcZwE4srdcRwngbhydxzHSSCu3B3HcRKIK3fHcZwEUmOjQorIDqCMn5SWSEuglL/WJhKXuW7g\nMtcNKiPzyapayj87jRpT7pVBRFaWZ8jLJOEy1w1c5rpBdcjsbhnHcZwE4srdcRwngdRW5f5gTReg\nBnCZ6wYuc93giMtcK33ujuM4TunUVsvdcRzHKYVap9xF5EIReVdENolIOX5yefQjIu1FZJmIrBOR\ntSLyj0F6CxF5WUQ2BvPjYvvcFFyDd0VkcM2VvnKISK6I/E1E/hCsJ1pmEWkuIk+LyAYRWS8iP6gD\nMk8Pnus1IrJARBokTWYRmSsi20VkTSytwjKKSB8R+d9g229FKvHfTlWtNROQC7wPnALkAW8DXWu6\nXFUg1wnA6cFyE+A9oCswG7gxSL8R+E2w3DWQvT5QEFyT3JqWI0vZZwC/B/4QrCdaZuBR4OpgOQ9o\nnmSZgXbAZuDYYH0RcEXSZAYGAKcDa2JpFZYRWA70AwRYAgzJtky1zXI/A9ikqh+o6gFgITCyhstU\naVR1m6r+NVjeC6zHXoqRmDIgmIf/vhsJLFTVb1V1M7AJuza1ChHJB4YBD8WSEyuziDTDlMDDAKp6\nQFX3kGCZA+oBx4pIPaAh8AkJk1lVXwU+T0uukIwicgLQVFXfVNP0j8X2qTC1Tbm3A7bG1j8K0hKD\niHQAegNvAW1UdVuw6VOgTbCclOtwLzATKIqlJVnmAmAHMC9wRT0kIo1IsMyq+jFwJ/AhsA34QlVf\nIsEyx6iojO2C5fT0rKhtyj3RiEhj4Bngp6r6ZXxbUJMnJrRJRIYD21V1VUl5kiYzZsGeDtyvqr2B\n/Vhz/TBJkznwM4/EKrYTgUYiMiGeJ2kyZ6ImZKxtyv1joH1sPT9Iq/WIyDGYYn9CVZ8Nkj8LmmoE\n8+1BehKuQ3/gRyKyBXOvnScij5NsmT8CPlLVt4L1pzFln2SZBwGbVXWHqh4EngXOItkyh1RUxo+D\n5fT0rKhtyn0F0FFECkQkD7gcWFzDZao0QY/4w8B6Vb07tmkxMDlYngy8EEu/XETqi0gB0BHriKk1\nqOpNqpqvqh2w+/hnVZ1AsmX+FNgqIp2CpPOBdSRYZswd009EGgbP+flYn1KSZQ6pkIyBC+dLEekX\nXKtJsX0qTk33MmfRKz0UiyZ5H7i5pstTRTKdjTXZ3gFWB9NQ4HhgKbAReAVoEdvn5uAavEsletSP\nhgkYSBQtk2iZgV7AyuBePw8cVwdkvg3YAKwB5mNRIomSGViA9SkcxFpoV2UjI1AYXKf3gfsIPjTN\nZvIvVB3HcRJIbXPLOI7jOOXAlbvjOE4CceXuOI6TQFy5O47jJBBX7o7jOAnElbuTeERkX02XwXGq\nG1fujlNFBANjOc5RgSt3p04iIiNE5K1gAK9XRKSNiOQEY2+3CvLkBONqtwqmZ0RkRTD1D/LcKiLz\nReR17AMdxzkqcOXu1FVeA/qpDeC1EJipqkXA48D4IM8g4G1V3QHMAe5R1b7AJaQOU9wVGKSqY6ut\n9I5TBt6MdOoq+cCTwYBOedgPJQDmYuN53AtcCcwL0gcBXWM/xmkajOIJsFhVv66WUjtOOXHL3amr\n/A64T1V7AD8BGgCo6lZsNL/zsJ9ELAny52CWfq9gaqeqYUft/mouu+OUiSt3p67SjGg41clp2x7C\n3DNPqeqhIO0lYFqYQUR6HfESOk4lcOXu1AUaishHsWkGcCvwlIisAnam5V8MNCZyyQBcDxSKyDsi\nsg64tjoK7jjZ4qNCOk4aIlKIdZ6eU9NlcZxs8Q5Vx4khIjcCU4giZhynVuKWu+M4TgJxn7vjOE4C\nceXuOI6TQFy5O47jJBBX7o7jOAnElbvjOE4CceXuOI6TQP4f00mp+G8KARsAAAAASUVORK5CYII=\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x7f92984789d0>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# initialise input again\n",
"x = np.random.randn(10,3,28,28)\n",
"x = (x - x.mean()) / x.std()\n",
"x = np.sqrt(input_var) * x + input_mean\n",
"\n",
"activations = [x]\n",
"for _ in tqdm(range(1000)):\n",
" W = conv_weights((f_out, f_in, ksize, ksize))\n",
" h = selu(conv_forward(activations[-1], W))\n",
" activations.append(h)\n",
"\n",
"plt.plot([a.var() for a in activations], label='variance')\n",
"plt.plot([a.mean() for a in activations], label='mean')\n",
"plt.xlabel('Layer')\n",
"plt.legend(loc='center right')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Final Thoughts\n",
"\n",
"SNNs are a great bit of a math! The name \"self-normalizing\" is slightly misleading as the activations are non-normal but do have fixed mean and variance.\n",
"\n",
"I wonder if the weight matrices keep the appropriate mean and variance throughout training. If not, could the weights just be normalized at each iteration or something like weight norm be applied?\n",
"\n",
"This seems very applicable to RNNs. The vanilla RNN is $h_t = \\sigma(Wh_{t-1} + Vx_t)$. If $h_{t-1}$ and $x_t$ have fixed moments, then $Wh_{t-1}$ and $Vx_t$ are both normal, their sum is normal, and the same math works. However, do you really want fixed mean and variance at every iteration of RNN?\n",
"\n",
"SNNs remove the much loved (or maybe secretly hated?) bias term. They also remove the learnable means and variances included in batch/weight/layer norm. Would there be any benefit to learning different means and variances per unit, where those means and variances would manifest themselves as different $\\lambda$ and $\\alpha$ parameters? If you wanted to go down this road, would it make sense to set $\\lambda$ and $\\alpha$ by solving the equations in the paper or just by learning (maybe initialized to the zero mean unit variance standard)?"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.13"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment