Skip to content

Instantly share code, notes, and snippets.

@manashmandal
Forked from eamartin/notebook.ipynb
Created July 17, 2017 09:18
Show Gist options
  • Save manashmandal/1f0d9fb8f7696a2f03f42ef570cc5318 to your computer and use it in GitHub Desktop.
Save manashmandal/1f0d9fb8f7696a2f03f42ef570cc5318 to your computer and use it in GitHub Desktop.
Understanding & Visualizing Self-Normalizing Neural Networks
Display the source blob
Display the rendered blob
Raw
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Self-Normalizing Neural Networks\n",
"\n",
"[Self-normalizing neural networks](https://arxiv.org/abs/1706.02515) (SNNs) are carefully constructed to have zero mean, unit variance activations without an explicit normalization scheme such as batch norm, weight norm, layer norm.\n",
"\n",
"SNN theory is based around analyzing the mapping from the mean and variance of one layer to the next. Let each component of $n$ component vector $x$ be iid distributed from some distribution with mean $\\mu$ and variance $\\nu$. SNNs exist for any $\\mu$ and $\\nu$, but to be concrete let $\\mu=0$ and $\\nu=1$. Let $z = w^T x$ where $\\sum_i w_i = \\omega$ and $\\sum_i w_i^2 = \\tau$. As $n$ grows, the central limit theorem states that $z \\sim \\mathcal{N}(\\mu \\omega, \\sqrt{\\nu \\tau})$. If the weight vector $w$ is normalized such that $\\omega = 0$ and $\\tau = 1$, the distribution of $z$ has identical mean and variance to the distribution of each component of $x$, which is a good first step towards normalized activations. For the moment, let's assume $w$ is meets these conditions.\n",
"\n",
"## Scaled Exponential Linear Unit (SELU)\n",
"\n",
"Neural nets need non-linear activation functions to be powerful function approximators. If there existed a non-linear function $s$ that could map $z \\sim \\mathcal{N}(\\mu, \\sqrt{\\nu})$ to $h$ such that $\\mathbf{E}[h] = \\mu$, $\\text{Var}[h]=\\nu$, then it would be possible to non-linearly map $x$ to $z$ to $h$ while preserving the first and second moments.\n",
"\n",
"Conveniently, such a function exists! The authors name it the scaled exponential unit (SELU):\n",
"\n",
"$\\begin{align*}\n",
"s(x) &= \\lambda \n",
"\\begin{cases} \n",
" x & x\\geq 0 \\\\\n",
" \\alpha e^x - \\alpha & x<0 \\\\\n",
" \\end{cases} \\\\\n",
"s'(x) &= \\lambda\n",
"\\begin{cases} \n",
" 1 & x\\geq 0 \\\\\n",
" \\alpha e^x & x<0 \\\\\n",
" \\end{cases}\n",
"\\end{align*}$\n",
"\n",
"Exponential linear units (ELUs) are a special case of SELU when $\\lambda = \\alpha = 1$. The authors compute that for $\\lambda\\approx 1.0507$ and $\\alpha \\approx 1.6733$, the $s$ preserves mean 0 and variance 1. Let's investigate this."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"%matplotlib inline\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"scale = 1.0507009873554804934193349852946\n",
"alpha = 1.6732632423543772848170429916717\n",
"def selu(x):\n",
" return scale * np.where(x >= 0, x, alpha * np.exp(x) - alpha)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAEACAYAAACqOy3+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd8VGXa//HPnQQIJfQqIAmPSBFduzQlS1EXBWwsTRR0\nWbvoWlZETGz7uKuirvvbItWyNAXdB2woGOnFJVIEQTqEFqUmkEbu3x8niQmEtJnMOTPzfb9e55Up\nZ85cmZCLO9e57+sYay0iIhJ8ItwOQEREKkYJXEQkSCmBi4gEKSVwEZEgpQQuIhKklMBFRIJUlK8H\nMMbsAI4Bp4Bsa+2Vvh5TRERK53MCBywQb6095IdjiYhIGfmrhGL8dBwRESkjfyRwC8wzxnxrjBnp\nh+OJiEgZ+KOE0tVau88Y0wj40hjzg7V2kR+OKyIiJfA5gVtr9+V9TTXGfARcCRQkcGOMmq2IiFSA\ntbbE8rRPJRRjTA1jTEze7ZrAtcC6YoLw/JaQkOB6DKESZzDEaK2le/furscQSp+n4vTvVha+jsCb\nAB8ZY/KP9W9r7TwfjykSEN98843bIYj4xKcEbq3dDlzsp1hERKQctBIzT3x8vNshlEkwxBkMMQaT\nYPk8FWfgmbLWWir8BsbYyn4PkYowxpS51igSaHn/Pks8iemPaYQVklc3Fw9SUhMJDq4lcFCi8KJw\n+o81ISHB7RBEfOJaCUV/vnqTfi4i3lCWEopOYoqIBCklcBGRIKUELiISpJTAi7F48WK6dOlC3bp1\nadCgAd26dePbb79lypQpREZGEhMTU7DVrl2b/fv3AxAbG8v8+fPPOF5ERATbtm0r8lhiYiLDhg0L\nyPcjIqFJCfw0x44d48Ybb2TUqFEcPnyYlJQUEhISqFatGsYYunbtyvHjxwu2Y8eO0bRpU8A56VDW\nWRzhNNvDqxITE90OQcQnSuCn2bx5M8YYBg4ciDGG6OhoevfuzYUXXliuJjOl0UwP9z333HNuhyDi\nEyXw07Rt25bIyEiGDx/O559/zuHDh90OSUSkWJ5N4Mb4ZyuvmJgYFi9ejDGGkSNH0rhxY/r378/B\ngwcBWL58OfXq1SvY2rRp4+fvXESkbDybwK31z1YR7dq1Y/LkyezevZv169ezd+9eHnnkEYwxdOrU\nicOHDxdsP/74Y6nHi4yMJDs7u8hj2dnZVKlSpWIBiojg4QTuFW3btuXOO+9k/fr1FT7Gueeey/bt\n24s8tn37dmJjY32MTkTCmRL4aTZt2sS4ceNISUkBYPfu3UybNo3OnTuX6fVZWVlkZGQUbDk5OQwc\nOJAXX3yRlJQUcnNz+eqrr5g7dy633XZbZX4rUgr1QpFgpwR+mpiYGFasWMFVV11FrVq16Ny5Mxdd\ndBGvvfYaAMuWLSsyDzwmJob//ve/Ba/v06cPNWrUKNief/55nn32Wbp06UK3bt2oX78+Tz31FFOn\nTqVDhw5ufZuCphFK8FMzKylCPxcRb1AzKxGREKYELiISpFy9oIOIiBRlLfzjH2XbVyNwCVs6iSle\nk54OgwbB5Mll218nMaWIcPq5hNP3Kt63bRvcdBNcdpkzAq9ePUAnMY0xkcaYZGPMHH8cT0QknMyb\nB507w+9/D5MmQXR02V7nrxr4KGADEOOn44mIhDxr4ZVX4I034IMP4Jpryvd6nxO4MaYF0Ad4CfiD\nr8cTEQkH6elw992wdSusWAEtW5b/GP4oobwOPAHk+uFYnjR8+HDGjh1bodd27NiRhQsX+jki2LVr\nFzExMarhigShbdugSxenVLJoUcWSN/g4AjfG3AgctNYmG2Piz7Zf4bP98fHxxMefdVdPKs+Vdk7n\nSxOswmJjY5k0aRI9evQAnAZZx48f98uxw5V6oYgbvvwSbr8dnnkGHnzwl7bXSUlJJCUlletYPs1C\nMcb8CRgG5ADRQG1glrX2jkL7BP0slBEjRtCiRQteeOGFMr8mJyeHqCj/TbOPi4tjwoQJ9OzZ02/H\nLE4w/VxEgom18OqrMG4cTJ8O3buXvH+lL6W31j5trW1prY0DBgELCifvYJWcnMyll15K7dq1GTRo\nEBkZGQXPzZ07l4svvph69erRtWtX1q1bV/BcbGwsf/nLX7jooouIiYnh1KlTxMbGsmDBAvbu3UuN\nGjWKXOEnOTmZRo0acerUKbZu3UqPHj1o2LAhjRo14vbbb+fo0aMADBs2jF27dtG3b19iYmJ49dVX\n2bFjBxEREeTm5jJjxgyuuOKKIt/D66+/Tv/+/QHIzMzk8ccfp1WrVjRt2pT77ruvyPckIpUrPR2G\nDIGZM2HlytKTd1n5eyFP0A/dsrKyuOmmm7jzzjs5fPgwAwYMYNasWRhjSE5O5u6772b8+PEcOnSI\ne+65h379+hW5WMP06dP57LPPOHLkCJGRkQWll3POOYfOnTsza9asgn2nTp3KgAEDiIyMBGDMmDHs\n27ePjRs3snv37oLS03vvvce5557L3LlzOX78OI8//niRmPv27cumTZvYsmVLkWMPHToUgKeeeoot\nW7awZs0atmzZQkpKCs8//3ylfH4iUtT27U69u1o1WLiw4vXu4nh2IY95zj9XbbcJ5fv+Fi5cyODB\ngwv6gQN07dqVHj168PPPP9OwYcMiya9du3aMHz+eq6++mri4OBISEhg+fHjB83FxcUycOJEePXow\nceJEpk6dyvz587HW0qpVK6ZOnUq3bt3OiOPjjz/m+eefZ/Xq1WccB2DHjh20bt2anJwcIiIiGDZs\nGOeffz5jx47lxx9/5LLLLuPgwYNUq1aNmJgY1q5dS+vWrQGnJe7QoUPZtm3bGe+rEoqI/+TXu8eM\ngYceKt9lHstSQvFsL5TyJl5/2bt3L82bNy/yWKtWrQDYuXMn77zzDm+99VbBc9nZ2ezdu7fgfssS\n/nu95ZZbeOihh9i/fz+bNm0iIiKiIHkfOHCAUaNGsXjxYo4fP05ubi7169cvc9xDhgzhscceY+zY\nsUydOpWbb76Z6OhoDh48yIkTJ7jssssK9rXWkpsbspOGRFxXuN49c6b/SianUy+U0zRr1qzI6Buc\nxA1Och4zZkyRa2KmpaUxcODAgn1Lmq1Sr149rr32WmbMmMHUqVMZPHhwwXNPP/00kZGRrF+/nqNH\nj/Lee+8VSbKlzYLp1asXqamprFmzhunTpzNkyBAAGjZsSPXq1dmwYUNBzEeOHOHYsWNl/1BClHqh\nSGXIr3fPmOHM766s5A1K4Gfo0qULUVFR/PWvfyU7O5vZs2ezatWqgqvU//Of/2TlypVYa0lPT+eT\nTz4hLS2tzMcfMmQI77zzDrNmzSpIsgBpaWnUrFmT2rVrk5KSwiuvvFLkdU2aNGHr1q1nPW6VKlUY\nMGAAjz/+OIcPH6Z3794AREREMHLkSB555BFSU1MBSElJYd68eeX5WELSc88953YIEmK2b4euXaFq\nVWd+97nnVu77KYGfpkqVKsyePZspU6bQoEEDZs6cya233grAZZddxvjx43nwwQepX78+bdq04d13\n3y3XHPF+/fqxZcsWmjVrxoUXXljweEJCAqtXr6ZOnTr07duXW2+9tchxR48ezYsvvki9evUYN24c\ncOaofMiQIcyfP58BAwYQEfHLj/bPf/4z5513Hp06daJOnTr07t2bzZs3V+jzEZHiffWV08/krrtg\nyhSoXr3y39OzJzHFHeH0cwmn71Uqj7Xw2mvONm0a+GudYlCfxBQR8boTJ+B3v4NNm5x6d2WXTE6n\nEoqISAXkz++OioLFiwOfvEEJXMKYeqFIReXXu0eMgHfeCUy9uziqgUsR+rmInJ21ztzuV1+FqVPh\n17+uvPdSDVxExE8K17uXL4e89X2uUglFRKQUO3Y487sjI516txeSN7g8Aq9oj20RkUCZPx+GDoXR\no+Hhh8vXz6SyuZbAVWcVES+zFl5/3blm5bRplVvvriiVUCRsqReKnM2JE04Xwfffd+rdXkze4OIs\nFBG3acaNFGfHDrj5ZujYEd5+28UpgpV9RR4RkVCyYAF06gR33gnvvute8i4rTSMUkbAXDPXu4iiB\ni0hYO3ECfv972LDBO/O7y0olFBEJW/nzu8Fb87vLSglcwpZ6oYS3/Hr3HXfAe+9BjRpuR1R+moUi\nImHFWnjjDfjzn51+JnnXCfcc9UIRESkkv979/fdOvTs21u2IfKMSioiEhZ07oVs3ZwS+ZEnwJ2/w\nMYEbY6KNMSuMMd8ZY9YbYxL9FJeIiN8sWABXXfXL6spgrHcXx+cauDGmhrX2hDEmClgMjLLWrij0\nvGrgIuIKa+HNN+Hll71d7y5OQFZiWmtP5N2sClQBcn09pkggqBdKaDt50plh8s47Tr07mJJ3Wflj\nBB4BrAb+B/ibtXb0ac9rBC6epF4ooWvnTqefSfv2MH58cJZMAjILxVqbC1xsjKkDfGSMucBa+33h\nfQqPdOLj44mPj/f1bUVEivX11zBkCDzxBDz6qLf6d5ckKSmJpKSkcr3Gr/PAjTFjgRPW2tcKPaYR\nuHiSRuChpXC9+9//hp493Y7IN5U+AjfGNARyrLVHjDHVgd7Ay74cU0SkvE6ehHvugXXrQmN+d1n5\nehKzGbDAGLMGWAnMs9Z+6ntYIiJls2uXM787Jyd05neXlU8jcGvtOuBSP8UiElDqhRL8kpJg8GB4\n/HH4wx+Cp97tL+qFIiJBx1r461/hf//XWZjTq5fbEfmfeqGISMjJr3evXQvLlkFcnNsRuUe9UEQk\naOzaBVdf7dS7ly4N7+QNSuAiEiSSkpx+JoMHO9MEg3Fxjr+phCIinmYtvPUW/OlPzoUXevd2OyLv\n0AhcwpZ6oXjfyZMwfDhMnOjUu5W8i9IsFAlbWonpbbt2wS23QJs2MGEC1KzpdkSBFZBuhCIi/vbN\nN069e9Agpw1suCXvslINXEQ8I7/e/dJLzvxulUxKpgQuIp5w8iTcey98951T727d2u2IvE8lFBFx\n3e7dzvzuzExnfreSd9kogUvYUi8Ub8ivdw8cCNOmqd5dHpqFIiKusBb+9jd48UXVu4ujXigi4kkZ\nGU69e/Vq1bt9oRKKiARUfr375Eklb18pgYtIwCxc6NS7BwyA6dNV7/aVSigiUumshf/3/+CFF5x+\nJtde63ZEoUEjcAlb6oUSGBkZcNdd8PbbTslEydt/NAtFwpZ6oVS+3budfiatW8OkSSqZlId6oYiI\na1TvrnyqgYuIX1kLf/87PP+86t2VTQlcRPwmIwPuvx++/dZZEv8//+N2RKFNJRQR8Ys9e+CaayAt\nzTlZqeRd+XxK4MaYlsaYr40x3xtj1htjHvZXYCKVTb1Q/GfRIrjySrj1VpgxQ/XuQPFpFooxpinQ\n1Fr7nTGmFvBf4CZr7cZC+2gWikiIKlzvfvdduO46tyMKHZXeC8Vaux/Yn3c7zRizETgH2FjiC0Uk\n6OXXu1etUr3bLX6rgRtjYoFLgBX+OqaIeNOePdC9Oxw/rnq3m/wyCyWvfPIhMMpam3b684VXvMXH\nxxMfH++PtxURFyxa5PTufvhh+OMfwZT4R76UVVJSEklJSeV6jc8rMY0xVYC5wGfW2jeKeV41cJEQ\nYC384x/w3HPwzjtw/fVuRxTaKn0lpjHGABOBDcUlbxEvUy+UssvIgN/9zkngS5YoeXuFr7NQugEL\ngbVA/oFGW2s/L7SPRuDiSeqFUjYpKU4/k3PPhcmToVYttyMKD2UZgauZlYQtJfDSLV7s1Lsfekj1\n7kDTJdVEpELy692JiU69+ze/cTsiKY4SuIgUkZkJDzwAy5c787vPO8/tiORs1AtFRAqkpDjzu48e\ndRK4kre3KYFLWHp83uNcPvhyt8PwlMWLnX4m/fvDzJk6WRkMlMAl7Hy44UNeX/46sTfHuh2KJ+TX\nu2+5BSZMgNGjdbIyWKgGLmFl/cH13PfJfTxz9TOs3r/a7XBcp3p3cNMIXMLGoZOHuGn6TYy7dhxd\nz+1KRk6G2yG5Kr/efeSI089EyTv4KIFLWMjJzWHwrMH0a9uPYb8aRnRUNJk5mW6H5ZolS5x6d79+\n8MEHEBPjdkRSESqhSFgY/dVocm0uf+n9FwCio6LDcgRuLfzrX/Dss5rfHQqUwCXkTVg9gY9++IgV\nv1tBVITzT75aZDV2frwTRrocXABlZsKDDzq17iVLoE0btyMSXymBS0j7fMvnPLPgGRaOWEiDGg0K\nHo+OiubgpwddjCywUlKcy501b+6csFTJJDSoBi4hK3lfMnd8dAezB87m/AbnF3kuOirapagCr3C9\n+8MPlbxDiUbgEpJ2Hd1F32l9+ccN/6BLyy5nPB8uCTy/3j15MvTp43Y04m9K4BJy9qftp9e7vXiy\n65Pc2uHWYvepFlUtwFEFVmam00FwyRJnhaXq3aFJJRQJKT+f+Jne7/Xmjl/dwcNXPXzW/UJ5BL53\nL8THw08/OfVuJe/QpQQuIeNoxlGue/86+pzXhzFXjylx32qR1aA7IdcPfOlSuOIKuPFG1bvDgUoo\nEhLSstK4cdqNXNX8Kl7u9TKmlGYexhiq9qpK5qnMkBmNv/02PPOMU+++4Qa3o5FAUAKXoHck4wh9\n/t2HDo068Faft0pN3vnyF/MEewLPr3cvXqz53eFGJRQJaj+f+Jme7/bk8nMu5+2+bxNhyv5POhSW\n0+/dC7/+tVPvXrFCyTvcKIFL0Nqftp/4d+LpFdeLN69/s1zJG4J/Of3Spc787j59VO8OV0rgEpS2\nHtrKNZOv4bb2t5Wp5l2capHVgjaBv/023HSTM8/7mWcgQr/JYUk1cAk6q1JW0W96P8ZeM5b7r7i/\nwsc5+vlRMn8bXCWUzEx4+GFYtMipeZ9/fumvkdClBC5BZe7muYz4zwgm9ptIv7b9fDrWwU8PBtUI\nfN8+p59JkyZOvVslE/H5Dy9jzCRjzAFjzDp/BCRSHGstf1v5N0bOGcncwXN9Tt75giWBL1vmzO/+\nzW9g1iwlb3H4YwQ+GXgLeNcPxxI5Q2ZOJvd/cj8r965kyV1LaF2vtd+OHQwJfPx4GDMGJk1yFuiI\n5DP+WIlmjIkF5lhrLyzmORtqq90kcPYe38stM26hRe0WTLlpCrWq+u9S6cYYIp+LJDIi0m/H9Lec\nHLC5EFVFFxoON1ljs7DWlvhTVwIXz/pmxzcMnT2U+y6/j6evfrpCM01KYowhI9ubI/B9+2HwIGjU\nGCZOgNq13Y5IAi26SnSpCTwgJzETExMLbsfHxxMfHx+It5UgdSr3FC8sfIF//fdfTO4/mevPu75S\n3ichIcGTXQmXLYMBA+D3v9cUwXCSlJREUlJSuV6jEbh4yp5jexg6eyhREVG8f/P7NItp5nZIAaV6\nt+QzxpQ6Atf/7eIZH3z/AZe/fTnXtr6WebfPC6vknZUF994L48Y5c7yVvKUsfC6hGGOmAd2BBsaY\n3cCz1trJPkcmYSM1PZUHPn2AtQfW8vGgj+nUopPbIQXUvn1w223QqJEzv1v1bikrn0fg1trB1tpz\nrLXVrLUtlbylPD7c8CEX/fMiWtVpRfI9yWGXvJcvd+Z3X3cdzJ6t5C3lo5WY4orth7fzyBePsOmn\nTcz+7Ww6t+zsdkgBN2ECPP00TJwIffu6HY0EI9XAJaAycjJ44ZsXuGL8FVzV/CrW3LvGteRdeHZU\nIGVlwX33wWuvOfVuJW+pKL/MQinxDTQLRXCWwn/646c88sUjdGzckTeue4NWdVu5GlPeWf6Avuf+\n/U69u0EDeO89lUzk7MoyC0UlFKl0q1JW8eRXT7I/bT9vXv8mfdr0cTskV6xY4STvkSM1v1v8Qwlc\nKs3WQ1sZs2AMi3YtIrF7IiMuGUFURHj+k5s4EUaPdure/fzTh0tECVz8b8uhLfxp0Z/4z6b/8Gin\nR5nYbyI1q9Z0OyxXZGXBI4/AggWwcCG0a+d2RBJKlMDFbzambuSlRS/x+ZbPefDKB/nxoR+pX72+\n22G5pnC9e8UKqFPH7Ygk1KgKJz6x1rJ412JunXkr3ad0p33D9mx9eCuJ8YmeT94JCQmVduwVK5z5\n3b17w0cfKXlL5dAsFKmQrFNZzPx+Jm8sf4OjmUd5+MqHGXHJCL+2ew1WkybBU0+p3i2+0SwU8bvt\nh7czKXkSE5Mn0qFRBxLjE+nTpk+5rwgfilTvlkBTApdSZeRk8PEPHzNh9QTWHFjD0AuHMm/YPDo2\n7uh2aJ6xf7/TArZePdW7JXCUwKVYp3JPsXjXYqavn84HGz7gkmaXMPLSkfRv15/oqGi3w/OUlSud\niw3ffTc8+6zmd0vgKIFLAWstK1NWMn39dGZumEmjGo0Y1HEQq0auIq5enNvheVJ+vXv8eOjf3+1o\nJNwogYe5zJxMknYkMWfzHOZsnkP1qOoM6jiIr4Z9RftG7d0Or1IlJiZWuB9KVhY8+ih89ZXq3eIe\nzUIJQ/vT9vP5ls+Zs3kO87fN54LGF9D3/L70Pb8vHRp18Pu1J72qor1QDhxw5nfXrQvvv696t1SO\nssxCUQIPA0czjpK0I4n52+czf/t89h7fS4+4HvQ9vy992vShcc3Gbofoiook8Px69113QUKC6t1S\neZTAw9SBtAMs27OMpbuX8s3Ob9iQuoFOLTrRM64nPeN6cmmzS4mMiHQ7TNeVN4FPngxPPunUu2+6\nqRIDE0EJPCxkn8pm/cH1BQl72Z5lHDp5iE4tOtGlRRe6nduNzi07a+ZIMcqawLOznXr3l1/Cxx9D\n+9A+NSAeoQQeYtKy0lh7YC3J+5JJ3u9sG1M3Els3ls4tOtO5ZWe6tOxCu4bttLCmDMqSwA8ccOZ3\n16mjercEllZiBqn0rHR++OkHNv60kY2pG9n400a+T/2e3Ud3c0HjC7ik6SVc1uwy7r7kbi5qclHY\ndvrzVWm9UFaudE5WDh8OiYmqd4v3aATukhPZJ9hxZAfbDm8r2PKTdmp6Km0atKF9w/bO1qg9HRp1\noG2DtlSJrOJ26GFB9W5xm0ooLrHWcjjjMCnHUkg5nkLKsRR2Ht3JtsPb2H5kO9sOb+PwycO0qtuK\n1vVa07pua+LqxdG2QVs6NOpAbN1YnWR0SXY2/OEPMG+e6t3iLpVQ/CwnN4efTvzEwfSDpKanknoi\nlQNpB5wknZeo9xzbw97je6kaWZXmtZvTPKY5zWs3p1WdVvRu3ZvW9ZxkfU7MOapTe0x+vbt2bad8\nonq3eJ3PI3BjzPXAG0AkMMFa++fTnvfcCDwnN4ejGUc5knGk2O1wxuGCBH0w/SCpJ1JJTU/leNZx\n6levT6MajWhUsxGNajSiSc0mRRJ1/le1VQ0uq1Y587tV7xavqPQSijEmEtgE9AJSgFXAYGvtxkL7\n+JTAc20umTmZnMw5SXpWOmlZaaRlpZGeXeh2ocfPeC47nWOZx4ok6BPZJ6hTrQ51o+tSr3o96kbX\ndbZqdQtu5yfowl/rV6+vUXMImjIFnngC3n4bbr7Z7WhEHIEooVwJbLHW7sh7w+lAf2Bj4Z2emPcE\nGTkZznYq45fbORlk5mQWuV/kuVOZZJ/KplpUNaKjoqlVtRY1q9SkVtVazu2qeber/HI7ploMTWs1\nLbJPTNWYIom6VtVaSsTC2LGJHDmSyBdfwDffQIcObkckUj6+JvDmwO5C9/cAV52+U+OajYmOii6y\n5SflMx6PLPp41ciqYdObQwInPR1efPE5brghkZUrnb4mIkHHWlvhDbgVGF/o/u3AW6ftY4vbEhIS\nbHESEhK0v/av1P1zc60dMsRawD77rPvxaH/tn5CQYL/++mubkJBQsAHWlpKDfa2BdwISrbXX590f\nDeTaQicyvXgSU8Lbm286de/vvqtYN0KRQAjEScwonJOYPYG9wEr8fBJTxJ+SkmDgQFi+HFq3VgIX\n76r0k5jW2hxjzIPAFzjTCCcWTt4iXrJjBwwaBFOnQpwuMCQhwOeFPNbaz4DP/BCLSKVJT3cueTZ6\nNPTs6TxWWi8UEa/TUnoJedbCb38LtWo517DUpCYJBlpKLwI89xzs2uXM9VbyllCiBC4hbdIkePdd\nWLoUonVNCwkxKqFIyPriC7jzTmfk3bat29GIlI9KKBK2kpNh2DD46CMlbwldaggiIWfzZrjxRvj7\n36Fr17Pvl5iYGLCYRCqDSigSUnbsgGuugYQEuPvukvct71XpRQKpLCUUjcAlZKSkOHO8n3ii9OQt\nEgqUwCUkpKZCr14wciQ89JDb0YgEhhK4BL19+yA+3lms89RTbkcjEjhK4BLUdu50at5DhzoLdkTC\niaYRStDassUpmzz6KIwaVf7XqxeKBDvNQpGgtGYN3HCDM9tk5Ei3oxHxP81CkZA0bx707g3jxil5\nS3hTApegMnky3HEHzJ7tnLQUCWeqgUtQyM11TlK+9556m4jkUwIXzzt2zGlKdfAgLFsGTZq4HZGI\nN6iEIp62eTN06gSNG8PXX/s3easXigQ7zUIRz5ozx1kS/8ILcM89/j++eqGIl6mdrASlzEz44x+d\nVrAffwxdurgdkYg3KYGLp2ze7Fw5PjbW6eldv77bEYl4l2rg4gnWOpc/69rVmds9a5aSt0hpNAIX\n1+3Z4yTt/fthwQK48EK3IxIJDhUegRtjBhhjvjfGnDLGXOrPoCQ85I+6L7nEqXOvXBnY5K1eKBLs\nKjwLxRjTDsgF/gU8Zq1dfZb9NAtFzvDDD07f7p9/hilT4KKL3I5IxFsqtReKtfYHa+3mir5ewlN6\nOoweDVdf7Vy3cuVKJW+RitJJTAkIa+HDD6F9e6fmvXat0wI2SmdhRCqsxF8fY8yXQNNinnraWjun\nrG9SeMVbfHw88fHxZX2phIAlS+DJJyEtzell0r272xGJeE9SUhJJSUnleo3PKzGNMV+jGrgUY+NG\n5xJn330HL77oXDUnQn/ziZRJIPuBl/gmEl5++MFp+dq9u3O5s02bYNgw7yVv9UKRYOfLLJSbgb8C\nDYGjQLK19jfF7KcReJhYuxZeegmSkpz69gMPQJ06bkd1duqFIl5WlhG4mlmJT6yFhQudq+OsXAmP\nPQb33gu1arkdWemUwMXL1MxKKk1GBkybBm++6dweNQqmT4fq1d2OTCR8KIFLuWzb5qyenDABLr4Y\nXn4Zrr23wksZAAAG+0lEQVTWe/VtkXCgBC6lOnnSae06YQKsW+fMJvn6a2dOt4i4RwlcipWbC4sX\nO2WRGTPg8sud2nb//lCtmtvR+Yd6oUiw00lMKWCtcyJy+nSYORMaNoSBA50Rd6tWbkcnEl50ElNK\nlZXlXOV97lz4z38gOtq5oMJXX6lEIuJ1SuBhKDUVPvvMuebkl19Cu3bQty/83/857VyNlmWJBAWV\nUMLA8ePOXO35850LJmzfDj17Okm7Tx//XuldRPxDC3nC1M8/w/LlsHSpM1tk7Vq48kro0cNJ3Jdf\nDlWquB2liJRECTwM5OY6TaOWLoVly5yve/c6CbtzZ6cfSdeuWmBTnMTERPVDEc9SAg8xGRmwfr3T\n3S852fm6di00beok6y5dnK8dO0JkpNvRep+W0ouXKYEHqaws+PFHZ2Sdv61bB1u2QJs2zjUk87df\n/Qrq1nU74uCkBC5epgTuYTk5sHu3szR9+3YnOecn61274NxznWl8+VvHjnDBBc40P/EPJXDxMiVw\nF2VkOLXolBTnEmI7dvySrLdtcx5v0gTi4qB1a2fLT9bnnRc6qx29TAlcvEwJ3M+sdS4Llpr6y7Zv\nn5OMT9+OH4dmzaB5c2eLi/slWcfFOSNsJWl3KYGLl2klZgkyM+HIkeK3Q4eKJumDB3+5HRkJjRr9\nsuUn6csvd/qE5Cfshg3Voc/r1AtFgl3QjMCzsiA93RkBp6X9crssjx09emaSPnXKOflXeKtX75ev\nhZN04a1GDT98KCIipfBMCeX55y0ZGZS4ZWaW/Dw4V3nJ32rWLPq1pMfq1DkzWVevriXjIuJdnimh\nZGU5I9f69Z1ZFIW3atXOfKy456OilHBFRAoLmhKKiEg4KcsIXKfZRESClBK4hC31QZFgV+ESijHm\nFeBGIAvYCoyw1h4tZj+VUMSTNA9cvKyySyjzgAustb8CNgOjfTiW65KSktwOoUyCIc5giDGYBMvn\nqTgDr8IJ3Fr7pbU2N+/uCqCFf0JyR7D8UIMhzmCIMZgEy+epOAPPXzXwu4BP/XQsEREpgxLngRtj\nvgSaFvPU09baOXn7jAGyrLVTKyE+ERE5C5/mgRtjhgMjgZ7W2oyz7KOzRCIiFVBpKzGNMdcDTwDd\nz5a8yxKAiIhUjC/TCH8EqgKH8h5aZq2931+BiYhIySp9Kb2IiFSOgK3ENMY8ZozJNcbUD9R7locx\n5gVjzBpjTLIx5gtjTDO3YyqOMeYVY8zGvFhnG2PquB1TcYwxA4wx3xtjThljLnU7ntMZY643xvxg\njPnRGPNHt+MpjjFmkjHmgDFmnduxlMQY09IY83Xez3u9MeZht2M6nTEm2hizwhjzXV6MiW7HVBJj\nTGReLppT0n4BSeDGmJZAb2BnIN6vgv5irf2VtfYSYC7wrNsBnUWwLKBaB9wMLHQ7kNMZYyKBvwHX\nAx2AwcaY9u5GVazJODF6XTbwqLX2AqAT8IDXPs+883S/ttZeDFwMXG+MucrlsEoyCtgAlFgiCdQI\nfBzwZIDeq0KstccL3a0F5J5tXzcFywIqa+0P1trNbsdxFlcCW6y1O6y12cB0oL/LMZ3BWrsIOOx2\nHKWx1u631n6XdzsN2Aic425UZ7LWnsi7WRWogkd/x40xLYA+wATA3W6Expj+wB5r7drKfi9fGWNe\nMsbsAobg3RF4YVpAVTHNgd2F7u/Je0x8ZIyJBS7BGVx4ijEmwhjzHXAAmGetXeV2TGfxOs4Mv1L/\ng/HLBR1KWPAzBudP/GsL7+6P96yI0hYmWWvHAGOMMU8BDwGJgYwvX7AsoCpLnB6lM/eVwBhTC/gQ\nGJU3EveUvL9cL847b/SRMeYCa+33bsdVmDHmRuCgtTbZGBNf2v5+SeDW2t5nCaYjEAesMc7ldFoA\n/zXGXGmtPeiP9y6Ps8VZjKnAJ7iUwEuLM28BVR+gZ0ACOotyfJ5ekwK0LHS/Jc4oXCrIGFMFmAW8\nb6392O14SmKtPWqM+Rrn/IKnEjjQBehnjOkDRAO1jTHvWmvvKG7nSi2hWGvXW2ubWGvjrLVxOL8k\nl7qRvEtjjGlT6G5/nDqe5xRaQNW/pAVUHuO1xVzfAm2MMbHGmKrAQOD/XI4paBlndDYR2GCtfcPt\neIpjjGlojKmbd7s6zqQKz/2OW2uftta2zMuXg4AFZ0veEPgLOnj5T9f/NcasM8asAXrhnAX2ordw\nTrJ+mTfN6O9uB1QcY8zNxpjdOLMSPjHGfOZ2TPmstTnAg8AXOGf6Z1hrPffLbIyZBiwFzjfG7DbG\njHA7prPoCtwO/Drv32Ry3kDDS5oBC/J+v1fi1MCD4fxRiTlTC3lERIKULqkmIhKklMBFRIKUEriI\nSJBSAhcRCVJK4CIiQUoJXEQkSCmBi4gEKSVwEZEg9f8BSjNEmvkmbVUAAAAASUVORK5CYII=\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x7ff8efa06cd0>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"x = np.linspace(-4, 4, 1000)\n",
"plt.plot(x, selu(x), label='SELU')\n",
"plt.plot(x[1:], np.diff(selu(x)) / np.diff(x), label='derivative')\n",
"plt.axhline(0, color='black', linestyle='--'); plt.axvline(0, color='black', linestyle='--')\n",
"plt.legend(loc='upper left')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$s$ is continuous, with a minimum value of $-\\lambda \\alpha$ and no maximum value. $s$ maps positive numbers to positive and negative to negative. $s'$ is not continuous, with maximum value of $\\lambda \\alpha$ at $x=0$ and a value of $\\lambda$ for $x\\ge 0$.\n",
"\n",
"Let's visualize how $s$ transforms a distribution."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAEKCAYAAADticXcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAG6JJREFUeJzt3Xu0XHV99/H3JwG8oUSKT8wFBAssBayE1cYIbRmKIiJN\ndPUpt1J5FCurVKX0olxac7z1QZ9HtJYl6hIwFgEpiOUiJREzCmJJUa5JENAEE0yCcgl3Gsy3f+zf\nJDuTuexzzszZM/t8XmvNyp49+/Kdndnf+c7v99v7KCIwM7NqmFJ2AGZm1jtO6mZmFeKkbmZWIU7q\nZmYV4qRuZlYhTupmZhXipF4BkuqSTi47jnYk1SStKTsOs8nASb0ASaslbZD00ty890laWmZcOZEe\nNoAk7SlpqaSnJa2UdHiHZV8k6UJJGyWtk3R67rUdJV0haZWkzZIObbH+pyX9Oj3O6dd76hVJH5B0\nm6TnJF1UYPnT03HZKOkCSTvlXut4nCW9VtK1kp6Q9CtJn+7Heyqbk3pxU4DTxrsRJT2Ix4bHpcCP\ngV2Bs4ErJO3WZtkR4LeBPYDDgA9Lelvu9R8AJwLrafoil3QKsAD4nfT44zRvwkjaYZSrPAR8Ariw\nwLbfBnwE+CPgNcBrgY/lFml7nFPyXwJ8F5gOzAIuHmWswyEi/OjyAFaRfZgeAXZJ894HLM0tczDw\nX8DjwDLgzbnX6sAngR8CT5OdtJuBvwTuA54APp7m3wJsBL4J7JjWnwZcCzwMPApcA8zKbX8p8N42\nsc8FbkvbXA98NvfavwHrUszfB/bLvfY14IvAd4AngZvJTobPpxhWAgfmll8NnAEsT69fCLwovVYD\n1uSWnQlcmd7Pz4EPdjj2o42j7bbTsfgR8BjwS+BfGsc4vb4ZOCX9nzwGnNeDz86+wHPAy3LzfgCc\n0mb5h4C35J5/HLi0xXJrgD9smncL8L7c8/cCPyoQ40zgGeCVuXlzgF8BU9Pn8nvAr9O8i0nnQe7/\n/sPAXcCzwJQxHKdPABd1WeYS4JO5538ErCtynIH3A9/vR34YtIcr9eJuI0vOf9f8gqRdgevIEs2u\nwLnAdZJemVvsRLIvgpcDv0jzjgAOAuaRfWl8GTgB2B04ADg+LTcFuICsetuD7MQ5r2Dc/wx8LiJ2\nIatsLs+9dh2wN/Aq4CfAN5rW/VOyimc34HmyhHgb8FvAFel95p2Q3tNvk51k/9AcjKQpZF9Kt5Ml\nk8OBv5Z0RIf3UCiOAtt+gezX1m8Bb06vn9q0r3cAv0tW6R7TqJIl/b6kxzo8Dm4T+/7AzyPi6dy8\nO9P85mPzSmBGer3hrlbLtrHfWNaNiF+SHdM/yc0+Afi3iPhNev6pFNvryT6fI02bOQ54OzAtIjan\nZo52x+rqFmEU+fXa6v1NT8et23GeBzwo6Tup6WWppAMK7HPoOKkXF8BHgQ+2+On8DuCnEfGNiNgc\nEZcB9wLzc+t+LSJWptc3pfmfiYinImIFcDdwQ0SsjogngOvJqiUi4tGIuCoinouIp4B/ArZrT23j\nv4F9JO0WEc9ExK1b3lDE1yLi6RTPx4A3Snp5LuZvRcTtEfE8cBXwbERcHFnpc3kjvtzy50XEQxHx\nGFkSOJ7t/R6wW0R8MiJeiIhVwFfJkkIro4mj47Yj4icRsSz9HzwIfKXFcTwnIp6IiDVkv4AOTOve\nHBGv7PC4pU38O5P9SsrbSPbl3mrZxuvdli2yr425bXZzCen/KzUPHpvmERE/i4gbI2JTRPwa+Bzb\nHrcAvpD+759P6xzd4VjNZ3tF+oRavT/Ijk+r4/wEW9//bLLPwT+TfTldB/y7pB0L7HeoOKmPQkQs\nJ2sGOYNtP4Qz2Vp9NzyY5je0Gv2xITf9bIvnOwNIeqmkL6cO241kTSW7FGybP5msal4paZmkd6Rt\nTpV0jqQH0jZXpeXzX1gP56afa3q+Jb427/EXbPv+G14DzMxXbsCZwP/q8B6KxtFx25L2TRXkuvSe\nP0VWteetz00/0+I9diRpuaQnU2fcIWRNRq9oWmwXsoTT7Kn0b375V6RtFPFUi3WfarNss28Bb5b0\nauAPgc0RcTOApOmSLpO0Nh23f2X74zbe0U1FPsut3h9kx6f5tcbrjWP3LHBTRNyQvvD/P9l7eN3Y\nQx5MTuqjtxD4C7KOloaHyBJK3mvS/IbxjE75W7LEPDc1oxxKdhJ0PREi4oGIOCEiXgV8mqzz6CVk\nP6/nA4enbe6VVhlPJ+4eTdO/bLHMGmBVU+X2iog4ehz7Lbrt84EVwN7pPZ9N93MgACT9QUrW7R6H\nAETE/hHx8rTfH6b9vVZS/svhjWR9D9vuKPuFs4706yC37D0F3//ysa6b9r2YrEI/gazTseGfgN8A\nB6Tj9udsf9yaO22v73CsrmsVQoEwW72/DSn25XQ+zvlmm8avkUpyUh+liPgZWSdmfiTM9cC+ko6X\ntIOkY8kqgGtzyxT5EKnN9M5klcbG1H6/sMu6W2dKJ0p6VXq6kezk2Zy2+TzwqKSXkZ24XbfXJfZT\nJc1KMZ4NXNZiuWXAk5I+LOkl6RfDAZJ+t8N2i+q27Z3JKrdnJL2OrKO623sSQETclJJ1u8cPW20g\nIu4D7gAWSnqxpHcBbyDrzG3l68A/SJqWYnwfWWdxFlA25PHF6Wl+urHu30iaKWkm8DdN666W9O4O\n7/cS4CSytvVLcvN3Juvgf0LSLODvO2wDgIh4e4dj9Y5cTFPTe9gBmJre39Q2m/06cLKk10uaRtZn\nc1HaX7fjfDEwT9Lhaft/Tdbpu7Lbexk2Tupj83HgpaTqIiIeAY4mq6h/TdaZenREPJpbp7kSaVWZ\nRNN04/nngZekbd9C9iVSZHsAbwPukfQkWVvocand8+tkTUQPkVVzP+qw/1bPW8V7CVm19zPgfrIR\nP9ssmzrejiaruH5OdmJ9he1/Oo86jgLb/juyKvSJNP+yFtvutO+xOo6s8/VRsi/PP0mfGST9maR8\nNb2Q7Pg9SNYx/5mIWJx7/adkzUIzgRuApyXtARARXybrKL47Pa6NiK+k/exE1on/nx3ivJqs43xd\nRNydm/8xsg79jWn7V9K76yL+Mb2fj5ANJniWrCBA0h6psp+d3t8NwGfI+joeJGsyzBc4bY9zSvon\nAl9Kr/8xMD8iXujR+xgYyvqa2ryYfYN+H3gR2TfpFRExImkvshNiV7JxoX8eEZskvYgsWRxENvzv\n2NQhZRUnaRVwckR8r+xYbHupeejUiPizsmOx/upYqUfEc8BhEXEgWfVzpKQ3kbXNfjYi9iEbz9u4\nRP1k4JE0/3NpOTMrWUT80Al9cuja/BIRz6TJnYAdyX52HUY2PhhgEfDOND0/PYfsJ1rby6HNzKz3\nuiZ1SVMk3UE23K7RXvp4RGxOizzE1pEgs0hDm1JbVaNjzyouIvZy04tZ+YpU6ptT88ts4E1UcFyn\nmVlVFL75TkRsVHZXwjcD0yRNSdX6bGBtWuwh0vhkZTf22aVpBAgAknxHQTOzMYiIjsN8O1bqknZL\n40FJF6y8lWxc51Ky+3FANq7139P01ek5wP8GbuwQ2MA/Fi5cWHoMVYlzGGJ0nI5z0B9FdKvUZwCL\n0mD9KcA3I+I6SSuAyyR9kuxGUBek5S8A/lXS/WRDGtvdz8PMzPqgY1KP7AKEg1rMX0XWvt48/3ng\nmJ5FZ9ZjzVeHF61+zIaFryjtoFarlR1CIcMQ52DF2P5C0cGKsz3H2VvDEmcRHa8o7dtOpXCFZGXI\nKvXGZ0+u1G2oSCLG01FqZmbDxUndzKxCRvtHYs0GljtBzVypW+X06m65ZsPJlbpZB67+bdi4Ujfr\nytW/DQ8ndTOzCnHzi9ko5Jtj3BRjg8iVutmouCnGBpuTuplZhTipm5lViNvUzfrEwyGtDE7qZvQz\nAW+9eZjZRHDzi9kW7gS14eekbmZWIU7qZmYV4qRuZlYh7ii1ymvuBC362mi265EtNihcqdsk0a4T\ndDydo+5YtcHjSt2sB1y126BwpW7WE67abTA4qZuZVYiTuplZhTipm5lViJO6mVmFePSLDYWx3HBr\nPGPQzYZVx0pd0u6SlkpaLukeSR9K80ckrZV0e3q8PbfOmZLul3SvpCP6/QZsMhntCBOPSLHJp1ul\nvgk4PSLukLQz8GNJS8jOlHMj4tz8wpL2A44F9gNmAd+VtG9EbO5D7GZm1qRjpR4R6yPijjT9FLCS\nLFlD6xtELwAujYhNEbEaeACY27twzcysk8IdpZL2BOYA/5lmfUDSnZIukDQtzZsJrM2ttpatXwJm\nQ0NSz9vkG9tsfpj1UqGknppergBOSxX7+cBrgQOBdcBnO6zuRk0bQv1oj89v0+391h9dR79I2hG4\nErg4Ir4NEBEP517/KnBNevoQsHtu9dlp3nZGRka2TNdqNWq12ugiNzOruHq9Tr1eH9U66jQ0TNlv\nw0XAIxFxem7+jIhYl6ZPB34vIk5IHaWXkLWjzwK+C+wdTTuR1DzLrKPso7j17322+vw0L1PWdCO2\novH4XLCiJBERHdvsulXqhwAnAndJuj3NOws4XtKBZJ/MVcApABGxQtLlwArgBeBUZ28zs4nTsVLv\n205dqdsouVI3602lbjaQPGrErDXf+8WGlEeSmLXipG5mViFO6mZmFeKkbmZWIe4oNesxd+JamZzU\nbWANb3LMD100m1hufrEB55EtZqPhpG5mViFO6mZmFeKkbmZWIe4oNStZvkPY94Gx8XKlblY6dwZb\n7zipm5lViJtfbKAM79h0s8HgSt0GkJsjzMbKSd3MrEKc1M3MKsRJ3cysQpzUzcwqxEndzKxCnNTN\nzCrESd3MrEKc1M3MKsRJ3cysQpzUzcwqxEndzKxCfEMvK51v4mXWO67UbUD4Jl5mvdAxqUvaXdJS\nScsl3SPpQ2n+rpKWSLpP0mJJ03LrfEHS/ZLulDSn32/AzMy26lapbwJOj4j9gXnAX0l6PXAGsCQi\n9gVuTM+RdBSwd0TsA7wfOL9vkZuZ2XY6JvWIWB8Rd6Tpp4CVwCxgPrAoLbYIeGeaXtCYHxG3AtMk\nTe9D3GZm1kLhNnVJewJzgFuB6RGxIb20AWgk7pnAmtxqa4HZ447SzMwKKTT6RdLOwJXAaRHxZNNf\nPw9J+R6u5qEMLXu/RkZGtkzXajVqtVqxiM3MJol6vU69Xh/VOoroPOJA0o7AtcD1EfH5NO9eoBYR\n6yXNAJZGxOskfQmoR8RlueUOzVX1jW1Gt/3a5JEVCY3Pw2Sf3srniDWTRER0HAPcbfSLgAuAFY2E\nnlwNnJSmTwK+nZv/7rTuPODx5oRuZp14aKeNT8dKXdLvAz8A7mLrJ+1MYBlwObAHsBo4JiIeT+uc\nBxwJPA28JyJ+0mK7rtRtC1fqrad9jlizIpV61+aXfnBStzwndSd1K6ZIUvdtAmzC+HYAo9M0IKHE\nSGyY+DYBNsHybcZuP+7Mx8dGz5W62RBp/rXjCt6auVI3Gzqu4K09J3UzswpxUjczqxAndTOzCnFS\nNzOrEI9+MRsCHuNvRblSNxsKHvFixTipm5lViJtfrOd8gYxZeVypW5+4ucCsDE7qZmYV4qRuZlYh\nTupmZhXipG5mViFO6mZmFeKkbmZWIU7qZmYV4ouPzIaY/46pNXOlbjbUfJGXbcuVuvWd7zBoNnFc\nqdsEcDVpNlGc1M3MKsRJ3cysQpzUzcwqxEndzKxCuiZ1SRdK2iDp7ty8EUlrJd2eHm/PvXampPsl\n3SvpiH4FboNF0paHmZWnSKV+EXBk07wAzo2IOelxPYCk/YBjgf3SOl+U5F8Dk4ZHuZiVrWvCjYib\ngMdavNSqJFsAXBoRmyJiNfAAMHdcEZqZWWHjqaI/IOlOSRdImpbmzQTW5pZZC8waxz7MzGwUxnpF\n6fnAx9P0J4DPAie3Wbbl7/GRkZEt07VajVqtNsZQzMyqqV6vU6/XR7WOitwESNKewDUR8YZOr0k6\nAyAizkmv/QewMCJubVonfPOhask6SBv/p54uY9rnVPVJIiI6jkYYU/OLpBm5p+8CGiNjrgaOk7ST\npL2AfYBlY9mHmZmNXtfmF0mXAocCu0laAywEapIOJCsTVgGnAETECkmXAyuAF4BTXZKbmU2cQs0v\nPd+pm18qx80v5U/7nKq+Is0vvvWuWcU0XwDmZD+5+MIgs0ryhWCTlSt1s4rwLRoMXKmbVYirc3NS\nNzOrFCd1M7MKcVI3M6sQJ3UzswpxUjczqxAndTOzCnFSNzOrECd1M7MKcVI3M6sQJ3UzswpxUjcz\nqxAndTOzCnFSNzOrEN9610bFt3c1G2xO6jYG7f60mg2i/Bex/wpS9bn5xazyfJ/1ycRJ3cysQpzU\nzcwqxEndzKxCnNTNzCrESd3MrEKc1M3MKsRJ3cysQnzxkdkk13yVsC9QGm5dK3VJF0raIOnu3Lxd\nJS2RdJ+kxZKm5V77gqT7Jd0paU6/AjezXvIFSlVRpPnlIuDIpnlnAEsiYl/gxvQcSUcBe0fEPsD7\ngfN7GKuZmXXRNalHxE3AY02z5wOL0vQi4J1pekFjfkTcCkyTNL03oZpZr0ja8rBqGWtH6fSI2JCm\nNwCNxD0TWJNbbi0we4z7sAHhBFBVbnKponF3lEZESMp/MprP/JafmpGRkS3TtVqNWq023lBsDDol\n6m07zHw3RrOJVq/Xqdfro1pHRXq6Je0JXBMRb0jP7wVqEbFe0gxgaUS8TtKXgHpEXJZb7tBcVd/Y\nXriHfTBkSb31rXQb/0edlvH0cE0X+T/1uTm4JBERHSursTa/XA2clKZPAr6dm//utPN5wOPNCd3M\nzPqna/OLpEuBQ4HdJK0BPgqcA1wu6WRgNXAMQER8R9JRkh4Angbe06/AzWz03C9SfYWaX3q+Uze/\nDAw3v3jazS/Do5/NL2ZmNoB8mwBryz/VzYaPK3XrwOOYzYaNk7qZWYU4qZuZVYiTuplZhbij1My2\nke8g9/DG4eNK3cyauIN8mDmpm5lViJO6mVmFOKmbmVWIk7qZWYU4qZuZVYiTuplZhTipm5lViJO6\nmVmFOKmbmVWIbxMwCfk+6WbV5aQ+aeX/lJlZa74PzPBx84uZdeD7wAwbJ3UzswpxUjczqxAndTOz\nCnFSNzOrEI9+MbNCPBJmOLhSN7OCPBJmGLhSr7jmC41cYZlVmyv1ScEVltlkMa5KXdJq4AngN8Cm\niJgraVfgm8BrgNXAMRHx+DjjNDOzAsZbqQdQi4g5ETE3zTsDWBIR+wI3pudmZjYBetH80nzzkPnA\nojS9CHhnD/ZhoyBpy8PMJpdeVOqLJd0m6S/SvOkRsSFNbwCmj3MfNiZuR7eJky8kXEyUa7yjXw6J\niHWSXgUskXRv/sWICEnOLGYV03rMuu/8OQjGldQjYl3691eSrgLmAhskvToi1kuaATzcat2RkZEt\n07VajVqtNp5QrCBXUdYbTuAToV6vU6/XR7WOxjpuWdJLgakR8aSklwGLgY8BbwEeiYhPSzoDmBYR\nZzStGx4v3T9Z4s6fdM3T3V73tKeLT0fEdp85n9/9IYmI6PhNOp5KfTpwVar8dgC+ERGLJd0GXC7p\nZNKQxnHsw8wGnH/9DZYxV+rj2qkr9b5ype7piZ92pT4R+l2pW8l8CwAza+bbBAw9D100s61cqZtZ\n3/jX5MRzpW5mfeZfkxPJlbqZ9ZxHxJTHlbqZ9YGr87I4qZuZVYiTuplZhbhNvULcjmlmrtQrxe2Y\nZpOdK3UzmzCtbtnrsey95aQ+BPyht+pod8te38q3V9z8MjTctGJm3blSH0LuEDWzdpzUB1TnxO2f\nqmbWmpP6QHPyturyL87+cJu6mZXE/UT94ErdzAZKq2GPVpyT+gDxz1EzcLPj+Dip99HYxpf7A21m\nY+ek3ndO0mbj5QvwinNHqZkNCXesFuFKvQRuOzcbH3emtuekXpp8s4ybaMxGZ+s54wS/LSf1CeQK\n3awfXBTluU19QrlN0Mz6y0ndzKxC3PxiZgOrn02WVR0m2ZdKXdKRku6VdL+kj/RjH/0kaZtHkWVa\nPcxsvEbXZDn68696TaI9T+qSpgLnAUcC+wHHS3p9r/fTa9t/GAJYWmCZ6DI9EeoTuK+xqpcdQEH1\nsgMoqF52AANq67k3WQusflTqc4EHImJ1RGwCLgMW9GE/fdCcjOsAHRL5oKiXHUAB9bIDKKhedgAF\n1csOYAhMzgTfjzb1WcCa3PO1wJt6tfFly5Zx8803b3k+b948Dj744I7rdPqP9P1YzCaD7udwL/4o\n9iC00/cjqff1Xdx4442cddZZ7Xfe9iC2vthnMnxzm9lW7c/54n8Ue5D/Mlk/kvpDwO6557uTVevb\n6Fcybb9dtZkuskwVp8vev6erN132/otO03WZbfNI0WQ+1uV7S73+eSBpB+CnwOHAL4FlwPERsbKn\nOzIzs+30vFKPiBckfQC4AZgKXOCEbmY2MXpeqZuZWXlKvU2ApL+VtFnSrmXG0Y6kT0i6U9Ltkm6Q\nNKPsmFqR9P8krUyxfkvSLmXH1IqkP5W0XNJvJB1UdjzNhuGiOUkXStog6e6yY+lE0u6Slqb/73sk\nfajsmJpJerGkWyXdkWIcKTumTiRNTbnomk7LlZbUJe0OvBV4sKwYCvhMRLwxIuYA1wIfLTugNhYD\n+0fEG4H7gDNLjqedu4F3AT8oO5BmQ3TR3EVkMQ66TcDpEbE/MA/4q0E7nhHxHHBYRBwIHAgcKaln\nw6/74DRgBV1GGJZZqZ8LfLjE/XcVEU/mnu4MbC4rlk4iYklENGK7FZhdZjztRMS9EXFf2XG0MRQX\nzUXETcBjZcfRTUSsj4g70vRTwEpgZrlRbS8inkmTOwE7MqDnuKTZwFHAV+kyVrKUpC5pAbA2Iu4q\nY/+jIelTkn4BnMDgVup57wW+U3YQQ6jVRXOzSoqlUiTtCcwhKzgGiqQpku4ANgCLI+K/yo6pjc8B\nf0+BL52+3aVR0hLg1S1eOpuseeCI/OL9iqObDnGeFRHXRMTZwNmSzgA+CIxMZHwN3eJMy5wN/HdE\nXDKhweUUiXNAecRAH0jaGbgCOC1V7AMl/cI9MPVDXSVp/4hYXnZceZKOBh6OiNsl1bot37ekHhFv\nbTVf0gHAXsCdaWD+bODHkuZGxMP9iqeddnG2cAlwHSUl9W5xSvo/ZD/PDp+QgNoYxfEcNIUumrPi\nJO0IXAlcHBHfLjueTiJio6SlZP0VA5XUgYOB+ZKOAl4MvELS1yPi3a0WnvDml4i4JyKmR8ReEbEX\n2YlzUBkJvRtJ++SeLiBrFxw4ko4k+2m2IHX+DINBuz/DbcA+kvaUtBNwLHB1yTENLWUV2wXAioj4\nfNnxtCJpN0nT0vRLyAZuDNw5HhFnRcTuKV8eB3yvXUKHwfjLR4P8s/f/Srpb0p3AW8h6nwfRv5B1\n5C5JQ56+WHZArUh6l6Q1ZKMhrpN0fdkxNUTEC0DjorkVwDcH8aI5SZcCtwD7Sloj6T1lx9TGIcCJ\nwGHpM3l7Kj4GyQzge+n8XkbWpj4M/VEdc6YvPjIzq5BBqNTNzKxHnNTNzCrESd3MrEKc1M3MKsRJ\n3cysQpzUzcwqxEndzKxCnNTNzCrkfwDraXTjYdfY4QAAAABJRU5ErkJggg==\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x7ff918139b90>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"z = np.random.randn(10000)\n",
"plt.hist(z, bins=100)\n",
"plt.title('Normal sample mean=%.3f, var=%.3f' % (z.mean(), z.var()))\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAEKCAYAAADticXcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGZFJREFUeJzt3X/0ZHV93/Hni9+RZVkQWfYXWQmQCDEu0RJasHyt1a6c\nBPD0iEKNQuKvQzh4NLUF28qX2BhsT5BQC5qKAv4AUQKCFbMbZaqGCCXl94JA64ZdAouYBXZXDbvs\nu398PrN7d3a+8+M7M9+Z+czrcc495879+bl3Zt7zvp/P595RRGBmZmXYY9gFMDOz/nFQNzMriIO6\nmVlBHNTNzArioG5mVhAHdTOzgjioW19IWi5puyR/psyGyF/AHkk6SdIdkp6T9FNJP5D0ujzvbEkv\nSdpUGV6QdFiev1bSG5tsc7ukIxqmTUv64twclXVL0sGSbpK0Ob+vZ7ZZ/pOSns3DJQ3zVkj6W0lb\nJN0t6TWVeQskXSNpQx4uGtQx9YukM/J3ZIuk2ztY/ixJf5fP5U2SDqrMm/E8SzpM0i2SnszfocMH\ndUyjzEG9B5LmA98E/gw4CFgCXAz8Y2Wxv46IAyrD/Ih4Os+LPHTCd4mNtv8O/AI4FPg3wJWSjmm2\noKT3A6cBv5GH38nTkLQP8A3gWmABcA3wDUl75dU/BewH/DJwPPC7ks4e0DE1payLVX4KXApc0m5B\nSccCnyGdw4XAz4ArKou0Os/bgW8B/7qLspUnIjzMcgBeB2xsMf9s4Pst5v8Y+BdNpm8HjmiYNg18\ncYbtHAn8L+A54CfA9ZV5fwY8ATwP3A2c1LDNrwFfBF4A7geOAi4ENuT13lRZvgb8CXBn3t7NwEF5\n3vJc7j3y6wOBq4C/B9YDH6/Pa1L+bssx47aBXwG+Czybz8WXgAMr664F/hC4L5+v64F9e/wc7E/6\nIT+yMu1a4E9mWP4O4D2V178H/E0efzOwvmH5vwPenMd/Ary2Mu9C4HsdlHHffLzHVqa9ghQ0DyEl\nJd8EngH+AbgVWNLw3v9n4K/zOke022eTMrwHuL3NMp8AvlR5fUQ+t/t3ep6BvfJn8fBev+PjODhT\n782PgJckXS1pZfUycY59HPh2RCwgXS1cXpl3F/Aa0pf2K8DXcjZY99ukL8ZBwD3AX+bpi4E/Aj7b\nsK/fBc4BFgHbGvZVdTXwIinIHkcKVu9pcQzdlKPdtv84l+9VwDLSj0ZdAG8D/hXwSlKmfDaApMMl\nbWwxvGOGsh8NbIuIxyvT7gOOnWH5Y/L8uvsryx6bXzPDfNj1CnsP4Ndn2M8OEfGPwI1AtVroDKAW\nEc8CIv1QHp6HnwOfbtjMO0nneR7whKQrWpyre9uVaQa7nJuI+H+k9/pouj/PE8lBvQcRsQk4iRQo\n/gfwjKRvSDq0stgJDR/2xwZQlBeB5ZKWRMSLEXFHpYxfjoiNEbE9Ii4lZWy/Wln3exGxOiJeAr5O\nyt4uya+/mrc7v7454NqIWBMRPwP+E3BG46W4pIXAW4APRcTPI+InwGXATEGx43K023ZE/N+I+E5E\nbM3B6lPAyQ37ujwino6IjaSMdEVe94mIOKjFcP0MZZ9HusKoegE4oMXyz1deP5+nNZtXn1/f1reB\nCyTNk3QkKcv/pRn20+gr7PoenJWnERH/EBE3RcQvImIzKWOunrcAro6Ih/NnaVtEnNviXK3osEyN\nWh1/t+d5Ijmo9ygiHomIcyJiGSljWkwKMnU/bPiwH9XBZl8C9m6YtjewdYbl/x0p07pL0oOSzqnP\nkPRvJa3JDbkbSVUXh1TWfaYy/nPg2cjXsPk17Aw4AOsq40/kclW3B6m+d2/gqfqPGame9BUzlL+b\ncrTctqSFkq6XtF7S86QqnZc37OvpyvjPG46vLUm3VRq+zwQ2A/MbFpsPbJphE43Lz8/Tms1r3Nb5\nucyPATeRgvKTHRa9BrxM0vGSlpOu4G7Kx/QySZ/NjY/Pk6rzDmz4wV7H4G0mfUar6sff7XmeSA7q\nfRQRPyI1bLW9HG7jCVLVQNUrSfXBzfa7ISLeFxFLgPcDV0g6QtLrgY8Ab4uIBRFxECnr6aaRq9Hh\nDeNbSfXXVetIdZ8vr/yYHRgRr55hm900Arfb9idIP4q/HhEHkqqLWn3Od+w7V79sajGcCRARb4md\nDd/XAY8Ce+XMue41wIMz7PMh8tVBk2UfIlUJVf1Gnk6+6npnRCzKx7wnqY2jrXzVcwOpCuZM4NaI\n2JJn/yGpeuP4fN5OJn1Oqp+VXd4nSZ9pca4eaFaEDor5EOl81PdxBOnq8lG6P88TyUG9B5J+VdKH\nJS3Jr5eRvix/08Vm9pG0X2XYi1Td8B8lLZG0h6R/Sapz/voM5XibpKX55XOkL8920mXpNuBZSftI\n+hi7ZzrdEPBOSa+S9DJSXffXKhk1ABHxFLAKuFTSAfkYfkXSP2+x3Y50sO15wBbghfy+fKSDY6pv\n+4nYtadS43DdDGXaAvwF8Ec54z0ROJV0ldDMtcCHJS2WtBj4MKmdAFI2/ZKk8yXtK+k80vv5XUhB\nTtLLJe0p6S3Ae0kNmOT5NbXu5livgtlR9ZLNI10BPC/pYKDZNnZ5nyLiAy3O1Y4f8Pwe7Ue6wtoj\nH1fjlWjdl0m9gU6StD/pM3ZjRGzp5Dzn/eyXX+6XX08UB/XebAJ+C7hT0mZSML+flPVA+jL+0yZZ\nzGsr2/gWqTdBffgY6YN8B/ADUk+ES4CzImLNDOV4HfBDSZtI3eHOj4i1pPrXb5MynLWkL+0TlfWa\ndals9TpIX6CrgaeAfUjVAc2WfVeevyYfw9eAw2Yof7flaLXti4HfJF2R3EpqHGyVIXbTrbSVc0l1\n28+QAtMHIuJhAEmvz+9N2mHEZ3PZHsjDNyPiz/O8F4HT8zFuJDXinh4R2/LqryV9xl4gNQifVd9P\ntpT0uWkqIu4iVWMsAm6rzLosl/9Z0mfvNtq/J516Fzu7Jr6e9Dnc0fCdvxMn5vKtAT5AOocbSD1e\nzq1sa8bznP2MdG4CeIT0Az9R1JBk7TozZZ7XkvqEBvDnEXG5pGlSK/hP8qIfjYjb8joXkhpvXiIF\nl1WDK77NJaUbR74YEZ8fdllsd/lq7fqIOGnYZbHh2avN/K2kXgb3SpoH/K2k1aQAf2nuTbGD0k0A\nbyd1S1oC/JWkoyNi+wDKbsPRS328DVBErCf1xrIJ1rL6JXf7ujePbwYeJgVraP7lPg24LncnWws8\nTrrrzcrhO1vNRljHdeq5C9RxwA/zpPMk3SfpKkkL8rTFpDv86taz80fAxlxEvMFVL2ajraOgnqte\nvg58MGfsV5Ju311BajD70xarO7MzM5sj7erUyV2PbiQ9j+FmgIh4pjL/c6SWfEg3QSyrrL6UJjdG\nSHKgNzObhYho2a7VMlPPd5NdBayJiMsq0xdVFnsrqVsWwC3AO3Kf6FeSHsp01wwFK3a46KKLhl4G\nH5uPz8dX3tCJdpn6iaSH+Nwv6Z487aPAmZJWkKpWfky6i5GIWCPpBlL/4W3AudFpSczMrGctg3pE\n/IDm2fxtTabV1/kE6VZtMzObY76jdACmpqaGXYSBKfnYwMc37ko/vk60vKN0YDuVXCtjZtYlSUQv\nDaVmZjZeHNTNzArioG5mVhAHdTOzgjiom5kVxEHdzKwgDupmZgVxUDczK4iDuplZQRzUzcwK0vZ5\n6tad9LTinfw4BDObS87UByLwHz6Z2TA4qJuZFcTVL224OsXMxokz9Y64OsXMxoODuplZQRzUzcwK\n4qBuZlYQB3Uzs4I4qJuZFcRB3cysIA7qZmYFcVA3MyuI7yi1vvHdt2bD50zd+sx335oNk4O6mVlB\nHNTNzArioG5mVhA3lM6SGwXNbBQ5U++JGwXNbLQ4U7eR46sgs9lzpm4jyldBZrPhoG5mVhBXvwyZ\nqxrMrJ9aZuqSlkm6XdJDkh6UdH6efrCk1ZIelbRK0oLKOpdLekzSfZKOG/QBlMFVDWbWH+2qX7YC\nH4qIY4ETgD+Q9CrgAmB1RBwNfCe/RtIpwJERcRTwPuDKgZXczMx20zKoR8TTEXFvHt8MPAwsAU4F\nrsmLXQOcnsdPq0+PiDuBBZIWDqDcNgYk7RjMbG503FAqaTlwHHAnsDAiNuRZG4B64F4MrKusth5Y\n2nMpbUy5WslsrnXUUCppHnAj8MGI2FTNvCIiJFW/uY1pWdNv9fT09I7xqakppqamOivxkHWbdTac\nq34Xx8wKVqvVqNVqXa2jdoFG0t7AN4HbIuKyPO0RYCoinpa0CLg9In5N0meAWkRcX1nu5EpWX99m\njEuAS0G5XtaZx+vH02r5ZsfcuPy4nJdmZnfsO810Dsf5nJj1kyQiomVm2a73i4CrgDX1gJ7dArw7\nj78buLky/V153ROA5xoDutmuXEVj1k8tM3VJJwHfA+5n5zfvQuAu4AbgcGAtcEZEPJfX+TSwEtgC\nnBMR/6fJdp2pz7D9cTkvzfTr2Es6J2b91Emm3rb6ZRAc1Gfe/ricl2Zan6ud2gXvks6JWT/1XP1i\n1j+uZjGbC35MgPVkWH3Q/XgFs+Yc1OdQud0bm1ezlLtfs9Hl6pc55SoIMxssZ+pD4lvnzWwQnKkP\njbN2M+s/B3Uzs4I4qJuZFcRB3cysIA7qZmYFce8Xm3Pu+WM2OM7UbQjc88dsUJypNzFqmeSk3xJf\n7p24Zv030UG9dbActVvQR608c2mSj92sOxMd1JOdAWPUMvSZTHLmOi7vkdmwTFxQbx0UxiUjHJdy\nDsIkH7tZexMX1BMHhl44WzYbXRMa1Ptv8gKdfxjNRpG7NPaNu+mZ2fA5qJuZFcTVLxNq0vu+m5XK\nmfpEc5WRWWkc1M3MCuLqFyvCJN+QZVbloD5iHJxmy10szcBBfQSN32MLzGx0uE59pLkh08y646Bu\nZlYQV79YcdwuYZPMQd1GRv/aENxoapPLQX3MlZWVOhib9cp16mPPjalmtpODuplZQVz9UiA/rMts\ncjlTL5arZcwmUdugLunzkjZIeqAybVrSekn35OEtlXkXSnpM0iOS3jyogpuZ2e46ydS/AKxsmBbA\npRFxXB5uA5B0DPB24Ji8zhWSfDVgZjZH2gbciPg+sLHJrGb9zk4DrouIrRGxFngcOL6nEpqZWcd6\nyaLPk3SfpKskLcjTFgPrK8usB5b0sA8zM+vCbIP6lcARwArgKeBPWyzr1jozszkyqy6NEfFMfVzS\n54Bb88sngWWVRZfmabuZnp7eMT41NcXU1NRsimJmVqxarUatVutqHXXSh1nScuDWiHh1fr0oIp7K\n4x8C/klEnJUbSr9CqkdfAvwVcGQ07ERS46Q5k/pwV29HL2e8fk4bj7HZue5kmZmM0zl0H30riSQi\nouVzNNpm6pKuA04GDpG0DrgImJK0gvTt+THwfoCIWCPpBmANsA04d2jRu8J/NmGNfIOWlaqjTL3v\nO53jTH2cMktn6v0dn+m4ejl+s2HpJFN3H3Izs4L42S8FcTXT7sp6NLFZew7qRRnc88jH9wfDz2i3\nyeLqF+uCHxJmNuoc1M3MCuKgbmZWENep2y7cf9tsvDmoT4Due4C4cdFsXLn6ZSK4gdNsUjiom5kV\nxEHdzKwgrlO3idHJDVRuKLZx56BuE6QaoHcN3rsGczcU2/hy9YsZ4MZkK4WDuplZQRzUzcwK4jp1\nsxb86F4bNw7qE2Z8H6E7LG40tfHi6peJ4wZBs5I5qJuZFcTVL9aSq2t2cv26jQNn6taGq2t28rmw\n0eegbmZWEAd1M7OCOKibmRXEQd3MrCAO6mZmBXGXRgPcddGsFM7ULXN3PbMSOKibmRXEQd3MrCAO\n6mZmBSm2odQNf2Y2iQrP1N34Z2aTpfCgbmY2WRzUzcwK0jaoS/q8pA2SHqhMO1jSakmPSlolaUFl\n3uWSHpN0n6TjBlVwMzPbXSeZ+heAlQ3TLgBWR8TRwHfyaySdAhwZEUcB7wOu7GNZzcysjbZBPSK+\nD2xsmHwqcE0evwY4PY+fVp8eEXcCCyQt7E9RzcysndnWqS+MiA15fANQD9yLgXWV5dYDS2e5DzMz\n61LP/dQjIiRV+w02dhB3n0IrVuP9EP7vUhu22Qb1DZIOi4inJS0CnsnTnwSWVZZbmqftZnp6esf4\n1NQUU1NTsyyK2bDVA7lveLP+qtVq1Gq1rtZRJ5mFpOXArRHx6vz6vwA/jYhPSroAWBARF+SG0vMi\n4hRJJwCXRcQJTbYXg85oUgZV/bJ53OONF5Sz31b987v752wnZ+3Wb5KIiJbZQ9tMXdJ1wMnAIZLW\nAR8DLgFukPT7wFrgDICI+JakUyQ9DmwBzuntEMxG08yPoXDWbsPVUabe9506U/f4UMfnZn/O1K3f\nOsnUfUepmVlBHNTNzArioG5mVhAHdTOzghT1Jxn+Ywwzm3QFZur+Ywwzm1xFZepmo6R65ejujTZX\nCszUzUaFrxpt7jmom5kVxEHdzKwgDupmZgVxUDczK4h7v5iNAP/ZhvWLM3WzkeHeMtY7B3Uzs4I4\nqJuZFcRB3cysIA7qZmYFcVA3MyuIg7qZWUHcT91sDviJjTZXHNTN5kQ9kMsB3gbK1S9mc843Gdng\nOFM3GyL/BaP1mzN1s6Fy1m795aBuZlYQB3Uzs4I4qJuZFWTsG0rd0GQlcrdHm61CMnU3Nllp/Jm2\n2SkkqJuZGTiom5kVxUHdzKwgDupmZgVxUDczK8jYd2k0K527N1o3egrqktYCLwAvAVsj4nhJBwNf\nBX4ZWAucERHP9VhOswm287G9Zu30Wv0SwFREHBcRx+dpFwCrI+Jo4Dv5tZmZzYF+1Kk3pg+nAtfk\n8WuA0/uwDzMz60A/MvVVku6W9N48bWFEbMjjG4CFPe7DzDJJOwazZnptKD0xIp6S9ApgtaRHqjMj\nIiQ1bdmZnp7eMT41NcXU1FSPRTGbBK5fnyS1Wo1ardbVOupXa7qki4DNwHtJ9exPS1oE3B4Rv9aw\nbPRxv+z6Qfe4x9uNj0o5eht3T5jJI4mIaPmLPuvqF0kvk3RAHt8feDPwAHAL8O682LuBm2e7DzMz\n604v1S8LgZty3d5ewJcjYpWku4EbJP0+uUtjz6U0M7OO9K36pauduvrF40MdH5VyuPrFutNJ9Yvv\nKDUbU77T1Jrxs1/Mxlaw69WHmYO6mVlRXP1iVoBuqmIab1xy1U1ZHNTNirCzAbWzAF9tfLWSuPrF\nrDiua59kDupmZgVxUDczK4iDuplZQdxQajYB/KjeyeFM3WxiuAF1Ejiom5kVxEHdzKwgrlM3m3B+\nMFhZnKmbTTzXtZfEQd3MrCBjF9QvvvjjHHDAoRxwwKHMn3/osItjViRJuww2PsYuqG/atIXNm9/H\n5s0PsnnzWcMujtlI6zYo77q8q2XG0dgF9WQecCiw/7ALYjbiug3MDuTjzr1fzGxW/Fz20TSmmbqZ\njQZn9qPGQd3MrCCufjGzjrknzOhzpm5mXXKVyyhzUDczK4iDuplZQVynbmZtdVKXPtMy7uo4txzU\nzawD9cDcKrhXl+lk+byE+7v3latfzGwEuPG1X5ypm9lI8fPde+NM3cxGjLP2XjhTN7M555uYBseZ\nupkNiTPyQXCmbmZzYljZ+aT1rnFQN7OB2jWodt7Vsb+Gtd+5N5DqF0krJT0i6TFJ/34Q+zCzceFq\nlrnU96AuaU/g08BK4BjgTEmv6vd+Rltt2AUYoNqwC2A9qQ27AF1p/K/UZkNVrVYbTkFHyCAy9eOB\nxyNibURsBa4HThvAfkZYbdgFGKDasAtgPakNuwBdqmb5zcerAd5BfTBBfQmwrvJ6fZ5mZjYAOwP8\nxRdf3PbPtmfK8vut1RXFIA0iqA+08kyCffe9mvnzf4d99vnqIHdlZmPnItqHoOZZfl0nVT2dVgcN\noz1B/e7eI+kEYDoiVubXFwLbI+KTlWXcamJmNgsR0TLtH0RQ3wv4EfBG4O+Bu4AzI+Lhvu7IzMx2\n0/d+6hGxTdJ5wF8CewJXOaCbmc2NvmfqZmY2PEN79ouk/yrpYUn3SfoLSQcOqyz9Jultkh6S9JKk\n3xx2efql5JvKJH1e0gZJDwy7LIMgaZmk2/Pn8kFJ5w+7TP0iaT9Jd0q6Nx/b9LDLNAiS9pR0j6Rb\nWy03zAd6rQKOjYjXAI8CFw6xLP32APBW4HvDLki/TMBNZV8gHVuptgIfiohjgROAPyjl/YuIXwBv\niIgVwApgpaTfGnKxBuGDwBradKcZWlCPiNURsT2/vBNYOqyy9FtEPBIRjw67HH1W9E1lEfF9YOOw\nyzEoEfF0RNybxzcDDwOLh1uq/omIn+XRfYC9ge0tFh87kpYCpwCfo80DbEbl0bu/B3xr2IWwlnxT\nWSEkLQeOIyVTRZC0h6R7gQ3Aqoj438MuU599CvgIHfxYDfQpjZJWA4c1mfXRiLg1L/MfgBcj4iuD\nLEu/dXJshXGLegEkzQO+DnwwZ+xFyFf9K3Lb3E2Sjo2Ih4Zdrn6Q9NvAMxFxj6SpdssPNKhHxJta\nzZd0NumS4o2DLMcgtDu2Aj0JLKu8XkbK1m1MSNobuBH4UkTcPOzyDEJEPC/pdlL7SBFBHfhnwKmS\nTgH2A+ZLujYi3tVs4WH2fllJupw4LTd0lKqUBzjfDRwlabmkfYC3A7cMuUzWIaX7168C1kTEZcMu\nTz9JOkTSgjz+S8CbSG0GRYiIj0bEsoh4JfAO4LszBXQYbp36fwPmAatzN50rhliWvpL0VknrSL0M\n/qek24Zdpl5FxDagflPZGuCrJd1UJuk64A7gaEnrJJ0z7DL12YnAO4E35O/bPTmxKsEi4LuS7iPd\nwb4qIkpuo2tZFeqbj8zMCjIqvV/MzKwPHNTNzArioG5mVhAHdTOzgjiom5kVxEHdzKwgDupmZgVx\nUDczK8j/B6tkt4OzDIr9AAAAAElFTkSuQmCC\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x7ff8ed326290>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"h = selu(z)\n",
"plt.hist(h, bins=100)\n",
"plt.title('SELU sample mean=%.3f, var=%.3f' % (h.mean(), h.var()))\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$h=s(z)$ is clearly not normally distributed despite the normal distribution of $z$. However, SNNs don't claim normally distributed activations, but only that the activations will have 0 mean and unit variance.\n",
"\n",
"Recall the special property of the activation function is to map $z \\sim \\mathcal{N}(\\mu \\omega, \\sqrt{\\nu \\tau})$ to $h$ with $\\mathbf{E}[h] = 0$ and $\\text{Var}[h] = 0$ when $\\mu =0,\\ \\omega=0,\\ \\nu=1,\\ \\tau=1$. In practice, none of these equalities will be exact because of issues such as sampling as well as the difficulty of maintaining appropriately normalized weight vectors. The power of SELU comes from the fact that mean and variance in the neighborhood of 1, $\\mathbf{E}[h]$ is closer to 0 than $\\mathbf{E}{z}$ and $\\text{Var}[h]$ is closer to 1 than $\\text{Var}[z]$. On the domain of near standard normal distributions, SELU maps to a (non-normal) distribution with closer to zero mean and unit variance.\n",
"\n",
"Stacking layers of nearly normalized matrices followed by SELU activations will lead to activations with moments approaching (0, 1). Each SELU layer is a [contraction](https://en.wikipedia.org/wiki/Banach_fixed-point_theorem) acting on the first two moments of the distribution.\n",
"\n",
"For SELU to move mean and variance closer to 0, 1, it is critical that it is applied to a normal distribution and not just any distribution with appropriate mean and variance. As an example, consider applying the SELU function repeatedly to a normal distribution. If each application contracted the mean and variance, after 10 iterations the mean and variance should be very close to 0, 1. Observe below that is not the case. Thankfully, multiplying by a weight matrix restores the activations to a normal distribution due to central limit theorem."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAEKCAYAAADzQPVvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAG4xJREFUeJzt3Xu0XGWd5vHvExLul4DYAZJw6RFGwIWAy0BLuzzILR1t\nwAs3ZQjIklkjNA7d2oDOQI4oLaIN0t0wSw0ZRK4DclFQkiEcB9omiIZrQEg3aZNAwjXhkkYS8ps/\n3rfITp2qOnVOcqpOeJ/PWrWy693v3vutXVXPfve7d50oIjAzs3e3Ud1ugJmZDT+HvZlZARz2ZmYF\ncNibmRXAYW9mVgCHvZlZARz2GxBJJ0u6t9vtGOm8n8z6KzrsJS2Q9PE8PewBIalP0qnDuP5TJT0h\n6VVJSyTdIWnLPO9/S/qjpNcqj7l53q6SVksaVbe+HkkLO/06bHAknSXpOUnLJU2XtHGLuodIelLS\nG5JmS9q5Mu+7kp7Kn58nJP2XumX3lfTbvOyDkj44nK9rXUn6hKT7JL2S988Pa9+HJvUvkPSopJWS\nzq+bt4Ok2yUtzt+VnRssf6ik30l6XdJCSccMx+saqqLDHlhvvyiTNLqT22uw/Y8B3wKOj4itgT2B\n6+u2fVFEbFV57DfEzQXD+FqsfZKOAM4GPg7sAvwp0Nuk7vbAzcDXgW2BB4EbKlVeBz6ZPz9Tge9L\n+rO87MbAbcCPgbHAVcBtksYMw8tqStJGg6i+NfANYEfS92E8cHGL+k8DXwXuoP/nezVwJ/CZJu3a\nC7gGODdvdx/gt4No6/CLiGIfwDOkL8mewJvAKuA14OU8fxPgu8C/A0uAK4BN87weYBHwt8BzpA//\nWODnwPPAy8DPgPG5/rfy+v8jb+OyXP5+YBbwEvAkcEylfe8BbgeWA3OAC4B7m7yWrwC3tHitM4Bv\nNJm3K+nDPKquvAdY2KD+PcAXmqxrCvA48GreP3+Ty7dttm/y/L78+v4575/b8+u/Jr/+B4BdKvVX\nA38F/CvwAvAdQHneydX91GofN2j/YNvR6v37BDA3L/cH4PwG+/yk/Pl6AfjaED7D1wLfrDz/OPBc\nk7qnAfdVnm8OrAD2aFL/NuCsPH04sKhu/r8DR7TRxiuAixus+7/n6XOA+fkz8zhwdKXeyfm9+Hvg\nxWaf4Tb31aeAR9qod3X1vaqbNzq/bzs3eB96h9q2TjxK79kDEBFPAP8V+JdIPd7t8qxvA+8DPpj/\nHQ+cV1l0HCnEds7LjwKm5+c7k4L9H/M2vg7cC5yet3GmpC1IQfET4L3A8cDlkvbM6/8n0pdxB+AL\nwCk071HfDxwhaZqkgyRt0qCO2t8rQzYdOC1S73BvYHZl2w33TcVxwImk/fyfgF/nZbYDngDOr6t/\nNPAhYH/gKNI+Wksb+7iRttrRxrpfB06MiG1Iwf/fJB1Vt62DgD2AQ4DzJP3nvO7P5eGHZo8Jefm9\ngIcr63sEGCdp2wava+9q3YhYQTpYfqDBftsM+DApfGvLPlJX7ZFcPpBrSfu0tu5tgcNYc+Y5H/jz\n/JnpBX4iaVxl+Um5nX8CXCjpzwfYNx9p0o6PAY+10d6hOACQpEckPSvp6ibvQdc47NdYKwglCfgi\n8NcRsSwiXgf+jvSFrllN6gGsjIg3I+LliLglT78OXEj6gDXbzieBZyLiqohYHREPAT8Fjsmnq58G\nzouI/4iIx0lnDw0DOyLuy/X3J/WgX5T0vco4vICv1H0pZgxyH7XjLWBvSVtHxPKImJvbN9C+CWBG\nRDwTEa8CvwDmR8TsiHgb+D9A/bDTRfm9WQhcCpzQoD1N93GT9g+mHS3XHRG/yu8bEfEoKdzqPw+9\nEfHHiHiEFMT75vrXRsS2LR6L8vJbks4camrTWzV4bVuQes9Vy/M66v0v4KGImNlkO7VlG22n3n1A\nSPpofv5Z4NcRsQQgIm6qTN9IGk45oLL8sxHxT3kfvxkR9w2wb35d3wBJh5HOos6rn7eeTCR1ED4N\n7A5sBvzDMG1rSNoZZy7Ve0mnub9NuQ+kwKweIF+IiLfemSltDlwCHEHq8QNsKUmRz/VYu2e+C3CA\npFcqZaNJ46Lb5+nqBdI/tGpwRPwS+GVuy8dJwfR74Ad5uxdHxGA+7KuARmOyY4CVTZb5DPA/gG9L\negQ4JyLub3PfLK2s503SkE/1eX0o1e+bnRq0p9U+bqbddrRct6QDSGeHewMbk4YFb6zb1pLK9ApS\nIA/G66Qx4pra9Gtt1K3VX6uupItJZwwHV4pfa7Js/cGjn4gISdeTDsb3Ap+jsv8lnQScRRragrR/\n31NZRb+bBAZD0oGkYbjPRMT8dVlXCytInYT5eZsXAv93mLY1JO7Zr1E/PPIiaahhr0qPYWw+1Wy2\nzN+QTskn5VP3j5EOEGpS/w/Ar+p6JVtFxOl5+6tIQx41/e4AaPpiImaThlDaOc1u5g/A9nm4Anjn\njGcX0nhto+0+GBFHkw6Wt7Im3AbaN/1W1Ub76vfN4iavodk+bkerdgy07mtJ+2BCRIwl9Zbb+s5J\n+rzWvnOq+ni1MozzOPlsIPsgsDQiXum/Vh7P82vb2II0TPV4payXdEA+PJ+BVZfdp259+1SXHcB1\nwGcl7UIalrk5b28XUmfkdGC7iNiWNNRS/Vys9R5I+miLffOapIMqdfcjXR84OSLuabOt/bbZhvoh\nrhHHYb/GUmBC7e6CiFgN/BC4VNJ7ASSNl3R4i3VsSTpALJe0Hf3HmJeSvlw1Pwf2kHSipDH58WFJ\n789DBj8FpknaLF/tn0qTD6GkIyUdJ2lbJZNIgXp/rQoDj9lvKumdB+kC6xzgIklb5OsAXyUN1dxf\nv3Bu/+clbZPb/xrwdpv7ptbGRtPNfEXSWEkTgTNZ+86Smjtoso9brLfddgy07i2BVyLirfx+fI6B\nQ0QAEXFNrH3nVPWxdWUY58fAqZL2lDSWdFbVbHjuFuADkj6d39/zgIcj4ikASeeSet+HNThY9AFv\nSzpT0iaSzsivZXZe9mRJzzR7UXmI60XgR8Av8xAZpDOZyPNGSTqFBtcQ6tZ1b4t9s1VE/HNu0wdI\nZ7pnRMSdrdaZ64/O+2UjYEz+HoyqzN8U2DQ/rX1HamYAp0jaLZ/FnkO6CWHEcNivcTepl7JEUu20\n/WzSxaP7JS0nXYzbo7JM/Rf3UtJY3Yuki3q/qKvzfVLv5mVJl+ae0+Gk6wCLSXf1/B3plB/gDFJg\nLAGuzI9mXiFdY3iKNJZ6NfCdiLiu0ta/resBPV+3jtdJp6O1Rw/pwtqf5P2wiHRq/4nq8FWdE4Fn\n8v46Dfh8m/uGuueNbu+sf34b6fa2uaQD5/T6ZSPiNVrv40baakcb6/4S8A1JrwL/k/4Ho0bBP6ge\nZUTcRboT6R7S2dYzVA6kkh6TdEKu+yJpmO1bpDuiPsza16C+RRp7nl/5jJyTl11JuiB+EumzdjLp\nrplVedmJpLH5Vq4l3S10baX984DvAf9C+px/oG4963Kb71+ThoOurLyeR2szJV0h6YpK/R+RPvfH\nk25PXUH6PNesIA1bBenOqzcqr2MG6cA7B1hA6ticOcR2D4varWrNK0gLSC/wbWBlREzKPbMbSKfz\nC4BjI2JZrn8Z8BekHXNy5At0kqaSdiCkW8VajZmatSRpNfC+iPi3brfFQNJdwJkR8ftut8Uaayfs\nnwE+FBEvV8q+A7wYEd+RdDawbUScI2kK6ZRpSr449f2IODAfHH5Duk0OUm/sQ7UDhNlgOezNBqfd\nYZz6ccsjSbcBkv89Ok8fVSuPiDnAWEk7kC74zIx0m9wy0nDI5HVpuBXPv+A1G4R2wj6AmUp/C+OL\nuWxcRNRuT1tK+nERpFvfqrdJLSL9MGWnPF1fbjYkEbGRe/Vm7WvnPvuDIuK5fEfKLElPVmfme2ir\nvaxO/ErTzMwGYcCwj4jn8r8vSLqFdI/sUkk7RMQSSTuy5kcni0lX5WsmkHrxi0l3dtRMZM3P6N9R\nd9AwM7M2RUTLjnbLYRxJm0vaKk9vQbrN7FHSH4eamqtNJf1whFx+Uq5/ILAsD/fMBA7P90TX/i7G\nXU0aPOIf559/ftfb4Ha6nRtqG93O9f9ox0A9+3HALUp/LmA0cE1EzJT0IHCj0t80XwAcm4P6TklT\nJM0n3YN6Si5/WdIFpDtyIP09EN+JY2bWIS3DPiKeYe2fYtfKXwYObbLMGU3KZ9D8l31mZjaM/Ava\nIejp6el2E9ridq5fG0I7N4Q2gtvZDQP+qKqTtNYfQOyca665hlNO+eI7z08//Utccsl3O94OM7Oh\nkESsywXaUrz99ttsvPHRrFz5IitX9rJiRbO/3mtmtmFy2Gfpv5DdnNZ/H8vMbMPksDczK4DD3sys\nAA57M7MCOOzNzArgsDczK4DD3sysAA57M7MCOOzNzArgsDczK4DD3sysAA57M7MCOOzNzArgsDcz\nK4DD3sysAA57M7MCOOzNzArgsDczK4DD3sysAA57M7MCOOzNzArgsDczK4DD3sysAA57M7MCOOzN\nzArgsDczK4DD3sysAA57M7MCOOzNzArgsDczK4DD3sysAA57M7MCOOzNzArQVthL2kjSXEk/y893\nkzRH0tOSrpc0JpdvIumGXH6/pF0q6zg3lz8p6fDheTlmZtZIuz37LwPzgMjPLwK+FxG7A68Ap+by\nU4GXcvkluR6S9gKOA/YCJgOXS/JZhZlZhwwYuJImAFOAHwHKxQcDN+Xpq4Cj8/SR+TnAzcAhefoo\n4LqIWBkRC4D5wKR1bbyZmbWnnd71JcBXgdUAkt4DLIuI1Xn+YmB8nh4PLASIiFXA8lx/J2BRZZ2L\nKsuYmdkwG91qpqRPAs9HxFxJPbXi9bTtaFQ4bdq0d6Z7enro6elpVM3MrFh9fX309fUNapmWYQ98\nBDhS0hRgU2Br4FJgrKRRuXc/gTW99sXAzsCzkkYD20TES5IWAxMr652Q6/ZTDXszM+uvviPc29s7\n4DIth3Ei4msRMTEidgOOB2ZHxInAPcAxudpU4LY8fXt+DvBZ4O5K+fGSNpa0G7A78EAbr8nMzNaD\ngXr29WpDL2cD10v6JvA7YHounw5cLelp4CXSAYKImCfpRtIdPauAL0VEw2EcMzNb/9oO+4j4FfCr\nPP0McECDOn8Ejm2y/IXAhUNrppmZrQvf625mVgCHvZlZARz2ZmYFcNibmRXAYW9mVgCHvZlZARz2\nZmYFcNibmRXAYW9mVgCHvZlZARz2ZmYFcNibmRXAYW9mVgCHvZlZARz2ZmYFcNibmRXAYW9mVgCH\nvZlZARz2ZmYFcNibmRXAYW9mVgCHvZlZARz2ZmYFcNibmRXAYW9mVgCHvZlZARz2ZmYFcNibmRXA\nYW9mVgCHvZlZARz2ZmYFcNibmRXAYW9mVgCHvZlZARz2ZmYFaBn2kjaVNEfSQ5IekzQtl++Wy5+W\ndL2kMbl8E0k35PL7Je1SWde5ufxJSYcP66syM7O1tAz7iHgTODgi9gX2BSZLOgC4CPheROwOvAKc\nmhc5FXgpl1+S6yFpL+A4YC9gMnC5JJ9VmJl1yICBGxEr8uTGwBgggIOBm3L5VcDRefrI/BzgZuCQ\nPH0UcF1ErIyIBcB8YNK6Nt7MzNozYNhLGiXpIWApMBP4V2BZRKzOVRYD4/P0eGAhQESsApZLeg+w\nE7CostpFlWXMzGyYtdOzX52HcSYABwDvX0/bjvW0HjMzG8DoditGxHJJ9wB/BoyVNCr37iewpte+\nGNgZeFbSaGCbiHhJ0mJgYmV1E3LdfqZNm/bOdE9PDz09Pe2/GjOzAvT19dHX1zeoZVqGvaTtgVUR\nsUzSZsBhpIuu9wDHADcAU4Hb8iK35+f3A58F7q6UXyvp70nDN7sDDzTaZjXszcysv/qOcG9v74DL\nDNSz3xG4StJGpCGfGyLiDknzgOslfRP4HTA9158OXC3paeAl4HiAiJgn6UZgHrAK+FJEeBjHzKxD\nWoZ9RDwK7N+g/BnS+H19+R+BY5us60LgwqE108zM1oXvdTczK4DD3sysAA57M7MCOOzNzArgsDcz\nK4DD3sysAA57M7MCOOzNzArgsDczK4DD3sysAA57M7MCOOzNzArgsDczK4DD3sysAA57M7MCOOzN\nzArgsDczK4DD3sysAA57M7MCOOzNzArgsDczK4DD3sysAA57M7MCOOzNzArgsDczK4DD3sysAA57\nM7MCOOzNzArgsDczK4DD3sysAA57M7MCOOzNzArgsDczK4DD3sysAA57M7MCOOzNzArQMuwlTZR0\nj6THJT0m6cxcvp2kWZKekjRT0tjKMpdJelrSw5L2q5RPzfWfknTS8L0kMzOrN1DPfiVwVkTsDRwI\nnC5pT+AcYFZE7AHcnZ8jaQrwvojYHTgNuCKXbwecB0zKj/OrBwgzMxteLcM+IpZExEN5+nXgCWA8\ncCRwVa52FXB0nj6qVh4Rc4CxknYAjgBmRsSyiFgGzAImr+fXYmZmTbQ9Zi9pV2A/YA4wLiKW5llL\ngXF5eidgYWWxRaSDw055ur7czMw6oK2wl7QlcDPw5Yh4rTovIgKIavX11zwzM1sfRg9UQdIYUtBf\nHRG35uKlknaIiCWSdgSez+WLgYmVxSeQevGLgZ5K+URgdqPtTZs27Z3pnp4eenp6GlUzMytWX18f\nfX19g1qmZdhLEjAdmBcRl1Zm3Q5MBS7K/95aKT8DuF7SgcCyiFgqaSZwYb4oK+Aw4OxG26yGvZmZ\n9VffEe7t7R1wmYF69gcBJwKPSJqby84Fvg3cKOlUYAFwLEBE3ClpiqT5wBvAKbn8ZUkXAL+ptS1f\nqDUzsw5oGfYRcR/Nx/UPbbLMGU3KZwAzBtU6MzNbL/wLWjOzAjjszcwK4LA3MyuAw97MrAAOezOz\nAjjszcwK4LA3MyuAw97MrAAOezOzAjjszcwK4LA3MyuAw97MrAAOezOzAjjszcwK4LA3MyuAw97M\nrAAOezOzAjjszcwK4LA3MyuAw97MrAAOezOzAjjszcwK4LA3MyuAw97MrAAOezOzAjjszcwK4LA3\nMyuAw97MrAAOezOzAjjszcwK4LA3MyuAw97MrAAOezOzAjjszcwK4LA3MyuAw97MrAADhr2kKyUt\nlfRopWw7SbMkPSVppqSxlXmXSXpa0sOS9quUT831n5J00vp/KWZm1kw7PfsZwOS6snOAWRGxB3B3\nfo6kKcD7ImJ34DTgily+HXAeMCk/zq8eIMzMbHgNGPYRcS/wSl3xkcBVefoq4Og8fVStPCLmAGMl\n7QAcAcyMiGURsQyYRf8DiJmZDZOhjtmPi4ileXopMC5P7wQsrNRbBIzP5YsalJuZWQeMXtcVRERI\nikqR1mV906ZNe2e6p6eHnp6edVmdmdm7Tl9fH319fYNaZqhhv1TSDhGxRNKOwPO5fDEwsVJvAqkX\nvxjoqZRPBGY3WnE17M3MrL/6jnBvb++Aywx1GOd2YGqengrcWik/CUDSgcCyPNwzEzhc0lhJ2wKH\nAXcNcdtmZjZIA/bsJV0HfAzYXtJC0l013wZulHQqsAA4FiAi7pQ0RdJ84A3glFz+sqQLgN/k1fbm\nC7VmZtYBA4Z9RJzQZNahTeqf0aR8Buk2TjMz6zD/gtbMrAAOezOzAjjszcwK4LA3MyuAw97MrAAO\nezOzAjjszcwK4LA3MyuAw97MrAAOezOzAjjszcwK4LA3MyuAw97MrAAOezOzAjjszcwK4LA3MyuA\nw97MrAAOezOzAjjszcwK4LA3MyuAw97MrAAOezOzAjjszcwK4LA3MyuAw97MrAAOezOzAjjszcwK\n4LA3MyuAw97MrAAOezOzAjjszcwK4LA3MyvA6G43wDpP0qCXiYiW66jNb1ZuZt3lsH8Xax28tWm1\nNd34AFGt27h8sAcWHxzMhofD/l1v6MHbfz31B4M1mq97fRxU6tbY4IDgMwqz1jo6Zi9psqQnJT0t\n6exObtsghepwheD6Wnd1PY2nJfV7tFOnnYfZu1XHwl7SRsA/ApOBvYATJO3Zqe0Pxg9+cFnLAOjr\n6+t8o97V+gZZv9kBoVWdVssN/SAx0mwon023s/M62bOfBMyPiAURsRK4Hjiqg9sfpOY91XfTB2Bk\n6Ot2AypaHRDO71c+1DOI4TpobCifTbez8zo5Zj8eWFh5vgg4oIPbH7Lql9Fjwba2wV2TWJdrFQ23\n3uAuqN7e3qZ3RzVa1srQybAf0Z+sVatms/XWf8lbb/0bb75ZP7f/l7LVF6qbX6KROLRg7ag/CAzl\nIBHANKC3QflAy3Zeb2/vkJf1gWrw1KmdJulAYFpETM7PzwVWR8RFlTp+B83MhiAiWh69Oxn2o4Hf\nA4cAzwIPACdExBMdaYCZWcE6NowTEasknQHcBWwETHfQm5l1Rsd69mZm1j0j7g+hSbpY0hOSHpb0\nU0nbdLtNjUg6RtLjkt6WtH+321O1ofx4TdKVkpZKerTbbWlG0kRJ9+T3+jFJZ3a7TY1I2lTSHEkP\n5XZO63abWpG0kaS5kn7W7bY0I2mBpEdyOx/odnsakTRW0k05M+fla6MNjbiwB2YCe0fEB4GngHO7\n3J5mHgU+Bfy/bjekakP68Rowg9TOkWwlcFZE7A0cCJw+EvdnRLwJHBwR+wL7ApMljeRbm78MzGNk\n36UXQE9E7BcRk7rdmCa+D9wZEXsC+wBNh8ZHXNhHxKyIWJ2fzgEmdLM9zUTEkxHxVLfb0cAG8+O1\niLgXeKXb7WglIpZExEN5+nXSl2mn7raqsYhYkSc3BsYAq1tU7xpJE4ApwI/o/1f0RpoR27486vHR\niLgS0nXRiFjerP6IC/s6XwDu7HYjNjCNfrw2vktteVeRtCuwH6kTMuJIGiXpIWApMDMiftPtNjVx\nCfBVRujBqCKAmZIelPTFbjemgd2AFyTNkPQ7ST+UtHmzyl0Je0mzJD3a4PGXlTpfB96KiGu70cZ2\n2zkCjeTT4g2WpC2Bm4Av5x7+iBMRq/MwzgTgAEl7d7tN9SR9Eng+IuYygnvN2UER8SHgL0jDdx/t\ndoPqjAb2By6PiP2BN4BzWlXuuIg4rNV8SSeTTvMO6UiDmhionSPUYmBi5flEUu/ehkjSGOBm4CcR\ncWu32zOQiFgu6R7S9ZDHu92eOh8BjpQ0BdgU2FrSjyPipC63q5+IeC7/+4KkW0hDpPd2t1VrWQQs\nqpzB3USLsB9xwziSJpNO8Y7KF502BCOph/IgsLukXSVtDBwH3N7lNm2wlP6mwHRgXkRc2u32NCNp\ne0lj8/RmwGG0uFjXLRHxtYiYGBG7AccDs0di0EvaXNJWeXoL4HDSTRkjRkQsARZK2iMXHUqLg/uI\nC3vgH4AtgVn5lqfLu92gRiR9StJC0h0ad0j6RbfbBOkiDVD78do84IaR+uM1SdcBvwb2kLRQ0ind\nblMDBwEnAgfnz+Pc3CEZaXYEZkt6mPTr9JkRsSFc7xqpw47jgHvzNZA5wM8jYmaX29TIXwHX5Pd9\nH+DCZhX9oyozswKMxJ69mZmtZw57M7MCOOzNzArgsDczK4DD3sysAA57M7MCOOzNzArgsDczK8D/\nB3CF84RQQUksAAAAAElFTkSuQmCC\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x7ff8ed687a10>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"def iter_compose(f, n):\n",
" '''Returns f composed with itself n times.'''\n",
" def g(x):\n",
" for _ in range(n):\n",
" x = f(x)\n",
" return x\n",
" return g\n",
"\n",
"selu10 = iter_compose(selu, 10)\n",
"h10 = selu10(z)\n",
"plt.hist(h10, bins=100)\n",
"plt.title('Iterated SELU sample mean=%.3f, var=%.3f' % (h10.mean(), h10.var()))\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's observe the moment contraction in action."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAEPCAYAAACgFqixAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXd4FMX/x9+TkB4SEiD00IsUaYogLWADBAFBkSIKCChi\nR4qNAEqzI375YQHEAoIiTUQEDWIDKSq9SA+RkgCBhPT5/THM7my5ktyFO+Dzep48udvd251tnz4z\njHMOgiAI4vomwNcNIAiCIHwPKQOCIAiClAFBEARByoAgCIIAKQOCIAgCpAwIgiAIeKgMGGNzGGMn\nGWPbXWx3M2MsjzF2ryfHIwiCIIoHTz2DuQA6OduAMRYIYBqA1QCYh8cjCIIgigGPlAHnfAOAsy42\newLAVwBOe3IsgiAIovgo1pwBY6wSgO4AZl1eRN2dCYIg/JDiTiC/A2AsF2NeMFCYiCAIwi8pUcz7\nbw5gIWMMAMoA6MwYy+WcL1c3YoyRx0AQBFEEOOdeMbKL1TPgnNfgnFfnnFeHyBs8ZlYEyrb0xznG\njx/v8zb4yx9dC7oWdC2c/3kTjzwDxtgCAO0BlGGMHQMwHkAQAHDOZ3vePIIgCOJK4JEy4Jz3LcS2\ngzw5FkEQBFF8UA9kPyMhIcHXTfAb6Fro0LXQoWtRPDBvx52K1AjGuD+0gyAI4mqCMQZ+NSSQCYIg\niKsDUgYEQRAEKQOCIAiClAFBEAQBUgYEQRAESBkQBEEQIGVAEARBgJQBQRAEAVIGBEEQBEgZEARB\nECBlQBAEQYCUAUEQBAFSBgRBEARIGRAEQRC4BpTBjz8CmZm+bgVBXFsUFAB//eXrVhBXkqteGdx2\nGzBrlq9bQRCuOX/e1y1wn+XLgaZNfd0K4kpy1SsDAMjN9XULCMI5mZlAqVK+boX7kLd9/eGRMmCM\nzWGMnWSMbXewvj9j7G/G2D+MsV8ZYzd6cjxH5OcXx14Jwnvk5Ij/eXm+bYeZTZtEOOj0aeNymnjQ\nvzl4EMjO9u4+PfUM5gLo5GT9QQDtOOc3ApgE4AMPj2cLKQPC35He66VLvm2HmVtuEeGguDjj8oKC\n4j8256R0ikrNmsCUKd7dp0fKgHO+AcBZJ+t/55zLSOlGAJU9OZ4jSBkQ/o604vxNGTjiSiiDgABg\n/vziP861Slqad/d3JXMGQwCsKo4dkzIg/B0ZJvI3ZRAcbL9cKoPittz37i3e/XsDxoDDh727T87F\nfv/4w7N9eJMS3t2dPYyxDgAGA2jtaJvExETtc0JCAhISEtzePykDwt/xV2UQFqa3TSUrS/zPzgZC\nQ713vFdfBX79FfjuO/FdVUZ5eUAJNyXSoUPA5s3Affd5r23OSE4GqlXz3v5OnBD/e/US+5YUFAiP\nyRFJSUlISkoCAGzc6L32AFdAGVxOGn8IoBPn3GFISVUGheVKuLQEAYgqm5EjgTlzCvc7f1YGsuRV\nWquAXk2UlWWvDP75R6xr0aJwx/vkE+DAAf27VAY//wy0by/2GRLiej8TJwLz5l25nENgoHvb/fGH\naFOrVs63k9f3xAldAaSnA9HRQsk1b27/O2koT5gg8j1//jnB/ZNwQbGGiRhj8QCWABjAOT/gavui\nQp6Bcy5dAqZP93Urrg0OHADmzi3872TO4OBB77bHU1RBr5ZoS6UlPQQzbdsKYVRYMjKM36Uy2LlT\n/N+82b39xMSI/1eq74a01jdtEl7NkiX2HlWrVkBrh/EPHfW3J0+K/0eO6McwV51lZgJnzhiXeVsR\nelpaugDAbwDqMsaOMcYGM8aGM8aGX97kFQAxAGYxxrYxxjZ52F5bSBk4Z9MmYMwYX7fi2kAKTDtB\n4Ay5fe/e3m2Ps+O5816EhemfVUGtegZ2FNUbd6QM5PHMJa6OkO1+7z3g22+L1hZ3kPdbXsvERKBL\nFxHe+b//A/bvt/7GHc9GLQs9dkz8l0phxAjg00+N2w8YAJQtK5Sf9N68jafVRH055xU558Gc8yqc\n8zmc89mc89mX1z/COS/NOW96+a+QTqV7+Fvttr9B5XuuKSgAtm1zvd25c+L/WYcBT3sKqzw8ITtb\nlIv27+96W1UZqB3NpGeQnm7/u6IqA3kMKVxl+EUudzeMdu6caPtnnwFvvFG4NhQUuH4nVq8WQveO\nO8R3ef/U+/jUU/bhHHdyLDk5QN26QIMGwMWLYplUBoA1dyIVxu7drvddVK7qHsjygbpwwbft8Hfk\nC+btTirXEp98AjRr5nj95s1COCxcKL4XVRmULl209rkLY0IY7dolQhmuUJWBVHSALpwbN7YP3Uhl\nIMM77iINNzmEjPzuyhMxk5oK3HADsG+fLijdpUYN4PnnnW/zzz/i//r14r+jd8dOKTpTBrNmAevW\niechLg6Ijxfv5+rVwvqXmKuXZJhq1Ch9mbdDZFe1MpAuXGFfzOsN+dA4svII4OhR8d+RZSqFx0cf\nif+FrfHOzhZW5oULQoC88ALw++9Fa6u7lCnjehtVcKkhD9VLsBvuRQrBhg2L1rYnnhD/pZIsrGeQ\nliaUAefA8eOF836PHAF27BCfGRO/N2MOsTlSBnaVP87CRCNGAFOniv0FBwtlfOkSsGaNcbtXXtHP\niXMR6gVEJZbk668dH6coXBPKwNudL641pMVHysAx0iKVSkFy4YJI3JmfMVka6C45OUBUFFCypNjf\nlCnAjBn6enNysCiYBaI7FTBSqNepA/zyi/g8e7YIv5i3sfudp6jKIDKycMqgXj3xOTtbeHZ5ea6V\nglyvjhNll9Q3h54dhfnsrrGrMFFgoNhfSIhQBllZ9vkdaeT++691Xa1a3q9MuyaUQXF7Bv/9p7uL\nVyNSCVxNo2bm5tq/BMWFtExPnTIu79dPJO7ks1a9OnDnnYXvhJSTIyzBChX0ahMpYLKzxTHMAmjW\nLL1qxh3Mv3fHWpbn9cgjwtKcPRt49FHjNtIq7tcP6HR58BlVeHlSwKEqg9KljWEiR3X0334LbNki\nPAPJoEFAUJDwuABh8ZuT1YAeUpZxekAoA3n/ORfviflaFsYzcKQMFi8W/4OC9OchNFSE2uz2L58x\nu+tbu7b9MTzhmlAGxe0ZDB8OFKIPnN8hXzB347H+wIwZwvrxNg0aAG++aV0uhYH5WZIvorTCUlOF\nFW32IFyRni48g9hYvc5ePr+y05F5pNAlS4xxfFeYLUV3CitycsQw8IMHC2/HrAgAXVCtWgV8/70Q\naqqiUQWrM+xi+6oyiI3Vz+HiRaBlS3sPpGtX8V8qg4oV9XW//24NPUmaNtXDLWfO6Ndn0CDg3nvF\n519/FV6DTObOny+OI/dpFsyOwkR2Vvv994v/QUF6mGj/fhE2kkpq7FjxbI0YASxaJJbZKYomTazL\nPOWqVwYxMcXvGcgX8s03r77KnH37gPHjxefCDPWdm+udc5UdmQqbvC6ue7prF/DVV9blly4J9918\nXClosrL0jkHlyxfeyzpzRsTw1R63UsBIIWkWXnahqP37HRs/ZgF08qSIPTsjNxeYNk1Y5ZUqGdet\nWQN07KgbETLZLIWaxN3wY3y8dZkjz0A+L3bFIWXLiv+1aok21aypr8vO1n+jegY5OWJ01hkzxHHO\nnjUaR3//Lf5LJbBnj/hfsSLQpo3eHrNBpYaJ5DabNgHh4dZ2SwICdM9AvpNSxkRHA1WqiCol2Raz\nh/Paa8AE7/U109vl/V1eOXJzhbWVl2fvEnqDDz8EfvtNfB41qvjGeedcCIy4OCAlxXv7XbBA/1wY\nZRAcLM7dGQ8+KGaac4a8L4cOuX9soPhqqQF7IZuZCUREAKNH68vy8vR7cemSXglUtqwQOOaknyNO\nnRIVJGXKGM9LCkL50pufLbvwwMSJerjh00+N29hZo67uYW6usFQBo1AFhKArU0YXco7CH3aK0d3Q\nkXwmpWfw5pvA//4neiQDVs8oP18I8txc8YwePGiM/2dn68pJlQmy/8IPPwhL36wMpHeTmir+JyeL\nTnW33iosfXmvzMrg5Emgb1/x2ZmBoF6P3FzRzpAQfX+yw5kkPFy/n2bZVq6cuGcREY6PVxSuamUg\ntWteXtFjaJcuORc8w4YZ3W1nbvt//wErVhStHV9+KYTM6dO6leIN1Lrkwta6b9gg/juy6j/7zHVv\nXGlp79tXuGMXJ+pYMJLMTCFE1E5Pb7whQiOAeNHlujJlROnhXXeJ4Qc+/9z58YYOBZKSrNU9UhBK\nQWR+6e2ey4sXdWE3cKCxb4RdGNAsoE6dAj7+WHgYq1dblcEjjwjDZO1a0cs4JES//46sXTX8k5cH\nLFsm6uRVQ8RR+bc6mqtUto8/rodtzKGl8+dFolnW4Zcvr7df7k9VBpmZ4nxee00Iz6ws4QGdPWtU\nnmZlcPw4MGmS8DxCQoCZM4GlS+0Vriw3Nnuc06bp3rXqcR45IuRKcLB+XHMSOyxMNw7U5+LwYXHf\n1TZ7i6taGagPclGt6cJWhThTBuPHA/fcU7R2qNUknTuLB8kbqBZ5YWeEO31avMShoeIlz8wUAuqR\nR/RtXOUh5Evg6/JfaemFhAgrzRzayMwUQlLFnEyWlCmjC4WRI4314XZIgWenDH79Vb9Hd9+tW/2A\nc2UgFbuqZO0ElXnZO++I+1enjnjOTp3S36FHHwWefFJ8vu02sTw0VG9/ZQcD0KvGy48/Aj16iM9r\n1+rLHd3/Dz4QYa/MTCHYzbRtK/5v2CDaevasNamuht6ysvR8zuuvCwWwbZtIxsfG6m3Jy9MHy1OR\n72FWln6cqChxnZ991nkEYtMmcU0lY8eKsCQg2lS7tnhWZAkv5+J+APp9ksojPFzImv/7P+Mxq1Y1\nKj9vcs0oA7OL6y7S2nPXrXXmCroTqvrhB2PCS2K+wW+95V57nNG1q54wAwqvDNLTdetj/379gVWF\npitlIOPbhU1eFzZMxJjzUFRysrgW2dkiiWyuVMrIAOrXF/Fc+UKa76f0+qKjdeHmTl5FJhnNYSLO\nRTz61VfF92PHjAPg2V2DjAxxX6SlLS1ZwHWp4T//WDuinT2rh3+aNAEaNTKuVz0D8/7/9z8xFo+q\nDNTEuvrZ7t2QvXvXrxfKwJyzUGnXDnj6aXEPzJ7qgw/qn7OzdYNMtkuGMm+6SWzbp4+o6DJfC86N\n4RoZfpL/Dx2y75MgOXIEeOwx4zIpX9auFdfq0Ud1i79sWTG0xQMPiO89eugeUXi4UICPPaYbrC++\n6PjY3uCaUAYLFjge5c8VK1eK/+4mOJ15Bu7kE9assfdizOPKeyNmbh6zpbDKIC9PxLoBoQzsKjtc\nXTd5vdypiWZMF25FSV47K/dUr2ft2lZlcOaMiMWGhIh1jAmrVUXGaNV6+K1bXbdLVQauWL1aP3d5\nvdX7dvGiMEikMrAbQkKya5dxBrPhw+3nD3BWF68qg/R0oSyio8X3xx4TFreqDFRPRW1bRoZQNt99\nJ6qEACEMBwwQ5+JIGbRpo39esAB45hmrN9+lixgzSB4nKEiMHSQt8OefF73LP/tMVAcNHiyWbd1q\nNMLeekuU15YsKb5LT8LduauPHRMel4o0Hr/4QiTe1Q5psgxWDo29cKEYogIwhuSkt1jcY7BdE8og\nNLToZZPS2nM3nu7MM3CmDO68UygeR7FTszKwK1nztIS2sMrg4EHd6pKutXlfrpSBq9EvJXLfngwt\n4mwcePXca9Y0DqOckiKs2HLlhMCXQxGYkcJBCgsVRy9qRoaeIDZ7Bo7IyhLW7J494pzUtsowkQxz\nqc+c+RpXry6UMediu6NHRXmleQ4AZz1mZZLzxAkh7DZt0mPkgOj4deiQfp/V4SnUZyMjQyjRTp30\nntcREUKxPPSQsKDtlMG5c+6NsSQV9blzIqxjNlxKlzYK2IoVxTErVNCXyaEeZLhKhons7nd8vPV9\nT0uzKnwpLy5cENdKXus6dXQlLKus1PugDhMijZw777S2w5tc98pAloZ56hnk5joPifzwg+jhKYWd\neVA0c5goJcX+gf7zT/faaWdZu6MM8vP1qQhVxScrOCTS7XelRB2V5JmRxypKUsw8sqSzbQDxwqox\n7LvuEv/Dw4XAUl/yWbOAl18WAunGG4UVHBlp3b+dEsvKAnr21Mf2cVb9oU6cMm2aiNkDYox/NUkv\nlYF81hx5Bjt2iPciIEAsr19fCPQNG0T9+vnzuvBz5hlERorfVapkP7dBSIhQrtLjUBO+av+WjAzr\n+UtlIFFzBvI4J04Iq1rdxs7LVPcdFWUUpoD1XZKC3m7CGnl/peK2C3FlZ4tjyBJPmYcyexHqcx0R\noRs927fr2zRtat2/PP86dUSuZMcOoEMH63be5KpUBitXCoGl9uIr6iBsUii44xmEhenKYNYsXeBe\nuCDaIUMPZotBnUJQfjYPimYnvNVez/IhGjlSdFJxhd0D7I4yOH5cWGrqMQERD1WvsXyZXFm67ioD\neV2lUC1MmMxRNY6Keu5yPBhJjRr658hIfT+NG4sY78SJIsTAmFAIUtDccou4LvHx4t6bldEzzwgj\nABAVNuZzUu/5jTfqn9Ua8uHDjQPFZWSI0J0UDOo5X7okzqV+fZEXAYTiO3lSj4VL6zgqSr+HzmYX\nq1xZH1ROPtft2hlzWlFRejtSU0WdPCDu+ZEj4nrdfbdRYH/8MfDcc0bhWbas8IbeeUfPXZi94Ro1\n7JWXuu+SJUW5p0w+A9b3S567DOuo832YLfCePUX7Af3Y0op//nnx+fx5cdzAQOOxVGUQGSmUav36\nxkhAy5bW0lKpJOWQG94uI7XD75VBWJjoeq4yZIgQWN7wDGR1gjvKpEIFcXOzskQPwYkThcsra6LT\n0sRNNO9LjkGTnu5YyNkJatUyl4mrTZtEGaorVA/mu+/ENZMK7+hRx1a0I6X4zjv2k5d7SxlIi1KG\nP+QL9eabrhW1VAaZmcKSVKuAcnPF8+NMGZQrJ6o2ABHXtuuFqyLPOTdXKIKjR0VycuFC/Z598om+\nT0AX9nb9DADraKZVq4r91ayp38v8fKuhsWuXCL1s2SKO3bGjMVTTuLHx/VGPL71iZ/dQDaNIwsOF\nopPIzlOcC2UgDR3zhCyqQBs8WCgaGXp77z3xv25dMTT0a6/Zd5hz5MWYPYOuXfX3ErA+71IJVagg\nFGafPuL7Z5+JqkA5TpPct0zs7tkjFL9cL889Lc1+6JAzZ4QBeOmSuG5xcfYjvZo75MXEiOspFRMp\nAwgh4mjSaE+VQX6+ECRxce55BhUqiBdT1gQnJooQgjraY1SUVRnIF+fcOcfttFMGaqWI6io3buy6\nrWrYolMn0S55jKpV9UoK85AFdmEaGYu3ExruKIOICBH3ljX7ZjjXLV0pAGS7Ro3SR5h0hDzX334T\nbn+HDiLM9tRT4sW+6SarMlCFanq6HhdWrUJXCTvzM3PqlBAytWrp1jQgkplVq4rP6vVSFZIUipKY\nGCGgoqN1BWOXk1q/XgwRcdNNYiRQ8/PVs6c+0qoZdwayk5apM+RYO7L0WI5kmpKid9gE7MNr3bqJ\nY4wcaVx+xx32vWwd5TfUoUuioqzrzWEiee4NGojcgSybjYsTz4d5tjJpnFStKrwT6f1IT+D8eeNx\nX3pJeAEpKeK6hIc7z2k5Qh6HlMFlzDFC1TILCjL25CsM0rULDtbd3PXrxfHMD0/PnmKWqnPndBdc\ntkEtaVSrL8yoHV3MloDdODJq5yj1szvhHnPIJChI/E5amcHBotwtKMhYWWOnDFq1Euc+aZLr45rJ\nzhYC7dtvdVfbjBoKkKWw6jV0pQzkuc6YIc4xPV14QzNm6L/NzRUW68GDRs8gK0tY4PJFVuPWrkbm\nlMpgxw4xiNvLL4vv//6rJ/2mTBGdkeyUpnreZmUgSxJlGStjwntyNswBYO0/0aSJ2Ffr1nr1isQd\nZVC5st5xURXsKsHB4lqkpgoPR5ZO33uvPqwDYN+PIC6ucBO2OFIG6jtpV2Zudy9TU3WPQApqu2Qx\n4Ly6LThYXHc1TzFpkvAyTpwQz35R+wbIhLQ5B1Ic+LUySEoS/1XBtnevbrE58gwuXtR76TlDhoi2\nbxcTe993nxiQLjxcvMQqS5aIpJ55DPrcXKMFr3ZdN3P8uLASZs2y3lxVwEuX9Jdf9DHL1bCPs7Fg\nNm/WO4ipSGUghczIkXqd908/6dvZKYOSJR1XSqntzsmxvnTZ2a5L82SISA48VlAAvP22vl6dAcoO\nsyEQGam3Q1aL5eYK67F6dXF///1XvOAykSdjuLK0D3DfM2jQQMS71edUKhezRaeO/6+GUMzKQMaw\n1T4NDRoY78NNN1nbZL5PoaG6cdOli3Gdu5O816vnfJJ3GSqRykDGu2vUMBpK3iiXdtSfqEQJ/Vms\nX9+63m5eYvM1nzrV8QBwrpTB+fPWd7pCBfEOP/BA4QYcVJHKqTiHZ5H4tTKwS5LVq6e/RDk5QsjF\nxooHUVrXmzZZ5xC1Iy3N+ECo3cl37rR6JA0bWkerzM01Cn9nnsG//wphGx1tte5zc0X9e/v2It78\n8MMi+di7t9if2jdBVQaMGSsTbr5ZDI/gyDOQIRW184xqedspg1KljA+jnE+5Vi0hiOVwDCEhoten\nSna29aUzc+yYSBj+/ruwHs1CytWgcFlZQlH36ye+R0RYr29Wlm6dhYWJ8tHly/WXXFqz8fH6S+3K\nM1CvsdmilM+AWThPmSKSrzJHINukXqO779bPxc4SltUncXHGxLPd8eR4+XbVPO4qA1fIMJFUBlIR\nVqxoVAaOlIkrZHjp5Elg8mTH28lEuBzVFBD3ePt293r0jxnjOCfhTBkEBYl30vxbtXOpO31M7Ljh\nBsdhPm/jkTJgjM1hjJ1kjG13ss0Mxth+xtjfjDGbIirXOIuzBwfrI/1JoeYqASix69ouyc/XBb+M\nIzOmu5PqsAHuKoOoKFE1oMbv1XPp1k14Q5GRxt6Go0cL633BAhF3NwtH2c7//hP/U1PFy1+zpj4z\nklQGdsJeFWp262NidMF44416NdNNNwkXf8AA3YqWnY7WrhVJ9uxsY0jMfG127xbFAK1aifuoxn63\nbhVDALgaFVMO+iVfvpAQa1+BQYPEqJWALux79BBjxLRvb7TYzZO0O0Idx8gcp5bWvOo1AkIAP/OM\nnneSSkEqg4AAawkh58a+BrKjW0GBsTQTsBow0jOQA/GplCtnf16FxRwmkm2KitKT+RMmALffXrT9\nV6wo3pu4OGt/HDMFBcZhM7p1K/psbCquPIOHHrJWPqkGgjvza9tRooQo/rgSeOoZzAXQydFKxlgX\nALU457UBDAMwy9G2ZtSejI6EqzochexEAugPoFng9uolKoAkzpRBXp4Qsh07igSdRD6MvXqJl1Z1\nTwF7ZdCggRBOJUuKF0b1DC5dEolo2XNSUquWOG50tO5ixseLUjhzxU3JkkL4yrDB+fPi5W/VSoy6\nKNudk2M/FlNWlrh2Awda+zE0bCiGSzD3hk1LEy+4ebA1aR29+64Ih2VlGZWBOYS2dKm4DzJRJv8D\nwgJu1MjeM1BfTqkM5Mv322/GSh6JXW3+339bhWS7duK/WZCrVK9u/G5WBhcuiHs1bJj976XFLxW4\nVAopKaLk0ozduEAFBboHAYj+EmqJJKCHUDMyrPmGXr28M0Jufr6w2B96SFyzVq1EwUNkpOcdJV9+\nWYy8ajf8tR3FFU5x5iVKD8tcHqpiVtr+iEfKgHO+AYCzIcjuAfDJ5W03AijFGHPLHlFjt+4og8hI\nXSBJC1G1ks6cEXF/tczuwgXxEttNVJKfLwSneRwheTzGRFw7M9PqGZgtSmmVSaGjegYbNwqhunev\nNcnUoYMQhMuWCQunWTPx20OHxIstzzcgQMT/ZZL5m2+sYQEZLpDD7apcuiSs+08/NVbBAMKai401\n5mkAoURV4SLbIq1uqbBUz0BO96ciq2xk6MPsaqvVNJIVK8Q5SwtZKgOzsJs+3RhWkG1r0MA4oJhZ\nGSxZIvbvrCghKUn3NACjFSiHgBg3zvEEPWYLV3oGjhKNISEiHKcq8/x84QXL8OhHH1mtb3nf7TwD\nxuyTuoVl506RKM7OFoZNUJB4zuyqhwrLxIm6cvYlt92md040I987u/4aL70k/rtK/PsDxZ0zqARA\nHYT2OAAHYx/aExHhWBl88IH+gqudheR/OQIjoFd3qAri0iXx+ypV9MSwtNDy8/UJSVTUl1UeUwrI\ngABhzXfrZvyNLC2TL2OZMsKC6tZNF6InTlgFgfx+/rwIl4SG6hbopUt67sT8shw+bLUEw8ONSkpO\nEAIIYeEofizDYlIZmBWfRAopaU3L3ERWlhDo330nrq1ZGeTmAt2765VGZiEZHW0NE8nqk+bNRbjl\n0CFxbcwJvBtuEAJZIp+jcuWMZa5mIVmihIjtP/ssHBIfbyzxlffl1CkxZDXgvAKEMZFclCE36aE6\nC4P062es+5fWqixvtPMepKcqe8AWB/I5q1vX+CxeiXLIK0WdOmLcKDvkqLV2PaNvuUX891Z+pjhx\n0vfQa5gdN9voW6IcaQpAu3YJABIAiGTq55+LB1pWF0l27RKJVkA8eFKwyv+ffCK68nMuEquAcRiC\nzEz9hZUPrhRmf/0lwg1mgaAKbHlMKeAcTXcnBbN0YaXLuHKlPu7KiRPOe4LK46oCWCaB1ZBJv34i\nn7Fnj7HcTq2tj4vTjxUVJQS2KmSef160q0kT/SGWgseRMpAdaebNE96FOqpm6dLCqlJndpJkZRkH\nU5PnKa3+qCirZ6C67LK6pGtX6wsnFdnHH4u4qyNL305oqZPcuINUBqGhunfjqhywdWvx9+yz+rNR\nmBJEdyalV5V5cQ19LO+pFHwS1TO4EtUwvuLTT0UZqd3z5e3zTkpKQpJZEHqJ4lYGyQCUKDAqX15m\nQVUGapxRWjaPPaaP0XLhgu6Wy/9qmEhNiKal6W70tGnGmZ+kZwBY3TgpaM2egWq5yWNKARkdrSsD\nGcKSg4SpMxepL6Vs6/799qWMYWGiZrlFC+s6tYZbMmeO6CuxZo1xnP3wcF3A7typh2W++EJYwRUr\nipzD5s3CwmvcWChfWWonlYUqzNWQjlqDPn++bs2fPKknKmXeQsU83o28NrJ90dGi3Vu3ijDZLbfY\nx/Ltyk+W3FpcAAAgAElEQVRl2wcPFsrArKhbtxYJdm9YsPI5DAvTz8eu85Md6jkXRmC7O4plcU/V\nKu+pDPlJVGV4tU0XW1hatLCv/PK2MkhISECCMiH7BC/Of1ncYaLlAAYCAGOsJYBznHMXVePGpJZU\nBqtX68NUqw+ZFPRSMOflGTtw/fefiO2HhAgvQnoGWVkiVm9WBuZpHM2JKzURJAW8DD9ERQnLGNDz\nEDk5wgovUUK3cNUX3txT2Exmpkgo2gkJu+GTQ0LES3nsmDGMER6uV6RERelWdFycuBaXLun9DqRw\nbN9eP19Z3qYK4uBgEeIpUUIoWemlDR9urwxkRZNKVpbxfpp7vMrjSw9q0ybr8NOAVdg88IAx39Os\nmbXn9ptvGo/hCVIBlCihn09hq3W2bi2c8HDHM7gSSGUwYoRx+dUQGvEWGzbow72r3Hxz0Se8utJ4\nWlq6AMBvAOoyxo4xxgYzxoYzxoYDAOd8FYCDjLEDAGYDGOFkdxpqbC4kRAjwtDS904z6kMnpLiMi\nhJVtjpWnpAiPIDtbhCvS04VAkjFutfYcMA6j26WLVUCrY8gEBBgHr5PD8XboIBJfBQXGRO6gQSKP\nob7wKSm6BSlDWe5irvyR0wyOGiUEtVoppV6T4GARQoqO1qtNLl3SR3C0m5UqIkKE5czz/i5dqivp\nggIxpkzLlrrLnJ6u5ydUz2DRIhHjNnsGQ4cay1vltWHMufAz96swW82//WbsXCfPCXBcUVYYVO9C\nfvZW6aYd99/vXsdKwHU/D0+RxpA5GV23rrEPzLVMcLC9wVa2rCgAuRrwtJqoL+e8Iuc8mHNehXM+\nh3M+m3M+W9lmJOe8Fue8MefcjalAjBZvaKgIh1SrJsatUSfSeOcdPcNfqpRQGKmp4uXevFlUVqih\nlMBAcXNSUvRRF6XQKllS1J2rwr55c6ulZg4byfK5YcP0XsvVqolQyf79eogIEJUF775r/P3q1WJM\nGXVkSncx19LLGZN69rQm3c1hsK+/Ft6DqgzCwkQpn6OqiRtu0L0HFSmU4uJEqOmPP/Qy16ZN9fxE\nUJCo7tmwQQjm5GRRfaIqA8aMglWuO31a/EVGCi/AnFTOyDAKX3OeISTE6sZLJeUNZRAfr/dzkefu\njf064ssvhQfmDmq5bnHgqKOWOk4R4f/4ZQ9kVYDLUs3q1YVQMfe4lMTFiZDQ3r2iG3zz5kJwm0MK\nFSsKwfv+++K7jCMHBoqSzJgYvc7bbgyg9983hpLk+PcvvKCPQS9fjpQU+56fKv/8I6z0wszUNn++\nsKpzc913xWUYRbqsoaFCAZqVwcSJxiGd3WHECKFYJk3SBa4cWkIN1QQGCo9g1Chjss3ZePqSM2eE\n0pSeglrKOWiQGF6jRw+9j4mrXsuArgycTe5SGGTCvmFDYbT4S9L0pZeMwzl7mx9/tA/dqfjLtSAc\ncyWqiQqNjKOXLau/qHYCSn3AZLngqlVCOABCUKs9NwFhtatWuDmpGBAAvPGGqN+2G763XDmjBSor\nJtTEsqzgOHxYhGIc1Rj36SOEll3tvzMefFC45EOGCK/EnbmXK1YUbTW7rLLk1By7LyyyrFHer9BQ\ncQ27d9e3kWGtTZuMczO7UgaZmSJkJ/uFSB54QAwyN3u27qJLAe/OWDDyPhVHctOR0eILevcWf8WF\n3ZzeKnFxeudHwn/xS2WQlSU6h8XHCwuwXj1jT0s71PJEKawjI60WS/36orcuIGKpHTva789RZyEz\nzpTBqVPGMJFKgwZiQvGixnNvu03E782T5DiiXDn72bhKltTnoPXGyIiqMpDzOLjC1XHDwoSCnzTJ\neL3uvFMoA7tYrTueASBKiCmUUby4GmiQ8A/8UhlkZwtrQ8bn7Ya4LV9en1gbMCZ+pcCIjNRHxJRW\nfvXqwjPo3ds4vlBRkcpAFUiyOuXUKXEsO8vJ1bDMrggIEEpSJlVlFVNhCQ4W4Rs59ounqMrAXcwT\nu9hRvrzw8tSRNx2VViYmuh+vd2duCIK4HvBLZWA316oZ85gq6vbysyw3nThR74kaFSUqWrxVYSGt\nWvX448YJS1uWLn72mXeOZYdUBnKqyqIgh7hw5e67g/SQHN2/qChr8tddZQAYQ3SOlMH48a73RxCE\nEb9MIMuxZgqLLDOVlTQyyRgTo1e0SEve0SQWhUUmmdUwUXi4cYwYu/HVvYU3as3lAH/eqH5x5Rmo\n5YcyieyOYi6MMiAIovD4nWfAedGVwb59IqksQ0ayxE/tVKT2WPYGjvajKgN3pg4sKq+95t7MZ64I\nDPROxYcrZVCvnj4irdzWHcUsvQdVGfhLpyuCuBbwO2UgJ6wpynyhgKiskaEbWa5pNwmJtzyDRx+1\nV1wyh5CQULxT1smJZjxFdljzFGfK4MABkQdSZz5bs8a98FRgoPidWp2kjmxLEIRn+J0yKKpXIFEr\ndwIDRe9jNenobc8gIUH8OcJubmN/w5ullc6UgRxU7rnn9HyKXSc2R5h7Rt9xx7U/5g1BXCn8Thm4\nkzwuDObRJ2WdureUgSuuBmXgTWTuxNGw4wCFdwjCH/E7ZeCpZ+AKGaJwNouVtxgwoHh7fvojAQGi\nH0ejRo63ocQvQfgf150yYEyMjXMlOhp9+mnxH8MfkcNyOII8A4LwP/xOGXg7TGRHmzbFu3/COaQM\nCML/8Lt+BsXtGRC+p107+ykaCYLwHaQMiCtOnz76MCEEQfgHfqcMrkSYiCAIgjDid8qAPAOCIIgr\nj98pA/IMCIIgrjx+pwzIMyAIgrjyeKwMGGOdGGN7GGP7GWOWkXIYY/GMsZ8YY1sZY38zxjo72x8p\nA4IgiCuPR8qAMRYIYCaATgDqA+jLGLvBtNlLABZyzpsBeADA/5ztk8JEBEEQVx5PPYMWAA5wzg9z\nznMBLATQ3bRNAQA5iHQpAMnOdkieAUEQxJXH0x7IlQCoFePHAdxi2iYRwBrG2BMAIgA4HayAPAOC\nIIgrj6fKwJ0BhPsBmMs5f5sx1hLAZwAamDdKTEwEAKxfD1SunAAgwcOmEQRBXFskJSUhKSmpWPbN\nuAcDwl8W7omc806Xv48DUMA5n6ZsswPAXZzz5Mvf/wVwC+f8jLINl+0YNUrMZvX880VuFkEQxHUB\nYwyccy/MUeh5zmAzgNqMsWqMsWAAfQAsN21zFMDtAHA5uRyqKgIzaWnem6yeIAiCcA+PlAHnPA/A\nSADfA9gF4EvO+W7G2ATGWLfLmz0HYChj7C8AXwB4yNk+09L0+W4JgiCIK4NHYSKvNeJymOjUKREi\nSkoC2rf3dasIgiD8G38KE3kVOZKlOxOkEwRBEN7Dr5RBbi5wyy1A7dq+bglBEMT1hV/NdEZ9DAjC\nv2HMKxEJoggUd0iflAFBEIXCH/KM1xtXQgn7VZiIlAFBEIRvIGVAEARBkDIgCIIgSBkQBEEQIGVA\nEARhYMOGDahXr56vm3HFIWVAEASh0LZtW+zZs8fXzbjikDIgCIK4TF5enq+b4DNIGRAEcdUzbdo0\n3HfffYZlTz31FJ566inMmzcP9evXR1RUFGrWrIkPPvhA2yYpKQmVK1fG9OnTUaFCBQwZMgRJSUmo\nUqWKts3UqVNRq1YtREVFoUGDBli6dKm2bt68eWjTpg2ef/55xMbGokaNGli9erW2Pi0tDYMGDUKl\nSpUQGxuLnj17autWrlyJJk2aICYmBq1bt8b27duL49K4DSkDgiCuevr27YtVq1bh4sWLAID8/Hws\nXrwY/fv3R1xcHL799lukp6dj7ty5eOaZZ7Bt2zbttydPnsTZs2dx9OhRzJ4927LvWrVq4ZdffkF6\nejrGjx+PAQMG4OTJk9r6TZs2oV69ekhNTcXo0aMxZMgQbd2DDz6IrKws7Nq1C6dOncKzzz4LANi2\nbRuGDBmCDz/8EGlpaRg+fDjuuece5OTkFNclcg3n3Od/ohmcDx/O+axZnCAIP0W+q47Xe+evKLRp\n04bPnz+fc875mjVreM2aNW2369GjB3/33Xc555z/9NNPPDg4mGdnZ2vrf/rpJ165cmWHx2nSpAlf\ntmwZ55zzuXPn8lq1amnrMjIyOGOMnzx5kp84cYIHBATwc+fOWfbx6KOP8pdfftmwrG7dunz9+vW2\nx3R03S8v94ocJs+AIAiv4S11UBT69euHBQsWAAC++OIL9O/fHwDw3XffoWXLlihdujRiYmKwatUq\npKamar8rW7YsgoODHe53/vz5aNq0KWJiYhATE4MdO3YYfl++fHntc3h4OADg4sWLOHbsGGJjYxEd\nHW3Z55EjR/Dmm29q+4yJicHx48eRkpJStJP3AqQMCIK4JujduzeSkpKQnJyMpUuXol+/fsjOzkav\nXr0wevRonDp1CmfPnkWXLl0M4ys5G/fnyJEjGDZsGN5//32kpaXh7NmzaNiwoVvjM1WpUgVpaWk4\nf/68ZV18fDxefPFFnD17Vvu7ePEi+vTpU7ST9wKkDAiCuCYoW7YsEhIS8PDDD6NGjRqoW7cucnJy\nkJOTgzJlyiAgIADfffcd1qxZ4/Y+MzIywBhDmTJlUFBQgLlz52LHjh1u/bZChQro3LkzRowYgXPn\nziE3Nxc///wzAGDo0KH4v//7P2zatAmcc2RkZODbb7/Vch6+gJQBQRDXDP369cO6devQr18/AEDJ\nkiUxY8YM3H///YiNjcWCBQvQvXt3w2/sPAO5rH79+njuuefQqlUrlC9fHjt27ECbNm0M25l/r37/\n9NNPERQUhHr16qFcuXKYMWMGAKB58+b48MMPMXLkSMTGxqJ27dqYP3++dy5CEfGraS8TEoDERCAh\nwccNIgjClsvTLPq6Gdcdjq67X017yRjrxBjbwxjbzxgb42Cb+xljOxljOxhjnzvaF3kGBEEQvsGj\nyW0YY4EAZgK4HUAygD8ZY8s557uVbWoDGAvgVs75ecZYGUf7I2VAEAThGzz1DFoAOMA5P8w5zwWw\nEEB30zZDAczknJ8HAM75GUc7u3SJlAFBEIQv8FQZVAJwTPl+/PIyldoA6jLGfmGM/c4Yu8vRzsgz\nIAiC8A2ezoHsTiapBIBaANoDqALgZ8ZYI+kpSBITE3HmDDBzJtC1awISKItMEARhICkpCUlJScWy\nb4+qiRhjLQEkcs47Xf4+DkAB53yass0sABs55/Muf18LYAznfIuyDeecIzoaOHIEKFWqyE0iCKIY\noWoi33A1VBNtBlCbMVaNMRYMoA+A5aZtlgJIAIDLyeM6AA7a7YzCRARBEL7BI2XAOc8DMBLA9wB2\nAfiSc76bMTaBMdbt8jbfA0hljO0E8COAUZzzs+Z9FRQAOTlASIgnLSIIgiCKgt90OsvM5IiJEd4B\nQRD+CYWJfMPVECbyGhQiIgiC8B2kDAiCIAj/UgZhYb5uBUEQVyPVqlXDG2+8gcaNGyMyMhKPPPII\nTp48ic6dOyMqKgp33HEHzp07BwD4448/cOuttyImJgZNmjTB+vXrtf3MnTvX5RSZb731FsqVK4eK\nFSti3rx5V/pUiw2/UgbkGRAEURQYY1iyZAnWrl2Lffv2YcWKFejSpQumTp2K06dPo6CgADNmzEBy\ncjK6du2KV155BWfPnsUbb7yBXr16aZPVlCtXzuUUmenp6Thx4gQ+/vhjPP7447bzFVyNeNrpzGuQ\nMiCIqx82wSu5TPDxhU9SP/HEEyhbtiwAoG3btihXrhwaN24MAOjZsyfWrVuHzz//HF26dEGnTp0A\nALfffjtuuukmfPvttxg4cCC6dOmi7a9du3a48847sWHDBjRt2hQAEBQUhFdeeQUBAQHo3LkzIiMj\nsXfvXrRo0cLTU/Y5pAwIgvAaRRHi3qJcuXLa57CwMMP30NBQXLx4EUeOHMHixYuxYsUKbV1eXh46\nduwIQEyROWHCBOzfvx8FBQXIzMzEjTfeqG1bunRpBAToAZXw8HCfTkjjTUgZEARxTWI3tWWVKlXw\n4IMPGnIBEjlF5meffYbu3bsjMDAQPXv2vG5KaSlnQBDENY8U6AMGDMCKFSuwZs0a5OfnIysrS5s3\n2dMpMq92SBkQBHFNok4/KaenrFy5MpYtW4bJkycjLi4O8fHxePPNN8E5L/IUmdcKftMD+YsvOJYv\nBxYs8HVrCIJwBPVA9g3UA5kgCIK4IviNMrh0iZMyIAiC8BF+owzIMyAIgvAdfqMMLmWRZ0AQBOEr\n/EYZZGeTZ0AQBOEr/EYZkGdAEAThO6gHMkEQheJarrW/nvEjZUCeAUH4O9TH4NrFb8JE5BkQBEH4\nDo+VAWOsE2NsD2NsP2NsjJPtejHGChhjzezWk2dAEAThOzxSBoyxQAAzAXQCUB9AX8bYDTbblQTw\nFIA/HO2LPAOCIAjf4aln0ALAAc75Yc55LoCFALrbbDcJwFQA2QBss09Z2eQZEARB+ApPlUElAMeU\n78cvL9O4HBaqxDlfdXmRbQaKPAOCIAjf4Wk1kdPSAsZYAIC3ADykLrbb9t8DE/HppyXw009AQkIC\nEhISPGwaQRDEtUVSUhKSkpKKZd8eDWHNGGsJIJFz3uny93EACjjn0y5/jwZwAICcF648gDQA3Tjn\nW5X98IZNsvDZvBBcnrKUIAiCcIE/DWG9GUBtxlg1xlgwgD4AlsuVnPPznPOynPPqnPPqEAlkgyKQ\n0KilBEEQvsMjZcA5zwMwEsD3AHYB+JJzvpsxNoEx1q0w+6IEMkEQhO/wm5nOypTPxI6/wlCunK9b\nQxAEcXXgT2Eir0GeAUEQhO/wH2VApaUEQRA+w2+UQV4eR3Cwr1tBEARxfeI3yiAkFKCRcQmCIHyD\n3yiD0FDfJ7IJgiCuV/xGGYSE+LoFBEEQ1y9+owzIMyAIgvAd/qMMwnzdAoIgiOsX/1EG5BkQBEH4\nDL9RBpQzIAiC8B1+owzIMyAIgvAdfqMMQqj3MUEQhM/wH2UQQp4BQRCEr/AbZUDjEhEEQfgOv1EG\nIZQzIAiC8Bn+owwoTEQQBOEz/EYZUJiIIAjCd/iNMggmz4AgCMJn+I0yCKVOZwRBED7DY2XAGOvE\nGNvDGNvPGBtjs/5ZxthOxtjfjLG1jLF4u/1QApkgCMJ3eKQMGGOBAGYC6ASgPoC+jLEbTJttBdCc\nc94YwFcAptvti4ajIAiC8B2eegYtABzgnB/mnOcCWAigu7oB5zyJc551+etGAJXtdkTVRARBEL7D\nU2VQCcAx5fvxy8scMQTAKrsVNBwFQRCE7yjh4e/dNucZYwMANAPwjN3671dOx5kjEQCAhIQEJCQk\neNg0giCIa4ukpCQkJSUVy74Z50UPzzDGWgJI5Jx3uvx9HIACzvk003a3A5gBoB3n/IzNfvgnX53E\nwF5xRW4LQRDE9QZjDJxz5o19eRom2gygNmOsGmMsGEAfAMvVDRhjTQH8H4BudopAQjkDgiAI3+GR\nMuCc5wEYCeB7ALsAfMk5380Ym8AY63p5s+kAIgB8xRjbxhhbarcvyhkQBEH4Do/CRF5rBGN81c8p\n6Ny2vK+bQhAEcdXgT2Eir0H9DAiCIHyH/ygD6oFMEAThM/xGGYSH+7oFBEEQ1y9+owzCwsgzIAiC\n8BWkDAiCIAj/UQYlPO0LTRAEQRQZv1EG3P2RLQiCIAgv4zfKgCAIgvAdfqMM/KHzG0EQviHlQgq2\npmz1dTOua/xGGRAEcX2Sm5+L/kv6o/kHzQEA+QX5uJR7ybBNZm4m8gvyfdG8YoFzjl+P/urrZhjw\nG2XgLGfAJjCsP7z+CraGuN5Yvnc5Pv/nc8OyR1c+ioNnD/qoRb4nMzfzihwn+NVg/HT4J+37M98/\ng/DJxo5HEZMj8NqG1yy/zczNxIG0A8XeRm+zKXkT2sxt43D9nG1z8PTqp7Xvufm5xd4mv1EGrtif\ntt/XTSCuYQZ+MxADvhlgWDZ7y2ws27PMK/vnnOPbfd9q34+cO2Kxfn1BcnoyTl48aVm+4cgGREyO\nQEZOhrbs8LnDRRa86dnpqPFuDbe23XFqh+3yval7LctG/zAatd+rjfTs9CK1y1fkFeQ5Xf/Gb2/g\n3Y3vAgBOZZxC8KvBxd4mv1EGrnIG11tOoYAX4N+0f33djOuGfG4fgsgtyHVole0+vdvt/WfkZqDr\ngq7ac1zt3WoYu3Zs4RuqsPrAanDOtf9FocaMGmj1cSvL8pMZQkFETonUvPJms5uh7sy6bu03MzfT\n0Kbj6cdx6NwhwzbL9izDqYxTlt86ihLYneOZTDEqfvTUaGTlZVnWv7fxPQxeNhhsglfGcsPUX6Zi\n1p+zivx7qQQYM7Zn0c5FOHvprO1vLmRfKPLxCoPfKINrhb/++8uy7N+0f7WH1l0W71yMWu/V8laz\nAAD3Lb4Pe87s8eo+nXElEoJ1Z9a1tWwLSwEvsF0+Zu0Y1Hu/nmX5gbQDqP+/+nhx3YvYe8Zqsebm\n5yI1MxWhr4qx2WW8+1Ke7g2kXkq1bYe74ZnOn3fG0fNH0fnzzjh87rDtNpxznM447XAfOfk5tu0I\nYLpoSLmYAkBY9+p12pe6zyBkZ/05C6cyTmHdwXWImByBL3d+aWiHmR5f9sDUX6Y6PsHLOAsRq0rc\nTmk/ufpJzP1rrmV5AS+wvS6bkjeh6eymDo83bt04jF47Gk+sesKh8OacY8epHYZzzszNxMbjGxE0\nKcj2N32+6oOPtn4EABiybAh2n7EaGsVtEPuNMnDVz8AX/RAKeAF6ftmzUDeh6eymFsFf671a6LGw\nBwDhan+w5QOXlkp2frZl2bHzx3Dvl/c6/Z2j0ENeQR6+2vWVIVThLpzzQifvcvJz0PyD5sgryMOv\nR3/F78d+x4kLJwp9bFfsS92Hfan7PN6PKuQ45wYBb5c3kFbo5F8m2wqbx1c9jjKvl0F2fjY458gt\nEILqYs5FfLz1Y3Ecm2f61Z9fRcTkCJftlc+kbHdgQKBlm2m/TEPXBV0R94bzGQQzcjLwxfYvDMtK\nBBh7ge49s9cgeO9bfJ/FSxixagRGfDsCt396OwCg79d9Uf/9+oZ2yv/y+pq9BfXcAHG9Ej5JEMuV\n65V0OAlPrHrC8Fzm5Oc4PU+V9ze9j7g34jB42WDDO7P24FrNoEs6nGT7nuYV5OGr3V/hq11fYePx\njYZ1/6b9i5mbZqLRrEYImBiADUc2AAA6ftIRLT9uqW3HwLRz3ZayDYB+D+f8NcdyPADYmLwRpzNO\no/Jbld0+z8LgN8rAFUXVinkFecjOswpWd0i7lIale5bifPZ5y7qc/ByLNSktkwJeYBF8/138DwBQ\n/d3qGL5yOABgxd4VDpVCaAnrbD8bjm7AN3u+cdrm8Mnh+OnQT4Zlu0/v1iwSRxawM0atGYVKb1Uq\n1G+k1ZWdl402c9vg1jm3auftbQpjKKRmptq63ep1+f3477begKPt5X3nnGPlvpX4Zvc3WH9Et2bV\nUNPpjNN4ZMUjDve7/dR2t85DGgvnss4Z2iCZvGEyNqdsxh/H/3C5r3yej/5L+mvhj6TDSYb1L6x7\nwXI9Vu1fZbuvs1lGa3n3md1Yvne51t6c/BwcTz+u7W/pHutcV+r9VL0kKQOOnDuCd/54BzP/nGl4\nH97/830ETtSV4qKdiwz7XbJ7CT7a+hF6LOyBf8+KEOzcv+ZqnwHjff3n5D8ARAHL/lQ9Z5lfkI/0\n7HQMWzkM7ea1Mxyj1nu18OTqJ7Xv0mOT+1LPAxC5gWYfNANgVcASee1afdwKQ5YPQfKFZJy9dNbr\n1VV+owxc5gyK6Bk8svwRVH67aJpUCrS0S2mWdWGvhWHMD2O07wt3LNQqIHLyc1DprUpITk/W1l/I\nuWDJAagPyKGzuoWUV5Bnmy8wPyyrD6zGnjN7LEk9s/I6ev6o9vnwucPYeWqnZd+Sl398Gbd+fCsA\nUdHQ56s++OXYLziZcRK5+bm4+4u7Hf5WRcaCM3L1BGRRFFFhmbxhMt7f9L7D9WVeL4PRP4y2LFdf\nLLuwU8qFFFzIvoDj6cfxwroXcDHnorbuaLq4vsfSj6Hbgm64d9G9Bm9l2Z5leP6H5wEAPb/sqS23\ne+bdtW6lkOy6oKvld5xzvPjji9hyYottHP3g2YO2xx6xagQazWqEDp90wHf7v9OWm613NoE5zKOc\nz7IaTt0Xdtee9ay8LJfJU9m2/IJ8QwJbyoBq71bDsr3WxP6WlC3aM8Y5R5+v+hjW91rUC0NXDMWy\nvcsMSmbRzkX47dhvAPRnNOVCiqZoAeDFH1/UPucW5Gq/d/eZVr2q7Pxs7Xej1+rP4uFzh7H5xGbL\nb1Vj9u+TfwMAYqfH4s7P7nTr2O7iN8rAjmm/TNMEsfrwDl8xHGwCw+3zb8eYH8ag79d9He5jY/JG\nt+L1dvFtKdBSM1PR88ue+HrX19q6Al6Abf9t077/fORn7SGXL+bZrLPaTT+VccqSA1BfihozauDv\n/8SNnrNtDl748QUARosvKMAYb+z8eWfc8P4NaPx/jTH+p/HaeUaFRAGA9iKpAuF/m/+HhrMaWs41\nvyAfC7YvwNK9S/H78d8BAN/s+QaLdi7SBOXZrLNYtX8V9qXuw8NLH9Z+a1fJcTpTKFJVaErX2B2m\n/zodjyx3bEHvPLUTd3x6hzhWxmkkJiWi6xdd8eKPL2LsurFYvnc5lu5ZitBXQ5F0OMlwrf/L+E97\nrgp4AYatGKa9rCv2rsCm5E2W41V8qyJ6fNkDVd6ugim/TEHrOa21dV/t+grz/prn0FKb+utUfL5d\nlK2qVXELdixAiw9bGLZ15MXmF+TjTOYZbD6xGcfTj2uhDemBSutxwfYF2v04n33eVrnUnFETaw+u\ntfVKZSWPmtuwQ4a9zNh50QAwZPkQAMD8v+cbDB8zLT9qiQ1HRWilxKQSaPC/Bk7boSLfjwJeYKsE\nVVRlMOnnSRiwZAC2pmzV7mHFtypifNJ4bZvFuxYDsD7DeQV5TkO+jDG89ftbhvswZ9scfLTtI8u2\nb//xNm7+8GbLcjVkrBp2Px760eFxi4LfKAM7y3/surEoPb20ZfnK/SsBAOsOrcP036Zj4Y6FOJN5\nRoe7d+sAAB1dSURBVLtQbALDsj3L8MvRX5CaKZJjmbmZqP9+fcROizUIdUCEcJp/0NygcE5lnMK3\n+0V8/aOtH2HpnqVYuHOh4XdSwBw9f9Qg9OQLPe3XaQa31Yz5gb2QcwHxb8dj3aF12jLZBsCxG5mZ\nm4mJP0/E8r3LAQCBLBB//fcXIqdEYuGOhejxZQ/Lb2b9OQtbTmzBXZ/dha0pW/H3yb/Rb0k/7Zye\nXv20Fk7ZkrIFgK6Y6s6si0/+/gQXsi/g6PmjiJ4ajdd/fR01Z9TU9i+9ooHfDNSWMca0pFu9mfWQ\nk5+DM5lnsHjnYkv7Fu1chI+3XY6t21ixa/5dg7UH1wIAei/ujQnrJ2jX6mLORXRf2B09v+yJ7Pxs\ndPikgyVcUnp6abzy0ytITErEh1s/1Jbfs/AeTP3VPqnpKEkLAIOWDXIoQJ0pwT9P/Kl9bjOnDb7/\n93sAwPifxiO/IB8PfPUAun7RFWPXjkXZ18vi5g9vRq9FvSzHSk5PRkZOBvot6adZuWmX0rT7uWT3\nEoNCdGVV2uWsHLH24Fp0/UJ4KK6S30+tfspisatsTDbG4NXzXLRzkdMIglTwgRMDkXwh2XYbmRi/\nkGMMFWbmZqL5B83x10lrAYiKXW7GFc+tec7w/fFVj2PeX/Pc+u3SPUsx/dfphT5mUfB4rFDGWCcA\n7wAIBPAR53yaaX0IgPkAmgFIBdCHc36ksMdR3SyzUIwKiUKXz7vgzxN/go8XD8vG5I2Y8ssUbZuU\nCylahr734t6oV6YetgzbgvTsdC0ks3DHQrSOb4346HgMXTFUE64fbP3AcFxZkSNfrqrvVDW0R4ZG\nPvvnM6fnZBY6pzNO41j6MRzbeUxb9sR3T+BC9gX0bdRX8zIyczNtY61SIQ1cOlDzahx5TSNWjdA+\nlwkvg/6N+gPQvRpZ46xiFkBRU6MQHiRCY3P+moODZw+i5JSSaF6huRYz//WYsZdl7PRYXBh3AXtT\n9+Jc1jmsPbgW/Zf0x2tpr2FM6zHay1a1VFVsSdmCsq+XxZnMM3i0+aOY1XUW8gvyceR8oR8f5BXk\nWQTJpJ8nFWofasjCjuPpx22XS2XqiJMXTyImLMZwrSb+PBGMMa0iR93HsfPHLHmPrgu6omP1jgBg\nm1DvtagXfh38K26tcqvTtkikEeUO0kMD7MNEZqTXWBSc5XJkXg6ArXcH6GGdY+ePGZbLUlpX/ShK\nBJSwDXNxzm3DyYXxhu1Qw4rFjUeeAWMsEMBMAJ0A1AfQlzF2g2mzIQBSOee1AbwNYBpskC8q5xwb\nj29EmznG3nlLdi/RMvfmcElwYLClPM7sHptv4J4zezBk+RBUeLOCFqfrt6QfBiwZgNz8XE0RqCzc\nsRDp2em44f0bbPcpUb0Ed6jydhUAwL2LrJVCF3MuYuDSgaj2TjXNwo+YHIH+S/pbtpXu99HzR126\nySpfbP8C49aNAwBDnBQAYsNitc92lUrSEpQC6GLORUPyVEW+LNtPiiRpena6ts8Xf3wRo9aMwi0f\n3YLk9GTt/snQ1x/Jf+DTvz/FjI0zUHNGTUudtivOXjqrWbt2itStfWTZlxJK7vrsLpf7CGRWy7L8\nm+UxaNkgy/IJ6ydon1VBl3IxRUs6qsjn+J2N79geO+VCCuq8V8dlGwHgh4M/uLWdGTVHVBw4qxxT\nw5VbTjhXwHZVTIBzZfDfxf8MJbcq3+z5BmVeL2NZrian/R1Pw0QtABzgnB/mnOcCWAigu2mbewB8\ncvnz1wBus9uRDBOtPrAaLT9uabEofzr8ExLXJ+LP5D8NIaWZnWcaBJjsCGQWXDIOriI9gqe/17t9\nbzi6wemLED01Wvu8MXmjrYBUKw/cwZFFCegdThy5vYbjutlLu2l5ax21DIGYa6c7VOugfbarR5e4\nk0iTidlb5wjrtPZ7tQ2VNe9sfAebkjeh8tuV8fsx4/1KzUzFwKUD8ebvbwIovMW1ct9Ki6IrLIUp\nXbQjJjTGYec2c2lnUZDC8PC5w4gOibas7724t/aMPNLUcT7masEsmNXQz4Gzzi18R8+Cs3tc4c0K\nDsNgS3YvsV2u5h38HU+VQSUAqr91/PIy220453kAzjPGYuEAZ5o5ryAPLT5qYaj7DgwIRHBgsHaT\n6v9P1DVLt09iZ3k56vbubsUMYB8KGrZymNu/d4Uj4WGHml9wxg8PWpWdFCRS0daIqYHzY8+jQVk9\ngdfhkw6W3xUGR9aYHWbFcyxdPGbSOncV2zUz5685qPBmhUL9xtvIkJq3qFiyIgAgLsLaj6Bx+cZO\nf3t/g/sdrmtV2doj2R8xt1MNndl59sWJLBC4mvFUGXitJ9i7U99FYmIiFv9vMeBAZtiVW5YNL4vM\n3EyDG10zpqZblrSrigkAaF+1vdP1rgS/XX+B4uCZls+4va0a+nFEaIlQRIVEaWWT3sCR9zDy5pH4\noOsHLn/fukprTembE3BVoqq41YYWlVq43ugykcGR+H2I0UOpVLJw/S2KkxvL3YjP7/3cNr5fMrik\n09/eUfMOPNT4Idt1T97ypO1yiZ1n6S7So/M0lg4A5SPLG76bi1Cqlapm+7s7atyBu2u7b/C5Qj3O\nyJtHOt22sOdtMSAOAfhJ+fMiniqDZADqW1gFwjswbxMPAIyxEgCiOeeWTMuTY55EYmIiKt1TCahu\nfzCzZflJj09w7w3WOHuHah0MNf6esLzvcqx/2HF3eFc398W2L2Jos6Eet2NF3xVO19eOrY3JHScb\nlg1pOsQ2xulOvD0kMAQA8FDjhzCpgzXR+nCThy3LmlWwxrHdoVmFZhja3P4a1Y6trX129gJLK9kO\nVQG8dedbbrerdmxttKzc0rDsgYYPWITMn0P/xEttX7LdR/VS+sNclL4y4UHhCA4Ug5SNazPOsC6v\nIA/9GvWz9R5DSoS43LddwvPwU4fRp4Hjah8AuK26baQXjcs1xgddP3CoMGd2nom9I0XPY7Mgl8sK\no2jKhpd1uj6sRJjt8tuq34aV/Vai1w293D4WALzW8TVsG77NsjwmNAalQksBALrXM0fJjdQuXdt2\nuV0uCdBzcj3rXU4kVwdKdykNdID48yKeKoPNAGozxqoxxoIB9AFg9s+WA5AmSG8A62ADB8e/af9i\n4Y6FdqttySvIsxVsdUrX0cIK7jC+/XisG2ht1qPNH0VUSBTaVW2HgY0H2vzS9QvetU5XzO462+22\nOKJRXCOn64MCgzCu7TiDoHqp3Uv4+eGfAdi/fJJasaL/gyp4pQBKqJaAl9oZBd1zrZ7DlNv0Sq2O\n1Ttidf/V+LDbh3BGzks5tiEIZyGLshFl8fE9osTULhwisXuxpeBXwwet41tbtnNEvTLWypU37nwD\nO0fsRP2y9bVlTcs3RY0YMSKnWRCqBkxmbqbFMxnURIQvDzxxQDuHG8roNRhxEXF4reNruDjuItrG\ntzVYirImXi2o6FCtAy6Ou4iQwBCUCbcmNFXshtmoWqoqGGOG+7uy70rDNt3qdrPdX5v4NhjafKjD\nJHLp8NKaV2pntac8l4Jhzd0LsaaPTXeZp3KUbJbeurNwWLmIcgCAqtGiUrB73e4Y3nw4mpRvohlK\nkszcTOwasQspz6Xgtuq3IbF9omG9vGf31b/PYXn4K+1fcXour3Z8FRfGiee4MKHjwuCRMricAxgJ\n4HsAuwB8yTnfzRibwBiTT8zHAEozxvYDeBqA7VCNOfk5mP/3fJQOs/YrWPvgWu1zndKiGuLdTu+i\ne12hhfc8bhx8ze4lkGGIiCDjuC/rBq5DYkKidvNVpt2hFz7ZVQ7tf8J1wja0RKimsBw9CJLokGj8\nNvg3y/Lsl7JRtZSxfLV5heaoGaPX9UvLQg0PlAothdbxrcHHc9twV/9G/fHPo/9o1poq/BzV1MdH\nx2PKbVMQGRypLasWXQ131brLZUw8KDAIPz1k9W0jgu3H4vl+wPf4ps83Wr8NR8m72LBYjLp1FJIe\nStKWRYVE4ZlWInTmKFm4YdAGw3epACWOQi3hQeHYOULvxR0YEIi6ZcQ4Pb8OdjxhSXp2uuG6AXro\nrGZsTe36qfH+NvFt0KR8E0QER6Bz7c7YNWKXtk4KhbQxaYZlEcERCCkRggqR1hzJi21fxOahouro\nz6F/WtZLxrYZq4XeOtXqpC3fOmwr2lVtZ1EQgH793rjjDVtBGxsWqykDR4o9JjTGYZtUQkuEIqfA\ncbJ3UodJDoWmVAbPtHoGC3tZjc9PenyCBb0WAAAaxokOmksfWIrS4UI2JT+bbJA5GbkZqFCyAspH\nlgdjDOMTjEljeV+/7P0lxrUZhydaPGE5ZlBAkK2C/PxekYsoG15We3bqlnZv5NjC4nGnM875d5zz\nupzzWpzzKZeXjeecr7j8OZtzfj/nvDbnvCXn/LDdfhrNaoSJP09EQrUE8PEcB57QE8kdq3fE7sd3\nY3LHydg6bCvOjz2PJ295Urs55hCB+YUDgMFNBwOAZtGVjyyPmNAYrTY7LMjqUsqevADw+h2v46Nu\nxl6DQQFBWNVvFXY/vht8PAcfz3FzRWMPQvngTbt9msHis+Pc2HNoVUV/iUJLhKJNfBuLkAKAzcM2\no3KUPsyGVDRqz1C1/fIlPPSUsFT3jtyLeT3moVG5RghgAbgw7oJ2jQBrAl4yoNEABAUGITI4UktE\nS+9IVQa3VLrF9vcyfNG4nBB4qsdyX/37tM+L71uM22vcjriIOC23c2O5G7X1Pzz4AyYkiNLLQBYI\nxpihhl4+A9Eh0YbrAEDzNOQ1+66/GHrBXLJsVlJd63S1PScAuKniTUhsn6iF5WJCYyz3rYAX4L3O\n7xmWqW3TBp1TQgZTbpuC22vcrn1Xwz/SQIkMjsTBJ4WVLzsGhgSGWM4bECGS5hXFjGJhQWGYfrvj\nDk2yPDkwIFCrTmpaQYRxYsKE0B7bWrftpMU8pNkQ/DbEatTEhsVqhpFqyDzX6jks7SPKfdV7LK1y\nAOhcqzMAkcMb23osSgSUMBhw87rP0z43jGuIl9q9hCdaPIHJHSdjWLNhhnBudKg4lwAWgLvrWEOP\nAxsPRIfqHZD9Urb2nKqUDi+tKX/Acf+T4c3FWFzyvjLGMODGAZjReYZl21qxtXDoqUOa8okMjsT2\nx7ajTbwosZfvLx/PHeZ6PMVveiBLpFVfM7amFvNjjKFemXoY13YcIoIjLA95yRCjBRdSIgSLei8y\nhFZkZ6bG5RqjQmQFJD2UhBPP6YPJuUr0VixZ0eIeBwcGo3PtzgaLetPQTYZyTPmCjG492qFl68hK\nqhJVxWK9VipZCZkviP3Irv+lQktp8Xp1+Ao1XyA9Lml91Cldx+CpRAZHai+XrTJtMhjLH1iOiR0m\nastUIQUYva5asbXwUbePsG34NoNlCYgH+q27RAhnx2N6Rdei+xbh0eaPAgB61++ttV+eU/tq7bWw\nSuWoytrxHrvpMQDC85Avyh9DRI/jvSP3YsOgDehcq7OWc5DPj/Q4pPA19y5VldvoW0c7zdsEBwZj\nfMJ4TUimjUnDuTHncPr500gfm649i2p4aeeInZh6+1QceVp0opt+x3SsGbAGHap10J4Js7cln6c2\n8W0MeYrqMSI3IY2Bhxo/hFfav4JhzYahQmQFbB221fYcR906CqXDSqNtfFuLp6AWWJg9Y6k41TyR\nWfk92eJJg2KTAu3cmHOY1HES7qp5F/Jfycfrd7yuxdrrlqmrGV1q50JprFWKqoQpt08BYwwTO0xE\nynMphvMH9Hd5RucZGNd2HGZ3m42HmugCVPU+IoMjsefxPZoQVgkODHaae+HjOeqVqWdQDCp9G4oO\nn3blqqpyynwhE73r9wYgoiB7Ht+DM8+fQcO4hogNi0Xt2NqG+1YpSoQizeEqT/G4B7K3cZUUcsXI\nm0fipoo3oWLJilrnG9XLKBlS0qAEJGqyqWe9npqbqCIfsqCAIOQW5CIo0H5scjVMpXocasXT9Nun\na4NUPX3L0+jbyNpT2Dz+S93SdVE+sry2T9kn4OyYsw5/I3Gngkgq1ZLBJS0d5z7u/rHL36uCK7cg\nF0OaifFoZnSaoQ2RIGlftT0GNxmsJd4kdtd0xM0jNKEzp/scvNPpHUSFRKGAF2DtobWY0EHvnDWv\nxzzM6zFP+14uUii4Vf31UTbli9i2alv8PuR3TVHO7DwTy/Yuw9e7xXAlqnKzU5B2RAZHar3gw4LC\ntHs1o/MMrVfsqVGnEPdGnKYY4qPjAQiDQ3q5Q5oNAZvALGFNKZym3jbVNv8hFaf0MO+seSdmQ89Z\nma83YwzJzyaDMWYR5mrHRankJFI4qYaZWXC+2/ldDGs+DJuSN2Hw8sHaMygt89UDVlvaH8ACMKTZ\nELyz8R1D6XdIYAh+G/yblt8ChLCWnqUaYrETkuoy8zWoW6Yutj+2HQt3LLT02HclcLcM22JbRCKf\nAQDo17AfKpQ0huy2DBODCKqRAEA8r/KZBcTztO8JY+6jSfkmAICsl7LAXvbOpD2AHyoDVWgVpvri\n4SYP47lWzxk0vHyYa8YKl3T0raMdJoJVzyC0RKitRSC3SX42GXFvxNmGbwD95VYfCMDYO1ONZ3ap\n3cXwME+7fRrGrB1jGRly2/BthoS5XY9Y+Zucl4zWiAypOaN+2fr4+v6vMWbtGJfb2hFaIhR3174b\nI24eYeifULt0bUsVRWBAoK2CsbumJUNK4o6a+pAHUgDVL1tfC/EUBulxBLAArVpI3qsutbsg9VIq\nkg4naQpoSNMh6NPQWmEzo9MM9KhnHffJjoRqCUiolgBAJMWlN+AM8/MD6MLJ7A0DwLBmw5xWdJ0c\nddLWC3Vk/X4/4HstxPHn0D8NCVvpKajK2+7eNYhroIXb7MJWjljUexE2Jm/EoGWD8PQtT6P/jf1x\nU8WbbLdNHZ1qUJqqMJWoQt1RjqpPgz7oUruLYZmr98adviOt41tb5I4MtxWF+Oh4nBvjWQdKO/xO\nGdjF7t1hbnfrBCMjbhphGNNFTQjbHTfrxSw0md1Ei0+aka6xtCwclYNN7DDRNq43u+tslAothbLh\nZbUe0Xkv51lc99GtR2PM2jEW99J8bUbcNMIyoJj0DMwWtl1i3kwAC8C9N9yL5XuXF6q3rlTajDGs\n7GdNLBaGPg36OO2R7Q3UkIKZ0uGlLUnuj+6xjjAJAE/cYk0Euos0GApLYEAgAlmgrWCd3c151Zqz\naiw7ZD4NsFajyZ73dUvXRXRINDJyMxzmieR742goBztuKHsDKpSsgEHLBmHybZOdyoXYsFhtOJvt\nj2237XPyQtsX0L5ae/Ra1Muhl8wYs1zXh5s87FEnvDtr3qnF/b2J9K68Cufc538AOBLF39xtc7nk\nqe+e4kOWDeH+yNu/v80LCgqK/Ptle5ZxJMLh+gXbF/Bv931b6P1GT4m23e+J9BM8/u14t/aRm5/L\ns3Kz3NoWieAPL324UG0kPCPitQiempnq0zacSD/B67xXx+3tM3IyCn2MC9kXOBLh9nuWnpXulW2u\nJoQI944cZtwPJppnjHEkis9L+yx12XGDcEz7ee1xOuM0dj2+y/XGXmDlvpVoWr6pltQiip8tJ7Zo\nFUHXMgW8AI+tfMylx3M9wxgD59wriQO/UQavrn8VCdUS0KpKq0K5k4SR7LxscPArNgwGQRC+45pU\nBv7QDoIgiKsJbyoDMsEJ4v/bu7cYu6o6juPfXy9cBKEtCMV2dJoAoYBgay0jmqBYSCHKk9wEM1yi\nmBghfVCrJsCLDzw0LUhEI5ZwidIKBTuJYAu0CcaAXEotLbWDFuwUOjTVGsXES/j7sNbh7HbOtHM5\n5+yZ2b9PsjP7rL3PzNrr/M/5z977rLXMzMnAzMycDMzMDCcDMzPDycDMzHAyMDMznAzMzAwnAzMz\nw8nAzMxwMjAzM0aRDCTNkLRe0g5J6yRNa7DPxyX9TtKrkjZLGnzmczMzK81ozgyWAusj4nTgaRpP\ndP8u8JWIOBtYDKyQNPQZLipo48aNZVdhzHBb1Lkt6twWrTGaZHAZcH9evx8YMOVTRPRGxJ/y+tvA\nO8Do5rWc4BzodW6LOrdFnduiNUaTDE6OiP683g8MnGuuQNJCYGotOZiZ2dhxyGkvJa0HZjbY9P3i\ng4gISYOOQS3pFOABoPEExGZmVqoRz2cgaTvw2YjYkz/sN0TEGQ32Ow7YAPwgItYM8rs8mYGZ2Qg0\naz6DQ54ZHMZaoBu4I/98/OAdJB0BPAY8MFgigOYdjJmZjcxozgxmAKuBjwBvAFdExH5JC4CbIuKr\nkq4FVgJbC0/tjog/jK7aZmbWTGNi2kszMytX6T2QJS2WtF1Sr6TvlF2fVpPUIWmDpK25M97NuXzQ\nTnyS7srts1nSvPJq33ySJkvaJKknP54j6fl8vA9LmprLj5S0Kpc/J+mj5da8uSRNk/SIpNckbZN0\nXoVjYkl+b2yR9PP82lciLiStlNQvaUuhbNhxIKk7779D0pC+uFNqMpA0Gbib1CHtTOBqSXPLrFMb\n/BdYEhFnAV3AN/IxN+zEJ+lS4NSIOA34GnBPOdVumVuAbUDtFPUOYFk+3r8BN+byG4F9uXx53m8i\nuRP4dUTMBc4BtlPBmJA0C/gm8ImI+BgwGbiK6sTFfaTPw6JhxUG+hH8rsDAvtzUaIWKAiChtAT4F\nPFl4vBRYWmadSmiDx4FFpDf/yblsJrA9r/8EuLKw//v7jfcFmA08BXwO6Mlle4FJeb2rFh/Ak8B5\neX0KsLfs+jexHY4H/tygvIoxMQv4CzA9v849wMVVigugE9gywjiYCVwN3FMo/zFw1eH+btmXiWYB\nuwqP+3JZJUjqBOYBzzN4J74PM7CNZrepiq22HPgW8B6ApBOA/RHxXt6+m3o8vB8rEfE/4O/5P6CJ\nYA6wV9J9kl6W9FNJx1DBmIiI3cAyUkJ4C9gPvEQ146JmOHEwK5f3NSg/pLKTQWXvXks6FngUuCUi\n/lHcFimdF9vm4K/ejvt2k/QF4J2I2ET9+Kr6FeMpwHzgRxExnzSm1wFjfVUhJgAkTScNddNJ+lA7\nFrikzDqNJUOIgxErOxnsBjoKjzs4MKNNSPnm16PAgxFR65/RL2lm3n4KaRwnGNhGs3PZeHc+cJmk\nncAvgAuBFcA0SbW4nE09HnaTvsaMpCnA8RHx1/ZWuWX6gL6IeCE/foSUHPZULCYgXTLdGRH78n/6\na4BPU824qBnOZ0Nfg/Ihfa6WnQxeBE6T1Jk7qF1J6sw2YUkS8DNgW0SsKGyqdeKDAzvxrSUP4yGp\ni3S63M84FxHfi4iOiJhDukH4TERcS+qtfnnerRv4VV4vts+XSDfSJoSI2APsknR6LlpE6pvTQ4Vi\nInsT6JJ0dH6v1NqicnFRMNzPhnXAxfkbatOBi4DfHPavjIGbJZcAfwReB75bdn3acLyfIV0jfwXY\nlJfFwAzSzdQd+cWcVnjO3bl9NgPzyz6GFrTJBcDavD6HdA+lF1hFGtwQ4EhSJ8de4Dmgs+x6N7kN\nzgVeyK/xGtJN5UrGBHA78BqwhTQi8tSqxAXpLPkt4D+k+wHXjyQO8vN689I9lL/tTmdmZlb6ZSIz\nMxsDnAzMzMzJwMzMnAzMzAwnAzMzw8nAzMxwMrAKkfTPsutgNlY5GViVtKVTTR4WwWxccTKwSpP0\nxTwpyst5ApGTJE3Kk4KcmPeZJOl1SSdI+lCehOb3eTk/73O7pAcl/ZbUa9ZsXHEysKp7NiK6Io0W\nugr4dqShkh8Crsn7LAI2RcQ+0iQ0yyNiIWksnHsLv+sM4PMRcQ1m44xPZ63qOiStJk0KcgSwM5ev\nJA2GdidwA2kGKkiJYW4aQw2AD+a5B4I0vtK/21Vxs2bymYFV3Q+BuyLiHOAm4CiAiOgjDR18IfBJ\n4Im8v0gza83LS0dEvJu3/avNdTdrGicDq7rjSKNEAlx30LZ7SZeLVkd9RMd1wM21HSSd2+oKmrWD\nk4FVyQck7SosS0jDJf9S0oukeXaL3zjqAY6hfokIUiJYIGmzpK2ks4kaDwFs45aHsDYbhKQFwLKI\nuKDsupi1mm8gmzUgaSnwdeDLZdfFrB18ZmBmZr5nYGZmTgZmZoaTgZmZ4WRgZmY4GZiZGU4GZmYG\n/B+VGdKQdxYt/gAAAABJRU5ErkJggg==\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x7ff8ed181f10>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"n_dims = 1000\n",
"input_mean = 1.1\n",
"input_var = .1\n",
"\n",
"x = np.random.randn(n_dims)\n",
"x = (x - x.mean()) / x.std()\n",
"x = np.sqrt(input_var) * x + input_mean\n",
"\n",
"W = np.random.randn(n_dims, n_dims)\n",
"W -= W.mean(axis=1, keepdims=True)\n",
"W /= np.sqrt(np.sum(W ** 2, axis=1, keepdims=True))\n",
"\n",
"assert np.allclose(x.mean(), input_mean)\n",
"assert np.allclose(x.var(), input_var)\n",
"assert np.allclose(W.sum(axis=1), 0)\n",
"assert np.allclose(np.sum(W ** 2, axis=1), 1)\n",
"\n",
"activations = [x]\n",
"for _ in range(1000):\n",
" h = selu(np.dot(W, activations[-1]))\n",
" activations.append(h)\n",
"\n",
"plt.plot([a.var() for a in activations], label='variance')\n",
"plt.plot([a.mean() for a in activations], label='mean')\n",
"plt.xlabel('Layer')\n",
"plt.legend(loc='center right')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Even with initialization far away from the fixed point, the random SNN normalizes the activations over 1000 layers. This is the key property and benefit of SNNs. Re-running this experiment with different activation functions such as as sigmoid, RELU, or ELU gives dramatically different results.\n",
"\n",
"## Dropout for SNNs\n",
"\n",
"The paper introduces a new type of dropout (called alpha dropout) for use with SNNs. Rather than dropping out to 0 value, they drop out to the minimum SELU value $\\alpha' = -\\lambda \\alpha \\approx -1.7580$. To keep the self-normalizing property, the dropout scheme must not alter the mean or variance of the activation. One can solve for an affine function $am(h) + b$ that preserves the moments of $h$ after setting units to $\\alpha'$ with probability $1-q$ through mask function $m$."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAEKCAYAAADzQPVvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xm4HFWd//H3JyEEWZMABkISAkKGZRDQn4CiEgQBGQVx\nBEGRiChuuDCjAurARdRBZ1RcRtAx4MomKIKCJmKCuICiyBaQRcISSEBIYhJgJOT7++OcTup2uvv2\n7Xt7u/15PU8/XV11qupUV/W3Tp2qPkcRgZmZjWyj2p0BMzNrPgd7M7Me4GBvZtYDHOzNzHqAg72Z\nWQ9wsDcz6wEO9i0i6VuSzmp3PswqkbRc0rR258Oax8E+k7RA0mJJGxbGvUPS3GFaReRXR5E0T9IJ\n7c5HJ5E0TdJcSSsl3SnpgBppx0o6X9IySY9KOrls+gGS7srL+qWkqfXO20oRsUlELGjX+oeLpAmS\nfiRpRf5NH1Mj7ThJ386/+8WSziib/jJJv5f0d0m3SNq3MG1/SbdKWiLpb5J+KGlSM7dtqBzs+xsF\nfLCJy1fFkdJ6TVznQDruBNQBLgL+CEwAPg5cJmmLKmn7gBcAU4H9gY9KOhggz3N5XsZ44Cbgknrm\nbZU2H3vN8D/AM8DzgbcA50rapUraLwIbANsCewFvlfQ2SCcN4Crgs8BmwOeAqySNy/PeARwUEeOB\nrYF7gHObsUHDJiL8Sv8ivh84BXgC2CyPewcwt5DmZcAfgKXA74GX1ljensCfgL8DF5MCyFl52gzg\nYeCjwKPAt4H1gXOAhfn1RWD9svSnAY/nvL65sK7NgO8AjwELSMFFeVof8N1C2mnAamA08GlgFfA0\nsBz4ch3f09uA3wBfAJYA9wEvzeMfBBYDxxXSjwX+G3gAWET6QWyQp40DfpLz/STpx7VNYd55wCeB\nX+fv8efA5k0+DqaTgsVGhXG/At5VJf1C4MDC508CF+XhE4FfF6ZtCDwFTB9o3gHyuHc+blQYdwRw\nSx7eC/hd3j+PAF8BxhTSrgbeSwpQ9xXGbZ+H/wW4GViW9+kZFY6f4/I+fRz4WGH6KOBjwL15n90E\nTM7TdgLmkH5jdwFHDvO+2wj4P2CHwrjvAP9ZJf3jwIsLn08DfpWHXwvcXpb+L8DbKyxnLPCf5ek7\n7eWSfX83kQLMh8sn5DP9T0kBeQIp2P00jy9Puz5wBSmIjwd+APwr/UvRE/O0qcC7gE+QfqS759de\neVwx/ebAJGAm8A1J0/O0rwCbANsB+5F+iMfnadVK7hERHweuB94X6TL+A1XSltsLuIX0PVxIKq2+\nmFRKPRb4aqE67Gxgh7xNOwDbAKfnaaOAWfk7mEo66Xy1bF3HkE4kzyedENfZNwCSpuZL6mqvo+vc\ntl2Bv0bEysK4W/L48nWWSnW3FEbfWki7a3FaRDxFOjnuWse8VUXEjcBKoFi99Gbg+3l4FekKdXPS\nifgAUnAvOhx4CVCp1LsCODYiNiMF/vdIOrwszb6kE+MBwOmS/imP/3fgaOA1EbEp6Th8StJGpED/\nPWDLnOZrknautI2SvlZjX/65ylczHVgVEfcWxlXcdwWjyob/uZiNCmnXLKt0zJFO4P9OKv13rnaf\nbTrlRSotv4q0M5cCW1Ao2QNvBW4om+e3wMwKy3olsLBs3G+AT+bhGaQSyPqF6fcChxQ+HwTcX0j/\nLPC8wvRLSCeD0XlZOxWmnVjIdx+VS/aj8ue5wAmD+J7eBtxd+LxbXt6WhXF/A15I+rGsIJcY87SX\nkoJppWXvATxZ+DyX/qXG9wDXNPk4eCvwu7JxnwIuqJB2St724n58dWG/zaKsVEm6SjluoHnryOdZ\nwKw8vEn+nqdUSfsh4IeFz6uBGWVp1pTsK8x/DvCFsuNnUmH6jcBRefgvwOsqLONN5FJzYdzXgdOH\ncd+9Ani0bNw7KVydl037LqmabWNSQeQ+4Ok8bXPSldHRwBhSAes54NwKyxlPukrfu5nH5lBfLtmX\niYg7SFULp9K/VDyJdElb9ACppFpuEukSvTxt0eMR8Y+yeYppHszjSpZExNNly9uadFCOqTBvpXxV\nM9h6+8WF4acBIuLxsnEbk0pwGwJ/LJXKgGtIJ1IkbSjp6/lG2jLgOmAzScUS1aIKyx02ku7IT6L8\nPd+AWw5sWpZsM1KVRLkV+b2YftO8DKosqzR9oHkHciHwhnwV+QbgjxHxUN6m6ZJ+km/6LiNV121e\nNv9D1RYsae98g/oxSUtJV57l8xf3y1Os3S+TSUGz3LbA3sUSOulqZGJdW1ufFVT/viv5AOmYugf4\nEek7XQgQEU+Qrn7+jbStBwO/IFWn9hMRS0hX8T+W1LExtWMz1mZnkEoExYC5kHTAFm1LhZ1Pqk8t\nD7bl85YH2EdIpaaSqXlcyfjik0J5eY+QStHPVpi3lK+VpIBbstUA+RhOfyP9mHaJiPH5NS7S5T2k\nS9/pwF6Rqgz2I10NVLyRXUu+pF5e41XxqYyI2DVSFdamEfEbYD6wvaTiSWV30g258nmXkPb1HmVp\nb8/Dd+TPpTxuRKrquqOOeWuKiDtJJ/jXkILmhYXJ5+bt2CF/rx9n3d96rf1+IakacnJEjAPOqzB/\nNQ+RSsnlHgSuKxwH4/P3/r5KC5F0Xo19eVuVdd8NrCepuP6q32lELImIYyNi64jYjXSVfGNh+q8i\nYq+I2Jx0NbYT6V5dJWNIVY3lJ5vO0e5Li055katxCp+/QbqRNDf6X9YdA6xHuix9EphQYVmlkvYH\n8vAbgH/QvxrnobJ5ziJV9WyRX78uS/8s8F95ea8glWJKN/q+C/yQVLraFriTfCMJOJB0I2oKqYT6\nY/pX41wEfLosL/Mo3JQrm/Y24PrC5x2A1WVpHgJelofPIVU5bZk/b0N6igHSkw5Xk25wTSCVrqpW\nMZWvu4nHwu/yd70B6cbnEqrcGCbdmJtHutm8E+kEXNq+LUlVgm/Iy/os8Nt65s3TVwOvrJHPj+bv\n6KnicUgKWP9BOmnuRKpaub5suduXLat4g3bNTXbS/ZnFwHfy52nFfVTYT6Xj7cOkevId8vpfmPft\nxqSHB44lHcNjSPcMdqq2fQ3uu4tIJ6sNSfcVlgI7V0m7Pel3PZp00ny8mJb0kMUYUgA/p+w7PIJU\nUBmV9/OlwE3NPjaH9N20OwOd8mLdYD+ZVCr9ZWHcvqSbuEtJT+W8rMbyXsy6T+MUg/eDZenHAl/K\nP/hH8sFVfBrnIdJTDo/nH81bCvOOIwX8x0glqE/Q/0mNr5IC1t2k+xDPsTag7pODwZPAOXncvcAB\nVbZrJoW61/yjfq4sTTHYjyVVI9xHerpjPnBSnrZ1DhTLSU9nnFiWtzVBpNK6m3gsbMvaIHpn2XHx\nFgpPXZBuGs/K27YI+FDZsg7Iy3gK+CUwtZ55SSfnZcD4Gvmckr+vq8rGvyKvcznpSaIzy/bZc6wb\n7NeMIz1MsCAfu1cBX6Z/sF+zj8r3Eyn4fRz4a57/RnL9Pik4lp6++hupWuSFw7zvxpMKDSvyNhxd\n9r0sL3w+knTFvpL0W3112bIuJP3Wl5J+v1sUpp2Ut3EF6QrtQqrcM+mUV+nxvEGRtAGpfnUsqZR7\nWUT0SdqOFNgmkJ5TfmtEPCtpLOkRqBeRSstviojyOmyrQtIM0k3WKS1Y12Tg4oh4ebPXZdVJegup\n+uvj7c6LjQwN1dlHxDPA/hGxB6nO8RBJe5MuUz8fETuSSpKlf2aeADyRx38xp7MOFBEPO9C3X0R8\n34HehlPDN2gjPTMM6VJ0DOmGz/7AZXn8t4HX5+HD8mdIjzpV/fu5VdXMG6lmNsI1HOwljcp/blgM\nzCbVyS6NiNU5yULWPpGyDflRr4hYBSyr9Gckqywi5kXE1IFTmplVNpSS/epcjTOZ9PftnYYtV2Zm\nNqyG3AhSRCxTahnypcA4SaNy6X4ya5/1Xkh+bjw3vLRZRDxZvixJrqowM2tARNT8f0pDJXtJW5Ra\nf5P0PNLfvO8kPYJ1ZE42k/RMN8CV+TPAG4Fra2S4a19nnHFG2/PQi3l3/tv/cv7b+6pHoyX7rYFv\nSxpNOmFcEhE/lTQfuFjSp0jPrc7K6WcB35V0D+nRy3obpTIzs2HQULCPiNtIz8yXj7+fVH9fPv7/\ngKMaWZeZmQ2d28YZRjNmzGh3FhrWzXkH57/dnP/O19A/aJtFUnRSfszMuoEkohk3aM3MrLs42JuZ\n9QAHezOzHjDSepZvuv6dKCW+z2Bmnc7BviHF4D7oTpXMzFrO1ThmZj2g0eYSpuQOie+QdLukD+Tx\nfZIelnRzfr2mMM9pku6RdJekg4ZrA8zMbGCN9lS1FbBVRPw5d8z8R1Lb9UeRuv36Qln6XUjddr2E\n1NzxL0j9p64uS9fxz9mnOvv+1TidnuehKL9HMZK31axbNe05+4hYFBF/zsMrSI2gldqur7TCw4GL\nIuLZiFhA6uN0r0bW3Y0k9Xt1n8B9p5h1tyHX2UuaRuqF/YY86iRJt0iaVWoZE5jE2uaOycPb0FO6\nJ2B294nJzCoZUrDPVTiXAR/MJfxzge1J/dI+Cny+xuzdEfl6VvecnMxsYA0/eilpDKk/2e9FxBUA\nEfFYYfo3gavyx4XAlMLsk/O4dfT19a0ZnjFjRk80UGRmNhjz5s1j3rx5g5qn0Ru0InUg/kREnFwY\nv3VEPJqHTwZeEhFvLtyg3Yu1N2h3KL8bO1Jv0Pafp/Nv6Jbnt5vybtaL6rlB22jJfl/gWOBWSTfn\ncR8DjpG0Byk63A+8CyAi5ku6FJgPrALe2/FRfRCKddsjaLPMbARxE8eDVKlkP1DJ1yV7M2umZpbs\nRyQ/Uz6wak/o+Lsy62wO9usolmhtXZVL/JX45GnWOdw2jjWZH+E06wQu2dfQyI3XbrxZ6z9PmY18\nPR/sawe6tdUU9QfEbq0GcrPNZiNZzwf7pJ4APfgg3ksl5m68ojHrJa6zb6peqq/upW016z4O9mZm\nPWC4Oy+ZIGmOpLslzS60eomkL+fOS26RtOdwbUA3GumtSlbbtu5v6tmsezVasn8WODkidgX2Ad4n\naWfgVGBOREwHrs2fkXQoqS2cHYETSa1j9rCRXuVRbfuixjQza6bh7rzkMFIDaeT31+fhw0vjI+JG\nYJykiUPIt5mZDcJwdl5yIzAxIhbnSYuBUkCfBDxUmO1hUjPHVoeRWv0xErfJrFMN6dHL3HnJ5aTO\nS5aXPX4Xkmo9vF3xWr7X2rOv/5HFbn1+v5Z1t8lNLJgNrGXt2cOazkt+AlwTEefkcXcBMyJikaSt\ngbkRsZOk84B5EXFxId1+hauA0jJb3uplrRYeBx5uZJ7BtyjZ7FYzB2rJc/iHK00rciubZoPRtA7H\nc+cls4D5pUCfXQnMzMMzgSsK44/L8+4DLC0P9LZudU1vVXH4xq1ZMzXaU9XLgV8Bt7L2F3oa8Hvg\nUmAqsAA4KiKW5nm+ChwCrASOj4g/VVhuS0r26wbQzijZ17uOkVmyd/v5Zo2qp2Tfk52XDK3qpv2B\n0cHezIrceYl1PXeWYjY8HOy7UG8FwPo7SzGz6nom2I+sG52VA6BbnjSzanqsIbSR/sTHSN8+M2tU\njwV7M7Pe5GBvZtYDeqbOvtfUc4/C9fpmvcPBfsQa+Bn2kXXT2sxqabS5hPMlLZZ0W2Fcn6SHJd2c\nX68pTDstd1xyl6SDhiPjNhy694Zu7zUnYTY0jdbZX0Bq+qAogC9ExJ75dQ2ApF2ANwG75Hm+Jsn3\nCmyI1p6oHPjNBtZo5yXXA0sqTKr0azscuCgino2IBcC9wF6NrNessu69QjFrleEuYZ+U+5idVeh/\ndhKps5KSh0m9WpmZWYsM5w3ac4FP5uGzgM8DJ1RJW7UY1mudl5iZDVarOy+ZBlwVEbvVmibpVICI\nODtP+xlwRu6Ltny+prV6OXwtXXZuC5Hds77mrcOPk1ovalrnJVVWtnXh4xFA6UmdK4GjJa0vaTtg\nR1K792Zm1iINVeNIugjYD9hC0kPAGcAMSXuQiln3A+8CiIj5ki4F5gOrgPe2vO9B6zmVnszxYWe9\nrGc6L3E1Tietr5nrKOq/vk461s2GU0urccw6gx/DNKvEzSVYzymv4nGJ33qBS/bWo3wFYL3Fwd7M\nrAe4Gsd6xkBt57h6x0YyB3vrIZWf2ukf5N2puY1MrsYxc/299QAHezOzHtBwsK/SgckESXMk3S1p\ndqHlSyR9OXdgcoukPYeacTMzq99QSvaVOjA5FZgTEdOBa/NnJB0K7BAROwInklrIbDp3amFmljQc\n7Kt0YHIY8O08/G3g9Xn48NL43NrlOEkTG1334Lg+1sxsuOvsJ0bE4jy8GCgF9EnAQ4V0DwOTh3nd\nZmZWRdMevYyIkFTe8lW/JJXmc+clZma1tbTzEli3AxNJdwEzImJRbt9+bkTsJOk8YF5EXFxIt1/h\nKqC0vGFt9bI5LV2OpBYivU31tqDpP1hZJ2tHq5dXAjPz8EzgisL443Km9gGWlgd6s87j+z02cjRc\njVOhA5PTgbOBSyWdACwAjgKIiKslHSrpXmAlcPxQM27WDm5SwbrViO68xNU4nbq+7tum0nFZfkx1\n0u/Hepc7LzEzM8ANoZnVxX/Ms27nkr1ZXXyz1rqbg72ZWQ8YUdU4vtQ2M6tsRAX7pNafds3MetMI\nDPZm7eFn8K2Tuc7ebFj5Rq51pqaU7CUtAP4OPAc8GxF7SZoAXAJsS/53bUQsbcb6zVrF94msWzSr\nZB+kBtH2jIi98riKHZuYdbegWmnenedYJ2lmNU75EV6tYxOzEcpVOtY5mlmyny3pJknvzOOqdWxi\nZmZN1qyncfaNiEclbQnMye3Xr1GhY5M13HmJjUTFqhw/pWND1fLOS+pagXQGsAJ4JxU6NilLO6RW\nL/u3SAjd0JpiZ6zD29TKdfRvQXMtnwSsUW1p9VLShpI2ycMbAQcBt1G9YxOzntL/pq3r9a01mlGN\nMxH4UT6Y1wO+HxGzJd1EhY5NzHpPscS/Lpf4rRmGPdhHxP3AHhXGPwkcOJzrevrpp7nuuuuGc5Fm\nHaL2CcFssLq6uYTFixfzutcdwUYb7Ycvhc3MquvqYA8wduxEli37GSnYu/UHM7NKuj7Ym40U1f5p\n68c2bTi4KGzWMao9meMndmzoXLI361J+ascGwyV7s67mUr/VxyV7sy7iFjStUQ72Zl1l4G4366ne\nqbcKyFVFI0dLq3EkHSLpLkn3SDqlleu2gcxrdwaGaF67M9B21ZphKLarP1BTDfWmKzfYRrk6Tbfn\nvx4tC/aSRgNfBQ4BdgGOkbRzq9ZvA5nX7gwM0bx2Z6AD1PM0Tz11/NU7ZKmm24Nlt+e/Hq0s2e8F\n3BsRCyLiWeBi4PAWrt/MqnCPWiNfK+vstwEeKnx+GNh7qAt99tllwBfwEwlmQ1FfWzy1Tghnnnlm\nWlId9whq5sT3BZqi6e3Zr1mR9K/AIRHxzvz5WGDviHh/IY33splZAwZqz76VJfuFwJTC5ymk0v0a\nA2XWzMwa08o6+5uAHSVNk7Q+8CZShyZmZtZkLSvZR8QqSScBPwdGA7Mi4s5Wrd/MrJe1rM7ezMza\nx23jmJn1AAf7TNK3JJ013GkbyMfbJF3fjGWbDZWkcyV9ot35sMHruWAvaZ6kJ/NN4qLB/GWw55oa\nlNQn6bvtzkcnkTRW0vmSlkl6VNLJA6Q/OadbJmlW8RjMDy7MlbRS0p2SDiibd3tJP5H0d0mPS/ps\ns7arloh4T0R8qh3rHipJb5b0gKQVkn4kaXyNtK+TdLuk5ZJ+U/y3f97vX5S0MMeS/5G0XmH69yQ9\nkvfzXySd0Oxtq0dPBXtJ04BXAKuBwyolaWV+Bqt4QFlH6ANeAEwF9gc+KungSgnz+FOAVwHbAtsD\nZxaSXAT8EZgAfBy4TNIWed71gTnAL4CJpD8ofm/4N6c2SV0bLyTtCpwHvIX0HT4FfK1K2h1J3++J\nwGbAVcCVhe0/FXgRsCswPQ8Xr3Y+A2wbEZuR4synJL1ouLdp0CKiZ17A6cCvgc8DV5VNuwA4Kw/P\nIP0H4DTgceB+4M1lab8K/AT4O3ADsH1h+peAB4FlpEdOX14jT5uTHkFdBtwInAVcX5i+GngvcA9w\nXx73zvz5CeDHwNZl6d8P3Jfz/jnW3ogX6aBcACwGvg1sWtjmh8rytgA4gNSe0f8B/wCWAzfX8V1P\ny3l5W/4ungTeBbwEuBVYAnylbJ63A/Nz2p8BU+v5TklB99K8PX8Hbgde3ILjaSFwYOHzJ4GLqqS9\nEPhU4fOrgEfz8HTgGWCjwvRfAe/KwycC1zWYxzuBfyl8Xi8fF3vkzz8AHgWWAtcBuxTSfgs4F7ga\nWJGPhW+x9ncynvQbeCzvs6uAbQrzz8vfya/zfvk5sHlh+suB3+Zj4UFgZh4/Fvhv4AFgUc7DBkPc\nV58Bvlf4vH0+pjeqkPYkCvGB9Lt5Ctg/f/4D8K+F6ccAD1ZZ7z8BjwBvbPbxONCra8/UDTqOdMb+\nPnCwpOfXSDuRFIgnATOBb0ianqcJOJoUZMYD9wKfLsz7e2D3PO1C4AcVqo1K/od0IG1FCnbHs24V\n0eGkILmLpFeRDtwjga1JP4iLy9K/HngxqcRxeF4uedkzSYF9e2Bj0kmrmgAiIn6W13lxRGwSEXvW\nmKfcXsAOpP9VfIl0An0VqVR0lKRXAkg6PE87AtgCuJ5U2i0Z6Dt9XU6/GenkWXW7JN0qaUmVV63v\no7iM8aTv/5bC6FvzdlWyS4W0E/NydgX+GhErC9NvKSxrH+ABSVfnKpy5kv65nnySvqtjCp8PBh6L\niD/nzz8l7Z8tgT+RfhtFx5CC+8akoF2swhQwi3RlMxV4mnW/92NIJ/znA+sDHwaQtC3pJPIl0v7e\nAyjl6eycp93z+zakghqSptbYd0skHV3le+j3/UfEX0mFl+kV0gb9az1G5W3957JxxeHJkjYpjZD0\nNUkrSSfbR/K2tle7zzatepFKEf8AJuTPdwIfKkwvL9k/CzyvMP0S4BN5+FvANwrTXgPcWWPdTwK7\nVRg/OudpemHcp1m3ZD+j8HkWcHbh80Z5GVML6Q8qTH8P8Is8fC3w7sK06XneUVQu2d8PvCoP9wHf\nHcT3PS3npXjV8TfgyMLny4AP5OFrgLcXpo0CVgJTBvpOc95mF6btAjzV5ONpSt6+9QvjXg3cXyX9\nvWX7ZUyefyrwVuB3Zek/DZyfh2fn/XQwqWT+YdKV25g68vkCUql6g/z5+6XjuELacTlPmxSO82+V\npVnzO6kw/x7Ak4XPc4GPlR2L1+Th04DLKyxDpKuI4pXyS0knw6Hsr18AJ5aNexh4ZYW0/5TzsB/p\nBPUfwHPAKXn6WaQT3xakQtqNefrECtuyL6labr1mHo/1vHqpZD+TFBCezJ8vyuOqWRIRTxc+P0Aq\nyUE68y8uTHuaVEoGQNKHJc2XtFTSElJpc4sK69iS9OMtNhD3YIV0xeml0nzKSCoNPkEq/VRK/yDp\n6mSdefO09UhXMc1S/j1V+962Bb5UKqGRtgnydtXxnRaX+xSwwXDWMUs6L9+sWy7pVFJ1FsCmhWSb\nFsaXW1EhLTl9+bTyZT1NKgD8PCJWRcR/k646dxoo3xFxH6lgc5ikDUlXQBfmbRot6WxJ90paRjq5\nw9rvNeh/LPUjaUNJX5e0IM9/HbCZ1K/Vs0WF4eL+ngL8tcJitwQ2BP5YOBauofLvZzBWkI6Zoor7\nKyL+QooNXyWVyjcnVS+Wmnf5NHAz6Urk18CPgFURsbhsORERvwEmk050bdUTwV7S84CjgP3y0xCP\nAh8Cdpf0wkLSYvXJ+PzjKNmWtOMHWtcrgI+QSrDjImI8qZ650s3fx4FVpNJdydQK6Yr5eoRUai6t\nbyPSwbiwyjKmFqb1mzdPW0UKlCtJP7LSckeTfniV8tAMD5JKXuMLr40i4oZBfqcDknRHIXCXvyre\ntIuId0eqwtokIs6OiKWkuu49Csl2J90vqOSOCmkXR8SSPG17SRuXTb8jDxerfygLpvW4iFSdcjgw\nP1IVBsCbSTcQD4h0M3G70ioGWF7pWPh30tXhXnn+/fK89eTvQdJVR7m/kU4KuxSOg3ERsSmsqcap\ntu+WSzqmwjIhfZe7lz5I2p50b+DuihsYcXlE7BYRW5CuHKeR6uqJiGci4v0RMTkidiBdZd5UY1vH\nkKpN26ongj2pDnsVsDNph++eh68n1eND5YP0TEljcrD5F9LNLCqkK9okr+tvktaXdDrrltoAiIjn\ngB8CfZKeJ2kXal9tQPrhHi9pd0ljSXXpN0RE8Yrgw5LGSZoCfIBUBVWa92Slx/w2Zm09/GrSQb+B\npEMljSHdyB1bWOYiYFox0Cg9jjl3gPwOpLS884CP5e8ASZtJOjJPq/s7rUdE7FoI3OWv9w5iUd8B\nPpG/652Ad5CqPqqlPUHSzpLGkb7fC3J+7iaVEs+QtIGkI4DdgMvzvN8D9pF0QD4Jf4hUULgT1vzv\n44Ia+byYVAX0bvrXyW9Mukn5ZC40fKZsvkrHefF3sjEpMC+TNAE4o0r6Si4EDpR0pKT1JG0uafd8\nLP4vcI6kLfP2bSPpIICIeLDGvtskIi6qsr7vA6+T9PK8rZ8kVSOtrJRY0ovzlc+WwDeAH+f9hKRJ\n+SVJ+5D25Rl52paSjpa0cZ7/YNL9vWur5KtleiXYH0eq/3w4Ih7Lr8Wky7Q35x9Q+bPzi0hPCTwC\nfJf0ZESpFFDpOfvS55/l192kp1mepnLVTMlJpB/NIuD8/Couu996IuJaUh3i5Tlv25EOpqIfkx7j\nu5n0tMT5efz5eVt+RbqEfor05A4RsYz01M83SZerK+h/CV860T0hqVSKmUK6jK2mnquByOu/Avgs\ncHGuEriNFKBg4O+01v5opjNIdecPkJ48+VxEzIZ+JdDJABHxc9KTUXNz+vvpHxyPBv4fqZT4GdLT\nHk/kee8GjiWdEJ8kVcUcFhGr8ryTqbEfImIR6amXl7L2xA/pBPQA6crvduB3rHvsVfpeS+POAZ5H\nKo3/llTdUms/rJk3F04OJV0dPEE6VktX2aeQ7nHckI+FOVS+kVq3iJjP2pPdYtK9rjUndqWb36cW\nZjmH9PsghZGrAAAPOElEQVS/K+fvnYVpLwB+Q/qNXECqy/9FYRvfTfrtPEna5x+MiJ8MJf/DoWbb\nOJLOJ5VoH4uI3fK4/wJeS7phdB9wfA4USDqN9OTHc6Qbb6UD/xDSlzca+GZEtOUPIfWSNIN0M3LK\nQGk7jaTVwA6FS/Vmrutm0g3cJc1el1Wm9ETSzcAL85WiWUUDlewvID1jXTQb2DUidieVtE4DyJff\nbyI9CXEI8LV8meO+Z0eoiNjTgb69IuIfuVrKgd5qqhnsI+J60qVMcdycXK8G6ZGjyXn4cNIfSp6N\niAWky7C96d6+Z1tRDdAM3ZpvM2uiodbZv521fxaYRP+epx4mPTY3iXX7ni0+JthxImJeRFR6Kqbj\nRcToVlThmFl3abitFUkfB/4RERcOV2bkPmjNzBoSA3Tr2lDJXtLbSHfS31IYXd7H7GRSKX7AvmeL\nhvIPsXa/zjjjjLbnoRfz7vy3/+X8t/dVj0EH+/xkzUeAwyPimcKkK4Gj83PQ2wE7ktozcd+zZmZt\nVrMaR9JFpH/FbSHpIdJzwaeR2ouYk/9f87uIeG9EzJd0KelvxauA90Y65bjvWTOzNqsZ7COi0l+P\nz68wrpT+M6z7Lzwi4hrSHy5GtBkzZrQ7Cw3r5ryD899uzn/n66gOxyVFJ+XHzKwbSCKacYPWzMy6\ni4O9mVkPcLA3M+sBDvZmZj3Awd7MrAc42JuZ9QAHezOzHlAz2Es6X9JiSbcVxk2QNEfS3ZJm5y7W\nStO+LOkeSbdI2rMwfmZOf7ek48rXY2ZmzTVQT1WvIHW99Z1Y21PV54C/RcTnJJ0CjI+IUyUdCpwU\nEYdK2hv4UkTsk/um/APw4rzYPwIvjtRhc/n6mv6nquuvv56nn356zecDDzyQUaN8gWNm3aueP1UN\n+A9aSdOAqwrB/i5gv4hYLGkrYF5E7CTp68AvI+KSQroZwP7AKyPiPXn8eXmeiyusq+nBftLUSawY\nu4JR649i2fxlPPPMM4wdO3bgGc3MOlSz/kE7MVJn3ZA67p2Yh6t1UlKtU5O2eC6eY/mhy1l21DJG\njxndrmyYmbVUw52XAERElHU4UvPMUo++vr41wzNmzOiJBorMzAZj3rx5zJs3b1DzNBLsF0vaKiIW\nSdoaeCyPr9V5yYzC+CnAL6stvBjszcxsXeUF4TPPPHPAeRqpxrkSmJmHZwJXFMYfByBpH2Bpru6Z\nDRwkaZyk8cCrSW3bm5lZiwy285LTgbOBSyWdACwAjgKIiKslHSrpXmAlcHwe/6Sks0hP5ACcWelJ\nHDMza56ea89+4pSJPPaGx2ACjP70aFYuX+mnccysq7k9ezMzAxzszcx6goO9mVkPcLA3M+sBDvZm\nZj3Awd7MrAc42JuZ9QAHezOzHtBwsJd0sqTbJd0m6UJJYyVtJ+nG3IHJxZLG5LRjJV2Sx98gadvh\n2wQzMxtIQ8Fe0jbA+0mdkOwGjAaOBj4LfD4idgSWACfkWU4Ansjjv5jTmZlZiwylGmc9YENJ6wEb\nAo+SOiq5LE//NvD6PHxY/gxwOXDAENZrZmaD1FCwj4iFwOeBB4FHgKWk7gaXRsTqnGwhazsp2Ybc\nsUlErAKW5e4KzcysBRrqvCQ3VXwYMA1YBvwAeM1wZMidl5iZ1dZI5yUNtXop6Ujg4Ih4R/78VuBl\nwBtJ3RaulvRS4PSIeI2knwF9EXFDrvZ5NCK2rLBct3ppZjZIzWz18gFgH0nPkyTgQOAOYC5wZE4z\nE/hxHi52ePJG4NoG12tmZg1otM7+96QbsX8Cbs2jvw6cAvybpHuA8cCsPG0WsHke/yHg1KFk2szM\nBsedl7gax8y6nDsvMTMzwMHezKwnONibmfUAB3szsx7gYG9m1gMc7M3MeoCDvZlZD3CwNzPrAUPp\nvGScpMsk3SlpvqS9JU2QNEfS3ZJmSxpXSP/l3HnJLZL2HJ7sm5lZPYZSsv8ScHVE7Ay8ELiL1AzC\nnIiYTmr/5lQASYcCO+TOS04Ezh1Srs3MbFAa7alqM+AVEXE+pDbqI2IZ/TspKXZecnhpfETcCIyT\nNHEoGTczs/o1WrLfDnhc0gWS/iTpfyVtRGreeHFOsxgoBfRJ5M5LsoeByQ2u28zMBqmhzkvyfC8C\nToqIP0g6h7KWLCMiJBVbNStvpKdii2fuvMTMrLZWdl6yFfC7iNguf345cBqwPbB/RCyStDUwNyJ2\nknQeMC8iLs7p7wL2K1wFlJbrVi/NzAapaa1eRsQi4CFJ0/OoUuclV7G2k5KZwBV5+ErguJypfUh9\n1fYL9GZm1jyNVuMAvB/4vqT1gfuA44HRwKWSTgAWAEcBRMTVkg6VdC+wMqc1M7MWaTjYR8QtwEsq\nTDqwSvqTGl2XmZkNjf9Ba2bWAxzszcx6gIO9mVkPcLA3M+sBDvZmZj3Awd7MrAc42JuZ9QAHezOz\nHjCkYC9ptKSbJV2VP28n6cbcScnFksbk8WMlXZLH3yBp2+HIvJmZ1WeoJfsPAvNZ24LlZ4HP505K\nlgAn5PEnAE/k8V/M6czMrEWG0i3hZOBQ4Jusbb54f+CyPFzsvKTYqcnlwAGNrtfMzAZvKCX7LwIf\nAVYDSNqc1Jrl6jx9IbBNHt6G3HlJRKwClkmaMIR1m5nZIDTUEJqk1wKPRcTNkmaURg9Hhtx5iZlZ\nba3svOQzwFuBVcAGwKbAj4CDSV0Trpb0UuD0iHiNpJ8BfRFxg6T1gEcjYssKy3XnJWZmg9TMzks+\nFhFTck9VRwO/jIhjgbnAkTnZTODHefhK1nZq8kbg2kbWa2ZmjRmu5+xLxfFTgH+TdA8wHpiVx88C\nNs/jP0RZf7VmZtZcQ+mpCoCIuA64Lg/fD+xdIc3/kXutMjOz1vM/aM3MeoCDvZlZD3CwNzPrAQ72\nZmY9wMHezKwHONibmfUAB3szsx7QULCXNEXSXEl3SLpd0gfy+AmS5ki6W9JsSeMK83w5t2d/i6Q9\nh2sDzMxsYI2W7J8FTo6IXYF9gPdJ2pn0z9g5ETGd1CTCqQCSDgV2yO3ZnwicO+ScW8tJqvoys87W\naNs4iyLiz3l4BXAnqRnjYrv1xfbsDy+Nj4gbgXGSJg4h311hMAGxa4JnX36VD5tZRxtynb2kacCe\nwI2kFi8X50mLgVJAn0Ruzz57GJg81HV3ipqBuo9+AXEwac3MhsuQ2saRtDGp56kPRsTyYvCKiJBU\nbK+4vLja3LaMW61v7fuAJfO+svcRoNI2l5qrLp/W7GaszWxdDQf73Jn45cB3I+KKPHqxpK0iYpGk\nrYHH8viFwJTC7JPzuHV0cucldQetvrL3KvNXG1c+viuCYx/rVPH027a+sndqnxS7YpvN2qSRzksa\n7alKpGaL50fEOYVJpXbrP5vfryiMPwm4WNI+pO4LF1NBMdh3pL6174OuW+9j3dJ8X9n7AOsZqLTc\nUSeJvrL3rOpJoDDcUdth1mHKC8JnnnnmgPM0Wme/L3AssL+km/PrEOBs4NWS7gZelT8TEVcDf5V0\nL/B14L0Nrrflat407aM1VTGF9fTLS63xQ9D0m8V9DJzHQppaTwF19M1ssw7SUMk+In5N9RPFgVXm\nOamRdXWEvrL3duore68xvp5AONiqqLboK7wXX8VpZlbTkDsvsQ7WV3ivNpx1cwl5MNVcZr3Kwb7H\nDXQTdZ00nahv7Xs922PWixzse11f2XujaTpFX9l7DS79Wy9xsK+g40uyNmhV92nf2vda/xUw63Zu\n9bKaPrqjJGv16WPgZh7K05iNIC7ZZy7NWyV+3t9GCgf7or6yd7O+sndc12/dycHerE6Dbf7BJwHr\nJC2ts5d0iKS7cicmp7Ry3WZD1kfFq76B/tXsKkLrBC0L9pJGA18FDgF2AY7JHZ6MHPe3OwM9rJ3f\nfR8D3vCtFPiL4wbbqFWncf47XytL9nsB90bEgoh4FriY1KlJW22wwQbDV/JaMDyLsQYsaHcGBtBH\nzXaM9t9//65uA6jbg2W3578erQz227BuBybbtHD91fW1OwPWU/pY95jbj8qPhhaGu/EkYJ2jlTdo\nO+Ju1SiNYuOfb8yo9UexfNXydmfHrH59hffCcKsCfiM3nH3TunOoVV9+bse+LyIOyZ9PA1ZHxGcL\naXwkmJk1ICJqnvVbGezXA/4CHAA8AvweOCYi7mxJBszMeljLqnEiYpWkk4CfA6OBWQ70Zmat0bKS\nvZmZtU9HNYQm6b8k3SnpFkk/lLRZu/M0GJKOlHSHpOckvajd+alXN//ZTdL5khZLuq3deWmEpCmS\n5ubj5nZJH2h3nuolaQNJN0r6c857X7vz1AhJo3PXqle1Oy+DJWmBpFtz/n9fK21HBXtgNrBrROwO\n3A2c1ub8DNZtwBHAr9qdkXqNgD+7XUDKe7d6Fjg5InYF9gHe1y3ff0Q8A+wfEXsAewCHSNq7zdlq\nxAeB+XTIE4ODFMCMiNgzIvaqlbCjgn1EzImI1fnjjcDkduZnsCLiroi4u935GKSO/LNbvSLiemBJ\nu/PRqIhYFBF/zsMrgDuBSe3NVf0i4qk8uD4wBlhdI3nHkTQZOBT4JtCtf1qoK98dFezLvB24ut2Z\n6AGd+2e3HiNpGrAnqaDTFSSNkvRnYDEwOyL+0O48DdIXgY/QZSepggBmS7pJ0jtrJWx5q5eS5gBb\nVZj0sYi4Kqf5OPCPiLiwpZmrQz357zLdeOk64kjaGLgM+GAu4XeFfCW+R76/9iNJu0bEHe3OVz0k\nvRZ4LCJuljSj3flp0L4R8aikLYE5ku7KV7vraHmwj4hX15ou6W2ky6oDWpKhQRoo/11oITCl8HkK\nqXRvLSJpDHA58L2IuKLd+WlERCyTNJd0/6Qrgj3wMuAwSYcCGwCbSvpORBzX5nzVLSIeze+PS/oR\nqVq2YrDvqGocSYeQLqkOzzd/ulm31P/dBOwoaZqk9YE3AVe2OU89Q6k9gVnA/Ig4p935GQxJW0ga\nl4efB7yadM+hK0TExyJiSkRsBxwN/LKbAr2kDSVtkoc3Ag4iPSRSUUcFe+ArwMaky5GbJX2t3Rka\nDElHSHqI9FTFTyVd0+48DSQiVgGlP7vNBy7ppj+7SboI+C0wXdJDko5vd54GaV/gWGD/fMzfnAs9\n3WBr4JeSbiH9I352RHTzfbZuq9KcCFyf75ncCPwkImZXS+w/VZmZ9YBOK9mbmVkTONibmfUAB3sz\nsx7gYG9m1gMc7M3MeoCDvZlZD3CwNzPrAQ72ZmY94P8DocZMSUdq1ZkAAAAASUVORK5CYII=\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x7ff8ed495c10>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"alpha_p = -scale * alpha\n",
"def alpha_dropout(h, q):\n",
" '''h is activation, q is keep probability'''\n",
" mask = np.random.binomial(1, q, size=h.shape)\n",
" dropped = mask * h + (1 - mask) * alpha_p\n",
" a = 1. / np.sqrt(q + alpha_p ** 2 * q * (1 - q))\n",
" b = -a * (1 - q) * alpha_p\n",
" return a * dropped + b\n",
" \n",
"keep_prob = .9\n",
"x = np.random.randn(10000)\n",
"h = selu(x)\n",
"hdrop = alpha_dropout(h, keep_prob)\n",
"\n",
"ax1 = plt.subplot(2, 1, 1)\n",
"plt.title('No dropout, mean = %.3f, variance = %.3f' % (h.mean(), h.var()))\n",
"plt.hist(h, bins=100)\n",
"plt.setp(ax1.get_xticklabels(), visible=False)\n",
"\n",
"plt.subplot(2, 1, 2, sharex=ax1)\n",
"plt.title('Alpha dropout, mean = %.3f, variance= %.3f' % (hdrop.mean(), hdrop.var()))\n",
"plt.hist(hdrop, bins=100, color='green')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The affine linear combination keeps the activation mean and variance steady."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Final Thoughts\n",
"\n",
"SNNs are a great bit of a math! The name \"self-normalizing\" is slightly misleading as the activations are non-normal but do have fixed mean and variance.\n",
"\n",
"I wonder if the weight matrices keep the appropriate mean and variance throughout training. If not, could the weights just be normalized at each iteration or something like weight norm be applied?\n",
"\n",
"This seems very applicable to RNNs. The vanilla RNN is $h_t = \\sigma(Wh_{t-1} + Vx_t)$. If $h_{t-1}$ and $x_t$ have fixed moments, then $Wh_{t-1}$ and $Vx_t$ are both normal, their sum is normal, and the same math works. However, do you really want fixed mean and variance at every iteration of RNN?\n",
"\n",
"SNNs remove the much loved (or maybe secretly hated?) bias term. They also remove the learnable means and variances included in batch/weight/layer norm. Would there be any benefit to learning different means and variances per unit, where those means and variances would manifest themselves as different $\\lambda$ and $\\alpha$ parameters? If you wanted to go down this road, would it make sense to set $\\lambda$ and $\\alpha$ by solving the equations in the paper or just by learning (maybe initialized to the zero mean unit variance standard)?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.12"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment