wassname/pixels to pong annotated.ipynb

## pixels to pong annotated.ipynb
{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reinforcement Learning, what is it good for?\n",
    "\n",
    "It can win atari games, learn to walk, and more:\n",
    "- video: pong https://www.youtube.com/watch?v=YOW8m2YGtRg\n",
    "- video: learning to walk https://www.youtube.com/watch?v=JzD8h4vtiuM\n",
    "- browser demo http://cs.stanford.edu/people/karpathy/reinforcejs/waterworld.html\n",
    "\n",
    "- [slides](http://www0.cs.ucl.ac.uk/staff/D.Silver/web/Teaching_files/intro_RL.pdf)\n",
    "\n",
    "More:\n",
    "- the article that goes with this code: [Karpathy: Pixels 2 Pong](http://karpathy.github.io/2016/05/31/rl/)\n",
    "- main RL course [cs229]( )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2017-07-13T11:30:30.383Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from matplotlib import pyplot as plt\n",
    "plt.rcParams['figure.figsize'] = [10,6]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-12T23:17:44.232545Z",
     "start_time": "2017-07-13T07:17:44.228465+08:00"
    }
   },
   "source": [
    "# How it works: Pixels to Pong\n",
    "\n",
    "Trains an agent with (stochastic) Policy Gradients on Pong. Uses OpenAI Gym.\n",
    "\n",
    "From: https://gist.github.com/karpathy/a4166c7fe253700972fcbc77e4ea32c5\n",
    "\n",
    "Please don't expect to understand everything, I can never keep up with code/maths in a lecture. Instead just try to absorb parts and get a general overview."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2017-07-13T11:30:58.984Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import cPickle as pickle\n",
    "import gym # open ai gym provides an evironment which response to actions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-13T11:16:21.644667Z",
     "start_time": "2017-07-13T19:16:21.639115+08:00"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# hyperparameters\n",
    "H = 200  # number of hidden layer neurons\n",
    "batch_size = 10  # every how many episodes to do a param update?\n",
    "learning_rate = 1e-4\n",
    "gamma = 0.99  # discount factor for reward\n",
    "decay_rate = 0.99  # decay factor for RMSProp leaky sum of grad^2\n",
    "resume = True  # resume from previous checkpoint?\n",
    "render = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# model initialization\n",
    "\n",
    "In keras this model would be\n",
    "```py\n",
    "from keras.layers import dense\n",
    "model = Sequential()\n",
    "model.add(Dense(H, bias=False, activation='relu'))\n",
    "model.add(Dense(H, bias=False, activation='softmax'))\n",
    "model.compile()\n",
    "```\n",
    "\n",
    "But we are using numpy so lets make matrices for the weights\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-13T11:16:23.557627Z",
     "start_time": "2017-07-13T19:16:22.250592+08:00"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "D = 80 * 80 # input dimensionality: 80x80 grid\n",
    "if resume:\n",
    "    model = pickle.load(open('save.p', 'rb'))\n",
    "else:\n",
    "    model = {}\n",
    "    model['W1'] = np.random.randn(H,D) / np.sqrt(D) # \"Xavier\" initialization\n",
    "    model['W2'] = np.random.randn(H) / np.sqrt(H)\n",
    "    \n",
    "grad_buffer = { k : np.zeros_like(v) for k,v in model.iteritems() } # update buffers that add up gradients over a batch\n",
    "rmsprop_cache = { k : np.zeros_like(v) for k,v in model.iteritems() } # rmsprop memory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Helpers functions we will use later\n",
    "\n",
    "These are puzzle pieces we will use later, so lets try to get a general idea of thier function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sigmoid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-13T11:16:23.561649Z",
     "start_time": "2017-07-13T19:16:23.559153+08:00"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def sigmoid(x): \n",
    "    return 1.0 / (1.0 + np.exp(-x)) # sigmoid \"squashing\" function to interval [0,1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-13T11:16:23.862829Z",
     "start_time": "2017-07-13T19:16:23.651847+08:00"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f2bd48ce610>]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlYAAAFpCAYAAABeYWb6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt8VPWd//H3Z2ZyJ0CAcMsFULnITYEAVlvt2qqAFtQq\nYGvXXml/u27bbWtrf+26VrvdX9u9dNuqrdv2162/toDWVlQUpV7bak1QbgGBGNFJSEi4Bcg9me/v\nj5nAEBIIMMmZy+v5eMxjzpzznZlPDmdm3pzzmTPmnBMAAADOnc/rAgAAAJIFwQoAACBGCFYAAAAx\nQrACAACIEYIVAABAjBCsAAAAYoRgBQAAECMEKwAAgBghWAEAAMQIwQoAACBGAl498YgRI9z48eO9\nenoAAIA+27Bhwz7nXP7pxnkWrMaPH6+ysjKvnh4AAKDPzOydvozjUCAAAECMEKwAAABihGAFAAAQ\nIwQrAACAGCFYAQAAxAjBCgAAIEYIVgAAADFCsAIAAIiR0wYrM/uFmdWZ2dZelpuZ/dDMKsxss5nN\njn2ZAAAA8a8ve6x+KWnBKZYvlDQxclkh6YFzLwsAACDxnDZYOedeknTgFEOWSPqVC3tV0lAzGxOr\nAgEAABJFLH4rsEBSMOp2VWReTQweGwAAnAPnnNo7nTpDTh2hkEJOkpNCzinknJzC085JLjLfSQqF\nIvPkFHLRY8K3nZxCofB8qevxji+XwuPDU121hJd3nxcZfWzmCeMjt05+LHds+vwRg1Q8PDuGa+3s\nDeiPMJvZCoUPF6q4uHggnxoAgAHR1hFSU1uHGts61dTa7bqtQ42tUdftHWpq7VR7Z0gdIaeOY9cu\nfB0KhQNRZ3j6hGWd4WXtoZA6O53aQ5Hw1PUYkemQO33Nie7OhVP0uSvO97oMSbEJVtWSiqJuF0bm\nncQ596CkByWppKQkBf6pAQCJpLG1Q9WHmlV1sEkHG9tPDkjdg1EPy9s7+/7xlpnmU3Z6QGl+U8Dn\nU8Bv8vtMaT5f+DpyO+DzKc3vU2aaKc0fvcyngM/Cl8hjnLTM37XcJ7+ZzCSfmXwmWdR113xT5Dpq\neffbppPvF74dmafo6/DfGr4VGdO1Ak6xzKyneSfesWvZ2KFZZ/Tv3J9iEazWSLrdzFZKmi+pwTnH\nYUAAQNxpautQ9cFmVR0Mh6eqg80KRq6rDjbrQGNbj/czk3LSA8pO9ysnI3KdHlBeTroK87rNj1qe\nneE/+X6R6+z0gPw+6/H5kLhOG6zM7LeS3i9phJlVSfpnSWmS5Jz7iaS1khZJqpDUJOkT/VUsAACn\n0tzWqepDTQpGh6cDx0PU/m7BKT3gU+HQLBXkZWl6wRAV5mWpMC9bBUOzlD8o41gwykzzHduDApzK\naYOVc+6W0yx3kv4+ZhUBANCLlvbOE/Y2dU0HDzar+mCT9h3tFpz8PhXkZakwL0tXjx2swrzsY+Gp\nKC9LIwZlyMdeI8TQgDavAwBwpnbva9TarTV6emutNlc1nLAszW8qGBoOSh+8cJQK87JUNOx4eMon\nOGGAEawAAHGnou6I1m6p1VNba7W95rAk6aLCIfrCByZq/IhsFeVlqzAvWyNzCU6ILwQrAIDnnHPa\nsTcSprbUaFfdUUlSybg8ffPaC7Vg+mgV5sXHeYqAUyFYAQA84ZxT+Z7DWrulRk9trdXb+xrlM2nu\n+GH61uJpumbaaI0ekul1mcAZIVgBAAaMc04bg4f01NZard1So6qDzfL7TO85b7g+/b4JunrqaOXn\nZnhdJnDWCFYAgH4VCjltePeg1m4JN6DXNLQozW+67IIR+vyVE3XV1FHKy0n3ukwgJghWAICY6+gM\n6bXdB/TUllqtK69V3ZFWpQd8unxivu64ZrI+cOEoDclK87pMIOYIVgCAmGjvDOmVt/brqa01eqZ8\nr/Y3tikzzae/mTxSC2eM0ZVTRmpQBh87SG5s4QCAs+ac04s76/Xk5ho9s22vGprblZPu15UXjtKi\n6aN1xeR8ZafzUYPUwdYOADgrTW0duuORzXpyc41yMwO66sJRWjhjjN43cYQy0/xelwd4gmAFADhj\n7+xv1Gcf2qAde4/ojmsm69Pvm6CMAGEKIFgBAM7Iizvr9fnfviFJ+uUn5umKSfkeVwTED4IVAKBP\nnHN64MW39P11OzR5VK5++rE5Gjc8x+uygLhCsAIAnFZja4fueGST1m6p1XUzx+h7N82kKR3oAa8K\nAMApvb2vUZ99qEwVdUf1vxdN0Wfed57M+OFjoCcEKwBAr55/s06fX/mG/D7Trz45X++dOMLrkoC4\nRrACAJwkFHK6/4UK/fuzO3Xh6MH66cfmqGhYttdlAXGPYAUAOMHR1g59efVGrSvfq+svHqt/vXGm\nstI5lQLQFwQrAMAxb9Uf1Wcf2qC39zXqn66bqk9eNp5+KuAMEKwAAJKk9dv26h9XbVRawKeHPjVP\nl55PPxVwpghWAJDiQiGnHz63Sz9Yv0vTCwbrpx8rUcHQLK/LAhISwQoAUtjhlnZ9adUmrd++VzfO\nLtB3bpjB7/wB54BgBQApqqLuiFY8tEHv7G/S3R+aqtsupZ8KOFcEKwBIQevKa/WlVRuVmebXrz89\nX5ecN9zrkoCkQLACgBQSCjn95/qd+tFzFbqocIgeuHWOxtJPBcQMwQoAUkRDc7u+uPINPb+jXjfP\nKdS910+nnwqIMYIVAKSAnXuPaMWvylR1sFn3Xj9dt84vpp8K6AcEKwBIck9tqdGXH96k7PSAfrvi\nEs0dP8zrkoCkRbACgCTVGXL692d26P4X3tKs4qF64KNzNHpIptdlAUmNYAUASaihqV2fX/mGXtxZ\nr1vmFenuxdOUEaCfCuhvBCsASDIt7Z266Sd/0e79jfrODTP0kfnFXpcEpAyCFQAkmQdfqtSuuqP6\nxcdLdOWUUV6XA6QUn9cFAABiJ3igSfc9X6FrZ44hVAEeIFgBQBL51uPb5PeZvnnthV6XAqQkghUA\nJInn3tyr9dv36gsfmKgxQzibOuAFghUAJIGW9k7dvWabLhg5SJ+4bILX5QApi+Z1AEgCP32xUu8e\naNJvPj1f6QH+zwx4hVcfACS4d/c36f4XKvShi8bq0gtGeF0OkNIIVgCQ4O55olwBn+kbi2hYB7xG\nsAKABLZ+216t316nL3xwIj9XA8QBghUAJKiW9k5964lyTaRhHYgbNK8DQIJ64IW3FDzQrN98Zr7S\n/Pw/GYgHvBIBIAG9s79RD7z4lhZfNFaXnk/DOhAvCFYAkGCcc7p7TbnSfKZvcIZ1IK4QrAAgwazf\nXqfnd9TrH6+apFGDaVgH4gnBCgASSEt7p771eLkmjRqk2y4d73U5ALqheR0AEsj9L7ylqoPNWrni\nEhrWgTjEqxIAEsTufY36yYtv6fqLx+qS84Z7XQ6AHhCsACABOOd09+PlSvf79L85wzoQtwhWAJAA\nnt22Vy9EGtZH0rAOxC2CFQDEuea2Tn3r8W2aMjpXt71nnNflADiFPgUrM1tgZjvMrMLM7uxhebGZ\nPW9mb5jZZjNbFPtSASA13f9ChaoPNetbi6cpQMM6ENdO+wo1M7+k+yQtlDRV0i1mNrXbsG9KWu2c\nmyVpuaT7Y10oAKSit/c16qcvVuqGWQWaT8M6EPf68l+feZIqnHOVzrk2SSslLek2xkkaHJkeImlP\n7EoEgNTknNM/rylXRsCnry+a4nU5APqgL8GqQFIw6nZVZF60uyXdamZVktZK+oeYVAcAKWxd+V69\ntDPSsJ5LwzqQCGJ1sP4WSb90zhVKWiTpITM76bHNbIWZlZlZWX19fYyeGgCST1Nbh+59Ityw/rc0\nrAMJoy/BqlpSUdTtwsi8aJ+StFqSnHOvSMqUdNLPrTvnHnTOlTjnSvLz88+uYgBIAfc9H25Yv2fJ\ndBrWgQTSl1drqaSJZjbBzNIVbk5f023Mu5I+IElmdqHCwYpdUgBwFirrj+rBlyp14+wCzZswzOty\nAJyB0wYr51yHpNslrZO0XeFv/5Wb2T1mtjgy7MuSPmNmmyT9VtLHnXOuv4oGgGTV1bCeGfDr6ws5\nwzqQaPr0I8zOubUKN6VHz7sranqbpMtiWxoApJ515bV6edc+3f2hqcrPzfC6HABniAP3ABAnmto6\ndM/j23ThmMG69RIa1oFERLACgDjx4+cqtKehRfcu4QzrQKLilQsAceCt+qP675crddOcQpWMp2Ed\nSFQEKwDwmHNOd68pV2aaX3cu5AzrQCIjWAGAx57aGm5Y/8rVkzViEA3rQCIjWAGAhxpbw2dYnzpm\nsD46v9jrcgCcoz6dbgEA0D9+9FyFahpa9OOPzKJhHUgCvIoBwCMVdUf0s5crdfOcQs0ZR8M6kAwI\nVgDgga4zrGen+/U1GtaBpEGwAgAPPLmlRn+u2K87rqFhHUgmBCsAGGCNrR369hPbNW3sYH1kPmdY\nB5IJzesAMMB++Nwu1R5u0f23zpbfZ16XAyCG2GMFAAOoou6Ifv7y21pWUqTZxXlelwMgxghWADBA\nnHO667Fy5WQE9NUFk70uB0A/IFgBwAB5YnON/vJWuGF9OA3rQFIiWAHAAPnJi29p8qhc3TKPM6wD\nyYpgBQADYGt1g8r3HNatlxTTsA4kMYIVAAyAlaXvKiPg0+KLC7wuBUA/IlgBQD9rbuvUYxv36NoZ\nYzQkK83rcgD0I4IVAPSzp7bW6EhLh5bOLfK6FAD9jGAFAP1sZWlQ44dna/4EfmgZSHYEKwDoR5X1\nR/Xa2we0bG6xzGhaB5IdwQoA+tHqsir5faYPz6FpHUgFBCsA6CftnSE9sqFKV04ZqZG5mV6XA2AA\nEKwAoJ8892ad9h1t1XKa1oGUQbACgH6yqjSoUYMzdMWkfK9LATBACFYA0A9qG1r0wo463TynSAE/\nb7VAquDVDgD94JENQYWctLSEw4BAKiFYAUCMhUJOq8qCuvT84Soenu11OQAGEMEKAGLslcr9Ch5o\n1jKa1oGUQ7ACgBhbVRrUkKw0XTNttNelABhgBCsAiKGDjW16emutbphVoMw0v9flABhgBCsAiKE/\nbKxWW2eIw4BAiiJYAUCMOOe0qjSoiwqH6MIxg70uB4AHCFYAECObqxr0Zu0RLWVvFZCyCFYAECMr\nS4PKSvNr8UVjvS4FgEcIVgAQA42tHVqzsVrXzhyj3Mw0r8sB4BGCFQDEwJNbatTY1skPLgMpjmAF\nADGwujSo8/JzNGdcntelAPAQwQoAzlFF3RGVvXNQy+cWycy8LgeAhwhWAHCOVpUGFfCZbpxd6HUp\nADxGsAKAc9DWEdKjr1frqqmjNGJQhtflAPAYwQoAzsEft+/V/sY2zl0FQBLBCgDOycrSoMYMydTl\nE/O9LgVAHCBYAcBZqj7UrJd21evmkiL5fTStAyBYAcBZe6SsSpJ08xya1gGEEawA4Cx0hpxWlwX1\n3gtGqGhYttflAIgTBCsAOAt/rtin6kPNWkbTOoAoBCsAOAurSoPKy07TVVNHeV0KgDhCsAKAM3Sg\nsU3PbKvVDbMKlRHwe10OgDjSp2BlZgvMbIeZVZjZnb2MWWpm28ys3Mx+E9syASB+PPp6ldo7HYcB\nAZwkcLoBZuaXdJ+kqyRVSSo1szXOuW1RYyZK+rqky5xzB81sZH8VDABecs5pVWlQs4qHavLoXK/L\nARBn+rLHap6kCudcpXOuTdJKSUu6jfmMpPuccwclyTlXF9syASA+vP7uIe2qO6rl7K0C0IO+BKsC\nScGo21WRedEmSZpkZn82s1fNbEGsCgSAeLK6NKjsdL+unTnW61IAxKHTHgo8g8eZKOn9kgolvWRm\nM5xzh6IHmdkKSSskqbi4OEZPDQAD42hrhx7fvEcfmjlWgzJi9fYJIJn0ZY9VtaTofd6FkXnRqiSt\ncc61O+felrRT4aB1Aufcg865EudcSX4+v6sFILE8sWmPmto6tWwehwEB9KwvwapU0kQzm2Bm6ZKW\nS1rTbcwfFN5bJTMbofChwcoY1gkAnltZGtSkUYM0q2io16UAiFOnDVbOuQ5Jt0taJ2m7pNXOuXIz\nu8fMFkeGrZO038y2SXpe0h3Ouf39VTQADLQdtUe0MXhIS0uKZMYPLgPoWZ+aBJxzayWt7Tbvrqhp\nJ+lLkQsAJJ1VpUGl+U03zuYHlwH0jjOvA8BptHZ06tE3qnT1tNEalpPudTkA4hjBCgBO45nyvTrU\n1M65qwCcFsEKAE5jdVlQBUOzdNn5I7wuBUCcI1gBwCkEDzTp5V37tLSkSD4fTesATo1gBQCn8HBZ\nUGbSzSU0rQM4PYIVAPSiM+T08IYqXT4xX2OHZnldDoAEQLACgF68tKteNQ0tNK0D6DOCFQD0YtVr\nQQ3PSdcHLhzldSkAEgTBCgB6UH+kVeu379WH5xQqPcBbJYC+4d0CAHrw6OtV6gg5LS3hMCCAviNY\nAUA3zjmtKguqZFyeLhg5yOtyACQQghUAdFP2zkFV1jdqGU3rAM4QwQoAuln5WlCDMgK6duYYr0sB\nkGAIVgAQ5XBLu57cskcfumisstMDXpcDIMEQrAAgyuOb9qilPcS5qwCcFYIVAERZVRrUlNG5mlk4\nxOtSACQgghUARJTvadDmqgYtn1skM35wGcCZI1gBQMTq0qDSAz5dP6vA61IAJCiCFQBIamnv1O/f\nqNaCaaM1NDvd63IAJCiCFQBIWldeq8MtHTStAzgnBCsAUPjcVcXDsnXJecO9LgVAAiNYAUh57+xv\n1CuV+7W0pFA+H03rAM4ewQpAyltdFpTPpJvmcBgQwLkhWAFIaR2dIT1cVqW/mTxSo4dkel0OgARH\nsAKQ0l7YUa+6I61aStM6gBggWAFIaStLgxoxKENXThnpdSkAkgDBCkDKqjvcoud31OmmOYVK8/N2\nCODc8U4CIGU98nqVOkNOyzgMCCBGCFYAUpJzTqtKg5o3YZgmjMjxuhwASYJgBSAlvVp5QO/sb+JM\n6wBiimAFICWtLgsqNzOghdPHeF0KgCRCsAKQchqa2rV2S42uv7hAWel+r8sBkEQIVgBSzmObqtXa\nEaJpHUDMEawApJyVrwU1bexgTS8Y4nUpAJIMwQpAStla3aBtNYdpWgfQLwhWAFLKytJ3lRHwafHF\nBV6XAiAJEawApIzmtk499sYeLZoxRkOy0rwuB0ASIlgBSBlrt9ToSGsHTesA+g3BCkDKWFUW1Pjh\n2Zo/YZjXpQBIUgQrACmhsv6oXnv7gJbNLZaZeV0OgCRFsAKQElaVBeX3mT48h6Z1AP2HYAUg6bV3\nhvS7DVW6cspIjczN9LocAEmMYAUg6T33Zp32HW3j3FUA+h3BCkDSW1Ua1KjBGbpiUr7XpQBIcgQr\nAEmtpqFZL+yo001zChXw85YHoH/xLgMgqT1SVqWQk5aWcBgQQP8jWAFIWqGQ0+oNQV16/nCNG57j\ndTkAUgDBCkDSeqVyv4IHmjnTOoABQ7ACkLRWlgY1JCtN10wb7XUpAFIEwQpAUjrY2KZ1W2t1w6wC\nZab5vS4HQIroU7AyswVmtsPMKszszlOM+7CZOTMriV2JAHDm/rCxWm2dIQ4DAhhQpw1WZuaXdJ+k\nhZKmSrrFzKb2MC5X0hck/TXWRQLAmXDOaeVrQV1UOEQXjhnsdTkAUkhf9ljNk1ThnKt0zrVJWilp\nSQ/j7pX0XUktMawPAM7YpqoG7dh7REvZWwVggPUlWBVICkbdrorMO8bMZksqcs49GcPaAOCsrCp9\nV1lpfi2+aKzXpQBIMefcvG5mPkn/IenLfRi7wszKzKysvr7+XJ8aAE7S2NqhNRv36NqZY5SbmeZ1\nOQBSTF+CVbWk6P3phZF5XXIlTZf0gpntlnSJpDU9NbA75x50zpU450ry8/nNLgCx9+SWGjW2dfKD\nywA80ZdgVSppoplNMLN0Scslrela6JxrcM6NcM6Nd86Nl/SqpMXOubJ+qRgATmFVaVDn5edozrg8\nr0sBkIJOG6yccx2Sbpe0TtJ2Saudc+Vmdo+ZLe7vAgGgryrqjmjDOwe1fG6RzMzrcgCkoEBfBjnn\n1kpa223eXb2Mff+5lwUAZ25VaVABn+nG2YVelwIgRXHmdQBJoa0jpN+9Xq2rpo7SiEEZXpcDIEUR\nrAAkhfXb9+pAYxvnrgLgKYIVgKSwsjSoMUMydflEvnEMwDsEKwAJr/pQs17eVa+bS4rk99G0DsA7\nBCsACe/hsvCPQ9w8h6Z1AN4iWAFIaJ0hp4fLqvTeC0aoaFi21+UASHEEKwAJ7U8V+1R9qFnLaFoH\nEAcIVgAS2urSoPKy03TV1FFelwIABCsAiWv/0VY9s61WN84uVEbA73U5AECwApC4fv9Gtdo7HYcB\nAcQNghWAhOSc08rSoGYVD9WkUblelwMAkghWABLU6+8eUkXdUS1nbxWAOEKwApCQVpW+q5x0v66b\nOdbrUgDgGIIVgIRzpKVdj2+q0XUzxyonI+B1OQBwDMEKQMJ5YnONmts7tWwehwEBxBeCFYCEs6o0\nqEmjBmlW0VCvSwGAExCsACSUN2sPa2PwkJbNLZYZP7gMIL4QrAAklFWlQaX5TTfMKvC6FAA4CcEK\nQMJo7ejU79+o1tXTRmtYTrrX5QDASQhWABLGM+V7daipnXNXAYhbBCsACWNVaVAFQ7N02fkjvC4F\nAHpEsAKQEIIHmvSnin1aWlIkn4+mdQDxiWAFICGsLgvKTLq5pNDrUgCgVwQrAHGvM+T0cFmVrpiU\nr7FDs7wuBwB6RbACEPde2lmv2sMtNK0DiHsEKwBxb2Xpuxqek64rp4zyuhQAOCWCFYC4Vn+kVX/c\nXqcPzylUeoC3LADxjXcpAHHt0der1BFyWlrCYUAA8Y9gBSBuOee0qjSouePzdMHIQV6XAwCnRbAC\nELdKdx9U5b5G9lYBSBgEKwBxa2XpuxqUEdC1M8d4XQoA9AnBCkBcOtzSrrVbarT44rHKTg94XQ4A\n9AnBCkBcWrNxj1raQ1rGYUAACYRgBSDudDWtTxmdq5mFQ7wuBwD6jGAFIO48s22vtlQ36NZLxsmM\nH1wGkDgIVgDiSnNbp+55fJumjM7lJ2wAJBw6QgHElfuer1D1oWat/ux7FPDzfz8AiYV3LQBxo7L+\nqB58qVI3zirQvAnDvC4HAM4YwQpAXHDO6Z/XlCsj4NOdi6Z4XQ4AnBWCFYC4sK68Vi/v2qcvXT1J\nI3MzvS4HAM4KwQqA55raOo41rH/sknFelwMAZ41gBcBzP36uQnsaWnTv9dNpWAeQ0HgHA+Cpyvqj\n+u+XK/Xh2YWaO56GdQCJjWAFwDNdDeuZaX7duZCGdQCJj2AFwDNPbw03rH/5qknKz83wuhwAOGcE\nKwCeaGrr0D1PbNOFYwbrVhrWASQJghUAT/zouQrVNLTo3iXTaFgHkDR4NwMw4CrqjupnL1fqpjmF\nKqFhHUASIVgBGFDOOd1NwzqAJEWwAjCg1m6p1Z8q9umOayZrxCAa1gEkF4IVgAHT2Nqhe5/Ypmlj\nB+uj82lYB5B8+hSszGyBme0wswozu7OH5V8ys21mttnM/mhmvGMCOMkPn9ul2sMtumfJdPl95nU5\nABBzpw1WZuaXdJ+khZKmSrrFzKZ2G/aGpBLn3ExJj0j6XqwLBZDYKuqO6Ocvv62lJYWaMy7P63IA\noF/0ZY/VPEkVzrlK51ybpJWSlkQPcM4975xritx8VVJhbMsEkMicc7rrsXJlp/v1tQU0rANIXn0J\nVgWSglG3qyLzevMpSU+dS1EAksuTW2r0l7f2645rJms4DesAklgglg9mZrdKKpF0RS/LV0haIUnF\nxcWxfGoAcepoVMP6R2hYB5Dk+rLHqlpSUdTtwsi8E5jZByV9Q9Ji51xrTw/knHvQOVfinCvJz88/\nm3oBJJgf/XGX9h5u1b3X07AOIPn1JViVSppoZhPMLF3ScklrogeY2SxJP1U4VNXFvkwAiWjX3iP6\n+Z/e1rKSIs0upmEdQPI7bbByznVIul3SOknbJa12zpWb2T1mtjgy7PuSBkl62Mw2mtmaXh4OQIro\naljPyQjoqwsme10OAAyIPvVYOefWSlrbbd5dUdMfjHFdABLc45tr9Erlfn37+uk0rANIGZx5HUDM\nHW3t0Lef2KYZBUN0yzy+qAIgdcT0W4EAIEn/tX6n6o+26sG/LaFhHUBKYY8VgJjaUXtEv/jzbi2f\nW6SLi4Z6XQ4ADCiCFYCYCTesb1VuZkB3XMMZ1gGkHoIVgJhZs2mP/vr2Ad1xzWQNy0n3uhwAGHAE\nKwAxcaSlXd9+crtmFg7R8rk0rANITTSvA4iJ/1q/S/uOtupnNKwDSGHssQJwznbUHtH//ctuLZ9b\nrItoWAeQwghWAM6Jc07/FGlY/+o1nGEdQGojWAE4J49t3KPX3j6gry2Yojwa1gGkOIIVgLN2uKVd\n/7J2uy4qGqplJUVelwMAnqN5HcBZ+8Gz4Yb1n99WIh8N6wDAHisAZ2d7zWH9zyu79ZF5xZpZSMM6\nAEgEKwBnoesM64MzA7qDhnUAOIZgBeCM/f6NapXuPqivLZiiodk0rANAF4IVgDPS0Nyu70Qa1pfS\nsA4AJyBYATgj//nsTu1vbNO3l0ynYR0AuiFYAeizbXsO61ev7NZH5xdrRuEQr8sBgLhDsALQJxV1\nR/T3v3ldQ7PT9ZWraVgHgJ5wHisAp/X01lp9efVGZaX79eDH5tCwDgC9IFgB6FVnyOkH63fqR89V\n6KKiofrJrbM1ZkiW12UBQNwiWAHoUUNzu7648g09v6NeS0sKdc+S6cpM83tdFgDENYIVgJPsqD2i\nzz5UpupDzfr29dP10fnFMuMbgABwOgQrACdYu6VGX3l4k3IyAvrtZy5RyfhhXpcEAAmDYAVAUrif\n6t+e2aEHXnhLs4qH6ie3ztGowZlelwUACYVgBUCHmtr0+ZUb9dLOet0yr1h3L56qjAD9VABwpghW\nQIrbXnNYn31og2oamvWdG2boI/OLvS4JABIWwQpIYY9v2qOvPrJZg7MCWrniPZozLs/rkgAgoRGs\ngBTU0RnS99ft0E9fqlTJuDzdf+tsjcylnwoAzhXBCkgxBxvb9A+/fUN/qtinj10yTv903VSlB/h1\nKwCIBYJPH7w7AAAMEklEQVQVkELK9zTosw9tUN3hVn3vwzO1dG6R1yUBQFIhWAEp4rGN1fra7zZr\naFa6Vn/uPbq4aKjXJQFA0iFYAUmuozOkf33qTf38T29r3oRhuu8js5Wfm+F1WQCQlAhWQBLbf7RV\nt//mDb1SuV8fv3S8vnHthUrz008FAP2FYAUkqa3V4X6q+qOt+rebL9JNcwq9LgkAkh7BCkhCj75e\npa8/ukXDc9L1u89dqhmFQ7wuCQBSAsEKSCLtnSH9y5Pb9cu/7NYl54X7qYYPop8KAAYKwQpIEvuO\nturvfv26Xnv7gD713gn6+sIpCtBPBQADimAFJIFNwUP63P/boAONbfrBsot1/awCr0sCgJREsAIS\nVHtnSK9W7tfaLbX63etVyh+Uod/9r0s1vYB+KgDwCsEKSCBtHSH9uWKf1m6p0bPb9+pQU7uy0/26\nbsYYffO6qRqWk+51iQCQ0ghWQJxrae/USzvr9fTWWj27fa+OtHQoNyOgD04dpYXTR+vySfnKTPN7\nXSYAQAQrIC41tXXohR31emprrZ7bvleNbZ0akpWmBdNGa9GMMbr0guHKCBCmACDeEKyAOHG0tUN/\n3L5XT2+t1fM76tTSHtLwnHQtvrhAi2aM1iXnDees6QAQ5whWgIcamtu1fttePbW1Vi/tqldbR0j5\nuRlaWlKkhdPHaO74PE6ZAAAJhGAFDLADjW16dlutntpaqz9X7FN7p9OYIZn66PxiLZoxRnOK8+Tz\nmddlAgDOAsEKGAD1R1q1rrxWT2+t1SuV+9UZcioalqVPXDZBC6eP1kWFQwlTAJAECFZAP3DOaU9D\ni9Zv26u1W2r02u4Dck6aMCJHn738PC2aMUbTxg6WGWEKAJIJwQo4C8457W9sU9XBZlUdbDrhOnig\nSdWHmtXSHpIkTRo1SP9w5UQtmjFak0flEqYAIIkRrIAeOOd04FhwigpNUSGqKzh1GZqdpsK8LE0c\nmau/mTxSRcOyddkFw3XByFyP/goAwEDrU7AyswWS/kuSX9LPnHP/p9vyDEm/kjRH0n5Jy5xzu2Nb\nKhA7zjkdbGo/YW9T8ED03qdmNbd3nnCfIVnh4HR+fo6umJSvorwsFeZlq3BYlgqGZik3M82jvwYA\nEC9OG6zMzC/pPklXSaqSVGpma5xz26KGfUrSQefcBWa2XNJ3JS3rj4KR2pxzausMqam1U41tHWpq\n61Rja7frto5TLq870qKqg81qajsxOA3ODKgwL1sTRuTo8kn5KuwKTnlZKsjL0mCCEwDgNPqyx2qe\npArnXKUkmdlKSUskRQerJZLujkw/IunHZmbOORfDWtGPnHNyTgo5J6fItZM6Qk4dnSF1hJw6Q07t\nnaHI9Ym3O0IhdXS68Phe7nN8edfYUGRseExja6ea2jrU2NapptaOXoNRR6jvm1VWml85GX5lpweU\nne5Xdrpf44fn6L0XdAWncHgqyMvSkCyCEwDg3PQlWBVICkbdrpI0v7cxzrkOM2uQNFzSvlgUeTZe\nrdyv7z795gnzuse8Hj+euw3qaUxPcdHJHZvv3PH7RWfLY8ujx0aNcVFPGH1/F3X/rvuGL06hyLzQ\nCfMi4SjkIo9/PCh1XR+/T/jaa2ZSTiT85GRErtMDGpaTrqK87BPnRy3PzghfRy/LSfcrOyOgrDS/\n/JzCAAAwgAa0ed3MVkhaIUnFxcX9+lxpftOgjJP/vO7fyOrpY7f7l7Z6HnPyXDvhvnZsOnq+RR7N\nrNu8E8baCc9rFj1tx8b4Io/hM4s8XmSeupbZ8eWSfL7IOHW/b2T5sfHH75vm88nvMwX8poDPp0Bk\n2u8zpfl9kWuT3+dTms9OGHvymKjbPp/8fgs/XuR+fFsOAJDo+hKsqiUVRd0ujMzraUyVmQUkDVG4\nif0EzrkHJT0oSSUlJf26n2TOuGF66FPdd6wBAAD0n778CFmppIlmNsHM0iUtl7Sm25g1km6LTN8k\n6Tn6qwAAQKo57R6rSM/U7ZLWKXy6hV8458rN7B5JZc65NZJ+LukhM6uQdEDh8AUAAJBS+tRj5Zxb\nK2ltt3l3RU23SLo5tqUBAAAklr4cCgQAAEAfEKwAAABihGAFAAAQIwQrAACAGCFYAQAAxAjBCgAA\nIEYIVgAAADFCsAIAAIgRghUAAECMEKwAAABixLz6rWQzq5f0Tj8/zQhJ+/r5ORIF6+I41sVxrIsw\n1sNxrIvjWBfHsS6kcc65/NMN8ixYDQQzK3POlXhdRzxgXRzHujiOdRHGejiOdXEc6+I41kXfcSgQ\nAAAgRghWAAAAMZLswepBrwuII6yL41gXx7EuwlgPx7EujmNdHMe66KOk7rECAAAYSMm+xwoAAGDA\nJEWwMrMFZrbDzCrM7M4elmeY2arI8r+a2fiBr7L/mVmRmT1vZtvMrNzMvtDDmPebWYOZbYxc7vKi\n1oFgZrvNbEvk7yzrYbmZ2Q8j28VmM5vtRZ39ycwmR/1bbzSzw2b2xW5jknabMLNfmFmdmW2NmjfM\nzJ41s12R67xe7ntbZMwuM7tt4KruH72si++b2ZuR7f/3Zja0l/ue8rWUaHpZF3ebWXXU62BRL/c9\n5edNoullXayKWg+7zWxjL/dNqu0iZpxzCX2R5Jf0lqTzJKVL2iRparcxfyfpJ5Hp5ZJWeV13P62L\nMZJmR6ZzJe3sYV28X9ITXtc6QOtjt6QRp1i+SNJTkkzSJZL+6nXN/bw+/JJqFT4XS0psE5IulzRb\n0taoed+TdGdk+k5J3+3hfsMkVUau8yLTeV7/Pf2wLq6WFIhMf7endRFZdsrXUqJdelkXd0v6ymnu\nd9rPm0S79LQuui3/d0l3pcJ2EatLMuyxmiepwjlX6Zxrk7RS0pJuY5ZI+p/I9COSPmBmNoA1Dgjn\nXI1z7vXI9BFJ2yUVeFtVXFsi6Vcu7FVJQ81sjNdF9aMPSHrLOdffJ+aNG865lyQd6DY7+v3gfyRd\n38Ndr5H0rHPugHPuoKRnJS3ot0IHQE/rwjn3jHOuI3LzVUmFA16YB3rZLvqiL583CeVU6yLyOblU\n0m8HtKgElwzBqkBSMOp2lU4OE8fGRN5EGiQNH5DqPBI53DlL0l97WPweM9tkZk+Z2bQBLWxgOUnP\nmNkGM1vRw/K+bDvJZLl6f4NMlW1CkkY552oi07WSRvUwJtW2DUn6pMJ7cHtyutdSsrg9clj0F70c\nIk617eJ9kvY653b1sjxVtoszkgzBCt2Y2SBJv5P0Refc4W6LX1f4UNBFkn4k6Q8DXd8Aeq9zbrak\nhZL+3swu97ogr5hZuqTFkh7uYXEqbRMncOHjGSn/1Wgz+4akDkm/7mVIKryWHpB0vqSLJdUofAgs\n1d2iU++tSoXt4owlQ7CqllQUdbswMq/HMWYWkDRE0v4BqW6AmVmawqHq1865R7svd84dds4djUyv\nlZRmZiMGuMwB4ZyrjlzXSfq9wrvxo/Vl20kWCyW97pzb231BKm0TEXu7DvlGrut6GJMy24aZfVzS\ndZI+GgmaJ+nDaynhOef2Ouc6nXMhSf+tnv/GVNouApJulLSqtzGpsF2cjWQIVqWSJprZhMj/ypdL\nWtNtzBpJXd/quUnSc729gSSyyPHwn0va7pz7j17GjO7qLzOzeQpvA0kXMs0sx8xyu6YVbtLd2m3Y\nGkl/G/l24CWSGqIOESWbXv/nmSrbRJTo94PbJD3Ww5h1kq42s7zIIaGrI/OSipktkPRVSYudc029\njOnLaynhdeuvvEE9/419+bxJFh+U9KZzrqqnhamyXZwVr7vnY3FR+NtdOxX+tsY3IvPuUfjNQpIy\nFT4EUiHpNUnneV1zP62H9yp8WGOzpI2RyyJJn5P0uciY2yWVK/xtllclXep13f20Ls6L/I2bIn9v\n13YRvS5M0n2R7WaLpBKv6+6ndZGjcFAaEjUvJbYJhcNkjaR2hfthPqVwf+UfJe2StF7SsMjYEkk/\ni7rvJyPvGRWSPuH139JP66JC4Z6hrveLrm9Pj5W0NjLd42spkS+9rIuHIu8DmxUOS2O6r4vI7ZM+\nbxL50tO6iMz/Zdd7RNTYpN4uYnXhzOsAAAAxkgyHAgEAAOICwQoAACBGCFYAAAAxQrACAACIEYIV\nAABAjBCsAAAAYoRgBQAAECMEKwAAgBj5/54M+9NsA39bAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f2bd49989d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# lets plot this to remind ourselves what a sigmoid is *cough*\n",
    "plt.plot(sigmoid(np.arange(-10,10)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# policy model\n",
    "\n",
    "In keras these would be handled by `model.train(X,y)`\n",
    "\n",
    "\n",
    "## Math\n",
    "For the people who read maths easier than code:\n",
    "\n",
    "Given weights $W_1$ $W_2$ and input $x$\n",
    "\n",
    "$ h = W_1 \\cdot x$\n",
    "\n",
    "$ h = \\max(0, h)$\n",
    "\n",
    "$ o = W_2 \\cdot h $\n",
    "\n",
    "$ o = \\frac{1.0}{1.0 + \\exp(-o)} $\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-13T11:16:23.981979Z",
     "start_time": "2017-07-13T19:16:23.977057+08:00"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def policy_forward(x):\n",
    "    \"\"\"Predict with our model\"\"\"\n",
    "    h = np.dot(model['W1'], x)    # Dense 1\n",
    "    h[h<0] = 0                    # ReLU nonlinearity\n",
    "    logp = np.dot(model['W2'], h) # Dense 2\n",
    "    p = sigmoid(logp)             # Sigmoid\n",
    "    return p, h # return probability of taking action 2, and hidden state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-13T11:16:24.123976Z",
     "start_time": "2017-07-13T19:16:24.119296+08:00"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def policy_backward(eph, epdlogp):\n",
    "    \"\"\" backward pass. (eph is array of intermediate hidden states) \"\"\"\n",
    "    dW2 = np.dot(eph.T, epdlogp).ravel() # ? weighted change to W2?\n",
    "    dh = np.outer(epdlogp, model['W2'])  # Backprop Dense 2\n",
    "    dh[eph <= 0] = 0                     # Backprop relu\n",
    "    dW1 = np.dot(dh.T, epx)              # Backprop Dense 1\n",
    "    return {'W1':dW1, 'W2':dW2}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## discount_rewards"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-13T11:16:24.412071Z",
     "start_time": "2017-07-13T19:16:24.408013+08:00"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def discount_rewards(r):\n",
    "    \"\"\" take 1D float array of rewards and compute discounted reward \"\"\"\n",
    "    discounted_r = np.zeros_like(r)\n",
    "    running_add = 0\n",
    "    for t in reversed(xrange(0, r.size)):\n",
    "        if r[t] != 0: running_add = 0 # reset the sum, since this was a game boundary (pong specific!)\n",
    "        running_add = running_add * gamma + r[t]\n",
    "        discounted_r[t] = running_add\n",
    "    return discounted_r"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-13T11:16:24.754280Z",
     "start_time": "2017-07-13T19:16:24.591595+08:00"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f2bd47dff50>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlYAAAF1CAYAAAAqdaQaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3XecVOX1x/HP2WWpUlTAQrXRuwiiUlQsIBawYqICMUaN\n5Rdjj2KLSYzGaMSoURMsKFZQUSxYAEVAQEQFERSUJr3Dwu7c5/fHc3cZlu3szN3d+b5fL3Rn5u69\nZ8rOnHmec89jzjlEREREZO+lRR2AiIiISGWhxEpERESkjCixEhERESkjSqxEREREyogSKxEREZEy\nosRKREREpIwosZJiMzNnZodHHYckjpkNMbNPo44jh5ndaWbPRx1HspjZSDP7czmIo6eZzY86jtIy\nsy1mdmjUceTHzJqH76VVCrj9VjN7KtlxSdlRYpWizOwWMxuf57oFBVx3QXKjK1/KW7IhkmjOucnO\nuZZRx1EcZvaJmV0af51zbh/n3I9RxbQ3nHN/cc5dCkUnYVI+KbFKXZOAY8wsHcDMDgIygM55rjs8\n3FYSJKo3zZznOdVF8Tgk+jk3T+/vIhHQH17q+gKfSHUKL/cEPgbm57nuB+fc8rjf6xuOYm0ws0fN\nzHJuMLNhZjbPzNab2Xtm1izuNmdmlxf0u3HbVTez7WZWP7z8JzPLNrM64eV7zOyh8OdqZvaAmf1s\nZivN7HEzq5HfnTWzdDP7h5mtMbNFZnZV/DdBM6trZk+b2QozW2Zmfw5/pzXwONAjnF7YUMD+h4b3\nfbOZ/WhmvyvogQ9HwD4zs3+a2VrgzsIePzO7y8weCX/OMLOtZnZ/eLmGmWWa2X7h5VfM7Bcz22hm\nk8ysbdxxR5rZY2b2jpltBY43s/3N7E0z22Rm04HDCon7GTP7Y/hzo/Dx+314+TAzW5fzYW5mA8xs\ndvhcTzGzDoXs92EzWxLGMNPMehawXXUze97M1ob7/cLMDghvOzi8H+vMbKGZ/baQ4+X3OBT4WjKz\niWZ2dvjzseH9Pi28fKKZzY57DD4K41tjZqPMrF7ccReb2U1mNgfYamZVzKyzmc0KXzcvAdULiTvn\ndTMifH6/M7MT427/xMzuNbPPgG3AoYW8rquFj2G7uN9vYP5vr6GZ9TGzpXG3tQ73v8HMvjWzM/Ic\n99I8cX4a/mzmX+erwuf36/hj5rl/Q62QvyEzOzN8TW0ysx/M7FQzuxf/PjXC/N/niHDb3LKF8DF4\n1sxWm9lPZnZb3Ot0iJl9Gj73682/N/TLc19+DGNaZGa/KiD2bmb2efj4rAifo6oFPZehYWa2PNz+\n+rh9xU9/53yp3RDevx5F7FPKASVWKco5txOYBvQKr+oFTAY+zXNd3tGqAcBRQAfgPOAU8G96wK3A\nIKBBuK8Xi/O7eeLKxCd9vcOregM/AcfGXZ4Y/vw3oAU+ETwcaAQML+Au/xboF27bBTgrz+0jgexw\nP52Bk4FLnXPzgMuBz8PphXrkb1V4/+oAQ4F/mlmXArYF6A78CBwA3FvE4zcR6BP+fBTwC7ueox7A\nfOfcuvDyeOAIoCEwCxiV57gXAvcCtfHP9aNAJnAQMCz8V5D4OHqH8feKuzzZOReYWWfgv8DvgP2B\nJ4A3zaxaAfv9Av+87Ae8ALxiZvklGJcAdYEm4X4vB7aHt40GlgIHA+cAfzGzEwq5L3kfh8JeS0Xd\n75zXowF/DWNoHcZ5Z57jDgZOA+rh33/HAs+F9/0V4OxCYgb/uvkBqA/cAbxuYVIdugi4LLxfP1Hw\n63oH8HoYT47zgInOuVXxBzSzDOAt4H386+pqYJSZFWeq8GT8Y9UC/9ydB6wtYNsC/4bMrBvwLHAD\n/rHrBSx2zv0J/7dyVfj3eVU++30kPPah+Ofr4nD/Obrjv1DWB/4OPB0mhLWAfwH9nHO1gWOA2QXE\nHgP+EO6jB3AicGXBDwsAx+P/Vk8GbjKzvvlsk/M6qxfev8+L2KeUB845/UvRf/g3/THhz1/h/8hP\nzXPdJXHbO+C4uMsvAzeHP48HfhN3Wxr+W3Ozon43n7juwb+hVcEnEdfiP/iq4z9I98d/iG0FDov7\nvR7AogL2+RHwu7jLfcOYquCTmx1AjbjbBwMfhz8PAT4t4WM7Fri2gNuGAD/nua7Axw+ogU9+9gdu\nxidgS4F9gLuAfxVwnHrhfawbXh4JPBt3ezqQBbSKu+4vBd1X/GjW+jC2x/GJ09LwtmeA68KfHwPu\nyfO784HexXzs1gMd416jz4c/DwOmAB3ybN8E/8FWO+66vwIjC9h/3seh0NcS/kNyTvjzu8ClwNTw\n8kRgUAHHOQv4Mu7yYmBY3OVewHLA4q6bAvy5kNdN3u2nAxeFP38C3B13W1Gv6774Eemc2z4DLg5/\n7hP33PbE/x2mxW37InBn3HEvzRPnp+HPJwDfA0fH/35J/4bwyfk/C9hut+OH1zl8MpkO7ATaxN32\nO+CTuFgXxt1WM/zdA4FawAZ8slujhLH/H+H7aD63NQ+PEf9393fg6Xxe8znbVinJ8fUv2n8asUpt\nk4Djwm+8DZxzC/Bv7MeE17VjzxGrX+J+3ob/cAefADwcDoVvANbhP7AaFeN388oZIegCfA18gP+m\neTT+TXAtflSnJjAz7pjvhtfn52BgSdzl+J+b4adFV8Tt6wn8t/NiMbN+ZjbV/FTUBqA//ttrQZbk\nuVzg4+ec2w7MwD8GvfCPzxT8KF7uiEk4xfO3cJpkE/6DnDxxxB+3AT6xjL/up4ICds79gE9AOuE/\nbMcBy8ORi/iRm2bAH3PuS3h/muCfgz2Y2fXhFNDGcNu65P/YPQe8B4wOp1D+Ho6mHAysc85tznM/\nGuWzjxx5H4fCXkufAy3MTzt2wo+cNDE/Xd2N8G/EzA4ws9Hmp9w2Ac/ncz/ij3swsMyFn6BxcRcm\nv+3jH9eSvK4/BmqaWXczax7etzH5HPNgYIlzLshz3MIeXwCccx8BI/Ajo6vM7D8WTuvnVcTfUBP8\nSF1J1cc/BvGPa97Yc9+XnHPbwh/3cc5tBc7Hj4yuMLO3zaxVAbG3MLNx5qfhN+G/oBT29w97/t3l\n+/chFY8Sq9T2Of5D7Lf4b6s45zbhvxX/FljunFtUzH0twY8I1Yv7V8M5N6UUcU0BWgID8VMTc4Gm\n+DfanA/vNfjRq7Zxx6vrnCsoWVsBNI673CRP7DuA+nH7quOcy6lPiv8g20M4xfUa8ABwgPPThe/g\nE6OC5N1nUY/fRPy3/874qbOJ+KnU3A92/PTWmfiRiLr4b7vkiSP+uKvx00Txj0XTwu5reNxzgKrO\nuWXh5UuAfdk1TbIEuDfPfanpnMs7NYz5eqob8VNE+4aP3Ubyeeycc1nOubucc23w0zID8NM6y4H9\nzKx2nvuxrJD7Ef84FPpaCj9sZ+JHTr9xfhp9CnAdfsRnTbifv4T7be+cqwP8Op/7EX/cFUAjs91q\nDYt6/PPbPr4GMn7/hb6unXMx/Mjx4PDfuDzJaY7l+EQy/vMi/vHdik9McxwY/8vOuX85544E2uCn\nBG/Ie4Bi/A0toeD6v8L+PtfgR2WbxV1X1GsjPvb3nHMn4afKvwOeLGDTx8Lbjwif+1sp/O8f9vy7\nW57PNoW+90j5pMQqhcWNhFyHr1PI8Wl4XUnOBnwcuMXCYumwYPTcUsaV80H2e3YlUlPw3xwnhtsE\n+De5f5pZw/CYjcxsj7qt0MvAteE29YCb4o63Al8/8g8zq2NmaeYLkXPqvFYCjQspRq0KVCNMVMLi\n15NLeLeLevwm4pOIueEH+yf4KalFzrnV4Ta18R+ka/EfdH8p7IDhB+vrwJ1mVtPM2uCTpMJMBK5i\n12vjk/Dyp+H+wD8vl4cjIWZmtczstDyJT47a+ORuNVDFzIbja2z2YGbHm1l782fxbcJ/YAbOuSX4\n18dfzRe4dwB+gx8xKlIxX0s59zvn9fhJnss592ULsNHMGpFPApHH5+F9v8b8SQmD8IlyYRrGbX8u\nvpbrnQLuV1Gva/A1becDvwp/zs80/AjzjeFx+wCn4+vawCfUg8LX0OH4xx4AMzsqfB1k4BOwTCBg\nT0X9DT0NDDV/skBa+PzkjB6txNdP5fcY5CSP95pZbfMnhFxHMV4b4QjkmWGt1Q78c5tf7OCf+03A\nljCuK4raP3B7+Ji1xdd8vZTPNqvDY5bLnlySPyVWMhH/Zh3fp2lyeF2xEyvn3BjgPvw0zSbgG3yx\n+N7ElYGvIcm5XDtPTDcBC4Gp4TEn4Ee68vMk/kNmDvAl/sMoG1+bAz5pqQrMxdf4vIr/lgq+Putb\n4BczW0Me4bf8a/Bv4OvxI0dvluTOFuPxm4Kvtcq5/3PxH1Lxj8ez+CmFZeHtU4tx6KvwU7K/4GuP\n/lfE9nmfh0/xSVxuHM65GfgRzxH4x2MhvpYlP+/hp92+D2PPZM9p0hwH4p+XTcC8MJbnwtsG40fo\nluOns+5wzk0o4r7EK+q1lPd+5/d6vAs/fb0ReBuftBYoTJAH4R+bdfgEp9DfwSc5R+BHYu4Fzgmn\nxgtS2Osa59w0fMJzML7Or6A4T8e/HtcA/8bXYn0XbvJPfB3TSnytXfwJE3Xwf3vr8c/vWuD+fI5R\n6N+Qc246YUE7/vGdyK5RqIeBc8yf1fevfO7C1eF9/BH/en0Bf3JFUdLwSdhy/PPTm4ITpuvDmDeH\n9ze/JCmvifjX3IfAA8659/NuEH7JvBf4LJzOPboY+5WI2e7T9SKpIfxG/LhzrlmRG4uUA2Y2BF+k\nfVzUsYhIwTRiJSnBfL+n/uZ7BzXCn6qeX6GuiIhIqSmxklRh+Kma9fipwHkU3PNKRESkVDQVKCIi\nIlJGNGIlIiIiUkaUWImIiIiUkYSusF6Y+vXru+bNm0d1eBEREZFimzlz5hrnXEGre+SKLLFq3rw5\nM2bMiOrwIiIiIsVmZkUtOQVoKlBERESkzCixEhERESkjSqxEREREykhkNVYiIiLlUVZWFkuXLiUz\nMzPqUCQC1atXp3HjxmRkZJTq95VYiYiIxFm6dCm1a9emefPmmFnU4UgSOedYu3YtS5cu5ZBDDinV\nPjQVKCIiEiczM5P9999fSVUKMjP233//vRqtVGIlIiKSh5Kq1LW3z70SKxEREdlN8+bNWbNmTdRh\nVEhKrERERMox5xxBECRs/9nZ2QnbdyoqMrEys+pmNt3MvjKzb83srny2qWZmL5nZQjObZmbNExGs\niIhIKli8eDEtW7bk4osvpl27djz33HP06NGDLl26cO6557Jlyxa++OILBg0aBMAbb7xBjRo12Llz\nJ5mZmRx66KEAPPnkkxx11FF07NiRs88+m23btgEwZMgQLr/8crp3786NN97I2rVrOfnkk2nbti2X\nXnopzrnI7ntFV5yzAncAJzjntphZBvCpmY13zk2N2+Y3wHrn3OFmdgFwH3B+AuIVERFJmrve+pa5\nyzeV6T7bHFyHO05vW+R2CxYs4JlnnuHwww9n0KBBTJgwgVq1anHffffx4IMPcuuttzJ79mwAJk+e\nTLt27fjiiy/Izs6me/fuAAwaNIjf/va3ANx22208/fTTXH311YA/+3HKlCmkp6dzzTXXcNxxxzF8\n+HDefvttnn766TK9z6mkyMTK+bR1S3gxI/yXN5U9E7gz/PlVYISZmVPKKyIiEftl8Tzq1D+ImvvU\nizqUEmnWrBlHH30048aNY+7cuRx77LEA7Ny5kx49elClShUOO+ww5s2bx/Tp07nuuuuYNGkSsViM\nnj17AvDNN99w2223sWHDBrZs2cIpp5ySu/9zzz2X9PR0ACZNmsTrr78OwGmnnca+++6b5HtbeRSr\nj5WZpQMzgcOBR51z0/Js0ghYAuCcyzazjcD+wJo8+7kMuAygadOmexe5iIhIUVZ9x74jezK76VC6\nD7u/xL9enJGlRKlVqxbga6xOOukkXnzxxT226dWrF+PHjycjI4O+ffsyZMgQYrEY99/v7+uQIUMY\nO3YsHTt2ZOTIkXzyySd77F/KVrGK151zMedcJ6Ax0M3M2pXmYM65/zjnujrnujZo0KA0uxARESme\nIAZv/J5qZOF2bI46mlI7+uij+eyzz1i4cCEAW7du5fvvvwegZ8+ePPTQQ/To0YMGDRqwdu1a5s+f\nT7t2/mN68+bNHHTQQWRlZTFq1KgCj9GrVy9eeOEFAMaPH8/69esTfK8qrxKdFeic2wB8DJya56Zl\nQBMAM6sC1AXWlkWAIiIipTL137Bshv/ZJe6sukRr0KABI0eOZPDgwXTo0IEePXrw3XffAdC9e3dW\nrlxJr169AOjQoQPt27fP7cV0zz330L17d4499lhatWpV4DHuuOMOJk2aRNu2bXn99dc1q7QXrKgy\nKDNrAGQ55zaYWQ3gfeA+59y4uG1+D7R3zl0eFq8Pcs6dV9h+u3bt6mbMmLH390BERCSvNQvh8WPh\nsBPY+N1EvjugP92vfKpYvzpv3jxat26d4AClPMvvNWBmM51zXYv63eKMWB0EfGxmc4AvgA+cc+PM\n7G4zOyPc5mlgfzNbCFwH3FyieyAiIlJWggDevAqqVIPTHiSGgc6lkiQpzlmBc4DO+Vw/PO7nTODc\nsg1NRESkFKb/B37+HM78N9Q5iIA0rAJPBUrFos7rIiJSeaxbBB/eBYefBJ0uBMBhFbrGSioWJVYi\nIlI5BAG8eTVYOpz+EJj55WAwDCVWkhzF6mMlIiJS7s38HyyeDKc/DHUbAxA4CEhTjZUkjUasRESk\n4lu3CN6/HQ7tA10uyb06CEesNBUoyaLESkREKrYgBmOvhLR0OGMEhD2cwA9UOYw0F4swwL1z5513\n8sADDwAwfPhwJkyYEFkss2fP5p133inx7/Xp04fy2GJpyJAhvPrqq2W6T00FiohIxTb13/DzFH8W\nYL0mu90UOEfgjD2XuK2Y7r777kiPP3v2bGbMmEH//v3LfN+xWCx37cJEyM7OpkqVxKc9GrESEZGK\na9U8+PAeaNk/9yzAeC63xqpiTQXee++9tGjRguOOO4758+fnXh8/wnLzzTfTpk0bOnTowPXXXw/A\nypUrGThwIB07dqRjx45MmTIFgAcffJB27drRrl07HnroIQAWL16cu/QNwAMPPMCdd94J+BGmm266\niW7dutGiRQsmT57Mzp07GT58OC+99BKdOnXipZdeYuvWrQwbNoxu3brRuXNn3njjDQC2b9/OBRdc\nQOvWrRk4cCDbt2/P9342b96cm266iS5duvDKK6/www8/cOqpp3LkkUfSs2dPvvvuO2KxGIcccgjO\nOTZs2EB6ejqTJk0C/FI8CxYsYPr06fTo0YPOnTtzzDHH5D5mI0eO5IwzzuCEE07gxBNPxDnHVVdd\nRcuWLenbty+rVq0qq6csl0asRESkYoplwZjLodo+vmA9bgowh8MRIw0rbfH6+Jvhl6/3MtA8DmwP\n/f5W4M0zZ85k9OjRzJ49m+zsbLp06cKRRx652zZr165lzJgxfPfdd5gZGzZsAOCaa66hd+/ejBkz\nhlgsxpYtW5g5cyb/+9//mDZtGs45unfvTu/evdl3330LDTM7O5vp06fzzjvvcNdddzFhwgTuvvtu\nZsyYwYgRIwC49dZbOeGEE/jvf//Lhg0b6NatG3379uWJJ56gZs2azJs3jzlz5tClS5cCj7P//vsz\na9YsAE488UQef/xxjjjiCKZNm8aVV17JRx99RMuWLZk7dy6LFi2iS5cuTJ48me7du7NkyRKOOOII\nNm3axOTJk6lSpQoTJkzg1ltv5bXXXgNg1qxZzJkzh/3224/XX3+d+fPnM3fuXFauXEmbNm0YNmxY\n0c9ZCSixEhGRimnyP2DFbDjvWdinYb6bBGGNVUVqtzB58mQGDhxIzZo1ATjjjDP22KZu3bpUr16d\n3/zmNwwYMIABAwYA8NFHH/Hss88CkJ6eTt26dfn0008ZOHAgtWrVAmDQoEFMnjw53/3GGzRoEABH\nHnkkixcvzneb999/nzfffDO3BiwzM5Off/6ZSZMmcc011wB+/cIOHToUeJzzzz8fgC1btjBlyhTO\nPXdXv/EdO3YAfrHpSZMmsWjRIm655RaefPJJevfuzVFHHQXAxo0bueSSS1iwYAFmRlZWVu4+Tjrp\nJPbbbz8AJk2axODBg0lPT+fggw/mhBNOKPQxKA0lViIiUvEs/xIm3Q/tz4M2Zxa42V6fFVjIyFKU\nqlSpwvTp0/nwww959dVXGTFiBB999FGJ9xEEux6XzMzM3W6vVq0a4BO07OzsfPfhnOO1116jZcuW\nJbwHu+QkfEEQUK9ePWbPnr3HNr169eKxxx5j+fLl3H333dx///188skn9OzZE4Dbb7+d448/njFj\nxrB48WL69Omzx/6TRTVWIiJSsWRl+inAWg2h/98L3TSnxqoiLWnTq1cvxo4dy/bt29m8eTNvvfXW\nHtts2bKFjRs30r9/f/75z3/y1VdfAX4q7bHHHgN8MfjGjRvp2bMnY8eOZdu2bWzdupUxY8bQs2dP\nDjjgAFatWsXatWvZsWMH48aNKzK22rVrs3nz5tzLp5xyCo888ggunGr98ssvc+/DCy+8AMA333zD\nnDlzitx3nTp1OOSQQ3jllVcAn7Tl3K9u3boxZcoU0tLSqF69Op06deKJJ56gV69egB+xatSoEeDr\nqgrSq1cvXnrpJWKxGCtWrODjjz8uMq6SUmIlIiIVy0f3wOrv4MxHoEbhdULOuXAqsOKcFdilSxfO\nP/98OnbsSL9+/XKnu+Jt3ryZAQMG0KFDB4477jgefPBBAB5++GE+/vhj2rdvz5FHHsncuXPp0qUL\nQ4YMoVu3bnTv3p1LL72Uzp07k5GRwfDhw+nWrRsnnXQSrVq1KjK2448/nrlz5+YWr99+++1kZWXR\noUMH2rZty+233w7AFVdcwZYtW2jdujXDhw/fo0asIKNGjeLpp5+mY8eOtG3bNrcYvlq1ajRp0oSj\njz4a8FODmzdvpn379gDceOON3HLLLXTu3LnA0TWAgQMHcsQRR9CmTRsuvvhievToUay4SsJcRN1o\nu3bt6spjTwsRESnHFn8GI0+DrkNhwD+L3Hzd1p2suK8r2bUb0/GG4vVfmjdvHq1bt97bSKUCy+81\nYGYznXNdi/pdjViJiEjFsGMzjL0C9m0GJ91TrF/RWoGSbCpeFxGRiuG9W2HDzzB0vG+xUAw5awWm\nVaAaK6nYNGIlIiLl39w3YdazcOy10Kz4dTG5NVZKrCRJlFiJiEj5tmk5vHUNHNQJjv9TiX7VQTgV\nWLJ64qjqjyV6e/vcK7ESEZHyKwh8a4XsHXD2U1Clasl+3bkSt1uoXr06a9euVXKVgpxzrF27lurV\nq5d6H6qxEhGR8mvqo7BoIpz+L6h/RIl/PXD4JW1KMGLVuHFjli5dyurVq0t8PKn4qlevTuPGjUv9\n+0qsRESkfFoxBybcBa0GQJeLS7WLnBqrknRez8jI4JBDDinV8UQ0FSgiIuXPzm3w2qVQqz6c8Ui+\nCywXh3MQOLVbkOTRiJWIiJQ/H9wOa+bDRWOh5n6l3k2Q28dK9VKSHBqxEhGR8mX+u/DFU9DjKjjs\n+L3aVUVcK1AqNiVWIiJSfmxeCW/8Hg5oDycO3+vdBRVwrUCp2JRYiYhI+eAcvHEl7NwStlaotte7\n9J3XjTQXK4MARYqmGisRESkfpj4GCydA/wegYasy2aUL+1ihEStJEo1YiYhI9JbNhA+GQ8vT4KhL\ny2y3OZ3XtVagJIsSKxERiVbmRnhlKNQ+EM4cUerWCvlRjZUkm6YCRUQkOs7BW9fCxqUwdPxetVbI\nTxCEZwWqj5UkiUasREQkOjP/B9+OgRNvh6bdy3z3DkcMw7TunySJEisREYnGL9/Au7fAYSfCMdcm\n5BDOgdOIlSSREisREUm+nVvh1aFQvS4MfALSEvNxpM7rkmyqsRIRkeR75wZYswAufgP2aZCww7iw\nj5U6r0uyaMRKRESSa/aLMHsU9L4RDu2d0EMFYR8rjVhJsiixEhGR5FmzAN7+IzQ7FnrdmPDDBQ4c\nRppqrCRJlFiJiEhyZG2HV4b4pWrOfgrSk1GN4gicEitJHtVYiYhIcrx9Paz8Bi58BeocnJRD+rUC\n09RuQZJGI1YiIpJ4s56F2c9DrxugxclJO2wQ5JwVqBErSQ4lViIiklgrvvKjVYf2gT63JPXQDrSk\njSSVEisREUmc7evhpYugVn04+2lIS0/q4XP6WKnGSpKlyMTKzJqY2cdmNtfMvjWzPdrjmlkfM9to\nZrPDf8MTE66IiFQYQQBjroBNy+DckT65SjLnIKZ2C5JExSlezwb+6JybZWa1gZlm9oFzbm6e7SY7\n5waUfYgiIlIhffYQfD8eTr0PmnSLJASn4nVJsiJHrJxzK5xzs8KfNwPzgEaJDkxERCqwRZPgo3ug\n7UDo/rvIwgicUx8rSaoS1ViZWXOgMzAtn5t7mNlXZjbezNqWQWwiIlIRbVoOrw6D/Q+HMx4Bs8hC\n0VqBkmzF7mNlZvsArwH/55zblOfmWUAz59wWM+sPjAWOyGcflwGXATRt2rTUQYuISDkVy4JXhsLO\nbXDJOKhWO9JwHH4qUCNWkizFGrEyswx8UjXKOfd63tudc5ucc1vCn98BMsxsjypF59x/nHNdnXNd\nGzRI3KKbIiISkQ/ugCVT4Yx/QcNWUUeD01mBkmTFOSvQgKeBec65BwvY5sBwO8ysW7jftWUZqIiI\nlHNzXoapj0K3y6D9OVFHA/gTE9XHSpKpOFOBxwIXAV+b2ezwuluBpgDOuceBc4ArzCwb2A5c4JxO\nwRARSRnLZ8ObV/vFlU/5S9TR5PJTgUaaEitJkiITK+fcp0ChlYfOuRHAiLIKSkREKpCta+ClX0PN\n/eHcZyA9I+qIcvni9TQtaSNJo0WYRUSk9GJZ8MoQ2LIKhr0L+5Sv+tldNVYasZLkUGIlIiKl9/7t\nsHgynPU4NOoSdTR7cA4Cl+YTK+cibf0gqUFrBYqISOl8NRqmPQbdr4BOg6OOJl+B8zVWgE+sRBJM\niZWIiJTc8i/hrWuheU84+Z6ooylQTo0VAE51VpJ4SqxERKRktqyG0b+GWg384srlqFg9L4dvt+Av\nKLGSxFO9ASj+AAAgAElEQVSNlYiIFF8sC165BLatgWHvQa09ekGXKznF6/6CEitJPCVWIiJSfO/e\nDD99BoOehIM7RR1NkQIlVpJkmgoUEZHimf4kfPEUHHM1dDgv6miKxTniaqxi0QYjKUGJlYiIFO2H\nj2D8TdCiH/S9K+poii1wqrGS5FJiJSIihVv9Pbw8BBq0grOfhLT0qCMqNk0FSrIpsRIRkYJtWwcv\nng9VqsKFo6Fa7agjKpndpgLVx0oST8XrIiKSv1gWvHwxbFwKl4yDek2jjqjENGIlyabESkRE9uQc\nvHO9X65m4BPQtHvUEZVKsNuIlRIrSTxNBYqIyJ6mPQEzR8Jxf4COF0QdTak5NGIlyaXESkREdrdg\nArx3C7QaACcMjzqavaIRK0k2JVYiIrLLqnnw6lBo2NZPAaZV7I+J+M7rLlAfK0m8iv0XIyIiZWfz\nLzDqXMioAYNfhGr7RB3RXnNxfaxcoBErSTwVr4uICOzYAi+c59srDH0H6jWJOqIyEThH4MLESlOB\nkgRKrEREUl0sG177DfzyNQweXSHWACyu+BqrIIhRcVqbSkWlxEpEJJU5B+/eBN+/C6f9A1qcEnVE\nZUo1VpJsqrESEUlln48IF1a+Bo66NOpoypxqrCTZlFiJiKSqb8fC+7dBm7Mq1MLKJeE7r/uPOiVW\nkgxKrEREUtHP0+D1y6BJ90rRVqEgDnKnAgMVr0sSVM6/JBERKdjaH2D0YKjbCC54ETKqRx1RwsSP\nWKlBqCSDEisRkVSyZTWMOscXH/3qVai1f9QRJZRzcSNWKl6XJNBZgSIiqWLHFnjhXNi0Ai5+A/Y/\nLOqIEm63swJjGrGSxFNiJSKSCrJ3wssXwYo5cMEoaNo96oiSQmsFSrIpsRIRqeyCAN64En74CM58\nFFr2izqipAmcy223oOJ1SQbVWImIVGbOwft/gq9fgROHQ+dfRx1RUsXXWLkgO+JoJBUosRIRqcw+\nexim/hu6Xw7HXRd1NEnn1MdKkkyJlYhIZTX7BZhwB7QdBKf8FcyijijpgrgRK5RYSRIosRIRqYy+\nfw/euAoO6Q0DH6+0DUCL4lCNlSRXav6liYhUZku+gJcvgQPbwfnPQ5VqUUcUmcBB4HLOClQfK0k8\nJVYiIpXJL9/4BqC1D/ANQKvXiTqiSAXxfawCF3E0kgqUWImIVBZrf4DnBkJGDd8AdJ+GUUcUPQex\n8KNOndclGdTHSkSkMti4FJ490093XTwO9m0edUTlQnwfK1ONlSSBEisRkYpuyyqfVGVuhEveggYt\no46o3Ah2WytQiZUknhIrEZGKbPt6eG4QbFwGF42BgztFHVG54uKWtHEasZIkUGIlIlJR7dgCo86D\n1d/BhaOhWY+oIyp34ovXUY2VJIESKxGRiigrE0ZfCMtmwLnPwOF9o46oXHJxNVbqvC7JoMRKRKSi\niWXBq8Ng0UQ46zFoc0bUEZVbjvipQI1YSeIV2W7BzJqY2cdmNtfMvjWza/PZxszsX2a20MzmmFmX\nxIQrIpLighiMuRzmvw397odOF0YdUbm221SgaqwkCYozYpUN/NE5N8vMagMzzewD59zcuG36AUeE\n/7oDj4X/FxGRshLEYOyV8M2r0PdO6H5Z1BGVe/FnBWoqUJKhyBEr59wK59ys8OfNwDygUZ7NzgSe\ndd5UoJ6ZHVTm0YqIpKoggDevgTmj4fjb4Lg/RB1RheAcqrGSpCpR53Uzaw50BqbluakRsCTu8lL2\nTL4ws8vMbIaZzVi9enXJIhURSVVBAG//AWY/D71vgt43RB1RheGcy62xAiVWknjFTqzMbB/gNeD/\nnHObSnMw59x/nHNdnXNdGzRoUJpdiIikFudg/A0wcyQcdx30uSXqiCqUwLncJW00YiXJUKzEyswy\n8EnVKOfc6/lssgxoEne5cXidiIiUlnPw7i3wxVNwzNVw4nAwizqqCsXF11ipeF2SoDhnBRrwNDDP\nOfdgAZu9CVwcnh14NLDRObeiDOMUEUktzsEHt8O0x+DoK+Gke5RUlULgwLmcBqFKrCTxinNW4LHA\nRcDXZjY7vO5WoCmAc+5x4B2gP7AQ2AYMLftQRURShHPw0T0w5RHodhmc8hclVaUUX2OlPlaSDEUm\nVs65T4FC/6Kdcw74fVkFJSKSspyDD++GTx+EI4dCv78rqdoLvkGoRqwkedR5XUSkvMiZ/pvyiE+q\nTntQSdVeUoNQSTYlViIi5UFOofq0x/z0n0aqykTgwOVOBSqxksRTYiUiErUggHeuhxlPw9G/h1Pu\nVVJVRlz8iFWgGitJPCVWIiJRCgIYdy3MehaOvRb63qWkqgzt3m7BRRyNpAIlViIiUQli8ObVMHsU\n9LoBjv+TkqoytnuNlUasJPGUWImIRCGWDWOvgK9fhj63Qp+boo6oUnJxNVZoxEqSQImViEiyxbLg\n9d/Ct2N8N/Wef4w6okprtyVtNGIlSaDESkQkmbK2w8uXwIL3fDf1Y6+JOqJKLXDxfaw0YiWJp8RK\nRCRZMjfBi4Php89gwD+h67CoI0oBDpdbvK4RK0k8JVYiIsmwbR08fzas+ArOfgranxN1RCnBj1j5\nqUBT53VJAiVWIiKJtvkXePYsWPcjXDAKWvaLOqKUsdtZgSixksRTYiUikkjrF8OzZ8LWNfDrV+GQ\nXlFHlFKcA0sLi9c1YiVJoMRKRCRRVs/3I1VZ2+DiN6Bx16gjSjmBc6RZTrsFJVaSeEqsREQSYfls\neH4QWDoMfQcOaBt1RCnJOUhPSyNwpsRKkiIt6gBERCqdHz+BkQMgoxYMe1dJVYQcjvQ083VWSqwk\nCZRYiYiUpW9eg+fPgXpN4Dfvwf6HRR1RSgsCSDPfy8opsZIkUGIlIlJWpj0Br/4GGh8FQ8dDnYOj\njijlBc5RJT3NL2ujJW0kCVRjJSKyt5yDD++GTx+EVgN8n6qMGlFHJfinJs3ML2ujBqGSBEqsRET2\nRiwb3roWZj8PRw6B0x6EtPSoo5KQr7FCNVaSNEqsRERKa+c2eHUofP8u9L4Z+twMZkX/niRN4KBK\nWhoBhimxkiRQYiUiUhrb1sEL58OyGX6U6qjfRB2R5CNwjrQ0cCpelyRRYiUiUlLrfvRn/m1cCuc+\nA23OiDoiKYBzkG7m1wtU53VJAiVWIiIlsWQ6vHiBr9e5+A1o1iPqiKQQzsX1sdJagZIEarcgIlJc\nc9+AZ06HanXg0g+VVFUAgUMNQiWplFiJiBTFOZjyCLx8CRzYAS6doMafFYTDkWaGI03F65IUmgoU\nESlMLBvevQm+eAranAUDH1ePqgokCKBKes6IlRqESuIpsRIRKciOLfDqMFjwHhxzDfS9C9I00F+R\nBM6RYWmaCpSkUWIlIpKfTct9kfovX0P/B6Dbb6OOSEopLS08K1CJlSSBEisRkbyWzYIXB8POLTB4\nNLQ4JeqIpJQC56iSZgROI1aSHEqsRETiffM6jL0SatWHYe/Bge2ijkj2QhCuFagRK0kWJVYiIuAL\nmyfeB5/8FZp0h/NHwT4Noo5K9lJ8HyvTIsySBEqsRESytvtRqm9fh46D4fSHoUq1qKOSMuDCPlYO\nw3RWoCSBEisRSW2bVsDowbB8tj/r79hrtZByJRLkjlilka6pQEkCJVYikrqWf+mL1DM3wQUvQKv+\nUUckZcyRs1agUUWJlSSBEisRSU1zXoE3r/ZF6r95X0XqlVSw21qBmgqUxFNiJSKpJZYNHwyHqY9C\ns2Ph3GdUpF6JBUFOjZXOCpTkUGIlIqlj61p4dQgsmgTdfgen3AvpGVFHJQmWlntWoBIrSTwlViKS\nGlbMgdG/gi0r4cx/Q+dfRR2RJEFug1AtaSNJosRKRCq/nHqqmvvBsPHQ6MioI5IkCZzLLV431VhJ\nEiixEpHKK5YNE+6Az0eE9VQjYZ+GUUclSeSc1gqU5CpymXYz+6+ZrTKzbwq4vY+ZbTSz2eG/4WUf\npohICW1ZBc+d5ZOqbpfBxW8oqUpBgSOcCkxT53VJiuKMWI0ERgDPFrLNZOfcgDKJSERkb/30Obw6\nFLavh7Meg04XRh2RRMQ5l1u8rhErSYYiR6ycc5OAdUmIRURk7zgHU0bAyNMgowZc+qGSqhSX0yBU\nS9pIspRVjVUPM/sKWA5c75z7toz2KyJSPJmb4I3fw7w3odUAOOvfUL1u1FFJxALnSDP8VCAasZLE\nK4vEahbQzDm3xcz6A2OBI/Lb0MwuAy4DaNq0aRkcWkQEWPktvHQRrF8MJ/8Zelyl9f4EgCBwWDhi\npalASYYipwKL4pzb5JzbEv78DpBhZvUL2PY/zrmuzrmuDRqo07GIlIHZL8KTJ8LOrTBkHBxztZIq\nyeXwLwdfY6WpQEm8vR6xMrMDgZXOOWdm3fDJ2tq9jkxEpDA7t8H4G+HL56B5Tzj7aah9QNRRSTnj\nHKSZX9JGU4GSDEUmVmb2ItAHqG9mS4E7gAwA59zjwDnAFWaWDWwHLnBOXwtEJIFWzvVn/a2eDz3/\nCH1uhXS15ZM9Bc6Fyy9rSRtJjiLfiZxzg4u4fQS+HYOISGI5B7OegfE3QbU6cNHrcNgJUUcl5Vhu\ng1BLA3VelyTQVzwRqRgyN8G4/4NvXoND+8DA/2jqT4oUOIeZRqwkeZRYiUj5t/xLeGUobPgZTrgd\njrsO0vb63BtJAc6BkdN5XYmVJJ4SKxEpv5yDaY/D+7fDPgfAkLehWY+oo5IKxOH7WKl4XZJFiZWI\nlE+bV/qGnws/gJb94cxHoeZ+UUclFUyQc1agaSpQkkOJlYiUP/Pf9UnVzi3Q/wE46lL1ppJS2VVj\npeJ1SQ4lViJSfuzcBu/fBjOehgPaw9lPQcNWUUclFZhz5HZe14iVJIMSKxEpH1Z8Ba9dCmu+90vS\nnDgcqlSLOiqpwHJaKqYZOFPxuiSHEisRiVYQwOcj4MO7oVZ9uGgsHHZ81FFJJRCEM3+5ZwVqKlCS\nQImViERnw8++lmrRJGg1AM54RAXqUmZ2G7EK0yuRRFNiJSLJ5xx8+Ty8ewvgfELV+SIVqEuZyhmx\nSkvzZwVqEWZJBiVWIpJcm1bAW9fAgvf94slnPgr7Nos6KqmEgt0SKfWxkuRQYiUiyeEcfP0qvHM9\nZO+AU++Dbpepg7okXJrldF7XiJUknhIrEUm8rWtg3B9g3pvQ+Cg463Gof3jUUUklF8TVWAWWRhqx\niCOSVKDESkQSa944eOta2LEJ+t4Jx1wDaelRRyUpIPesQPP/0YiVJIMSKxFJjO3rYfzNMGc0HNgB\nBr4FB7SJOipJIbvOCjStFShJo8RKRMre3Dd9LdXWNdD7Zuh1PaRnRB2VpJhdI1Y5ndc1YiWJp8RK\nRMrO5l98QjXvLTiwPVz4MhzcKeqoJEXljFgZYed1jVhJEiixEpG9l9OX6v0/QVYmnHgHHHO1Rqkk\nUjkDVLsahGrEShJPiZWI7J11i3xx+qKJ0OxYOP1fOuNPyoXcswLTTGsFStIosRKR0gliMPUx+OjP\nkFYFBvwTugxRXyopN3atFQhorUBJEiVWIlJyK7+FN66C5bOgRT847R9Qt1HUUYnsxoWJlOmsQEki\nJVYiUnw7t8Gkv8OUR6B6PTjnv9B2kNb4k3JpV42VXytQZwVKMiixEpHimT8e3rkRNv4MnX4FJ/8Z\nau4XdVQiBcqpsTIDZ+nqvC5JocRKRAq3YQmMvwnmvw0NWsPQ8dDsmKijEilS/FmBgc4KlCRRYiUi\n+YtlweePwsT7/OW+d0GP36uFglQYu0as/FmBaTifbWnqWhJIiZWI7OmnKTDuOlg9D1qeBv3+BvWa\nRh2VSIm4+LMCc5IpJVaSYEqsRGSXrWvgg+EwexTUbQqDR0PLflFHJVIq8cXrAWEbEBcAagkiiaPE\nSkQglg0z/+d7Uu3cAsf9AXrdAFVrRR2ZSKntahAKWHxiJZI4SqxEUt2iSTD+Zlj1LRzSG/r9HRq2\nijoqkb2WW2OFKbGSpFFiJZKqNvwM798Gc9/w9VPnPQetT1f9iVQaOecAWrhWoL9SiZUklhIrkVSz\ncxt89jB89hBgcPyf/ILJGTWijkykTLmcqUDTiJUkjxIrkVThHMwdC+/fDhuXQLuz4aS7oW7jqCMT\nSYjctQI1YiVJpMRKJBX88rWvo/rpUzigPQx8ApofG3VUIgm1+5I2GrGS5FBiJVKZbVoOH93r2yfU\nqAenPQhHDoG09KgjE0m43LMCwyVtACVWknBKrEQqox2b4bN/+cWSXcx3TO91PdTYN+rIRJImyF10\nWTVWkjxKrEQqk1g2fPkcfPwX2LoK2g6CE4fDfodEHZlI0sWvFbir87oSK0ksJVYilYFzsOAD+OB2\nWP0dNDkaLngBmhwVdWQikdmtxgqNWElyKLESqehWzPH9qBZNhP0OVT8qkdCuRZjRVKAkjRIrkYpq\n3SI/5ff1K74w/dT7oOswqFI16shEyoWcCis/YqWpQEkOJVYiFc3mX2Di32HWM5CWAcde69f2q1Ev\n6shEyhWNWEkUlFiJVBTb18OnD8G0JyDIgi6X+IWS6xwUdWQi5ZLLTax0VqAkT5GJlZn9FxgArHLO\ntcvndgMeBvoD24AhzrlZZR2oSMrauRWmPubbJ+zYBO3PgeNv9fVUIlKg3c8KVGIlyVGcEauRwAjg\n2QJu7wccEf7rDjwW/l9E9kb2Tpg5Eibd71sntDgVTrgdDtzj+42I5CPYrfN6To2VK/gXRMpAWlEb\nOOcmAesK2eRM4FnnTQXqmZnmJkRKK5YFs56FEUfC+Bug/hEw7D248CUlVSIlkFtjBbtGrIJYZPGk\nuqxlXzHt6T+ybcvGqENJqLKosWoELIm7vDS8bkXeDc3sMuAygKZNm5bBoUUqkVgWfDXaj1Bt+AkO\n7uyXoDm8r1oniJSCy12E2UBL2kRnxRyYeB8Z342jO/DVtK50PHFw1FElTFKL151z/wH+A9C1a1eN\nx4qA75Y+J0yo1i+GgzpBv79Di1OUUInsBbfbWoGqsUq6MKHiu3FQrS6rDj+Phgtfxu3cHnVkCVUW\nidUyoEnc5cbhdSJSmFg2zHkpTKgWwUEdYfBoX0ulhEpkrwW7jVgpsUqaPAkVfW6B7pfz8/xFNFz4\nMmQrsSrKm8BVZjYaX7S+0Tm3xzSgiIRi2fD1y74X1fpFcGAHuOBFaNlPCZVIGXLsGrHSWoFJUEBC\nldNjbydh8+KszAiDTLzitFt4EegD1DezpcAdQAaAc+5x4B18q4WF+HYLQxMVrEiFlr0DvnrR96Ja\nvwgObO/X82vZXwmVSALEj1hprcAEWjYTJj9YYEKVYwfV/A+pPmLlnCu0wsz5Sezfl1lEIpXNzq2+\nbcKUR2DzCl+UfvIoaHWaEiqRBIrvvG6aCixbzsHiyTD5H/DjJ1C9LvS+GY6+osBVIHamZQBg2Sk+\nYiUipbR9PUx/Cqb+G7avg+Y94ax/w6HHK6ESSYbd+ljlJFY6b2qvBAEseM8nVEu/gFoN4aS74cih\nUL1Oob+6M6hCzBlpSqxEpES2rILPH4Uvnoadm30x+nHXQVP1zRVJpsDF11hpxGqvxLJh7lg/5bfq\nW6jXFE77B3T6NWRUL9Yusp1jO9WwVJ8KFJFiWr8YpoyAL5+D2E5oO9Avjnxg+6gjE0lJuTVWmIrX\nSytvbWiDVjDwP9BuEKRnlGhXWTFHJlVJi2nESkQKs2ymr5+a+4ZvQtjxAp9Q7X9Y1JGJpDQXX2OV\npgahJbJtHcz4L0z/D2xZCQd3gZP/7E+2SSty0ZZ8ZYeJlWqsRGRPQQAL3ocp/4KfPoNqdeCYq/2Z\nMHUOjjo6EaGgtQK1pE2h1i3yi75/+RxkbfMrP/R4Ag7ts9e1odlBwA6XQXpsR5mEWl4psRIpiaxM\n39Tz8xGw5nuo0xhO+Qt0vqjIwk0RSa74ESstaVOEpTP8yPu8N/1j1eE86PF7OKBtmR0iZyqwuqYC\nRYRt63wx+vQnYOtqXzc16Cloe1aJ6wxEJDlyzv9LU+f1/AUBfD/eJ1Q/f+5bJhx7LXT7HdQ5qMwP\nlx0LyKQqNTViJZLCVs2DaY/DVy/5pnaH9/VTfof0VssEkXJut7MC1SB0lx2b/YLv0x6HtQuhblM4\n9W/Q+ddQrXbCDpsdODJdVaoEGrESSS1BzNdPTX0MFk2E9GrQ4Vw4+soyHRYXkcTa1XmdXQXXqZxY\nrfsRpj8JXz4POzZBoyPhnP9B6zMgPfHpQFYsYDtVqRLbmvBjRUmJlUiOzI3w5Sg/3bd+MdQ+GE4c\nDl2GQK39o45OREpoV41V/FRgijUIdQ5+/BimPQHfvwdp6b4VTPfLoXHXpIaSc1ZgerAuqcdNNiVW\nImsW+mRq9guwcws06Q4n3gGtT1f9lEgF5uLOCky5PlY7t4bTfU/AmvlQqwH0vtF3SE9A/VRxZAUB\nO6hKlUA1ViKVTyzbT/fNeBoWToC0DGh3NnT/HTTqEnV0IlIGctcKhNQ5K3DtD77/1JfP+VH4gzrB\nwCf8KFWVapGGlh3zNVYZSqxEKpFNK/wbzsyRsGkZ1D7Ir8R+5FCofUDU0YlIGYrvY2WVecQqlgXz\n3/EJ1Y+fQFoVP+Le/Qpo0q3cnGiTc1ZghlNiJVKxBYEvQp/xX/jubd8g8NDjod990KJfUoo2RST5\ndutjVRk7r29YArOegVnP+u7odZvACbdB54vL5RfFrMCRSYZGrEQqrG3rYPYomPE/WPcD1NgPelzp\nR6e03IxIpefizwrMGbUJKnjn9SDmyxdm/NeXMzgHLU6BrsN8O5icBLIcyo4FbHfVSCfmR9kqaQ2r\nEiupXJyDpV/4Zp7fjoHYDmhyNPS+CdqcWexV2EWk4tvVx8qwil5jtXllWMbwDGz8GfY5AHr+Ebpc\nDPWaRh1dseScFQhA1nYlViLl2tY1/gyYL5+H1fOg6j6+2V3XYXBgu6ijE5EIxHdedxWx83osG374\nyCdU89+BINs3Jz75Hmh1WoVLTPxUYJhYZWcClXMZMCVWUnHlvuk8C/PfhSALGnWFAQ9B+3MS2kFY\nRMq/IK7GyipSH6u1P/gviV+9CJtXQM36vu/UkUOh/uFRR1dq2bGAHYTJYNb2aINJICVWUvGs/cHX\nTs1+Ie5N53d+hKph66ijE5FyIr7zupX3EaudW2HuGz6h+ukz39D0iJOh//1wxClQpWrUEe61rLDd\nAhCOWFVOSqykYti5Le5N51P/pnP4SdDv79Di1ErxpiMiZSyuxsqFS9o4F6N8NB9gV03ol8/BN6/7\nBsX7HeYbFHccHFkjz0TJDoLda6wqKSVWUn45Bz9PhTmj/ZvOjk2w36F+mZmOF1a6Nx0RKVu5I1bs\nGrFyQRB9YrVpBXz9sl9Ca818yKjlG3h2/jU0Pbrc9J0qa9kxRwyNWIkk35qFPpma8xJs+Bkyavoz\n+jpfBM2OqbRvOiJStuLPCoy88/qOzTBvnH9v+3Ei4PzyWWc84pOqFKgJzYoFxHKmArO2RRtMAimx\nkvJh6xr45jWfTC2b6af6Du0Dx9/mz36ptk/UEYpIBZPfWoGBC0hLVgCxbL8A8lejfXPi7O1Qrxn0\nugE6nF+hC9FLIztw7MidCtSIlUjZy9ruTyH+6iXf8M7F4MD2cPK9/qy+2gdGHaGIVGCB2zUXaGHj\nTJfoBqHOwfIv/ZfEb16Drauhej3oNNgnU026p+yoe86SNv6CaqxEykYs2xefz3nFF6Pv3Ax1GsEx\nV/s3nQPaRB2hiFQSu0asiOu8nqCpwPWL4etX/BfFtQsgvao/sabD+XDESZEvgFweZO3WIFQjViKl\nFwTw8+fw7es+mdq6GqrW9nVTHc6D5seV62UYRKRicuzZed2VZWK1cSl8O9a/ty2b6a9rdiwcc5V/\nf6uxb9kdqxLIDgJ2OI1YiZSOc7B0hn/D+XYsbF4OVWr4Na3aDfL9WTJqRB2liFRiu/ex8iNWzu3l\nVODmX/wXxG9ehyVT/XUHdoC+d0LbQbBvs73bfyWWrRErkRJyDlZ8FSZTY/wZfelVfb+pdvf4YXEV\noYtIkux2VmBa+HFXms7rW9f4ZOrbMbD4U8BBw7b+5Jp2g7SoezFlBQHbVWMlUgTn4JevYd6b/hvc\nuh/8G9ihx0OfW6FVf6heN+ooRSQFubgRK19oVYLi9W3r4Ltx/n1t0SR/ck39Fn5B97YDoWGrxARd\niWXHHNlUIdulUUUNQkXiBAEsm+GTqXlv+aJNS4NDesGx10Lr06HmflFHKSIpzuWsFUgxa6w2Lfdt\nEea9CYs/88nUvofAcf/np/kOaJuyZ/SVhayYfz4yqco+mgqUlBfL8utXzXvLN7nb8gukZcBhx0PP\nP0LL/lCrftRRiojkCuLOCrS0AtYKXPuDH5ma95ZfXgagfks47g/+S+JBHZVMlZHsMKnNpCq1srZH\n3wE/QZRYScGyMn1zu3lv+X5T29f7LuiH94XWZ0CLkzXNJyLl1m4NQokrXv/lm/BL4luw6lu/0UGd\n4ITbfTLVoGU0AVdy2XEjVk6JlaSMbetgwQc+kVo4wS8KWq0utOzn33AOOwGq1ow6ShGRIuUUr/sV\nbfxUYI0pD8CkPwPml8g65a/QegDUaxphpKkhK+ZHrHa4DIKs7cnrgJ9kSqwE1iyA+eP9vyVT/VD5\nPgdAu7OhzRnQvBdUqRp1lCIiJZJbY2WGS8tgSqwNXZsfSNX2Z/ryhX0aRhxhaskOHNUz0thONZyK\n16VSiWX7BConmVr3g7/+gHa+XqpFPzi4M6RV1u8TIpIKHLknA2KWxoVZtzF90Ik0rFM90rhSURA4\nYoGjTvUqZGZXVR8rqQS2b4AfPvKJ1IL3IXODLz4/pCccfYVv3KmhcBGpRALnchuDpuUuwhxlRKkr\nKyxcr5GRTmZWBi5rW8QRJY4Sq8oqCOCXr3yd1IIJ/mwXF4Ma+/l6qRan+nqp6nWijlREJCECt2vE\nKuf/OcvcSHLlFK5Xr5pO5raqoKlAqRC2rvVn8S34AH740K/JB/5sl+P+4M/ma9JN6/KJSEpwbtdS\nNtzD8/EAABALSURBVLlrMCuvikROYlUjI90va5O9OeKIEkeJVUUWxGDZLD8qtXBCuAio86NSh53g\nV1Q/7AQVaIpISnLO5Z7Sn5NgBcqsIhE/FbiDqli2aqykvNi4DBZN9InUDx/53lIYNO4KfW72o1IH\nd9aolIikvMC53NqqNDX5jFTuiFXVdDKdEivM7FTgYSAdeMo597c8tw8B7geWhVeNcM49VYZxpq7t\nG/yinz9+4hOqNd/762s18HVSh/f1o1JaQkZEZDcunxqroDSLMMtey+lhVbNqOttTfcTK/AJLjwIn\nAUuBL8zsTefc3DybvuScuyoBMaaWrExYMs0nUT9+Asu/9H2lMmr6ZnZdLoZDevvWCGqHICJSoEA1\nVuVGdrB7jVVaKidWQDdgoXPuRwAzGw2cCeRNrKQ0ghis+GpXIvXzVMjOBEv303u9bvCJVOOj1KRT\nRKQEfLsF/3POVKDTiFUkssMRq9ypQJft16BNz9i7HTvH9/PnUK9GNRo2a1UGke694iRWjYAlcZeX\nAt3z2e5sM+sFfA/8wTm3JJ9tJIjBym/8yuk/fean+TI3+NsatoGuw3wi1ewYtUIQEdlLabkjVupj\nFaWsnHYLOWcFgm+5UJrEav1PsHgyLJoMiz+lxaalTN//TBpe/WwZRlx6ZVW8/hbwonNuh5n9DngG\nOCHvRmZ2GXAZQNOmKdKMMpbtR6R++tQnUz9PhR0b/W37NvdrVB3SBw75//buN0auq7zj+O+Z2fXG\niU1C7GDijf9ExCLYgExYkpgEBAiBkxcsSPxJQOBWqdIXMX/UvEn6gkJe0aotUlWK6oIJRLQh/Inw\nC4sUUSSQCE5sCE3sKK3rkNhL4vwxJITsemfmPn1x7p25u5mNN56794zv/X4ka2fujNeP5/h6Hz3n\nnOe8U1q5JmakAFAp+YpVtnSdilUc7dyuwOezxKo9I+kUBQR36cQR6bFfSI/fGxKqPzweXjt7lbTx\nat320Pu0eu12Xb504b8ii0mspiStyz2/SL1F6pIkd3829/Rrkv6u3zdy912SdknSxMRENf91t2fD\nuqgskTq6LxxkLEmrLpG2fFDaeLW04Srp3PG4sQJAhfXbFVjNHzzDr5XrY/W00ipVvyahSSI9dShN\npH4Rvr5wPLy2/Pwwm7Ntp7TxHdIFl8rN9I1f79WnV24s5y+yCItJrO6XtMnMLlZIqK6T9PH8G8zs\nQnd/In36AUkPFxrlMDv5QugfdXRfmNY7ep/UTv+xXHCp9OaPSRuvConUytfGjRUAaoRdgcMjv8Zq\n2sfSizNhndXvHghLYx6/N/yaSWd1XjUeZnM2vF1a/3bpgtf3diGkTrY6cg8d3YfFKRMrd2+b2U5J\n9yi0W9jt7gfN7DZJ+919j6TPmNkHJLUlnZD0Z0sYc1zPHQvTeUfvCwcZP/lQOCpGJq3ZEnbtZYnU\nOatjRwsAtRXWU83bFZhEC6fWsl2Bc9ZYff8vpGcPS9m5gas2SZsnw8/P9dvC+bWn6D820+pICpWw\nYbGoNVbuvlfS3nnXPp97fKukW4sNbQh02mGh+dF9vWTq+WPhtdGzpfG3Su/4K2ndlWEH3/Lz4sYL\nAOhy926lyrpTgVSsYsj6WC0fbepRf63azeUhAbnsU2lFattpnRIyfaYmVrXx4ol0Wi+tRh07ILX+\nFF5buVZaf4W07tPh65o3Dr5NFACwZMJU4Lw1VuRVUeQ7r/+fj+vu7ffpI28bfBPb9Gyn+32HRX0T\nq9aM9OSDIZGa2i8d2y/9/tHwmjVC4vSWT0jrrgi/zlv38t8PADBU+u0KZI1VHPldgZLULmgYsorV\nWVSsSpYkYbtmlkBNHQhJVdIKr69cK41fJr11hzQ+Ia3dKo2tjBszAGAgSb5i1ehdQ/nyfayk3mL2\nQZ2xa6zOSM9NSQduD8nU1IHeLoNlK8IhxdtuCmukLpqQXrU2aqgAgOK5chUrOq9H1a1YpVN2WaI1\nqOnZud93GFQ3sWq9KP3876XXbJG2fCgkUeMTYbtmY3gGAACwNNzVZyowWji1lu9jJfUSrUGxeL1M\nqy6Rbj0mLTsndiQAgAj6NgilYhVFe15iVVjFagjXWDViB7BkzEiqAKDG+u4KjBlQjWUVqrOWhbSj\nXVBiNTOEuwKrm1gBAGotce9OAfYahJJaxZBVqMaaTZlVeyqQxAoAUElz1lh1j7SJF0+dZbsAR5qm\n0Uaj8KnAs6lYAQCwtFz9DmEms4ohO9JmpGkaaVph7RayBqFjI8OTzgxPJAAAFChJXrorkLXrcWRH\n2ow2GhppWDfRGtRMq6Plo81uO41hQGIFAKikObsC00MD6bweR7sTzm1sNEyjzUY30RrUdKszVAvX\nJRIrAEBFuXqNQbPDmMmr4mgliUaaIeUIU4FFNQjtDNXCdYnECgBQUZ7bFZhNBlKxiqPdcY2m2e1I\no6FWgbsCzxodrlRmuKIBAKAgiffOCKRiFVe706tYjRZYsZphKhAAgHJ4v87r7AqMopW4RptpxarZ\nKLSPFVOBAACUIHH1aRAaLZxaa3cSjaTlw5GGFXgIc2eojrORSKwAABWVuOcWr7PGKqZ2xzWSVqxG\nm43i+li1EipWAACUpTGv8zppVRxhKjC3K7CgPlbTs23WWAEAUIZ8HytLJwWdilUUYSowrVg1Cu5j\nRcUKAICll++8nu0O5KzAOFodX7I+VqyxAgCgBK6XrrGiYBVHO0nm7ApsFXakTcJUIAAAZUg8t8aq\ne43MKoZ2x3NTgcUcwtzuJJrtsHgdAIBShM7r6RordgVG1eoUf6TNTDskZyRWAACUwPt0Xkcc7XkN\nQos40mZ6tiNJOoupQAAAlt6cXYFUrKLKNwgNU4EFVKxaIbGiYgUAQAny66MbdF6PqtWZd6RNAWus\npkmsAAAoj0t9zgpEDO0kV7FqWiG7ArOpwOXLhiuVGa5oAAAoSDiEee41pgLjyB9pM9IotmJFHysA\nAEow56zABp3XY2olydwjbQpYY8VUIAAAJfJcH6vsK3lVHHP6WBW0K3CmOxVIYgUAwJJLvLcbMOtn\nxZE2ccw50qagXYFUrAAAKFFoEBp0dwVSsopi/pE27cQHnpYlsQIAoERhKnBuHyvSqjjCVGCvj5UU\nmoYOggahAACUKHHvdl637horUqsYWp25FStJA08H0iAUAIASJbmzArPKVcIiqyjaSa/dQpZgDbqA\nfbrV0UjDursNh8VwRQMAQEFcvUpVd1dgtGjqy93VSXpTgdnuwEErVtOzydBVqyQSKwBARc1ZY8Wu\nwGhaaQL10qnAwStWw7a+SiKxAgBUVGgQGh5b+tOONVbla6dTfllC1ZsKHHyNFRUrAABKkq9Ydc8K\nJK8qXVaxyqYAsynBgStWsyRWAACUJl+xoo9VPFkClT/SRuolXKfrjJ4KNLPtZvaImR02s1v6vD5m\nZt9JX99nZhuLDhQAgFfCvbe2ijVW8WT9qrLEalm2xqqAXYHLR4evPnTKiMysKekrkq6RtFnS9Wa2\ned7bbpD0e3e/RNKXJf1t0YECAPBKuHu3UtXtY8W+wNK1Otkaq+L7WA3jVODIIt5zuaTD7n5Ekszs\nTkmTkg7l3jMp6Qvp4+9J+mczM4+4SvDI0y/o5u/+JtYfDwCI7OkXTuamAsODO+59TD8+dDxiVPVz\nspVNBWaJVfh6812/0dljp58Y/c/xP+o9l75m8AALtpjEalzS0dzzY5KuWOg97t42s+ckrZL0TP5N\nZnajpBslaf369acZ8uI0G6YVY4v56wEAqmjb61br2jddKCn8UN+xbYOOPPOnyFHVz4ox6b1vWKOJ\nDedLkt40fq7ev2WNXkyPpDldb9t4via3jhcRYqFKzTzcfZekXZI0MTGxpNWsDavO0R03zM//AAB1\nZGb64uQbY4cBSatXjOlfPzkRO4wls5hVX1OS1uWeX5Re6/seMxuRdK6kZ4sIEAAA4EyxmMTqfkmb\nzOxiM1sm6TpJe+a9Z4+kHenjD0v6r5jrqwAAAGI45VRgumZqp6R7JDUl7Xb3g2Z2m6T97r5H0tcl\n3WFmhyWdUEi+AAAAamVRa6zcfa+kvfOufT73eEbSR4oNDQAA4MwyfJ21AAAAzlAkVgAAAAUhsQIA\nACgIiRUAAEBBSKwAAAAKQmIFAABQEBIrAACAgpBYAQAAFITECgAAoCAW60g/M3ta0mMl/FGrJT1T\nwp+D/vj842MM4mMM4mMM4jvTx2CDu19wqjdFS6zKYmb73X0idhx1xecfH2MQH2MQH2MQX13GgKlA\nAACAgpBYAQAAFKQOidWu2AHUHJ9/fIxBfIxBfIxBfLUYg8qvsQIAAChLHSpWAAAApahsYmVm283s\nETM7bGa3xI6nLszst2b2oJk9YGb702vnm9mPzex/06+vjh1nlZjZbjN7ysweyl3r+5lb8E/pffHf\nZnZZvMirY4Ex+IKZTaX3wgNmdm3utVvTMXjEzN4fJ+rqMLN1ZvZTMztkZgfN7LPpde6DkrzMGNTu\nPqhkYmVmTUlfkXSNpM2SrjezzXGjqpV3u/vW3LbaWyT9xN03SfpJ+hzFuV3S9nnXFvrMr5G0Kf11\no6SvlhRj1d2ul46BJH05vRe2uvteSUr/L7pO0pb09/xL+n8WTl9b0s3uvlnSlZJuSj9n7oPyLDQG\nUs3ug0omVpIul3TY3Y+4+6ykOyVNRo6pziYlfTN9/E1JH4wYS+W4+88knZh3eaHPfFLStzz4paTz\nzOzCciKtrgXGYCGTku5095Pu/qikwwr/Z+E0ufsT7v6r9PEfJT0saVzcB6V5mTFYSGXvg6omVuOS\njuaeH9PLDzCK45L+08wOmNmN6bU17v5E+vhJSWvihFYrC33m3Bvl2plONe3OTYEzBkvIzDZKeouk\nfeI+iGLeGEg1uw+qmlghnqvd/TKFUvtNZvbO/IsetqGyFbVEfObRfFXS6yRtlfSEpH+IG071mdkK\nSd+X9Dl3fz7/GvdBOfqMQe3ug6omVlOS1uWeX5RewxJz96n061OS7lYo7R7Pyuzp16fiRVgbC33m\n3Bslcffj7t5x90TSv6k3zcEYLAEzG1X4gf5td/9Bepn7oET9xqCO90FVE6v7JW0ys4vNbJnCArk9\nkWOqPDM7x8xWZo8lvU/SQwqf/Y70bTsk/TBOhLWy0Ge+R9Kn0l1RV0p6LjdVggLNW7PzIYV7QQpj\ncJ2ZjZnZxQoLqO8rO74qMTOT9HVJD7v7P+Ze4j4oyUJjUMf7YCR2AEvB3dtmtlPSPZKakna7+8HI\nYdXBGkl3h/tLI5L+3d1/ZGb3S7rLzG6Q9Jikj0aMsXLM7D8kvUvSajM7JulvJH1J/T/zvZKuVVgo\n+qKkPy894ApaYAzeZWZbFaaffivpLyXJ3Q+a2V2SDinspLrJ3Tsx4q6QqyR9UtKDZvZAeu2vxX1Q\npoXG4Pq63Qd0XgcAAChIVacCAQAASkdiBQAAUBASKwAAgIKQWAEAABSExAoAAKAgJFYAAAAFIbEC\nAAAoCIkVAABAQf4fGQGzIM+HwRoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f2bd48a3590>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot this, we see it assigns a reward to previous actions\n",
    "x = np.array([0] * 200 + [3.0] + [0] * 50 + [1.0] + [0] * 10 + [ 0.5])\n",
    "y = discount_rewards(x)\n",
    "plt.plot(x, label='reward')\n",
    "plt.plot(y, label='discounted reward')\n",
    "plt.title('When we get a reward we also reward previous actions a bit')\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-13T11:16:24.759248Z",
     "start_time": "2017-07-13T19:16:24.755556+08:00"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def prepro(I):\n",
    "    \"\"\" prepro 210x160x3 uint8 frame into 6400 (80x80) 1D float vector \"\"\"\n",
    "    I = I[35:195] # crop\n",
    "    I = I[::2,::2,0] # downsample by factor of 2\n",
    "    I[I == 144] = 0 # erase background (background type 1)\n",
    "    I[I == 109] = 0 # erase background (background type 2)\n",
    "    I[I != 0] = 1 # everything else (paddles, ball) just set to 1\n",
    "    return I.astype(np.float).ravel() # make it 1D"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-13T11:16:26.393597Z",
     "start_time": "2017-07-13T19:16:24.910815+08:00"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-07-13 19:16:24,920] Making new env: Pong-v0\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWcAAAF1CAYAAAA0pisdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X20HVWZ5/HvjxCTCGgSI9eQRIIYGcHWCGnE8S2Kyksz\nBmbNYDIuCS8amIZRerGWTdBRum260falYWyxgyBBJRBBIDIgYhpkuZwgATEGAhIgMYk3CQkIKIE2\nN8/8UftCcbgn99zzdqvq/j5r1bpVu172rsPluTvP3lVHEYGZmRXLHsPdADMzezkHZzOzAnJwNjMr\nIAdnM7MCcnA2MysgB2czswJycB5Gkm6RNL/OvumSQtKe3W6XDY2k8yV9b7jbYdXi//GHUUQcM9xt\nsOEl6QpgY0R8brjbYsXinrNVhv+VYVXi4NxBkv5W0rU1ZRdJujit3yHpE2l9lKSvSNom6VHgr2rO\ne7WkyyT1Stok6R8kjUr79pD0OUnrJW2VdKWkV++mXZ9J1/m9pE+k9Mkb076/kvQrSU9L2iDp/Nx5\n/amWU9K+JyWdIekvJa2S9AdJ36ip61RJa9Kxt0rav06bxkr6nqTt6Tp3S+pJ+yZK+k5q75OSbkjl\nsyVtTJ/zZuA7qfw4Sfel6/xC0ltz9ewn6TpJj0t6TNKncvvOl7Q0fX7PSLpf0qya/56b0r6HJB2Z\nu4VX7Oa8N6f/1n9I+z6SyhcAHwM+I+mPkn5U77+ZjUAR4aVDC7A/8CywT9oeBfQCR6TtO4BPpPUz\ngAeBacBE4HYggD3T/uuBfwP2AvYFfgmcnvadCqwF3gDsDfwQ+G6dNh0NbAYOAV4JfC/V88a0fzbw\nF2R/uN8KbAGOT/ump2O/BYwFPgw8B9yQ2jQF2Aq8Lx0/J7XrzWQptM8Bv6jTrtOBH6U2jQIOA16V\n9v1f4BpgAjA6d/3ZwE7gS8AYYBzw9tSGd6TrzAfWpf17APcAnwdekT6vR4Gj0vXOT/dzbDr3n4AV\nad9BwAZgv9xncWAD541On8F5qc4PAM8AB6X9VwD/MNy/q16Ktwx7A6q+AD8HTkrrHwIeye27gxeD\n878DZ+T2fTgFwj2BHuB5YFxu/zzg9rS+HPjr3L6DgD+TAntNey4H/im3/UZywXmA4/8F+Hpan56O\nnZLbvx34aG77OuDstH4LcFpu3x5kf6z2H6CeU4FfAG+tKZ8M7AImDHDObOA/gLG5skuAL9Yc9xDw\nvhSwf1ezbyHwnbR+PvDT3L6DgR25z2kr8EFgdM01dnfee8j+GO6R278EOD+tOzh7GXBxWqPzriIL\npAD/I20PZD+ynlm/9bn1/cl6YL3pn8Z/IOtF75s7d33Nuf1BfbB68utIeoek29M/+58i69FPqrnG\nltz6jgG29861+6Jcm58ARNbDrvVd4Fbg6pS++LKk0WT/kngiIp4c4ByAxyPiudz2/sA5/XWmeqel\n+94f2K9m33m89HPanFt/Fhgrac+IWAucTRaIt0q6WtJ+g52X6t0QEbty+9fX+QzMXuDg3Hk/AGZL\nmgqcQP3g3EsWRPq9Pre+gaznPCkixqflVRFxSNr/e7LAkz93Jy8Nmvl6pua2p9XsvwpYBkyLiFeT\npTBU7+YGsYEs9TI+t4yLiF/UHhgRf46Iv4uIg4H/DBwHnJSuMVHS+Dp11L5WcQNwQU2dr4yIJWnf\nYzX79omIYxu5mYi4KiLeTfZZB1k6ZTC/B6ZJyv+/9npgU532mwEOzh0XEY+TpS++QxYY1tQ5dCnw\nKUlTJU0Azs1doxf4CfBVSa9KA4AHSnpfOmQJ8DeSDpC0N/CPwDURsbNOPaekQapXAv+7Zv8+ZD3V\n5yQdTtbbb9a3gIWSDoEXBjX/+0AHSnq/pL9Ig5xPk6VldqV7vwX4pqQJkkZLeu9u6rwUOCP9C0CS\n9kqDnPuQ5emfSQN745QNwr5F0l8OdiOSDpL0AUljyPLLO8jSLYO5i6wn/ZnU9tnAfwGuTvu3kOW+\nzV7Cwbk7riLLVdbrNUMWVG4Ffg3cSzaol3cS2YDSA8CTwLVk+VjI8sjfBe4EHiMLHv9roEoi4hbg\nYrIBx7XAirTr+fTzr4G/l/QM2cDZ0kZusE5d15P1Lq+W9DSwGqg3t/t16Z6eBtYAP0v3BPBxsmD9\nIFne9+zd1LkS+CTwDbLPaS1wctrXR9Yjn0n2OW0Dvg3UndmSMwa4MJ2zmSyltHCwkyLiP8iC8THp\n3G+SjUE8mA65DDg4pVluaKAdNkIowv+qGskkvZksaI6p09M2s2HgnvMIJOkESWNS+uRLwI8cmM2K\nxcF5ZDqdLD3wCNAH/M/hbY6Z1epYWkPS0cBFZJPyvx0RF3akIjOzCupIcE4j7r8le+hiI3A3MC8i\nHmh7ZWZmFdSptMbhwNqIeDSNVl9N9iivmZk1oFNv8ZrCS58820j26OwL0ktfFqTNw4Zy8X17xrbU\nODPrrq1bntsWEa/tVn1HvX+v2P5EX9Pn37Pq+Vsj4ug2NmnIhu0VixGxCFgE0PO6cfHR+QcOV1M6\n7pOzDxn8oAZdesf9bbuWdc5zz9/aluuMHXNUW64z3P7Pl+9fP/hR7bPtiT7uunXq4AfWMXryI7Wv\nLOi6TqU1NvHSx4Kn8uLjqmZmNohO9ZzvBmZIOoAsKM+ltceAzcyGIOiLRp6uL66OBOeI2CnpLLLH\nkUcBl0eE/z1uZl0RwK6Sv1OqYznniLgZuLlT1zcz251dDb2Xqrj8hKCZWQH5CzGHUb2ZF+2c3WHF\nUm/2Rbtmd1gmCPpK/lI3B2czqyTnnM3MCiaAPgdnM7PiKXvP2QOCZmYF5J6zmVVOgAcEzcyKqNyz\nnB2czayCgvCAoJlZ4QT0lTs2e0DQzKyI3HM2s8rJXnxUbg7OZlZBog8NdyNa4uBsZpUTwC7nnM3M\nrN3cczazSnJaw8ysYLIXHzk4m5kVzq5wcDYzKxT3nK0l/saTkcffeGKNcnA2s8oJRF/JJ6M5OJtZ\nJTnnbGZWMM45m5kVkuiLcqc1yt16M7OKcs/ZzConeytdufueDs5dcOkd9w93E6zLxo45aribMOI5\n52xmVjARzjmbmVkHuOdsZpW0a6SmNSRNA64Eesjy74si4iJJ5wOfBB5Ph54XETfv7lp/3LmTFVu2\nN9sUM7OXyOY5dy4xsJv4NxG4BpgOrANOjIgnJQm4CDgWeBY4OSLu3V0drfScdwLnRMS9kvYB7pF0\nW9r39Yj4SgvXNjNrQcdzzvXi38nA8oi4UNK5wLnA3wLHADPS8g7gkvSzrqaDc0T0Ar1p/RlJa4Ap\nzV7PzKxdOj2Vbjfxbw4wOx22GLiDLDjPAa6MiABWSBovaXK6zoDa0npJ04G3A3elorMkrZJ0uaQJ\ndc5ZIGmlpJU7nyv79+Sa2UhVE/96cgF3M1naA7LAvSF32kYG6cy2HJwl7Q1cB5wdEU+TddcPBGaS\n/WX56kDnRcSiiJgVEbP2HOtJI2bWXn2hphdgUn/nMS0LBqpjgPj3gtRLbvprZluarSFpdGrY9yPi\nh6lBW3L7LwVuaqUOM7OhasMrQ7dFxKzdHTBQ/AO29KcrJE0GtqbyTcC03OlTU1ldTbc+jT5eBqyJ\niK/lyifnDjsBWN1sHWZmzdoVezS9DKZe/AOWAfPT+nzgxlz5ScocATy1u3wztNZzfhfwceA3ku5L\nZecB8yTNJOvOrwNOb6EOM7Mh6/RUOurHvwuBpZJOA9YDJ6Z9N5NNo1tLNpXulMEqaGW2xs9hwFne\nu53TbGZWdruJfwBHDnB8AGcOpQ4/IWhmlRO8MLBXWg7OZlZJfmWomVnBROC30pmZWfu552xmFaSR\n+1Y6M7OiCsqf1nBwNrNK6vA8544rRHDee889OaLnNcPdDDPrkLvZ3NX6ArGr5FPpyv2nxcysogrR\nczYzazenNczMCiagoRcYFZmDs5lVkOjzVDozs2KpQs+53K03M6so95zNrJKc1jAzK5gIlT6t4eBs\nZpVU9se3y916M7OKcs/ZzConwG+lMzMrHpU+reHgbGaVk81zds/ZzKxwyv5ujXK33sysotxzNrPK\nqcL7nAsXnFds2f7Cul/Ab2bN2lXyxEDhgrOZWasioM89ZzOz4il7WqPc/X4zs4pyz9nMKicbECx3\n39PB2cwqacS/MlTSOuAZoA/YGRGzJE0ErgGmA+uAEyPiyUau5xkaZtaqKjwh2K5+//sjYmZEzErb\n5wLLI2IGsDxtm5lZgzqVlJkDLE7ri4HjO1SPmdkAspxzs0sRtKMVAfxE0j2SFqSynojoTeubgZ7a\nkyQtkLRS0sodO/ra0AwzsxftQk0vRdCOAcF3R8QmSfsCt0l6ML8zIkJS1J4UEYuARQA9rxv3sv1m\nZs3yQyhARGxKP7dKuh44HNgiaXJE9EqaDGxttR4zs6EoSnqiWS21XtJekvbpXwc+DKwGlgHz02Hz\ngRtbqcfMbKRptefcA1wvqf9aV0XEjyXdDSyVdBqwHjixxXrMzBo24t9KFxGPAm8boHw7cGQr1zYz\na0VRBvaa5ScEzaxy/BCKmZl1hHvOZlZJZZ+t4eBsZtUTI3xA0MysiAIPCJqZFVLZe87lTsqYmVWU\ne85mVjlVmErn4GxmleTgbGZWMCP+8W0zs6Iq+2wNDwiamRWQe85mVj3hnLOZWeF4toaZWUGVPTg7\n52xmVkDuOZtZ5XgqnZlZQYWDs5lZ8ZR9nrODs5lVTlRgKp0HBM3MCsg9ZzOrJOeczcwKx7M1zMwK\nqew9Z+eczaxy+h/fbnYZjKTLJW2VtDpXdr6kTZLuS8uxuX0LJa2V9JCkoxq5BwdnM7OhuwI4eoDy\nr0fEzLTcDCDpYGAucEg655uSRg1WgYOzmVVPZNPpml0GvXzEncATDbZmDnB1RDwfEY8Ba4HDBzvJ\nwdnMKmkXanoBJklamVsWNFjtWZJWpbTHhFQ2BdiQO2ZjKtstDwiaWeUELQ8IbouIWUM85xLgi6n6\nLwJfBU5ttgFNB2dJBwHX5IreAHweGA98Eng8lZ/Xn3sxM6uqiNjSvy7pUuCmtLkJmJY7dGoq262m\n0xoR8VB/4hs4DHgWuD7tfllS3Myse5qfqdHs/GhJk3ObJwD9MzmWAXMljZF0ADAD+OVg12tXWuNI\n4JGIWC+Ve26hmVVDIwN7zZK0BJhNlpveCHwBmC1pJllaYx1wetaOuF/SUuABYCdwZkT0DVZHu4Lz\nXGBJbvssSScBK4FzIuLJNtVjZtaQTj6EEhHzBii+bDfHXwBcMJQ6Wp6tIekVwEeAH6SiS4ADgZlA\nL1lSfKDzFvSPhO7YMegfETOzhmVT4tT0UgTtmEp3DHBvfzI8IrZERF9E7AIupc58vohYFBGzImLW\nuHGDzsc2MxtR2pHWmEcupSFpckT0ps18UtzMrGtG9IuPJO0FfIiU+E6+PFBS3Mysmzo5INgNLQXn\niPgT8Jqaso+31CIzszYoSu64WX5C0MwqJyjOwF6z/G4NM7MCcs/ZzCqp5ClnB2czq6BwztnMrJhK\n3nV2ztnMrIDcczazSnJaw8ysgEb0QyhmZkXUhm9CGXYOzmZWPQGUPDh7QNDMrIDcczazSnLO2cys\niByczcyKpvwvPnJwNrNqKnnP2QOCZmYF5J6zmVWPX3xkZlZQJU9rODibWUWVu+fsnLOZWQG552xm\n1eS0hplZATk4m5kVTAVefOTgbGaVVPZ3a3hA0MysgNxzNrNqKnnP2cHZbIR47vlbBywfO+aoLrek\nS5xzNjMrHrnnbGZWMEHp0xoeEDQzK6CGgrOkyyVtlbQ6VzZR0m2SHk4/J6RySbpY0lpJqyQd2qnG\nm5kNTFnOudmlABrtOV8BHF1Tdi6wPCJmAMvTNsAxwIy0LAAuab2ZZmZDFC0sBdBQcI6IO4Enaorn\nAIvT+mLg+Fz5lZFZAYyXNLkdjTUza9hICM519EREb1rfDPSk9SnAhtxxG1OZmZk1qC2zNSIipKFN\nXJG0gCztwT6vGt2OZpiZvaggPeBmtdJz3tKfrkg/t6byTcC03HFTU9lLRMSiiJgVEbPGjRvVQjPM\nzGr0v/hoBAwIDmQZMD+tzwduzJWflGZtHAE8lUt/mJl1haL5pQgaSmtIWgLMBiZJ2gh8AbgQWCrp\nNGA9cGI6/GbgWGAt8CxwSpvbbGY2uIIE2WY1FJwjYl6dXUcOcGwAZ7bSKDOzkc5PCJqZFZDfrWFm\nlVSU3HGzHJzNrJoKMuuiWQ7OZlY9BXrSr1nOOZuZFZB7zmZWTSXvOTs4m1kleUDQzKyISh6cnXM2\nMysg95zNrJpK3nN2cDazyinSC4ya5eBsZtXkh1DMrAzGjjlquJvQXSXvOXtA0MysgNxzNrNKcs7Z\nzKyIHJzNzAqmArM1nHM2MysgB2czq6ZoYRmEpMslbZW0Olc2UdJtkh5OPyekckm6WNJaSaskHdpI\n8x2czayaOhicgSuAo2vKzgWWR8QMYHnaBjgGmJGWBcAljVTg4GxmldT/lGAzy2Ai4k7giZriOcDi\ntL4YOD5XfmVkVgDjJU0erA4HZzOz9uiJiN60vhnoSetTgA254zamst3ybA0zs5ebJGllbntRRCxq\n9OSICKm1+SIOzmZWTa1NpdsWEbOGeM4WSZMjojelLbam8k3AtNxxU1PZbjmtYWbV00K+uYX+7jJg\nflqfD9yYKz8pzdo4Angql/6oyz1nM6umDj6EImkJMJss/bER+AJwIbBU0mnAeuDEdPjNwLHAWuBZ\n4JRG6nBwNrNq6mBwjoh5dXYdOcCxAZw51Dqc1jAzKyD3nM2sckT5363h4Gxm1VTy4DxoWqPOM+T/\nLOnB9Jz49ZLGp/LpknZIui8t3+pk483MBjQ8szXaqpGc8xW8/Bny24C3RMRbgd8CC3P7HomImWk5\noz3NNDMbWQYNzgM9Qx4RP4mInWlzBdmkajOz4ujsi486rh2zNU4FbsltHyDpV5J+Juk9bbi+mdnQ\nlTw4tzQgKOmzwE7g+6moF3h9RGyXdBhwg6RDIuLpAc5dQPb6PPZ51ehWmmFm9jJFyR03q+mes6ST\ngeOAj6VJ1kTE8xGxPa3fAzwCvGmg8yNiUUTMiohZ48aNarYZZmYDK3nPuangLOlo4DPARyLi2Vz5\nayWNSutvIHu59KPtaKiZ2UgyaFqjzjPkC4ExwG2SAFakmRnvBf5e0p+BXcAZEVH7Qmozs84qUA+4\nWYMG5zrPkF9W59jrgOtabZSZWavKnnP2E4JmVk0OzmZmxVP2nrPfSmdmVkDuOZtZNZW85+zgbGbV\nMxJma5iZlY3SUmbOOZuZFZB7zmZWTU5rmJkVT9mn0jk4m1k1OTibmRVQyYOzBwTNzArIPWczq54C\nfVFrsxyczayaHJzNzIrHPWczsyIqeXD2gKCZWQG552xmleS0hplZ0fitdGZmBVXy4Oycs5lZAbnn\nbGaVI5xzNjMrJgdnM7PiUZQ7Ojs4m1n1VGC2hgcEzcwKyD1nM6skDwiamRWRg7OZWfG452xmVkQl\nD86DDghKulzSVkmrc2XnS9ok6b60HJvbt1DSWkkPSTqqUw03M6uyRnrOVwDfAK6sKf96RHwlXyDp\nYGAucAiwH/BTSW+KiL42tNXMrDEV+JqqQXvOEXEn8ESD15sDXB0Rz0fEY8Ba4PAW2mdm1pxoYSmA\nVuY5nyVpVUp7TEhlU4ANuWM2prKXkbRA0kpJK3fscMfazNqn/90azS5F0GxwvgQ4EJgJ9AJfHeoF\nImJRRMyKiFnjxo1qshlmZtXU1GyNiNjSvy7pUuCmtLkJmJY7dGoqMzPrrpK/W6OpnrOkybnNE4D+\nmRzLgLmSxkg6AJgB/LK1JpqZDV3Z0xqD9pwlLQFmA5MkbQS+AMyWNJMsdb4OOB0gIu6XtBR4ANgJ\nnOmZGmbWdQUa2GvWoME5IuYNUHzZbo6/ALiglUaZmbVKu4a7Ba3xW+nMzArIj2+bWTVVPa1hZlZG\nRRnYa5aDs5lVT1D6qXQOzmZWSWXvOXtA0MysgNxzNrNqKnnP2cHZzCqn/8VHZebgbGbVE1H6AUHn\nnM3MCsg9ZzOrJKc1zMyKyMHZzKx4Ot1zlrQOeAboA3ZGxCxJE4FrgOlkb+w8MSKebOb6zjmbWfUE\nsCuaXxr3/oiYGRGz0va5wPKImAEsT9tNcXA2M2ufOcDitL4YOL7ZCzk4m1k1tfbt25P6v4A6LQvq\n1PATSffk9vdERG9a3wz0NNt855zNrJJazDlvy6Uq6nl3RGyStC9wm6QH8zsjIqTmW+Ges5lVU/+D\nKM0sDV0+NqWfW4HrgcOBLf3fsZp+bm22+Q7OZlZJnfyCV0l7Sdqnfx34MNkXXS8D5qfD5gM3Ntt+\npzXMzIauB7heEmRx9KqI+LGku4Glkk4D1gMnNluBg7OZVU+Hv307Ih4F3jZA+XbgyHbU4eBsZpWT\nvZWu3I8IOjibWTXtGu4GtMYDgmZmBeSes5lVktMaZmZF0+EBwW5wcDazCir/N6E4OJtZJZX9Zfse\nEDQzKyD3nM2smqqe1pB0OXAcsDUi3pLKrgEOSoeMB/4QETMlTQfWAA+lfSsi4ox2N9rMbLcCVPJ5\nzo30nK8AvgFc2V8QER/tX5f0VeCp3PGPRMTMdjXQzKwpVe85R8SdqUf8Msre+nEi8IH2NsvMbGRr\ndUDwPcCWiHg4V3aApF9J+pmk99Q7UdKC/m8Z2LGjr8VmmJnVaO2bUIZdqwOC84Alue1e4PURsV3S\nYcANkg6JiKdrT4yIRcAigJ7XjSvIx2FmVTFinxCUtCfwX4HD+ssi4nng+bR+j6RHgDcBK1tsp5nZ\n0IzU4Ax8EHgwIjb2F0h6LfBERPRJegMwA3i0xTaamQ1NUP230klaAvw/4CBJG9Mb/gHm8tKUBsB7\ngVWS7gOuBc6IiCfa2WAzs5Ggkdka8+qUnzxA2XXAda03y8yseSJGbs7ZzKzQHJzNzArIwdnMrGBG\nwoCgmZl1n3vOZlZJHhA0MysiB2czs6Ip/9dUOedsZlZA7jmbWfUEpe85OzibWTWVfCqdg7OZVZJn\na5iZFVHJg7MHBM3MCsg9ZzOrngB2lbvn7OBsZhVU/nnODs5mVk0OzmZmBVTy4OwBQTOzAnLP2cyq\nxwOCZmZFFBDlfkTQwdnMqsk5ZzMzazf3nM2sepxzbo8/7tzJii3bh7sZZlYlJU9rFCI4m5m1nYOz\nmVnRlP/xbQ8ImpkVkHvOZlY9AezyPGczs+IpeVrDwdnMqqnkwXnQnLOkaZJul/SApPslfTqVT5R0\nm6SH088JqVySLpa0VtIqSYd2+ibMzF4qsnnOzS4F0MiA4E7gnIg4GDgCOFPSwcC5wPKImAEsT9sA\nxwAz0rIAuKTtrTYzq7hBg3NE9EbEvWn9GWANMAWYAyxOhy0Gjk/rc4ArI7MCGC9pcttbbmZWT0DE\nrqaXIhhSzlnSdODtwF1AT0T0pl2bgZ60PgXYkDttYyrrzZUhaQFZz5pX7OUZfWbWZgVJTzSr4eAs\naW/gOuDsiHha0gv7IiIkDemTiIhFwCKAvSaNLvenaGbFU/UBQQBJo8kC8/cj4oepeEt/uiL93JrK\nNwHTcqdPTWVmZtagRmZrCLgMWBMRX8vtWgbMT+vzgRtz5SelWRtHAE/l0h9mZp0XkT2E0uxSAI2k\nNd4FfBz4jaT7Utl5wIXAUkmnAeuBE9O+m4FjgbXAs8ApbW2xmVkjSp7WGDQ4R8TPAdXZfeQAxwdw\nZovtMjNrSRSkB9wsPyFoZhXkt9KZmVkHuOdsZtXjr6kyMyuogjzp1ywHZzOrnACi5D1n55zNrHoi\nsp5zs0sDJB0t6aH0Bs5zBz9jaByczcyGSNIo4F/J3sJ5MDAvva2zbZzWMLNK6nBa43BgbUQ8CiDp\narI3cj7QrgocnM2smjo7IDjQ2zff0c4KChGcn92+c9vdizf/Cdg2TE2Y5Lpd9wioezjr37+blT3D\nk7f+NK6d1MIlxkpamdtelN6k2TWFCM4R8VpJKyNi1nDU77pd90iouwj1d0tEHN3hKjr+9k0PCJqZ\nDd3dwAxJB0h6BTCX7I2cbVOInrOZWZlExE5JZwG3AqOAyyPi/nbWUaTg3NV8jut23SOw7iLUXxkR\ncTPZK5I7QlHyNzeZmVWRc85mZgVUiODc6ccga+qaJul2SQ9Iul/Sp1P5REm3SXo4/ZzQofpHSfqV\npJvS9gGS7kr3fk0aXOgISeMlXSvpQUlrJL2zi/f9N+nzXi1piaSxnbp3SZdL2ippda5swPtMX6d2\ncWrDKkmHdqDuf06f+SpJ10san9u3MNX9kKSj2l13bt85kkLSpLTd1vu29hv24NyNxyBr7ATOiYiD\ngSOAM1N95wLLI2IGsDxtd8KngTW57S8BX4+INwJPAqd1qF6Ai4AfR8R/At6W2tHx+5Y0BfgUMCsi\n3kI2gDKXzt37FUDtVKp693kMMCMtC4BLOlD3bcBbIuKtwG+BhQDp924ucEg655vp/4d21o2kacCH\ngd/litt939ZuETGsC/BO4Nbc9kJgYRfrvxH4EPAQMDmVTQYe6kBdU8kCwweAm8i+/msbsOdAn0Wb\n63418BhpnCFX3o377n+aaiLZIPRNwFGdvHdgOrB6sPsE/g2YN9Bx7aq7Zt8JZN9i/7LfdbKR/3e2\nu27gWrI/xuuASZ26by/tXYa958zAj0FO6UbFkqYDbwfuAnrixW8J3wz0dKDKfwE+A/Q/V/oa4A8R\nsTNtd/LeDwAeB76T0irflrQXXbjviNgEfIWs59YLPAXcQ/fuHerfZ7d//04FbulW3ZLmAJsi4tc1\nu4bt/ztrTBGC87CQtDdwHXB2RDyd3xdZV6Kt01gkHQdsjYh72nndIdgTOBS4JCLeDvyJmhRGJ+4b\nIOV355D9gdgP2IsB/vndLZ26z8FI+ixZWu37XarvlcB5wOe7UZ+1VxGCc8cfg6wlaTRZYP5+RPww\nFW+RNDntnwxsbXO17wI+ImkdcDVZauMiYLyk/vnmnbz3jcDGiLgrbV9LFqw7fd8AHwQei4jHI+LP\nwA/JPo9u3TvUv8+u/P5JOhk4DvhY+uPQjboPJPuD+Ov0ezcVuFfS67pQt7WoCMG5449B5kkScBmw\nJiK+ltsQ/P+gAAABQ0lEQVS1DJif1ueT5aLbJiIWRsTUiJhOdo//HhEfA24H/lun6s3VvxnYIOmg\nVHQk2esNO3rfye+AIyS9Mn3+/XV35d6Teve5DDgpzV44Angql/5oC0lHk6WzPhIRz9a0aa6kMZIO\nIBuc+2W76o2I30TEvhExPf3ebQQOTb8LHb9va9FwJ71TJ+JYslHsR4DPdriud5P9k3YVcF9ajiXL\n/y4HHgZ+CkzsYBtmAzel9TeQ/Q+5FvgBMKaD9c4EVqZ7vwGY0K37Bv4OeBBYDXwXGNOpeweWkOW2\n/0wWkE6rd59kg7L/mn73fkM2o6Tdda8ly+/2/759K3f8Z1PdDwHHtLvumv3reHFAsK337aX9i58Q\nNDMroCKkNczMrIaDs5lZATk4m5kVkIOzmVkBOTibmRWQg7OZWQE5OJuZFZCDs5lZAf1/JdMPWDHz\nmuMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f2bd4705210>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbQAAAF1CAYAAACAgB9XAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAH2hJREFUeJzt3X2wZVV55/HvzwaxQQ1omw52t6HVjhmwDGIXIUXioJgI\nxNhkyjIwloJS07EGE2NMKWgmZpJyKokvRCuRVEcITYrwEkChLHxhGBMmU2m0QQTkRRsE6U5L86Jg\nAkG7+5k/9r7meLlvfc4995yz/X6qdt2z195nr3U23Pv0s9ba66SqkCRp0j1t1A2QJGkxGNAkSZ1g\nQJMkdYIBTZLUCQY0SVInGNAkSZ1gQJMkdYIBTZLUCQY0CUiy36jbIGkwBjR1XpI1Sa5M8mCSh5P8\nRZLTk/y/JOckeRj4wyRPS/L7Se5LsivJhUl+or3GYUkqycYk/5JkZ5LfG/FHk9TDgKZOS7IM+Axw\nH3AYsAq4pD3888A9wErgg8Dp7fYq4IXAM4G/mHbJVwHrgF8B3pvkNcNsv6SFi2s5qsuS/AJwNXBo\nVe3uKT8d+KOqekFP2XXAFVX1iXb/JcBtwHJgNfBN4D9V1Z3t8T8DnltVZyzRx5E0BzM0dd0a4L7e\nYNbj/mn7z6fJ5KbcB+xHk8HN9J772vdIGgMGNHXd/cALZpn0Mb174l+An+7ZfwGwG3igp2zNtOP/\nshiNlDQ4A5q67kvATuBPkhyU5BlJjp3l3IuBdyVZm+SZwP8CLp2W3f2PJAcmOQJ4K3DpUFsvacEM\naOq0qtoD/BrwYuBbwHbgN2Y5/Xzgb4HracbL/h34rWnn/COwDbgO+HBVfWEIzZbUByeFSAuQ5DCa\nILf/LONxkkbMDE2S1AkGNElSJwwtoCU5IcldSbYlOWtY9UhLoarurarY3SiNr6GMobWrM3wd+GWa\nQfgvA6dW1e2LXpkkSQwvQzsa2FZV91TV92mWGtowpLokSWJYK4yv4kdXVNhOs27ejJYfuF89+9n7\nD6kpkjR5HnvsBzzx+O4sVX2vfdVB9fAjewa6xo23PPn5qjphkZq0z0b2lRlJNgIbAZ717P35jdNe\nNKqmSNLYuXTz3Uta30OP7OGGz68e6Br7H3r3itmOJVkDXEizlFwBm6rqY0k+RPOs6PeBu4G3VtV3\n20dl7gDuai+xparePlf9w+py3MGPLhG0ui37oaraVFXrq2r98uXLhtQMSdKY2A28u6oOB44Bzkxy\nOHAt8NKqehnN3Iuze95zd1Ud2W5zBjMYXob2ZWBdkrU0gewU4L8OqS5J0sCKPbV3eFev2kmzDB1V\n9b0kdwCrpq22swV4Q791DCVDa6c2vwP4PE3KeFlVfW0YdUmSBlfAXmqgDViRZGvPtnGmutruxJcD\nN0w79Dbgsz37a5N8Jck/Jvml+T7D0MbQquoa4JphXV+StLj2MnCG9lBVrZ/rhHbh7yuA36mqx3rK\n30/TLXlRW7QTeEFVPZzkFcCnkxzR+57pXClEkrQkkuxPE8wuqqore8pPB14HvKnah6Or6smqerh9\nfSPNhJGfmev6I5vlKEkaH0WxZ4iL1ScJcB5wR1V9tKf8BOA9wH+uqsd7yp8HPFJVe5K8EFgH3DNX\nHQY0SRLA1DjYsBwLvBm4NcnNbdn7gI8DBwDXNjHvh9PzXwn8UZIfAHuBt1fVI3NVYECTJFHAniEG\ntKr6J2CmB8VnnGtRVVfQdE8umAFNkgQMPUMbOieFSJI6wQxNktR0OQ5xUshSMKBJkgAGfwptxAxo\nkqRm2v6Ej6EZ0CRJULBnsuOZk0IkSd1ghiZJahcnnmwGNEkSEPbM+Nzz5BjLgLblgYd/ZP+Ylc8d\nUUsk6cdDAXsdQ5MkafTGMkOTJC09uxwlSROvWZzYgCZJ6oC9ZUCTJE24LmRoTgqRJHWCGZokiSLs\nmfAcx4AmSQIcQxsKH6SWpKXVhTG0sQxokqSlFvbUZHc5TnbrJUlqmaFJktrV9ic7xzGgSZIAx9Ak\nSR1Q5RiaJEljwQxNkgTAXrscJUmTrnkObbI77QxokiR+rJ9DS7ImyReT3J7ka0ne2ZY/J8m1Sb7R\n/jxk8ZorSRqGqWn7g2yjNkgLdgPvrqrDgWOAM5McDpwFXFdV64Dr2n1Jkoaq7y7HqtoJ7Gxffy/J\nHcAqYANwXHvaZuAfgPcO1EpJ0tDtmfDFiRclR0xyGPBy4AZgZRvsAL4NrJzlPRuTbE2y9Ykn9ixG\nMyRJfZr6+phBtrns6zBVGh9Psi3JLUmOmu8zDBzQkjwTuAL4nap6rPdYVRVN1+xTVNWmqlpfVeuX\nL182aDMkSQPaW08baJvHvg5TnQisa7eNwLnzVTBQQEuyP00wu6iqrmyLH0hyaHv8UGDXIHVIkoZv\natr+sDK0qtpZVTe1r78H9A5TbW5P2wyc3L7eAFxYjS3AwVOxZTaDzHIMcB5wR1V9tOfQ1cBp7evT\ngKv6rUOSNFFWTA0ltdvGmU5a4DDVKuD+nrdtb8tmNchzaMcCbwZuTXJzW/Y+4E+Ay5KcAdwHvHGA\nOiRJS6DIYkwKeaiq1s91wvRhqiY3attQVUlmHKZaiEFmOf4TzLpOyvH9XleSNBrDfpZsrmGqqto5\nbZhqB7Cm5+2r27JZjf5JOEnSyFXBnnraQNtc+himuhp4Szvb8Rjg0Z6uyRm59JUkaSns6zDVNcBJ\nwDbgceCt81VgQJMkARnqavv7OkzVPvZ15r7UYUCTJDXT9id8cWIDmiQJ8OtjJEkdUIS9ruUoSdLo\nmaFJkgC7HCVJHVCwkAWGx5oBTZIEhD1DnLa/FAxokqROZGiT3XpJklpmaJIkALscJUmTryoT3+Vo\nQJMkAZO/9NVkt16SpJYZmiSpmeXoGJokafJl4rscDWiSpPY5NDM0SVIHTPpajpPdekmSWmZokqRO\nfB+aAU2SBMDeCe+0M6BJkqiCPWZokqQumPQux8nOLyVJapmhSZLaSSGTneMY0CRJgF8fI0nqgC6s\nFDLZ+aUkSS0zNEkSOIYmSeqKSf/6mIHDcZJlSb6S5DPt/tokNyTZluTSJE8fvJmSpGGaerB6kG3U\nFiO/fCdwR8/+nwLnVNWLge8AZyxCHZKkIdtbTxtoG7WBWpBkNfCrwCfb/QCvBi5vT9kMnDxIHZKk\nbkhyfpJdSW7rKbs0yc3tdm+Sm9vyw5I80XPsr+a7/qBjaH8OvAd4Vrv/XOC7VbW73d8OrJrpjUk2\nAhsBnvXs/QdshiRpEEu02v4FwF8AF/6w3qrfmHqd5CPAoz3n311VRy704n1naEleB+yqqhv7eX9V\nbaqq9VW1fvnyZf02Q5K0SPaSgbb5VNX1wCMzHWt7+N4IXNxv+wfJ0I4FXp/kJOAZwLOBjwEHJ9mv\nzdJWAzsGqEOStAQW6cHqFUm29uxvqqpNC3zvLwEPVNU3esrWJvkK8Bjw+1X1f+e6QN8BrarOBs4G\nSHIc8HtV9aYkfw+8AbgEOA24qt86JEkT5aGqWt/ne0/lR7OzncALqurhJK8APp3kiKp6bLYLDGNa\nynuB302yjWZM7bwh1CFJWmSjmuWYZD/gvwCXTpVV1ZNV9XD7+kbgbuBn5rrOojxYXVX/APxD+/oe\n4OjFuK4kaYnUkkwKmc1rgDuravtUQZLnAY9U1Z4kLwTWAffMdZHRPzggSRq5YviTQpJcDPwz8JIk\n25NMPad8Ck+dDPJK4JZ2Gv/lwNurasYJJVNc+kqSBAx/tf2qOnWW8tNnKLsCuGJfrm+GJknqBDM0\nSVInvg/NgCZJAgxokqQOWKKlr4bKgCZJAvw+NEmSxoIZmiQJyjE0SVIHOMtRktQZkx7QHEOTJHWC\nGZokyWn7kqTuKAOaJKkLJv05NAOaJInqwLR9J4VIkjrBDE2SBDiGJknqBGc5SpI6wgxNkjTxurD0\nlZNCJEmdYIYmSYJqpu5PMgOaJAnwwWpJUgcUkz8pxDE0SVInmKFJkvA5NElSZzgpRJLUCZM+hmZA\nkyRRNfkBzUkhkqROMEOTJAE/5ktfJTk4yeVJ7kxyR5JfSPKcJNcm+Ub785DFaqwkaXiqBttGbdAu\nx48Bn6uqnwV+DrgDOAu4rqrWAde1+5KkMVeVgbb5JDk/ya4kt/WU/WGSHUlubreTeo6dnWRbkruS\nvHa+6/cd0JL8BPBK4DyAqvp+VX0X2ABsbk/bDJzcbx2SpKVRDBbMFjih5ALghBnKz6mqI9vtGoAk\nhwOnAEe07/lEkmVzXXyQDG0t8CDwN0m+kuSTSQ4CVlbVzvacbwMrZ3pzko1JtibZ+sQTewZohiRp\nElTV9cAjCzx9A3BJVT1ZVd8EtgFHz/WGQSaF7AccBfxWVd2Q5GNM616sqkoyY89qVW0CNgGs/Knl\nY9D7KunHyb8/+fk5jz/jgHl7uDpnEf4Qr0iytWd/U/u3fj7vSPIWYCvw7qr6DrAK2NJzzva2bFaD\nZGjbge1VdUO7fzlNgHsgyaEA7c9dA9QhSVoKtShjaA9V1fqebSHB7FzgRcCRwE7gI/1+hL4DWlV9\nG7g/yUvaouOB24GrgdPastOAq/qtQ5K0hGrArZ8qqx6oqj1VtRf4a/6jW3EHsKbn1NVt2awGfQ7t\nt4CLkjwduAd4K02QvCzJGcB9wBsHrEOS1FFJDu2Zd/HrwNQMyKuBv0vyUeD5wDrgS3Nda6CAVlU3\nA+tnOHT8INeVJC29YS99leRi4DiasbbtwAeA45IcSZPj3Qv8ZtOW+lqSy2h6/nYDZ1bVnDMIXSlE\nkgQM/+Hoqjp1huLz5jj/g8AHF3p9A5okqRPfWG1AkyS1EW2yA5qr7UuSOsEMTZIEjMcCw4MwoEmS\nGgY0SdLkW/ACw2PLgCZJakx4huakEElSJ5ihSZJ+uDjxJDOgSZIaE97laECTJLUmO0NzDE2S1Alm\naJKkhl2OkqROMKBJkiZeBxYnNqBJkoDJX8vRSSGSpE4wQ5MkNSY8QzOgSZIajqFJkrogZmiSpIlX\nTHyXo5NCJEmdYIYmSQLiGJokqSMmvMvRgCZJakx4QHMMTZLUCWZokn4sPeOA1466CeNnwjM0A5ok\nycWJJUnd4YPVkqRumPCANtCkkCTvSvK1JLcluTjJM5KsTXJDkm1JLk3y9MVqrCRJs+k7oCVZBfw2\nsL6qXgosA04B/hQ4p6peDHwHOGMxGipJmmxJzk+yK8ltPWUfSnJnkluSfCrJwW35YUmeSHJzu/3V\nfNcfdNr+fsDyJPsBBwI7gVcDl7fHNwMnD1iHJGkJpAbbFuAC4IRpZdcCL62qlwFfB87uOXZ3VR3Z\nbm+f7+J9B7Sq2gF8GPgWTSB7FLgR+G5V7W5P2w6s6rcOSdISqgy2zXf5quuBR6aVfaEnZmwBVvfb\n/EG6HA8BNgBrgecDB/HUyDvX+zcm2Zpk6xNP7Om3GZKkxVCLsMGKqb/r7bZxH1vxNuCzPftrk3wl\nyT8m+aX53jzILMfXAN+sqgcBklwJHAscnGS/NuKuBnbM9Oaq2gRsAlj5U8snfG6NJAl4qKrW9/PG\nJO8HdgMXtUU7gRdU1cNJXgF8OskRVfXYbNcYZAztW8AxSQ5MEuB44Hbgi8Ab2nNOA64aoA5J0lIZ\nPEPrS5LTgdcBb6qqAqiqJ6vq4fb1jcDdwM/MdZ1BxtBuoJn8cRNwa3utTcB7gd9Nsg14LnBev3VI\nkpbOEkwKeWqdyQnAe4DXV9XjPeXPS7Ksff1CYB1wz1zXGujB6qr6APCBacX3AEcPcl1J0ggMefAn\nycXAcTRjbdtp4sfZwAHAtU1nH1vaGY2vBP4oyQ+AvcDbq+qRGS/ccqUQSdKSqKpTZyiesRevqq4A\nrtiX6xvQJEmNCZ+eZ0CTJA00DjYuDGiSpIZfHyNJ6oQJz9AGXctRkqSxYIYmSQIcQ5MkdYUBTZI0\n8Towy9ExNElSJ5ihSZIaE56hGdAkSQ0DmiSpCxxDkyRpDBjQJEmdYJejJKkx4V2OBjRJUieeQzOg\nSZIaBjRJUidMeEBzUogkqRPM0CRJBMfQJEldYUCTJE28DsxydAxNktQJZmiSpMaEZ2gGNElSw4Am\nSeqCSR9DM6BJkhoTHtCcFCJJ6gQzNElSk51NeIZmQJMkAY6hSZK6YsID2rxjaEnOT7IryW09Zc9J\ncm2Sb7Q/D2nLk+TjSbYluSXJUcNsvCRp8aQG2+a9/pDjyUImhVwAnDCt7CzguqpaB1zX7gOcCKxr\nt43AuQu4viTpx8MFDDGezBvQqup64JFpxRuAze3rzcDJPeUXVmMLcHCSQ+erQ5I0BmrAbb7LDzme\n9Dttf2VV7WxffxtY2b5eBdzfc972tkySNM4GDWZNQFuRZGvPtnEBNS9aPBl4UkhVVbLvc2PaD7oR\n4FnP3n/QZkiSBpB2G9BDVbW+3zf3G0+m9JuhPTCV+rU/d7XlO4A1Peetbsueoqo2VdX6qlq/fPmy\nPpshSZpwA8eTKf0GtKuB09rXpwFX9ZS/pZ2dcgzwaE8qKUkaZ0MeQ5vFosWTebsck1wMHEfTN7od\n+ADwJ8BlSc4A7gPe2J5+DXASsA14HHjrPnwoSdIIDfvB6mHHk3kDWlWdOsuh42c4t4Az57umJGkM\nDTmgDTueuFKIJKnR9ZVCJEmaBGZokiRY4PJV48yAJklqGNAkSV1ghiZJ6oYJD2hOCpEkdYIZmiQJ\nsMtRktQFgy1fNRYMaJKkxoQHNMfQJEmdYIYmSWq+D23CMzQDmiSpYUCTJHVBarIjmgFNktSJWY5O\nCpEkdYIZmiQJcFKIJKkrDGiSpC4wQ5MkdcOEBzQnhUiSOsEMTZIEZZejJKkrDGiSpEnXhbUcHUOT\nJHWCGZokqeFajpKkLpj0LkcDmiSpE4sTG9AkSQBk76hbMBgnhUiSOsEMTZLUsMtRktQFw5wUkuQl\nwKU9RS8E/gA4GPhvwINt+fuq6pp+6pi3yzHJ+Ul2Jbmtp+xDSe5MckuSTyU5uOfY2Um2JbkryWv7\naZQkaYkVzbT9Qba5Ll91V1UdWVVHAq8AHgc+1R4+Z+pYv8EMFjaGdgFwwrSya4GXVtXLgK8DZwMk\nORw4BTiifc8nkizrt3GSpKWTGmzbB8cDd1fVfYvZ/nkDWlVdDzwyrewLVbW73d0CrG5fbwAuqaon\nq+qbwDbg6EVsryRpfK1IsrVn2zjLeacAF/fsv6Pt8Ts/ySH9Vr4YsxzfBny2fb0KuL/n2Pa27CmS\nbJz60E88sWcRmiFJGkgNuMFDVbW+Z9s0vYokTwdeD/x9W3Qu8CLgSGAn8JF+mz9QQEvyfmA3cNG+\nvreqNk196OXL7ZWUpFGaWpx4CbocTwRuqqoHAKrqgaraU1V7gb9mgF69vmc5JjkdeB1wfNUPRwN3\nAGt6TlvdlkmSxtkCJnYsklPp6W5McmhV7Wx3fx24bcZ3LUBfGVqSE4D3AK+vqsd7Dl0NnJLkgCRr\ngXXAl/ptnCSpO5IcBPwycGVP8Z8luTXJLcCrgHf1e/15M7QkFwPH0Qz2bQc+QDOr8QDg2iQAW6rq\n7VX1tSSXAbfTdEWeWVUOkEnSBBj24sRV9W/Ac6eVvXmxrj9vQKuqU2coPm+O8z8IfHCQRkmSRsCV\nQiRJXeDXx0iSJl8Beyc7ornaviSpE8zQJEmNyU7QDGiSpIZjaJKkbliaB6uHxoAmSQImP0NzUogk\nqRPM0CRJvSvmTywDmiSpXW1/siOaAU2S1Ng76gYMxjE0SVInmKFJkgC7HCVJXeCkEElSNyzZN1YP\njQFNkgT4YLUkSWPBDE2S1LDLUZI08Qoy4c+hGdAkSY0Jz9AcQ5MkdYIZmiSpMdkJmgFNktRwpRBJ\nUjcY0CRJE69wtX1JksaBGZokiVCOoUmSOsKAJknqBAOaJGnidWBSiAFNkrQkktwLfA/YA+yuqvVJ\nngNcChwG3Au8saq+08/1neUoSQKaB6sH2RboVVV1ZFWtb/fPAq6rqnXAde1+X+YNaEnOT7IryW0z\nHHt3kkqyot1Pko8n2ZbkliRH9dswSdISqxps688GYHP7ejNwcr8XWkiGdgFwwvTCJGuAXwG+1VN8\nIrCu3TYC5/bbMEnSUhowmDUBbUWSrT3bxqdWwheS3NhzbGVV7WxffxtY2e8nmHcMraquT3LYDIfO\nAd4DXNVTtgG4sKoK2JLk4CSH9jRWktRdD/V0Jc7kF6tqR5KfBK5NcmfvwaqqJH2nen2NoSXZAOyo\nqq9OO7QKuL9nf3tbNtM1Nk5F8See2NNPMyRJi6UYepdjVe1of+4CPgUcDTyQ5FCA9ueufj/CPge0\nJAcC7wP+oN9KAapqU1Wtr6r1y5cvG+RSkqTFsHfAbQ5JDkryrKnXNENWtwFXA6e1p53Gj/b67ZN+\npu2/CFgLfDUJwGrgpiRHAzuANT3nrm7LJEljbshLX60EPtXGjf2Av6uqzyX5MnBZkjOA+4A39lvB\nPge0qroV+Mmp/fa5gvVV9VCSq4F3JLkE+HngUcfPJGlCDDGgVdU9wM/NUP4wcPxi1LGQafsXA/8M\nvCTJ9jaKzuYa4B5gG/DXwH9fjEZKkjSfhcxyPHWe44f1vC7gzMGbJUlaUgXsdS1HSdLEG+jh6LFg\nQJMkNQxokqROmPCA5uLEkqROMEOTJDkpRJLUFQU12d/waUCTJDUcQ5MkafTM0CRJjqFJkjpkwrsc\nDWiSpIYBTZI0+SZ/6SsnhUiSOsEMTZLUTgrxOTRJUhdMeJejAU2S1DCgSZImX038c2hOCpEkdYIZ\nmiSpnbXvpBBJUhdMeJejAU2S1JjwSSGOoUmSOsEMTZLUZGc+WC1J6oQJ73I0oEmSACgzNEnS5HO1\nfUmSxoIZmiSpXW1/sjM0A5okqTHhK4XY5ShJooDaWwNtc0myJskXk9ye5GtJ3tmW/2GSHUlubreT\n+v0MZmiSpGZCyHAztN3Au6vqpiTPAm5Mcm177Jyq+vCgFRjQJElDV1U7gZ3t6+8luQNYtZh1jEVA\n+9fdu9nywMOjboYkjY1/3b17yeucr9twAVYk2dqzv6mqNk0/KclhwMuBG4BjgXckeQuwlSaL+04/\nlY9FQJMkjYHBuxwfqqr1c52Q5JnAFcDvVNVjSc4F/phmGO+PgY8Ab+un8tQYPEiX5EHg34CHRt2W\neazANi6GcW/juLcPbONiGec2/nRVPW+pKkvyOZr7MYiHquqEOerYH/gM8Pmq+ugMxw8DPlNVL+2n\n8rHI0KrqeUm2zhfZR802Lo5xb+O4tw9s42KZhDYulbkC0WJIEuA84I7eYJbk0HZ8DeDXgdv6rWMs\nApokqfOOBd4M3Jrk5rbsfcCpSY6k6XK8F/jNfiswoEmShq6q/gnIDIeuWaw6xunB6qfMhBlDtnFx\njHsbx719YBsXyyS0UQs0FpNCJEka1DhlaJIk9W0sAlqSE5LclWRbkrPGoD2zrTn2nCTXJvlG+/OQ\nMWjrsiRfSfKZdn9tkhvae3lpkqePuH0HJ7k8yZ1J7kjyC+N2H5O8q/3vfFuSi5M8Y9T3Mcn5SXYl\nua2nbMb7lsbH27bekuSoEbXvQ+1/51uSfCrJwT3Hzm7bd1eS1w67fbO1sefYu5NUkhXt/pLfQy2+\nkQe0JMuAvwROBA6nmfFy+Ghb9cM1xw4HjgHObNt0FnBdVa0Drmv3R+2dwB09+39Ksy7ai4HvAGeM\npFX/4WPA56rqZ4Gfo2nr2NzHJKuA3wbWt8++LANOYfT38QJg+jTq2e7bicC6dtsInDui9l0LvLSq\nXgZ8HTgboP3dOQU4on3PJ9rf+1G0kSRrgF8BvtVTPIp7qEU28oAGHA1sq6p7qur7wCXAhlE2qKp2\nVtVN7evv0fwRXtW2a3N72mbg5NG0sJFkNfCrwCfb/QCvBi5vTxlpG5P8BPBKmmdPqKrvV9V3GbP7\nSDPbd3mS/YADadabG+l9rKrrgUemFc923zYAF1ZjC3BwkkOXun1V9YWqmlqvaQuwuqd9l1TVk1X1\nTWAbze/9UM1yDwHOAd5DM018ypLfQy2+cQhoq4D7e/a3s8gLVg5i2ppjK3seAPw2sHJEzZry5zS/\nmFPr1TwX+G7PH5VR38u1wIPA37Tdop9MchBjdB+ragfwYZp/re8EHgVuZLzu45TZ7ts4/g69Dfhs\n+3ps2pdkA7Cjqr467dDYtFH9G4eANramrznWe6ya6aEjmyKa5HXArqq6cVRtWID9gKOAc6vq5TTL\nm/1I9+IY3MdDaP51vhZ4PnAQM3RTjZtR37e5JHk/Tbf9RaNuS68kB9I8yPsHo26LhmMcAtoOYE3P\n/uq2bKTaNceuAC6qqivb4gemuiHan7tG1T6ap+5fn+Remm7aV9OMVx3cdp3B6O/ldmB7Vd3Q7l9O\nE+DG6T6+BvhmVT1YVT8ArqS5t+N0H6fMdt/G5ncoyenA64A31X88EzQu7XsRzT9cvtr+3qwGbkry\nU4xPGzWAcQhoXwbWtbPKnk4zeHz1KBvUjkU9Zc0xmnad1r4+Dbhqqds2parOrqrVVXUYzT37P1X1\nJuCLwBva00bdxm8D9yd5SVt0PHA7Y3Qfaboaj0lyYPvffaqNY3Mfe8x2364G3tLO1DsGeLSna3LJ\nJDmBpgv89VX1eM+hq4FTkhyQZC3NxIsvLXX7qurWqvrJqjqs/b3ZDhzV/n86FvdQA6qqkW/ASTSz\nou4G3j8G7flFmu6cW4Cb2+0kmjGq64BvAP8beM6o29q29ziaFaoBXkjzx2Ib8PfAASNu25E033F0\nC/Bp4JBxu4/A/wTupFkU9W+BA0Z9H4GLacb0fkDzh/eM2e4bzXJCf9n+/txKM2NzFO3bRjMONfU7\n81c957+/bd9dwImjuofTjt8LrBjVPXRb/M2VQiRJnTAOXY6SJA3MgCZJ6gQDmiSpEwxokqROMKBJ\nkjrBgCZJ6gQDmiSpEwxokqRO+P8aBsr+PZJ4owAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f2bd4577750>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAa4AAAF1CAYAAACwDA52AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHLNJREFUeJzt3XuwXWd53/HvL7IdwBBsSY4qJBO7QTWhmVoGxbEHQsGC\nYCiDPZRx7dKiMk7UP2iC28wEk7ShdNLWTDuAe2NGxYDogMEYiF2GAI7CZZqLQDaG2BaOL9ixHF0s\nGcdcwsXS0z/2Ej4+SD5b+3LOfrW+n5k1e932Xs/Z2/bj533f9a5UFZIkteKnljoASZKOhYlLktQU\nE5ckqSkmLklSU0xckqSmmLgkSU0xcWnRJflAkt9f6jgWS5KXJNm11HFIxwsTlySpKSYuSVJTTFya\nuiTnJLklybeTfBR4ypxjv57k7iQPJ7kxybO6/W9P8t+79ROTfDfJf+m2n5rk+0mWJzkjSSXZlOSv\nkuxP8rtzPv/cJDuSPJpkb5J3zjn2sSR7kvxNki8l+ftzjn0gyf9K8odJvpPkT5L8nSTvTvKtJN9I\ncs6c8+9L8tYkd3TH35/kx3/nvO/jWUk+nuShJN9M8psT/Lql456JS1OV5CTgD4D/AywHPgb84+7Y\nBcB/Bi4BVgP3Ax/p3vpF4CXd+i8Be4AXd9vnA3dW1cNzLvUi4CxgI/B7SX6h2381cHVV/Qzw88B1\nc97zh8A64GeBW4APzQv/EuDfAiuBHwB/1p23ErgeeOe8818PvKK7zt/r3jv/+/gp4P8CXwPWdPFe\nkeQV88+VdGQmLk3becCJwLur6kdVdT3wle7Y64H3VdUtVfUD4K3A+UnOYJAk1iVZwSBhXQOsSfJ0\n4B8ySGxzvb2q/raqvsYgKZzd7f8R8JwkK6vqO1X154ffUFXvq6pvd9f+98DZSZ455zM/WVU3V9X3\ngU8C36+qD1bVQeCjwDk80f+oqge6hPofgcuO8H38EnBaVf2HqvphVd0L/G/g0oW+SEkDJi5N27OA\nB+uJsznfP+fY4XWq6jvAAWBNVf0tsINBknoxg0T1p8ALOXLi2jNn/XvA07v1yxlUP99I8pUkrwZI\nsizJVUnuSfIocF93/so5n7N3zvrfHmH76TzRA/P+xmfxk34OeFaSRw4vwO8Aq45wrqQjOGGpA9Bx\nbzeDSilzktezgXuAv2bwH3IAkpwMrAAe7HZ9EbiAQWXzlW77FcC5wJeGuXhV3QVc1jXRvRa4vqvi\nXgtcBLyMQdJ6JvAtIKP+ocDpc9afzeDvm+8B4JtVtW6M60i9ZsWlafsz4DHgN7tBFq9lkHgArgXe\nmGR9kp8G/hOwvaru645/EXgDcEdV/RD4AvBrDP7D/9AwF0/yz5KcVlWHgEe63YeAZzDotzoAPK27\n9rjelGRtkuXA7zJoTpzvy8C3k7ylG2SyLMkvJvmlCVxf6gUTl6aqSzivBf4F8DDwT4BPdMf+CPh3\nwMcZVGY/zxP7ev4UeCqPV1d3AN9nyGqrcyFwe5LvMBiocWnXDPlBBs15D3af++dH/4ihfRj4HHAv\ng4ryJ26y7vrHXg2sB74J7Afey6DikzSE+CBJaXxJ7gN+rUvGkqbIikuS1BQTlySpKTYVSpKaMlbF\nleTCJHd2U/ZcOamgJEk6mpErriTLgL8EXg7sYnCfzWVVdcfkwpMk6YnGuQH5XODubsoaknyEwQ2d\nR01cK5cvqzNOP3GMS0rS8eW+B37E/ocPjnPj+zF5xUtPrgMPHxz5/Td//QefraoLJxjSMRsnca3h\niVPc7AJ++cnecMbpJ/Llz57+ZKdIUq+c+4oHFj5pgvY/fJDtn1078vtPXH3PyoXPmq6pjypMsrl7\nrMSOhw6MnuUlSYLxKq4HeeLcbGt5fI65H6uqLcAWgA1nP8UhjJK0pIqDdWipgxjLOInrKwweO3Em\ng4R1KfBPJxKVJGkqCjhE2zXEyImrqh5L8q+AzwLLGDxX6faJRSZJmopD9Lfioqo+DXx6QrFIkrQg\nn8clST1SFAcbnzHJxCVJPdPbPi5JUnsKOGjikiS1pPWKy8eaSJKaYsUlST1S4OAMSVJb2r6Ly8Ql\nSb1SlIMzJEkNKTjYdt5ycIYkqS1WXJLUI4NJdttm4pKkXgkHWbQHLk/FkiauVzxr/RO2P/vXty5R\nJJLUDwUcso9LkqTFY1OhJPWMTYWSpGYMJtk1cUmSGnKoTFySpEYcDxWXgzMkSU2x4pKkHinCwcZr\nFhOXJPWMfVxj8IZjSVpcx0MflxWXJPVKOFhtNxW2Hb0kqXesuCSpRwazw7dds5i4JKln7OOSJDWj\nyj4uSZIWlRWXJPXMIZsKJUmtGNzH1XZjm4lLknql/T4uE5ck9cjxMBx+weiTvC/JviS3zdm3PMlN\nSe7qXk+dbpiSJA0Mk3Y/AFw4b9+VwLaqWgds67YlSQ04WBl5mQULJq6q+hLw8LzdFwFbu/WtwMUT\njkuSNAWHH2sy6jILRu3jWlVVu7v1PcCqo52YZDOwGeDZa+xSk6SldqjvgzOqqpLUkxzfAmwB2HD2\nU456niRp+o6H4fCjRr83yWqA7nXf5EKSJOnoRk1cNwKbuvVNwA2TCUeSNE3F6AMzZmVwxoJNhUmu\nBV4CrEyyC3gbcBVwXZLLgfuBS6YZpCRpclq/j2vBxFVVlx3l0MYJxyJJmrIqmp85o+3oJUm9Y+KS\npF4Jh8ZYFvz0Y5htKQP/LcndSb6e5PnD/AUmLknqkWLQVDjqMoQPMPxsS68E1nXLZuA9w1zAxCVJ\nPTPNmTOOcbali4AP1sCfA6ccvtXqyTiVhST1SBEOjTesfWWSHXO2t3QTTTyZo822tAZ4YM55u7p9\nu3kSJi5J0rHYX1UbRn3zQrMtDcPEJUk9swRTPu1Nsrqqds+bbelB4PQ5563t9j0p+7gkqUeKwSS7\noy4jOtpsSzcCb+hGF54H/M2cJsWjsuKSpF4JB4cY1j7ypx/bbEufBl4F3A18D3jjMNcwcUlSjxyu\nuKb2+ccw21JVFfCmY72GTYWSpKZYcUlSz0yzqXAxmLgkqUeq4hOQJUltcXZ4SZIWkRWXJPVIwVCz\nvM8yE5ck9Uqabyo0cUlSjwzu47LikiQ1ZAnmKpyotqOXJPWOFZck9cgEnse15ExcktQzhxpvbDNx\nSVKPVMFBKy5JUktabypsu16UJPWOFZck9chgcEbbNYuJS5J6xseaSJKacTzMnNF2vShJ6h0rLknq\nFfu4JEmN8bEmkqRmHA83IC9YLyY5Pcnnk9yR5PYkb+72L09yU5K7utdTpx+uJGlch+qnRl5mwTBR\nPAb8VlU9DzgPeFOS5wFXAtuqah2wrduWJGmqFmwqrKrdwO5u/dtJdgJrgIuAl3SnbQW+ALxlKlFK\nkiaid7PDJzkDOAfYDqzqkhrAHmDVUd6zGdgM8Ow1dqlJ0lJrfXDG0A2WSZ4OfBy4oqoenXusqorB\nfW0/oaq2VNWGqtpw2oplYwUrSRrP4RuQR11mwVCJK8mJDJLWh6rqE93uvUlWd8dXA/umE6IkSY8b\nZlRhgGuAnVX1zjmHbgQ2deubgBsmH54kadJaH1U4TKfTC4F/DvxFklu7fb8DXAVcl+Ry4H7gkumE\nKEmamBlq8hvVMKMK/x8ctSdv42TDkSRNU9H+4AyH+UlSz7Recc1Gg6UkSUOy4pKkHjkensdl4pKk\nnjFxSZKa0bspnyRJ7Wt9VKGDMyRJTbHikqQ+Kfu4JEkNcVShJKk5rScu+7gkSU2x4pKkHnE4vCSp\nOWXikiS1pPX7uExcktQjdRwMh3dwhiSpKVZcktQz9nFJkhrS/qhCmwolqWeqMvIyjCRvTnJbktuT\nXNHtW57kpiR3da+njhq/iUuSeuTwlE+jLgtJ8ovArwPnAmcDr07yHOBKYFtVrQO2ddsjMXFJkibp\nF4DtVfW9qnoM+CLwWuAiYGt3zlbg4lEvYOKSpD6pwZD4URdgZZIdc5bN865wG/ArSVYkeRrwKuB0\nYFVV7e7O2QOsGvVPcHCGJPXMmDcg76+qDUc7WFU7k7wD+BzwXeBW4OC8cypJjRqAFZck9Ugx/cEZ\nVXVNVb2gql4MfAv4S2BvktUA3eu+Uf8GE5ckaaKS/Gz3+mwG/VsfBm4ENnWnbAJuGPXzbSqUpF5Z\nlPu4Pp5kBfAj4E1V9UiSq4DrklwO3A9cMuqHm7gkqWdq5N6lYT+/fuUI+w4AGyfx+SYuSeoZp3yS\nJDVjMKy97cTl4AxJUlOsuCSpZ1qfZNfEJUk9M+3BGdO2YFNhkqck+XKSr3Uz/b69239mku1J7k7y\n0SQnTT9cSdK4pn0D8rQN08f1A+CCqjobWA9cmOQ84B3Au6rqOQzujL58emFKkiahGD1pNZO4auA7\n3eaJ3VLABcD13f6xZvqVJGlYQ40qTLIsya0M5pa6CbgHeKSbsh5gF7DmKO/dfHgW4YcOHDzSKZKk\nRVRjLLNgqMRVVQeraj2wlsHDwZ477AWqaktVbaiqDaetWDZimJKkiaj2+7iOaVRhN9/U54HzgVOS\nnNBVXWuBB6cRoCSN44OPrnzS42/4mf2LFMkMmZXSaUTDjCo8Lckp3fpTgZcDO4HPA6/rThtrpl9J\nkoY1TMW1GtiaZBmDRHddVX0qyR3AR5L8PvBV4JopxilJmpBZafIb1YKJq6q+DpxzhP33MujvkiQ1\npPUbkJ05Q5J65PATkFtm4pKkPimg8cTl7PCSpKZYcUlSz9jHJUlqi4lLktSO2ZkBY1QmLknqm8Yr\nLgdnSJKaYsUlSX1S3sclSWpN402FJi5J6p22Ky77uCRJTbHikqS+salQktQUE5ckqRnHwSS7Ji5J\n6pnW5yp0cIYkqSlWXJLUN41XXCYuSeob+7gkSS2JFZckqRlF802FDs6QJDXFikuSeiX2cUmSGtN4\nU6GJS5L6pvHEZR+XJKkpVlyS1DeNV1wmLknqEyfZlaTZ9oaf2b/UIcwcb0CWJLWl8cTl4AxJUlOG\nTlxJliX5apJPddtnJtme5O4kH01y0vTClCRp4FgqrjcDO+dsvwN4V1U9B/gWcPkkA5MkTUdq9GUW\nDJW4kqwF/hHw3m47wAXA9d0pW4GLpxGgJGnCKqMvM2DYwRnvBn4beEa3vQJ4pKoe67Z3AWsmHJsk\nadL6MDt8klcD+6rq5lEukGRzkh1Jdjx04OAoHyFJakiSf53k9iS3Jbk2yVMmOS5imKbCFwKvSXIf\n8BEGTYRXA6ckOVyxrQUePNKbq2pLVW2oqg2nrVg2apySpEmpMZYFJFkD/Cawoap+EVgGXMoEx0Us\nmLiq6q1Vtbaqzugu/sdV9Xrg88DrutM2ATeMGoQkafEswuCME4CndsXN04DdTHBcxDj3cb0F+DdJ\n7mbQ53XNGJ8lSVosU6y4qupB4L8Cf8UgYf0NcDMTHBdxTDNnVNUXgC906/cC5456YUlSk1Ym2TFn\ne0tVbTm8keRU4CLgTOAR4GPAhZMMwCmfJKlvxhtVuL+qNjzJ8ZcB36yqhwCSfILBWIlTkpzQVV1H\nHRcxDKd8kqQeGad/a8g+rr8CzkvytO6e343AHUxwXISJS5L6Zoo3IFfVdgaDMG4B/oJBntnCBMdF\n2FQoSX0z5RuQq+ptwNvm7Z7YuAgrLklSU6y4JKlnZmWy3FGZuCSpb0xckqRmzNDjSUZlH5ckqSlW\nXJLUN41XXCYuSeobE5ckqSX2cUmStIhMXJKkpthUKEl903hToYlLkvrkOLiPy8QlSX1j4pIkNaXx\nxOXgDElSU6y4JKlHgn1ckqTWmLgkSc04DkYV2sclSWqKFZck9U3jFZeJS5L6xsQlSWpJ631cJi5J\n6pvGE5eDMyRJTbHikqQ+KZqvuExcktQz9nFJktpi4pIktaQXFVeS+4BvAweBx6pqQ5LlwEeBM4D7\ngEuq6lvTCVOSpIFjGVX40qpaX1Ubuu0rgW1VtQ7Y1m1LkmZdjbHMgHGGw18EbO3WtwIXjx+OJGmq\nxklajSWuAj6X5OYkm7t9q6pqd7e+B1g18egkSROVMZdZMOzgjBdV1YNJfha4Kck35h6sqkqO3N3X\nJbrNAM9e41gQSdJ4hqq4qurB7nUf8EngXGBvktUA3eu+o7x3S1VtqKoNp61YNpmoJUmjO96bCpOc\nnOQZh9eBXwVuA24ENnWnbQJumFaQkqTJSY2+zIJh2u5WAZ9Mcvj8D1fVZ5J8BbguyeXA/cAl0wtT\nkjQxM5KARrVg4qqqe4Gzj7D/ALBxGkFJkqao8cTl7PCSpKY4zE+S+mSG+qpGZeKSpL4xcUmSWmLF\nJUlqS+OJy8EZkqSmWHFJUs/YVChJascMTd00KhOXJPVN44nLPi5JUlOsuCSpR4J9XJKk1jSeuGwq\nlKSeSdXIy4KfnZyV5NY5y6NJrkiyPMlNSe7qXk8dNX4TlyT1yTgPkRyiUquqO6tqfVWtB14AfI/B\nA4ivBLZV1TpgW7c9EhOXJGlaNgL3VNX9wEXA1m7/VuDiUT/UPi5J6plFHJxxKXBtt76qqnZ363sY\nPKR4JFZcktQ34zUVrkyyY86y+UiXSHIS8BrgYz9x+aqxboO24pKknhmz4tpfVRuGOO+VwC1Vtbfb\n3ptkdVXtTrIa2DdqAFZcktQ3UxycMcdlPN5MCHAjsKlb3wTcMGr4Ji5J0kQlORl4OfCJObuvAl6e\n5C7gZd32SGwqlKQ+qekPzqiq7wIr5u07wGCU4dhMXJLUN43PnGHikqQeOR7mKrSPS5LUFCsuSeqb\nIeYcnGUmLknqmdabCk1cktQnY81ZMRtMXJLUMzm01BGMx8EZkqSmWHFJUt/YVChJaomDMyRJ7Sia\nHw4/VB9XklOSXJ/kG0l2Jjk/yfIkNyW5q3s9ddrBSpLGlxp9mQXDDs64GvhMVT0XOBvYCVwJbKuq\ndcC2bluSpKlaMHEleSbwYuAagKr6YVU9AlwEbO1O2wpcPK0gJUkTtDjP45qaYSquM4GHgPcn+WqS\n93bPWllVVbu7c/YAq4705iSbDz/i+aEDBycTtSRpJIcn2T3emwpPAJ4PvKeqzgG+y7xmwao6ai6u\nqi1VtaGqNpy2Ytm48UqSxlE13jIDhklcu4BdVbW9276eQSLbm2Q1QPe6bzohSpL0uAUTV1XtAR5I\ncla3ayNwB3AjsKnbtwm4YSoRSpImqvWmwmHv4/oN4ENJTgLuBd7IIOldl+Ry4H7gkumEKEmaqBlJ\nQKMaKnFV1a3AhiMc2jjZcCRJ0zYrldOonDlDkvqkgENtZy5nh5ckNcWKS5L6pu2Cy8QlSX1jH5ck\nqS0zciPxqExcktQzrVdcDs6QJDXFikuS+mSGZnkflYlLknpkMDt825nLxCVJfXNoqQMYj31ckqSm\nWHFJUs/YVChJaoeDMyRJbZmdJxmPysQlST3jDciSJC0iKy5J6hubCiVJzShI4/dxmbgkqW8ar7js\n45IkNcWKS5L6pu2Cy8QlSX3jzBmSpLaYuCRJzSicHV6SpMVkxSVJPRLKPi5JUmNMXJKkppi4JEnN\ncHCGJEmLy8QlST2TqpGXoT4/OSXJ9Um+kWRnkvOTLE9yU5K7utdTR43fxCVJfVM1+jKcq4HPVNVz\ngbOBncCVwLaqWgds67ZHsmDiSnJWklvnLI8muWKS2VOStFjGSFpDJK4kzwReDFwDUFU/rKpHgIuA\nrd1pW4GLR/0LFkxcVXVnVa2vqvXAC4DvAZ9kgtlTktSMlUl2zFk2zzt+JvAQ8P4kX03y3iQnA6uq\nand3zh5g1agBHOuowo3APVV1f5KLgJd0+7cCXwDeMmogkqRFUIw7HH5/VW14kuMnAM8HfqOqtie5\nmnmFTVVVkpGDONY+rkuBa7v1obJnks2HM/NDBw6OGKYkaWIOjbEsbBewq6q2d9vXM0hke5OsBuhe\n940a/tCJK8lJwGuAj80/VlXFUZ7wUlVbqmpDVW04bcWyUeOUJE3INEcVVtUe4IEkZ3W7NgJ3ADcC\nm7p9m4AbRo3/WJoKXwncUlV7u+29SVZX1e5xs6ckaRFNf+aM3wA+1BU89wJvZFAoXZfkcuB+4JJR\nP/xYEtdlPN5MCI9nz6sYM3tKko4fVXUrcKR+sI2T+PyhElc3IuTlwL+cs/sqJpQ9JUmLpIBDPZir\nsKq+C6yYt+8AE8qekqTFckw3Es8kJ9mVpL4xcUmSmtJ44nKuQklSU6y4JKlP+jI4Q5J0vCiotp8k\naeKSpL6xj0uSpMVjxSVJfWIflySpOY03FZq4JKlvTFySpHa0P+WTgzMkSU2x4pKkPingkPdxSZJa\n0nhToYlLkvrGxCVJakc1fx+XgzMkSU2x4pKkPikoJ9mVJDWl8aZCE5ck9U3jgzPs45IkNcWKS5L6\npMobkCVJjWm8qdDEJUk9U1ZckqR2ODu8JEmLyopLkvqk8D4uSVJjnDlDktSKAsqKS5LUjKrmKy4H\nZ0iSmmLFJUk9Y1OhJKktjTcVphbxRrQkDwH3AyuB/Yt24dHMeoyzHh8Y46TMeoyzHh/Mdow/V1Wn\nLdbFknyGwfcxqv1VdeGk4hnFoiauH1802VFVGxb9wsdg1mOc9fjAGCdl1mOc9figjRg1PAdnSJKa\nYuKSJDVlqRLXliW67rGY9RhnPT4wxkmZ9RhnPT5oI0YNaUn6uCRJGpVNhZKkpixq4kpyYZI7k9yd\n5MrFvPbRJHlfkn1Jbpuzb3mSm5Lc1b2eusQxnp7k80nuSHJ7kjfPWpxJnpLky0m+1sX49m7/mUm2\nd7/5R5OctFQxdvEsS/LVJJ+a0fjuS/IXSW5NsqPbNzO/cxfPKUmuT/KNJDuTnD9LMSY5q/v+Di+P\nJrlilmLUeBYtcSVZBvxP4JXA84DLkjxvsa7/JD4AzL8n4UpgW1WtA7Z120vpMeC3qup5wHnAm7rv\nbpbi/AFwQVWdDawHLkxyHvAO4F1V9RzgW8DlSxgjwJuBnXO2Zy0+gJdW1fo5w7dn6XcGuBr4TFU9\nFzibwfc5MzFW1Z3d97ceeAHwPeCTsxSjxlRVi7IA5wOfnbP9VuCti3X9BWI7A7htzvadwOpufTVw\n51LHOC/eG4CXz2qcwNOAW4BfZnDT5wlH+mdgCeJay+A/WBcAnwIyS/F1MdwHrJy3b2Z+Z+CZwDfp\n+sdnMcZ5cf0q8CezHKPLsS+L2VS4Bnhgzvaubt8sWlVVu7v1PcCqpQxmriRnAOcA25mxOLtmuFuB\nfcBNwD3AI1X1WHfKUv/m7wZ+Gzg8380KZis+GDx14nNJbk6yuds3S7/zmcBDwPu7Jtf3JjmZ2Ypx\nrkuBa7v1WY1Rx8jBGQuowf+ezcTQyyRPBz4OXFFVj849NgtxVtXBGjTPrAXOBZ67lPHMleTVwL6q\nunmpY1nAi6rq+Qya1N+U5MVzD87A73wC8HzgPVV1DvBd5jW5zUCMAHT9la8BPjb/2KzEqNEsZuJ6\nEDh9zvbabt8s2ptkNUD3um+J4yHJiQyS1oeq6hPd7pmLE6CqHgE+z6Dp7ZQkhydzXsrf/IXAa5Lc\nB3yEQXPh1cxOfABU1YPd6z4G/TLnMlu/8y5gV1Vt77avZ5DIZinGw14J3FJVe7vtWYxRI1jMxPUV\nYF03iuskBiX8jYt4/WNxI7CpW9/EoE9pySQJcA2ws6reOefQzMSZ5LQkp3TrT2XQB7eTQQJ7XXfa\nksVYVW+tqrVVdQaDf/b+uKpePyvxASQ5OckzDq8z6J+5jRn6natqD/BAkrO6XRuBO5ihGOe4jMeb\nCWE2Y9QoFrNDDXgV8JcM+j5+d6k7+LqYrgV2Az9i8H+TlzPo+9gG3AX8EbB8iWN8EYNmja8Dt3bL\nq2YpTuAfAF/tYrwN+L1u/98FvgzczaDJ5qdn4Dd/CfCpWYuvi+Vr3XL74X9HZul37uJZD+zofus/\nAE6dwRhPBg4Az5yzb6ZidBl9ceYMSVJTHJwhSWqKiUuS1BQTlySpKSYuSVJTTFySpKaYuCRJTTFx\nSZKaYuKSJDXl/wPPtu5aFvHtzAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f2bd4845390>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAasAAAF1CAYAAABWJcUyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHC1JREFUeJzt3X/wXXWd3/Hny/Ajgvw0momEFTpGLbUaNMuP0VEEfwTq\nEDq1DLjtxh2m6bSylVV3hd0Ou2v7h3Y7su4s1WaFNd2qiKhLaqmIEcd2Z0WCIEKQJaJIEIggCGpF\nSN79457U63eT7/fm+70395yc52PmzPeecz733Pc3N+TN+/P5nM9JVSFJUps9a9oBSJI0F5OVJKn1\nTFaSpNYzWUmSWs9kJUlqPZOVJKn1TFaSpNYzWUmSWs9kpf1ekgOmHYOkhTFZqbOSvCDJZ5L8MMl3\nk/y75vgfJbkmyX9P8gTw9iQnJfnbJI8neTDJnyc5qGmfJJcl2Z7kiSTfSvKy5tzBSf5zku8neTjJ\nR5I8e4q/ttRLJit1UpJnAf8D+CZwDHAGcFGSNzdN1gDXAEcCHwd2AL8DLAFObdr/26btm4DXAi8G\njgDOBR5tzr2/Ob4SeFHzWZdO8FeTtBtxbUB1UZKTgU9X1a8NHbuEQWK5Dzi9ql47y/svAl5XVf80\nyenAR4DfBL5eVTubNgF+Ary8qr7THDsV+ERVHT+hX03SbtiXr656IfCCJI8PHVsE/G8Gyer+4cZJ\nXgx8EFgFHMLg7/4tAFX15SR/DlwOvDDJZ4H3AIubtrcM8tbgUs3nSNqH7AZUV90PfLeqjhzaDquq\ns5rzM7sMPgx8G1hRVYcDv88g8QwaV/1ZVb0KOIFBdfa7wCPA/wX+0dBnHFFVz5nw7yZpBpOVuurr\nwJNJ3pvk2UkWJXlZkl/fQ/vDgCeAnyR5KfBvdp1I8utJTk5yIPBT4OfAzqY78C+Ay5I8v2l7zNC4\nmKR9xGSlTqqqHcBbGEx8+C6DKuijDCZI7M57gLcBTzJIQJ8aOnd4c+wxBl2IjwJ/0px7L7AV+Foz\ns/BLwEvG+btImpsTLCRJrWdlJUlqPZOVJGlsklzZ3GB/xx7OJ8mfJdma5PYkrxzluiYrSdI4fQxY\nPcv5M4EVzbaOwUzdOS0oWSVZneTuJkNevJBrSZK6r6q+CvxoliZrgP9WA18DjkyybK7rzjtZJVnE\n4CbKMxncm3J+khPmez1JUi8cw6/etL+tOTarhaxgcRKwtaruBUhyFYOMuWVPbzgoB9diDl3AR0rS\n/uXn/JRf1FOZu+V4vPn1h9ajP9ox7/ffcvtTdzK4F3GX9VW1fsGBzWEhyWp32fHk2d6wmEM5OWcs\n4CMlaf9yU23ap5/3yI92cNP1y+f9/gOXfefnVbVqASE8ABw7tL+8OTariU+wSLIuyeYkm5/mqUl/\nnCSp3TYCv9nMCjwF+HFVPTjXmxZSWY2UHZvycD3A4TnaO5AlaaqKHYMHC0xEkk8CpwFLkmwD/hA4\nEKCqPgJcB5zFYGWYnwG/Ncp1F5KsbgZWJDmeQZI6j8FyNpKklipg599b53mM1686f47zBbxjb687\n72RVVc8kuRC4nsEjE66sqjvnez1J0r6xk8lVVpOyoOdZVdV1DEo6SZImxocvSlKPFMWODi5gbrKS\npJ6Z5JjVpJisJKlHCthhspIktV0XKytXXZcktZ6VlST1SIETLCRJ7de9u6xMVpLUK0U5wUKS1HIF\nO7qXq5xgIUlqPysrSeqRwUK23WOykqReCTvYZw8mHpupJqvrf3Dbr+y/+QUrpxSJJPVDATsds5Ik\nafzsBpSknrEbUJLUaoOFbE1WkqSW21kmK0lSi3W1snKChSSp9aysJKlHirCjg3WKyUqSesYxq73k\nTcCStG91dczKykqSeiXsqO51A3YvYklS71hZSVKPDFZd716dYrKSpJ5xzEqS1GpVjllJkjQRVlaS\n1DM77QaUJLXZ4D6r7nWqmawkqVe6OWZlspKkHunq1PU5I05yZZLtSe4YOnZ0khuS3NP8PGqyYUqS\n+myU9PoxYPWMYxcDm6pqBbCp2ZckdcCOyry3aZkzWVXVV4EfzTi8BtjQvN4AnDPmuCRJE7DrESHz\n3aZlvmNWS6vqweb1Q8DSPTVMsg5YB7CYQ+b5cZKkcdnZxwkWVVVJapbz64H1AIfn6D22kyRNXlen\nrs834oeTLANofm4fX0iSJP2q+SarjcDa5vVa4NrxhCNJmqRi/pMrpjnBYs5uwCSfBE4DliTZBvwh\n8H7g6iQXAPcB504ySEnS+HTxPqs5k1VVnb+HU2eMORZJ0oRV0ckVLLoXsSSpd1xuSZJ6Ja66Lklq\nt6Kb3YAmK0nqmS7eZ2WykqQeKcLOKU5Bn6/upVdJUu9YWUlSz9gNKElqtaKnC9lKkrok7HDquiSp\nzbpaWXUvYklS71hZSVLP2A0oSWq1qtgNKElqvx31rHlvo0iyOsndSbYmuXg3538tyY1Jbk1ye5Kz\n5rqmyUqSNDZJFgGXA2cCJwDnJzlhRrN/D1xdVScC5wH/Za7r2g0oST1SMOlV108CtlbVvQBJrgLW\nAFtmhHF48/oI4AdzXdRkJUm9kkmvun4McP/Q/jbg5Blt/gj4YpLfBg4F3jDXRe0GlKQeGdxnlXlv\nwJIkm4e2dfMI43zgY1W1HDgL+Ksks+YjKytJ6pkFrg34SFWtmuX8A8CxQ/vLm2PDLgBWA1TV3yZZ\nDCwBtu/polZWkqRxuhlYkeT4JAcxmECxcUab7wNnACT5h8Bi4IezXdTKSpJ6ZNLPs6qqZ5JcCFwP\nLAKurKo7k7wP2FxVG4F3A3+R5HcY9Ey+vapqtuuarCSpZ3ZOuFOtqq4Drptx7NKh11uAV+/NNU1W\nktQjVbCjg08KNllJUs/4WHtJkibAykqSemQwwaJ7dYrJSpJ6xkeESJJabdcKFl3TvVpQktQ7VlaS\n1CuOWUmSOmDCjwiZCJOVJPVIV28KnrMWTHJs8/jhLUnuTPLO5vjRSW5Ick/z86jJhytJWqid9ax5\nb9Myyic/A7y7qk4ATgHe0Tyi+GJgU1WtADY1+5Ikjd2c3YBV9SDwYPP6ySR3MXgS5BrgtKbZBuAr\nwHsnEqUkaSwmver6pOzVmFWS44ATgZuApU0iA3gIWLqH96wD1gEs5pD5xilJGpP9eoJFkucAnwEu\nqqonkl/+slVVSXb7LJKqWg+sBzg8R8/6vBJJ0mTt1zcFJzmQQaL6eFV9tjn8cJJlzfllzPI4YkmS\nFmKU2YABrgDuqqoPDp3aCKxtXq8Frh1/eJKkcevibMBRugFfDfxL4FtJbmuO/T7wfuDqJBcA9wHn\nTiZESdLY1H46waKq/g/scTTujPGGI0mapGI/n2AhSdo/dLGy6t5qhpKk3rGykqQe6erUdZOVJPWM\nyUqS1Gq9WG5JktR9XZwN6AQLSVLrWVlJUp+UY1aSpJZzNqAkqRO6mKwcs5IktZ6VlST1iFPXJUmd\nUCYrSVLbdfE+K5OVJPVIdXTquhMsJEmtZ2UlST3jmJUkqeWcDShJ6gArK0lSq3V1uSUnWEiSWs/K\nSpL6pAbT17vGZCVJPeNNwZKkViu6OcHCMStJUutZWUlSr3iflSSpA5xgIUlqvS6OWZmsJKlHqrqZ\nrJxgIUlqPSsrSeoZJ1hIklqvixMs5uwGTLI4ydeTfDPJnUn+uDl+fJKbkmxN8qkkB00+XEnSQlVl\n3tu0jDJm9RRwelW9AlgJrE5yCvAB4LKqehHwGHDB5MKUJI1DMf9E1epkVQM/aXYPbLYCTgeuaY5v\nAM6ZSISSpN4baTZgkkVJbgO2AzcA3wEer6pnmibbgGP28N51STYn2fw0T40jZknSAtQCtmkZaYJF\nVe0AViY5Evgc8NJRP6Cq1gPrAQ7P0R0c1pOk/UhH77Paq9mAVfV4khuBU4EjkxzQVFfLgQcmEaAk\nLcT1P7ht1vNvfsHKfRRJi3SwbBhlNuDzmoqKJM8G3gjcBdwIvLVptha4dlJBSpK6I8nqJHc3s8Uv\n3kObc5NsaWaZf2Kua45SWS0DNiRZxCC5XV1Vn0+yBbgqyX8EbgWu2IvfRZI0JZPsBmxyxeUMCptt\nwM1JNlbVlqE2K4BLgFdX1WNJnj/XdedMVlV1O3Dibo7fC5w0+q8gSWqDCd8UfBKwtckRJLkKWANs\nGWrzr4DLq+qxQTy1fa6LujagJPXIricFL+A+qyW7Zng327oZH3EMcP/Q/u5mi78YeHGSv0nytSSr\n54rb5ZYkqU8KWFg34CNVtWqBURwArABOYzBB76tJ/nFVPb6nN1hZSZLG6QHg2KH93c0W3wZsrKqn\nq+q7wN8xSF57ZLKSpJ4ZPNNqftsIbgZWNOvHHgScB2yc0eavGVRVJFnCoFvw3tkuarKSpL6Z4BIW\nzb23FwLXM7jN6eqqujPJ+5Kc3TS7Hni0mVV+I/C7VfXobNd1zEqSemXyC9JW1XXAdTOOXTr0uoB3\nNdtITFaS1Df74woWkiRNm5WVJPVJHxaylSTtBzrYDWiykqTe6V5l5ZiVJKn1rKwkqW/sBpQktZ7J\nSpLUagtfyHYqTFaS1DMTfp7VRDjBQpLUelZWktQ3HaysTFaS1DeOWUmS2i5WVpKkVhvxuVRt4wQL\nSVLrWVlJUq/EMStJUgd0sBvQZCVJfdPBZOWYlSSp9aysJKlvOlhZmawkqU9cyFaS2ufNL1g57RBa\nx5uCJUnt18Fk5QQLSVLrjZyskixKcmuSzzf7xye5KcnWJJ9KctDkwpQk9dneVFbvBO4a2v8AcFlV\nvQh4DLhgnIFJkiYjNf9tWkZKVkmWA/8E+GizH+B04JqmyQbgnEkEKEkas8r8tykZdYLFnwK/BxzW\n7D8XeLyqnmn2twHHjDk2SdK47a+rrid5C7C9qm6ZzwckWZdkc5LNT/PUfC4hSeq5USqrVwNnJzkL\nWAwcDnwIODLJAU11tRx4YHdvrqr1wHqAw3N0B/O5JO1nOvgv8ZyVVVVdUlXLq+o44Dzgy1X1G8CN\nwFubZmuBaycWpSRpbPbbCRZ78F7gXUm2MhjDumI8IUmSJqoWsE3JXq1gUVVfAb7SvL4XOGn8IUmS\n9KtcbkmS+qaDY1YmK0nqkWmPPc2XyUqS+sZHhEiSWq+DlZWrrkuSWs/KSpJ6xjErSVL7mawkSa3W\n0dmAjllJklrPykqS+qaDlZXJSpL6xmQlSWo7x6wkSZoAk5UkqfXsBpSkvulgN6DJSpL6pKP3WZms\nJKlvTFaSpNbrYLJygoUkqfWsrCSpR4JjVpKkLjBZSZJaraOzAR2zkiSNVZLVSe5OsjXJxbO0+2dJ\nKsmqua5pspKkvqkFbHNIsgi4HDgTOAE4P8kJu2l3GPBO4KZRQjZZSVLfTDBZAScBW6vq3qr6BXAV\nsGY37f4D8AHg56Nc1GQlST2Tmv8GLEmyeWhbN+PyxwD3D+1va4798vOTVwLHVtX/HDVmJ1hIUt8s\nbILFI1U15xjTniR5FvBB4O178z4rK0nSOD0AHDu0v7w5tsthwMuAryT5HnAKsHGuSRZWVpLUJ6OP\nPc3XzcCKJMczSFLnAW/7/x9f9WNgya79JF8B3lNVm2e7qJWVJPXMAsesZlVVzwAXAtcDdwFXV9Wd\nSd6X5Oz5xmxlJUl9M+GbgqvqOuC6Gccu3UPb00a5pslKknqmiytYjJSsmkGwJ4EdwDNVtSrJ0cCn\ngOOA7wHnVtVjkwlTktRnezNm9fqqWjk0ZfFiYFNVrQA2NfuSpLab7E3BE7GQCRZrgA3N6w3AOQsP\nR5I0UQtJVB1IVgV8McktQ3crL62qB5vXDwFLxx6dJGmsssBtWkadYPGaqnogyfOBG5J8e/hkVVWy\n+yG7JrmtA1jMIQsKVpLUTyNVVlX1QPNzO/A5BgsVPpxkGUDzc/se3ru+qlZV1aoDOXg8UUuS5m9/\n7AZMcmizlDtJDgXeBNwBbATWNs3WAtdOKkhJ0vhM8qbgSRmlG3Ap8Lkku9p/oqq+kORm4OokFwD3\nAedOLkxJ0tjsj/dZVdW9wCt2c/xR4IxJBCVJmqAOJivXBpQktZ7LLUlSn0x57Gm+TFaS1DcmK0lS\n21lZSZLar4PJygkWkqTWs7KSpJ6xG1CS1G5TXjZpvkxWktQ3HUxWjllJklrPykqSeiQ4ZiVJ6gKT\nlSSp7VLdy1YmK0nqk47OBnSChSSp9aysJKlnnGAhSWo/k5Ukqe2srCRJ7dfBZOUEC0lS61lZSVKf\n+Fh7SVInmKwkSW3W1bUBHbOSJLWelZUk9Y1rA0qS2q6L3YAmK0nqk44uZGuykqSeyc5pR7D3nGAh\nSWo9KytJ6hu7ASVJbecEC0lSuxWdnLo+0phVkiOTXJPk20nuSnJqkqOT3JDknubnUZMOVpK0cKn5\nb9My6gSLDwFfqKqXAq8A7gIuBjZV1QpgU7MvSdLYzZmskhwBvBa4AqCqflFVjwNrgA1Nsw3AOZMK\nUpI0RrWAbUpGqayOB34I/GWSW5N8NMmhwNKqerBp8xCwdHdvTrIuyeYkm5/mqfFELUmal10L2e6P\n3YAHAK8EPlxVJwI/ZUaXX1XtMedW1fqqWlVVqw7k4IXGK0laiKqFbVMySrLaBmyrqpua/WsYJK+H\nkywDaH5un0yIkqS+mzNZVdVDwP1JXtIcOgPYAmwE1jbH1gLXTiRCSdJYdbEbcNT7rH4b+HiSg4B7\ngd9ikOiuTnIBcB9w7mRClCSNVfdusxotWVXVbcCq3Zw6Y7zhSJImzRUsJEntVsDO7mUrV12XJLWe\nlZUk9U33CiuTlST1jWNWkqT2219XXZck7T8mfZ9VktVJ7k6yNcnfW+Q8ybuSbElye5JNSV441zVN\nVpKksUmyCLgcOBM4ATg/yQkzmt0KrKqqlzNYFek/zXVdk5Uk9clCVlwfrbI6CdhaVfdW1S+Aqxg8\npeOXIVTdWFU/a3a/Biyf66KOWUlSjwxWXZ/omNUxwP1D+9uAk2dpfwHwv+a6qMlKkvpm54LevSTJ\n5qH99VW1fj4XSvIvGKyO9Lq52pqsJEl745Gq2t3ye7s8ABw7tL+8OfYrkrwB+APgdVU158MOTVaS\n1DMT7ga8GViR5HgGSeo84G2/8vnJicB/BVZX1UiPlzJZSVKfTPjx9FX1TJILgeuBRcCVVXVnkvcB\nm6tqI/AnwHOATycB+H5VnT3bdU1WktQrk3/ib1VdB1w349ilQ6/fsLfXNFlJUs90cbkl77OSJLWe\nlZUk9U0H1wY0WUlSnxRkYfdZTYXJSpL6poOVlWNWkqTWs7KSpL7pXmFlspKkvpnwChYTYbKSpL4x\nWUmSWq1Y6KrrU+EEC0lS61lZSVKPhHLMSpLUASYrSVLrmawkSa3mBAtJkibDykqSesYJFpKk9utg\nspqzGzDJS5LcNrQ9keSiJEcnuSHJPc3Po/ZFwJKkhWgeaz/fbUrmTFZVdXdVrayqlcCrgJ8BnwMu\nBjZV1QpgU7MvSdLY7e0EizOA71TVfcAaYENzfANwzjgDkyRNQNHJympvx6zOAz7ZvF5aVQ82rx8C\nlu7uDUnWAesAFnPIfGKUJI3T/jx1PclBwNnAp2eeq6piD09Iqar1VbWqqlYdyMHzDlSSNB6pmvc2\nLXvTDXgm8I2qerjZfzjJMoDm5/ZxBydJmoAOdgPuTbI6n192AQJsBNY2r9cC144rKEmSho00ZpXk\nUOCNwL8eOvx+4OokFwD3AeeOPzxJ0lgVsLN791mNlKyq6qfAc2cce5TB7EBJUmdMtztvvlzBQpL6\nxmQlSWq9DiYrV12XJLWelZUk9cn+PMFCkrS/KKjuLWFhspKkvnHMSpKk8bOykqQ+ccxKktQJHewG\nNFlJUt+YrCRJ7dbN5ZacYCFJaj0rK0nqkwJ2ep+VJKntOtgNaLKSpL4xWUmS2q06eZ+VEywkSa1n\nZSVJfVJQLmQrSWq9DnYDmqwkqW86OMHCMStJUutZWUlSn1R5U7AkqQM62A1ospKknikrK0lSu7nq\nuiRJE2FlJUl94mPtJUmd4AoWkqQ2K6A6WFk5ZiVJfVI1qKzmu40gyeokdyfZmuTi3Zw/OMmnmvM3\nJTlurmuarCRJY5NkEXA5cCZwAnB+khNmNLsAeKyqXgRcBnxgruuarCSpZ2pnzXsbwUnA1qq6t6p+\nAVwFrJnRZg2woXl9DXBGksx2UZOVJPXNZLsBjwHuH9rf1hzbbZuqegb4MfDc2S66TydYPMljj3yp\nrrkPWAI8si8/ex7aHmPb4wNjHJe2x9j2+KDdMb5wX37Ykzx2/ZfqmiULuMTiJJuH9tdX1fqFxjWX\nfZqsqup5AEk2V9WqffnZe6vtMbY9PjDGcWl7jG2PD7oR475SVasn/BEPAMcO7S9vju2uzbYkBwBH\nAI/OdlG7ASVJ43QzsCLJ8UkOAs4DNs5osxFY27x+K/DlqtnXgPI+K0nS2FTVM0kuBK4HFgFXVtWd\nSd4HbK6qjcAVwF8l2Qr8iEFCm9W0ktXE+zfHoO0xtj0+MMZxaXuMbY8PuhHjfqOqrgOum3Hs0qHX\nPwf++d5cM3NUXpIkTZ1jVpKk1tunyWquJTimIcmVSbYnuWPo2NFJbkhyT/PzqCnHeGySG5NsSXJn\nkne2Lc4ki5N8Pck3mxj/uDl+fLOcytZmeZWDphVjE8+iJLcm+XxL4/tekm8luW3X9OA2fc9NPEcm\nuSbJt5PcleTUNsWY5CXNn9+u7YkkF7UpRu29fZasRlyCYxo+BsycynkxsKmqVgCbmv1pegZ4d1Wd\nAJwCvKP5s2tTnE8Bp1fVK4CVwOokpzBYRuWyZlmVxxgsszJN7wTuGtpvW3wAr6+qlUNTrdv0PQN8\nCPhCVb0UeAWDP8/WxFhVdzd/fiuBVwE/Az7Xphg1D1W1TzbgVOD6of1LgEv21efPEdtxwB1D+3cD\ny5rXy4C7px3jjHivBd7Y1jiBQ4BvACczuBHzgN39HZhCXMsZ/CN1OvB5IG2Kr4nhe8CSGcda8z0z\nuB/muzTj3W2McUZcbwL+ps0xuo227ctuwFGW4GiLpVX1YPP6IWDpNIMZ1qxOfCJwEy2Ls+liuw3Y\nDtwAfAd4vAbLqcD0v/M/BX4P2LVmzHNpV3wweILDF5PckmRdc6xN3/PxwA+Bv2y6Uz+a5FDaFeOw\n84BPNq/bGqNG4ASLOdTgf8NaMWUyyXOAzwAXVdUTw+faEGdV7ahB18tyBotZvnSa8QxL8hZge1Xd\nMu1Y5vCaqnolg+7ydyR57fDJFnzPBwCvBD5cVScCP2VGd1oLYgSgGX88G/j0zHNtiVGj25fJapQl\nONri4STLAJqf26ccD0kOZJCoPl5Vn20Oty5OgKp6HLiRQbfakc1yKjDd7/zVwNlJvsdgFejTGYy9\ntCU+AKrqgebndgbjLCfRru95G7Ctqm5q9q9hkLzaFOMuZwLfqKqHm/02xqgR7ctkNcoSHG0xvBTI\nWgZjRFPTLJ1/BXBXVX1w6FRr4kzyvCRHNq+fzWBM7S4GSeutTbOpxVhVl1TV8qo6jsHfvS9X1W+0\nJT6AJIcmOWzXawbjLXfQou+5qh4C7k/ykubQGcAWWhTjkPP5ZRcgtDNGjWpfDpABZwF/x2As4w+m\nPWDXxPRJ4EHgaQb/13gBg7GMTcA9wJeAo6cc42sYdFncDtzWbGe1KU7g5cCtTYx3AJc2x/8B8HVg\nK4PumINb8J2fBny+bfE1sXyz2e7c9d9Im77nJp6VwObmu/5r4KgWxngog4VRjxg61qoY3fZucwUL\nSVLrOcFCktR6JitJUuuZrCRJrWeykiS1nslKktR6JitJUuuZrCRJrWeykiS13v8DcCwkLhiCIfAA\nAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f2bb7f0a990>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f2bd43a6190>]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlYAAAF1CAYAAAAqdaQaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X+8XHV95/HXh4SA8isK0QIBAhVd0/oDGlG3VqlYBbTQ\ndm0L1aIuLXVbaF3tWqiuIup2/VHd6rJVrFbEKiJt3bTGRVpBaxUkFEEBwRB+JAgkhB8FgsSEz/5x\nvpcMc+/NzJ37nTs/7uv5eNxHZs458z3f+d6ZzHvO+Z7PjcxEkiRJs7fToDsgSZI0LgxWkiRJlRis\nJEmSKjFYSZIkVWKwkiRJqsRgJUmSVInBSmoREcsiIiNi4aD7MioiYpeIuC4i9i33nxAR/xAR90fE\nFyu0/wsRccPsezp8IuLIiFg/6H5MGLb+AETEaRHxvkH3Q+qWwUrzUkTcEhEPR8SDLT/7zbCNSR9C\nEXFmRHy2bm8f1/4tEfGyWTz+3RHxvYjYGhFntq17fURsaxmPmyPiryPi6R2aPQX4RmbeUe6/Gngq\nsHdm/nqvfZ2Qmf+Smc+YbTvdKKH6aTtY//qI+OZc9EWP+QTwmoh4yqA7InXDYKX57Jczc/eWnx8N\nukNzYA3wVuDL06z/dmbuDuwFvAx4GLgyIn52B22+ETiv5f5BwI2ZuXWmnfNI4fzT6XeemT8GvgKc\nNDc9kmbHYCXtQES8ISKuj4gHImJtRPxeWb4bzX/2+7Uc4fkt4E+B3yz3ry7b7hURn4yIOyLi9oh4\nT0QsKOteHxHfjIgPRsS95SjRMdP05TzgQOAfSvtvLcuPi4hrI+K+iLg0Ip453fPJzHMz8yvAAzt6\n3pm5LTNvyszfB74OnDlNnw4EDgEuL/ffBbyjZQxOjoidIuLtEXFrRGyIiM9ExF5l+4lTrydHxG3A\n16bYx+OODJajdn8cEdeU041fiIhdW7eNiD+NiLvLtq9peeylEfE7LfcfOwIVEd8oi68uff/Ntn48\nE/gY8MKy/r6yfK/ynDaW5/j2iJjy/9ZymvTT5Xd9HfC8tvX7RcTflrZujog/bFl3ZkRcUPb1QPmd\nr2hZ/yfl9fVARNwQEUeV5TtFxOkRcVNEbCptPHmq/k3R34nHPRDN6d5fLcsXRcQ9EfGslm2fEhGb\nI2JJuf+qiPhueV1+KyKe3fY7/JOIuAZ4KCIWTtf/4lLgld30WRo0g5W0YxuAVwF7Am8APhwRh2fm\nQ8AxwI9ajnh9DvgfwBfK/eeUNj4NbAWeBhwGvBz4nZZ9PB+4AdgHeD/wyYiI9o5k5m8Dt7H9SNv7\nozlN93ngTcASYBVN8FpUcQz+DviFadY9C1g7cXQqM9/J48fgk8Dry88v0oSw3YH/3dbOS4BnAq/o\nsk+/ARwNHAw8u7Q/4adoxnJ/4HXAORHR8VRiZr643HxO6fsX2tZfT3N07ttl/eKy6qM0R/gOKc/j\nJJrXylTeCfx0+XlF6R/QBCDgH4CrS9+PAt4UEa1jchxwPrAYWEkZx/L8TgWel5l7lLZvKY85DfiV\n0rf9gHuBszuNR3ETze9+L+BdwGcjYt/M3FL68dqWbU8E/jkzN0bEYcCngN8D9gY+DqyMiF3atn9l\neS4/vYP+A1wPPAdpBBisNJ99qXybvi8ivjTVBpn55XLkJjPz68BXmT5kTBIRTwWOBd6UmQ9l5gbg\nw8AJLZvdmpmfyMxtwLnAvjRzlLrxm8CXM/PizPwJ8EHgCcB/7LaPXfgRMN0RjsV0OPoFvAb4UGau\nzcwHgTOAE+Lxp4DOLOPzcJd9+khm/igz76EJI89tW//fM/OR8jv7Mk0Qqy6aI48nAGdk5gOZeQvw\n58BvT/OQ3wDem5n3ZOY64CMt654HLMnMszJzS2aupZlf1Ppa+WZmriqvlfPYHja2AbsAyyNi58y8\nJTNvKuveCLwtM9dn5iM0Rx9fHV2cds3ML5ZxfrQEzR8CR5TV5wIntnwJ+G22nxI+Bfh4Zl5ejn6e\nCzwCvKCl+Y9k5rryO99R/6F5je3Vqb/SMHA+g+azX8nMf9rRBtGclnsn8HSaLyJPBL43g30cBOwM\n3NFyEGonYF3LNndO3MjMzWW73btsfz/g1pbHPxoR62iOeNSyP3DPNOvuBfbo8PjH9bHcXsjjw+M6\nZubOltubyz4e61M5oti6vxldmDAD+9D8ftuf33Tjvx+Pf66tjzuI5tTyfS3LFgD/0nK//XnvGhEL\nM3NNRLyJJjT9TERcBLy5zBs8CPj7iHi05bHbaMb/9h09uYg4CXgzsKws2p3mOZOZl0fEZuDIiLiD\n5ojsypbn8rqIOK2luUU8/vfw2Dh06D80r7H7d9RXaVh4xEqaRjlt8bc0R4GeWk79rAImElJO8bD2\nZetovqnvk5mLy8+emfkzPXarvf2JD86JPgdwAB0+MGfoV3n8h3ura4CDOxz9eFwfaeaJbQXualk2\n1Vj26knRzIFr3d/EB/RDNOF4wk/NsO32ft4N/ITJz2+68b+D5vfTuu2EdcDNLa+TxZm5R2Ye21XH\nMj+XmS8qfUlgokTBOuCYtnZ3zcxOoeogmiNmp9Jc4bkY+D7bX//QHLV6Lc3RqgvLRPOJfb63bZ9P\nzMzPt3a5y/5Dc5r46m7GQRo0g5U0vUU0pyc2AlvL0auXt6y/C9g7ykTslmXLJiYvlxIEXwX+PCL2\nLBOJfzoiXtJjn+6imcsz4QLglRFxVETsDLyFJsh9a6oHR8TO0Uz03glYGBG7ltNZ7dstiIiDI+Kj\nwJE082smycz1NFcaHjHV+uLzwH8t7e3O9jlYM75qcAbeVSZY/wLNHLmJelrfBX4tIp4YTVmFk9se\n1z6+7e4Clk7MYSun5C4A3hsRe5Qw8mZgupIbFwBnRMSTImIpzfynCd8BHiiTuJ9Qfgc/GxHPm7qp\n7SLiGRHx0vJl4Mc0V3NOHKH6WOnfQWXbJRFxfKc2gd1oAs7G8rg3AO1Xh36WJni/FvhMy/JPAG+M\niOdHY7eIeGVETHl0s0P/oZkf9pUu+iwNnMFKmkZmPgD8Ic2H4b3Ab7H9VAeZ+QOa0LC2zNPaj+0f\n4Jsi4t/K7ZNoQtp1pZ0LaeZR9eLPgLeX/f1xZt5A86H2UZqjJ79MM7l9yzSP/wTNh9aJwNvK7db5\nQC+MiAeBf6e5EmtPmgnFOzr9+XGmn1MEzSTm84BvADfTfHCetoPtZ+tOmnH+EfA3wBvL7wqa+W1b\naALSuWV9qzOBc8v4TjUv62vAtcCdEXF3WXYazZGwtcA3gc/RPOepvIvm9N/NNIH7sTIVJaS9ima+\n2M00v8+/oru5RbsA/7M85k7gKTRz2QD+guZ1+9WIeAC4jOaCiR3KzOto5ot9m2a8ngX8a9s264B/\nowlg/9KyfDXwuzST6++lCd+v76X/5YvAsTS/L2noRWbNI/CS5ptylOEq4KjcXiR0UH05EvhsZi4d\nZD/mk4j4FM3VsW/vU/unAQdk5lv70b5Um5PXJc1KudJs+aD7obkXEcuAX6MpI9IXmfnRfrUt9YOn\nAiVJMxYR76aZzP6BzLx50P2RhoWnAiVJkirxiJUkSVIlBitJkqRKBjZ5fZ999slly5YNaveSJEld\nu/LKK+/OzCWdthtYsFq2bBmrV68e1O4lSZK6FhG3dt7KU4GSJEnVGKwkSZIqMVhJkiRVYrCSJEmq\nxGAlSZJUicFKkiSpEoOVJElSJQYrSZKkSgxWkiRJlXQMVhHxqYjYEBHfn2Z9RMRHImJNRFwTEYfX\n76YkSdLw6+aI1aeBo3ew/hjg0PJzCvCXs++WJEnS6OkYrDLzG8A9O9jkeOAz2bgMWBwR+9bq4Kh6\neMs21t+7ecp1azY8SGbOcY+kqd1y90P8ZNujk5bf89AWNj34yKTlW7c9ys13PzRlW7dueogtWye3\nde9DW7h7ira2PZqs3fjglG3dtmkzj2zdNmn5fZu3sPGByW1J0jCoMcdqf2Bdy/31ZdkkEXFKRKyO\niNUbN26ssOvh9YZPf4cXve+SScuvvPVeXvahr/Ppb90y952S2tz17z/myA9eynv+8bpJ6w5/98X8\n3Hv+adLyP/vKD/jFD17K7fc9/Ljl923ewks+cClv+/vvTXrMYe++mBVTtPXhi2/kpX/+9Unh6uEt\n23jxBy7hLRdcPekxzz3rYp733sltSdIwmNPJ65l5TmauyMwVS5Ysmctdz7nL1k59kO/WTc03/WvW\n3z+X3ZGmdO/mLcD0r9epXH7zJgDueXDL45Y/+MhWAL5106au2/rOLc1+N7QdgZo46vWNG8f7C5ik\n8VMjWN0OHNByf2lZJkmSNK/UCFYrgZPK1YEvAO7PzDsqtCtJkjRSFnbaICI+DxwJ7BMR64F3AjsD\nZObHgFXAscAaYDPwhn51VpIkaZh1DFaZeWKH9Qn8QbUeSZIkjSgrr0uSJFVisBoQ61hJkjR+DFZz\nLGLQPZAkSf1isJIkSarEYCVJklSJwUqSJKkSg5UkSVIlBitJkqRKDFaSSLov/9GPSiHtbc6kP5I0\nTAxWkiRJlRisJBF0X2CtH7XY2tucSX8kaZgYrCRJkioxWEmSJFVisJIkSarEYCVJklSJwUqSJKkS\ng9WAWKVHkqTxY7CaY15GLknS+DJYSZIkVWKwkiRJqsRgJUmSVInBSpIkqRKDlSRJUiUGqwHx2kAN\nk5xBAZDsQ62Q9jZn0h9JGiYGqwHxY0OSpPFjsJI0o/pq0YfDre1tWu9N0qgyWEmSJFVisJIkSarE\nYCVJklSJwUqSJKkSg5Ukyy1IUiUGqwHpx4eTJEkaLIPVHOvHperSbFluQZLqMFhJkiRVYrCSJEmq\nxGAlSZJUicFKkiSpEoPVgDiJXcPEcguSVIfBakAstyBJ0vgxWEmy3IIkVWKwkiRJqsRgJUmSVInB\nSpIkqRKDlSRJUiUGqz5LL//TCKhZbqGX17zlFiSNC4OVJElSJQarAfH7uMZVWP1W0jxmsJJkHStJ\nqsRgJUmSVInBSpIkqZKuglVEHB0RN0TEmog4fYr1B0bEJRFxVURcExHH1u+qJEnScOsYrCJiAXA2\ncAywHDgxIpa3bfZ24ILMPAw4Afg/tTs6bpxBIknS+OnmiNURwJrMXJuZW4DzgePbtklgz3J7L+BH\n9bo42ixjpVFQs45VT/u3jpWkMbGwi232B9a13F8PPL9tmzOBr0bEacBuwMuq9G6M+bEhSdL4qTV5\n/UTg05m5FDgWOC8iJrUdEadExOqIWL1x48ZKu5Y0W5ZbkKQ6uglWtwMHtNxfWpa1Ohm4ACAzvw3s\nCuzT3lBmnpOZKzJzxZIlS3rrsSRJ0pDqJlhdARwaEQdHxCKayekr27a5DTgKICKeSROsPCQlSZLm\nlY7BKjO3AqcCFwHX01z9d21EnBURx5XN3gL8bkRcDXweeH3614clSdI8083kdTJzFbCqbdk7Wm5f\nB/x83a5JkiSNFiuv95mH7TQKLLcgSXUYrKR5bDZX3026kq+HywU7PaKXNiVpkAxWA+IUNA2D2RwZ\nmnSUqYfXdKdH+D6RNGoMVnPMb+AaRsNWx0qSRpXBSpIkqRKDlSRJUiUGK0mSpEoMVn023eRb51pp\nmAxduQXnrEsaUQYraR6z3IIk1WWwGhAvI9cwsNyCJNVlsJJkuQVJqsRgJUmSVInBSpIkqRKDlSRJ\nUiUGqz5z6q1GQc1yCz1NYm+fCD/jFiRpOBisJEmSKjFYDYjfyDWurD0laT4zWEmSJFVisJpjfpfX\nMLKOlSTVYbCSJEmqxGAlSZJUicFqQDzzIUnS+DFY9Zl/Q1ajoGYdq572X+EPOkvSMDBYSfPYTCat\nT3pstN+feVudHmHpBkmjxmA1IH4f1zCYyZGqSY+tcJSp0yM8ciVp1BisJFluQZIqMVhJkiRVYrCS\nJEmqxGAlSZJUicGqz2YzOViaK0NXbqH+LiRpThispHnMcguSVJfBalD8Sq4hYLkFSarLYDXH/AKu\nYWS5BUmqw2AlSZJUicFKkiSpEoPVoHjqQ5KksWOw6jPn3moUDF25Bd83kkaUwUqaxyy3IEl1Gazm\n2GPfxP1GriFguQVJqstgJUmSVInBao55ZkPDyDpWklSHwUqSJKkSg5UkSVIlBitJw1duwas7JI0o\ng5U0j1luQZLqMlhJkiRVYrAaEE91aBhYx0qS6jJYzbHZnHqR+sVyC5JUh8FKkiSpEoPVgHjkSpKk\n8WOw6jOniGgU1Cy30NNcq/aH+L6RNKK6ClYRcXRE3BARayLi9Gm2+Y2IuC4iro2Iz9XtpiRJ0vBb\n2GmDiFgAnA38ErAeuCIiVmbmdS3bHAqcAfx8Zt4bEU/pV4cl1VOzjtX25TOYCN9xH54ylzRaujli\ndQSwJjPXZuYW4Hzg+LZtfhc4OzPvBcjMDXW7OT4mTrlYbkHDoG65hV7232kfvk8kjZZugtX+wLqW\n++vLslZPB54eEf8aEZdFxNFTNRQRp0TE6ohYvXHjxt56LEmSNKRqTV5fCBwKHAmcCHwiIha3b5SZ\n52TmisxcsWTJkkq7Hi1eDahhZB0rSaqjm2B1O3BAy/2lZVmr9cDKzPxJZt4M3EgTtCRJkuaNboLV\nFcChEXFwRCwCTgBWtm3zJZqjVUTEPjSnBtdW7KckSdLQ6xisMnMrcCpwEXA9cEFmXhsRZ0XEcWWz\ni4BNEXEdcAnw3zJzU786PUqcpK5RULOOVU/7b58IX38XkjQnOpZbAMjMVcCqtmXvaLmdwJvLj6QR\n0Y9yCzPbf6d9OPlK0mix8rokSVIlBqsBsTyPhkHNOla97b/TPnyjSBotBqs55pkNDSPLLUhSHQar\nAfGDRJKk8WOwkiRJqsRg1WdOEdEoGLpyC75vJI0og5U0j1luQZLqMlhJkiRVYrAaEE91aBjULLfQ\ny2vacguSxo3BSpLlFiSpEoOVJElSJQYrSZKkSgxWfeYMEY2CoSu34DtH0ogyWEnzmOUWJKkug5Uk\nSVIlBqsB8SpyDYOq5RZ6aMtyC5LGjcFKkiSpEoPVHJuYMeLUEQ0T61hJUh0GK0mSpEoMVn3mHBGN\ngprlFnp5zdf48ziSNAwMVpIkSZUYrKR5rB91rGZSe8o6VpLGjcFKkiSpEoPVgDiHRMOgah2rHpqy\njpWkcWOwkmS5BUmqxGAlSZJUicGqzzyRoVEwdOUWZtyCJA0Hg5UkSVIlBitpHrPcgiTVZbCSJEmq\nxGAlzWNVyy30tP9O+3C2laTRYrCSJEmqxGA1x5wyomFkHStJqsNgJUmSVInBqs+cIqJRULOOVU/7\nn/TncXzjSBpNBitpHutHuYWZ7b/TPjxHKGm0GKwkSZIqMVhJkiRVYrAakNnUD5JqqVrHqpe/Edhx\nH75PJI0Wg5Ukyy1IUiUGK0mSpEoMVv3mmQyNgOErt1B/H5I0FwxWknpiuQVJmsxgJUmSVInBSpIk\nqRKDlTSPzWYu06R5Ub200XEfTraSNFoMVnPOOSMaPpZbkKQ6DFaSJEmVGKz6zArrGgU1yy30VIHd\nt4mkMWGwkiRJqqSrYBURR0fEDRGxJiJO38F2/ykiMiJW1OuipGE03byomdSeso6VpHHTMVhFxALg\nbOAYYDlwYkQsn2K7PYA/Ai6v3UlJkqRR0M0RqyOANZm5NjO3AOcDx0+x3buB9wE/rti/MdRMJnFO\niYZB1XILPbRluQVJ46abYLU/sK7l/vqy7DERcThwQGZ+eUcNRcQpEbE6IlZv3Lhxxp2VJEkaZrOe\nvB4ROwEfAt7SadvMPCczV2TmiiVLlsx21yPKOSMaPtaxkqQ6uglWtwMHtNxfWpZN2AP4WeDSiLgF\neAGw0gnsDc9kaBQMW7kF3zeSRlU3weoK4NCIODgiFgEnACsnVmbm/Zm5T2Yuy8xlwGXAcZm5ui89\nliRJGlIdg1VmbgVOBS4CrgcuyMxrI+KsiDiu3x2UNJwstyBJky3sZqPMXAWsalv2jmm2PXL23ZIk\nSRo9Vl6XJEmqxGAlzWOz+VuWkyeY9zBpveM+nMUuabQYrOaYU0Y0jCy3IEl1GKwkSZIqMVj1mScy\nNApq1rHqaf/tdax850gaUQYrSQNjuQVJ48ZgJUmSVInBSpIkqRKD1RybmEviDBINg9nMl2qfB9VL\nW5ZbkDRuDFaSJEmVGKzmmHNxNYysYyVJdRis+sxTGRoFQ1duwbeNpBFlsJI0MJZbkDRuDFaSJEmV\nGKwkSZIqMVgNiCc4JEkaPwarAXFurkbd5L/v10MbHffhO0XSaDFYzTGPVGkYWW5BkuowWPWZ37c1\nCoau3EL9XUjSnDBY9YnfwKXOLLcgadwYrCRJkioxWEmSJFVisJIkSarEYNUn003wzQ7rpbk0m9dh\n+0N7actyC5LGjcFKkiSpEoNVn7V/4fYaJ427GkeZPFIlaVQZrCRJkioxWPWJ5Xc0X82k9pR1rCSN\nG4OVJElSJQarAfGLuCRJ48dgNSDOzdUwmMnfCJz02LYXcS9tWW5B0rgxWEmSJFVisOqz9m/xTsbV\nuKtSbqFCPyRpEAxWkiRJlRis+sTjUpqvLLcgaT4zWEmSJFVisJIkSarEYCVJklSJwapPOl/V5HVP\nGrzZXMDX/tBe2rKOlaRxY7CSRMzgcot+zCd3jrqkcWGw6je/cGsEzKRqej8OIrW36YEqSaPKYNUn\nfgGXOrPcgqRxY7CSJEmqxGA1MH4TlyRp3BisJEmSKjFYDYyzczV4s3kV1phwbrkFSePGYCVJklSJ\nwarP2r9vO7NK6oZHqiSNJoNVn3iZuNSZ5RYkjRuDlSRJUiVdBauIODoiboiINRFx+hTr3xwR10XE\nNRHxzxFxUP2uSpIkDbeOwSoiFgBnA8cAy4ETI2J522ZXASsy89nAhcD7a3dUkiRp2HVzxOoIYE1m\nrs3MLcD5wPGtG2TmJZm5udy9DFhat5uSJEnDr5tgtT+wruX++rJsOicDX5lNp8ZBp/o7lufRMJhd\nnahsuzfztqxjJWncLKzZWES8FlgBvGSa9acApwAceOCBNXc9tPxc0HzTSxhqD2W+bySNqm6OWN0O\nHNByf2lZ9jgR8TLgbcBxmfnIVA1l5jmZuSIzVyxZsqSX/kqSJA2tboLVFcChEXFwRCwCTgBWtm4Q\nEYcBH6cJVRvqd3P0WH9H89VMXvvWsZI0bjoGq8zcCpwKXARcD1yQmddGxFkRcVzZ7APA7sAXI+K7\nEbFymuZU+HkhSdL46WqOVWauAla1LXtHy+2XVe6XJEnSyLHyuiRJUiUGqwHxqicNg1kVW8gd36+x\nf8stSBo1Bqs+66W2jzTKegpD7SGtTlckac4ZrOaYk9YlSRpfBqs+MT9pvrLcgqT5zGAlSZJUicFK\nkiSpEoOVJElSJQarPpnuqqaJC6a86knDYDbVDGq8hi23IGncGKz6zM8FzTe9hKH2R/i+kTSqDFZz\nzIucJEkaXwarPjE/ab6y3IKk+cxgNSB+XEiSNH4MVpIkSZUYrCRJkioxWEmSJFVisBoQrybXcOj9\nldheEqGXEgnWsZI0bgxWfdb+seBFTtJkk0KaXz0kjSiDVZ8YoKTOLLcgadwYrCRJkioxWEmSJFVi\nsJIkSarEYCVJklSJwapPprtKfGK5l5FrGMzmZdj+Gu7lSj7LLUgaNwarPvODQfNNL6/59lDm20bS\nqDJYzTGvHpckaXwZrPqkU4CyPo/G1Uxe29axkjRuDFaSJEmVGKwkSZIqMVhJkiRVYrCSJEmqxGDV\nZ9PXs/J6cg3ebF6F7Y/tpUbbxJbtD7Hem6RRZbCaY9HxOihJkjSqDFZ9YoDSfGW5BUnzmcFKkiSp\nEoOVJElSJQYrSZKkSgxWkiRJlRis+iSnuZB9YrkXkWsYbC9rMPvHbC+d0EO5hUnLc8ZtSdIwMFhJ\nkiRVYrDqk+nKLUws9yJyDZOZVDXotK3lFiTNZwYrSZKkSgxWkiRJlRisJEmSKjFYSZIkVWKwGhAv\nItcwmE05g/aSIr201ekRlluQNGoMVn3m54LUWXuA8n0jaVQZrPpluqvEvXpceozlFiSNG4OVJElS\nJQYrSZKkSgxWkiRJlXQVrCLi6Ii4ISLWRMTpU6zfJSK+UNZfHhHLandUkiRp2HUMVhGxADgbOAZY\nDpwYEcvbNjsZuDcznwZ8GHhf7Y5KkiQNu+hUJyYiXgicmZmvKPfPAMjMP2vZ5qKyzbcjYiFwJ7Ak\nd9D4ihUrcvXq1RWewtQuW7uJ9/2/H/St/U6uuu0+AJbvuye77Lw9v256cAu33bMZgMMOXDyQvkkT\nHvzxVn644UFg8utx4jU83fKnPWV39th14WPLH3pkKzfe1VtbhyzZjb2esPNjyx/eso0f3PnAjNqS\nNH+d9MKD+NXDlvZ1HxFxZWau6LTdwk4bAPsD61rurweeP902mbk1Iu4H9gbubuvUKcApAAceeGAX\nu+7dzguC3Xfp5un1x7K9n8gtmzaz9+6LHrd8910Wcts9m3nOAYsH2j8JmtfjDzc8yLP232vS63HB\nTsGjmZOWP2fpXly9/n723WvXSW3deNeDLN93z0mPWbRwJ7ZsfXTS8sMOXMxVt93H/oufMKmtH9z5\nAM946h6THvPERQvYvGWb7x9Jj9l5wfBMGZ/T/5ky8xzgHGiOWPVzXz930JM57+T2/CdJktQ/3US8\n24EDWu4vLcum3KacCtwL2FSjg5IkSaOim2B1BXBoRBwcEYuAE4CVbdusBF5Xbr8a+NqO5ldJkiSN\no46nAsucqVOBi4AFwKcy89qIOAtYnZkrgU8C50XEGuAemvAlSZI0r3Q1xyozVwGr2pa9o+X2j4Ff\nr9s1SZKk0TI80+glSZJGnMFKkiSpEoOVJElSJQYrSZKkSgxWkiRJlRisJEmSKjFYSZIkVWKwkiRJ\nqsRgJUmSVEkM6k/6RcRG4NY+72Yf4O4+72OcOX69c+x659jNjuPXO8eud/Nh7A7KzCWdNhpYsJoL\nEbE6M1cMuh+jyvHrnWPXO8dudhy/3jl2vXPstvNUoCRJUiUGK0mSpErGPVidM+gOjDjHr3eOXe8c\nu9lx/Hrn2PXOsSvGeo6VJEnSXBr3I1aSJElzZmyDVUQcHRE3RMSaiDh90P0ZBhHxqYjYEBHfb1n2\n5Ii4OCIjG1zcAAAEkElEQVR+WP59UlkeEfGRMn7XRMThLY95Xdn+hxHxukE8l7kWEQdExCURcV1E\nXBsRf1SWO34dRMSuEfGdiLi6jN27yvKDI+LyMkZfiIhFZfku5f6asn5ZS1tnlOU3RMQrBvOM5l5E\nLIiIqyLiH8t9x65LEXFLRHwvIr4bEavLMt+3XYiIxRFxYUT8ICKuj4gXOnZdyMyx+wEWADcBhwCL\ngKuB5YPu16B/gBcDhwPfb1n2fuD0cvt04H3l9rHAV4AAXgBcXpY/GVhb/n1Suf2kQT+3ORi7fYHD\ny+09gBuB5Y5fV2MXwO7l9s7A5WVMLgBOKMs/BvyXcvv3gY+V2ycAXyi3l5f38i7AweU9vmDQz2+O\nxvDNwOeAfyz3Hbvux+4WYJ+2Zb5vuxu7c4HfKbcXAYsdu84/43rE6ghgTWauzcwtwPnA8QPu08Bl\n5jeAe9oWH0/z5qH8+ystyz+TjcuAxRGxL/AK4OLMvCcz7wUuBo7uf+8HKzPvyMx/K7cfAK4H9sfx\n66iMwYPl7s7lJ4GXAheW5e1jNzGmFwJHRUSU5edn5iOZeTOwhua9PtYiYinwSuCvyv3AsZst37cd\nRMReNF/GPwmQmVsy8z4cu47GNVjtD6xrub++LNNkT83MO8rtO4GnltvTjeG8H9tyeuUwmiMvjl8X\nyqms7wIbaP5jvQm4LzO3lk1ax+GxMSrr7wf2Zp6OHfC/gLcCj5b7e+PYzUQCX42IKyPilLLM921n\nBwMbgb8up6H/KiJ2w7HraFyDlXqQzXFbLxPdgYjYHfhb4E2Z+e+t6xy/6WXmtsx8LrCU5kjJfxhw\nl0ZCRLwK2JCZVw66LyPsRZl5OHAM8AcR8eLWlb5vp7WQZurIX2bmYcBDNKf+HuPYTW1cg9XtwAEt\n95eWZZrsrnK4lvLvhrJ8ujGct2MbETvThKq/ycy/K4sdvxkopxIuAV5Ic6pgYVnVOg6PjVFZvxew\nifk5dj8PHBcRt9BMaXgp8Bc4dl3LzNvLvxuAv6cJ9r5vO1sPrM/My8v9C2mClmPXwbgGqyuAQ8uV\nM4toJnGuHHCfhtVKYOIqjdcB/7dl+UnlSo8XAPeXw78XAS+PiCeVq0FeXpaNtTJP5ZPA9Zn5oZZV\njl8HEbEkIhaX208AfolmjtolwKvLZu1jNzGmrwa+Vr4ZrwROKFe+HQwcCnxnbp7FYGTmGZm5NDOX\n0fw/9rXMfA2OXVciYreI2GPiNs377fv4vu0oM+8E1kXEM8qio4DrcOw6G/Ts+X790FyhcCPNXI63\nDbo/w/ADfB64A/gJzbeRk2nmX/wz8EPgn4Anl20DOLuM3/eAFS3t/Geaya9rgDcM+nnN0di9iOaQ\n9zXAd8vPsY5fV2P3bOCqMnbfB95Rlh9C8+G+BvgisEtZvmu5v6asP6SlrbeVMb0BOGbQz22Ox/FI\ntl8V6Nh1N2aH0FwNeTVw7cRnge/brsfvucDq8t79Es1VfY5dhx8rr0uSJFUyrqcCJUmS5pzBSpIk\nqRKDlSRJUiUGK0mSpEoMVpIkSZUYrCRJkioxWEmSJFVisJIkSark/wPLUkKISDeMMAAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f2bd47a6a10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# get test data\n",
    "env = gym.make(\"Pong-v0\")\n",
    "I = env.reset()\n",
    "\n",
    "plt.title('video game screenshot')\n",
    "plt.imshow(I)\n",
    "plt.colorbar()\n",
    "plt.show()\n",
    "\n",
    "I = I[35:195] # crop\n",
    "plt.title('crop')\n",
    "plt.imshow(I)\n",
    "plt.colorbar()\n",
    "plt.show()\n",
    "\n",
    "I = I[::2,::2,0] # downsample by factor of 2\n",
    "plt.title('downsample')\n",
    "plt.imshow(I)\n",
    "plt.colorbar()\n",
    "plt.show()\n",
    "\n",
    "I[I == 144] = 0 # erase background (background type 1)\n",
    "I[I == 109] = 0 # erase background (background type 2)\n",
    "I[I != 0] = 1 # everything else (paddles, ball) just set to 1\n",
    "plt.title('erase')\n",
    "plt.imshow(I)\n",
    "plt.colorbar()\n",
    "plt.show()\n",
    "\n",
    "\n",
    "I=I.astype(np.float).ravel()\n",
    "plt.title('Flatten to 1D (for input to dense layers)')\n",
    "plt.plot(I)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-12T23:48:36.445956Z",
     "start_time": "2017-07-13T07:48:36.442517+08:00"
    }
   },
   "source": [
    "# Environments\n",
    "\n",
    "\n",
    "```python\n",
    "class Environment\n",
    "        def step (self, action='move_left'):\n",
    "            ...\n",
    "            return observation # e.g. screenshot\n",
    "            \n",
    "env = Environment()\n",
    "env.reset()\n",
    "screenshot = env.step('left')\n",
    "```\n",
    "\n",
    "We use the populat OpenAI Gym\n",
    "- prototype: https://github.com/openai/gym/blob/master/gym/core.py#L13\n",
    "- docs: https://gym.openai.com/docs#Environments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-13T11:16:26.591428Z",
     "start_time": "2017-07-13T19:16:26.395136+08:00"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-07-13 19:16:26,396] Making new env: Pong-v0\n"
     ]
    }
   ],
   "source": [
    "env = gym.make(\"Pong-v0\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-12T23:30:51.976941Z",
     "start_time": "2017-07-13T07:30:51.974816+08:00"
    }
   },
   "source": [
    "# Train <80 lines of code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-13T11:16:42.276129Z",
     "start_time": "2017-07-13T19:16:42.273962+08:00"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "render = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-13T11:16:42.493307Z",
     "start_time": "2017-07-13T19:16:42.490068+08:00"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "prev_x = None # used in computing the difference frame\n",
    "xs,hs,dlogps,drs = [],[],[],[]\n",
    "running_reward = None\n",
    "reward_sum = 0\n",
    "episode_number = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "start_time": "2017-07-13T11:16:42.670Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ep 0: game finished, reward: -1.000000\n",
      "ep 0: game finished, reward: -1.000000\n",
      "ep 0: game finished, reward: -1.000000\n",
      "ep 0: game finished, reward: -1.000000\n",
      "ep 0: game finished, reward: -1.000000\n",
      "ep 0: game finished, reward: -1.000000\n",
      "ep 0: game finished, reward: -1.000000\n",
      "ep 0: game finished, reward: -1.000000\n",
      "ep 0: game finished, reward: -1.000000\n",
      "ep 0: game finished, reward: -1.000000\n",
      "ep 0: game finished, reward: -1.000000\n",
      "ep 0: game finished, reward: -1.000000\n",
      "ep 0: game finished, reward: -1.000000\n",
      "ep 0: game finished, reward: -1.000000\n",
      "ep 0: game finished, reward: -1.000000\n",
      "ep 0: game finished, reward: -1.000000\n",
      "ep 0: game finished, reward: -1.000000\n",
      "ep 0: game finished, reward: -1.000000\n",
      "ep 0: game finished, reward: -1.000000\n",
      "ep 0: game finished, reward: -1.000000\n",
      "resetting env. episode 1 reward total was -21.000000. running mean: -21.000000\n",
      "ep 1: game finished, reward: -1.000000\n",
      "ep 1: game finished, reward: -1.000000\n",
      "ep 1: game finished, reward: -1.000000\n",
      "ep 1: game finished, reward: -1.000000\n",
      "ep 1: game finished, reward: -1.000000\n",
      "ep 1: game finished, reward: -1.000000\n",
      "ep 1: game finished, reward: -1.000000\n",
      "ep 1: game finished, reward: -1.000000\n",
      "ep 1: game finished, reward: -1.000000\n",
      "ep 1: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 1: game finished, reward: -1.000000\n",
      "ep 1: game finished, reward: -1.000000\n",
      "ep 1: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 1: game finished, reward: -1.000000\n",
      "ep 1: game finished, reward: -1.000000\n",
      "ep 1: game finished, reward: -1.000000\n",
      "ep 1: game finished, reward: -1.000000\n",
      "ep 1: game finished, reward: -1.000000\n",
      "ep 1: game finished, reward: -1.000000\n",
      "ep 1: game finished, reward: -1.000000\n",
      "ep 1: game finished, reward: -1.000000\n",
      "ep 1: game finished, reward: -1.000000\n",
      "ep 1: game finished, reward: -1.000000\n",
      "resetting env. episode 2 reward total was -19.000000. running mean: -20.980000\n",
      "ep 2: game finished, reward: -1.000000\n",
      "ep 2: game finished, reward: -1.000000\n",
      "ep 2: game finished, reward: -1.000000\n",
      "ep 2: game finished, reward: -1.000000\n",
      "ep 2: game finished, reward: -1.000000\n",
      "ep 2: game finished, reward: -1.000000\n",
      "ep 2: game finished, reward: -1.000000\n",
      "ep 2: game finished, reward: -1.000000\n",
      "ep 2: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 2: game finished, reward: -1.000000\n",
      "ep 2: game finished, reward: -1.000000\n",
      "ep 2: game finished, reward: -1.000000\n",
      "ep 2: game finished, reward: -1.000000\n",
      "ep 2: game finished, reward: -1.000000\n",
      "ep 2: game finished, reward: -1.000000\n",
      "ep 2: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 2: game finished, reward: -1.000000\n",
      "ep 2: game finished, reward: -1.000000\n",
      "ep 2: game finished, reward: -1.000000\n",
      "ep 2: game finished, reward: -1.000000\n",
      "ep 2: game finished, reward: -1.000000\n",
      "ep 2: game finished, reward: -1.000000\n",
      "ep 2: game finished, reward: -1.000000\n",
      "resetting env. episode 3 reward total was -19.000000. running mean: -20.960200\n",
      "ep 3: game finished, reward: -1.000000\n",
      "ep 3: game finished, reward: -1.000000\n",
      "ep 3: game finished, reward: -1.000000\n",
      "ep 3: game finished, reward: -1.000000\n",
      "ep 3: game finished, reward: -1.000000\n",
      "ep 3: game finished, reward: -1.000000\n",
      "ep 3: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 3: game finished, reward: -1.000000\n",
      "ep 3: game finished, reward: -1.000000\n",
      "ep 3: game finished, reward: -1.000000\n",
      "ep 3: game finished, reward: -1.000000\n",
      "ep 3: game finished, reward: -1.000000\n",
      "ep 3: game finished, reward: -1.000000\n",
      "ep 3: game finished, reward: -1.000000\n",
      "ep 3: game finished, reward: -1.000000\n",
      "ep 3: game finished, reward: -1.000000\n",
      "ep 3: game finished, reward: -1.000000\n",
      "ep 3: game finished, reward: -1.000000\n",
      "ep 3: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 3: game finished, reward: -1.000000\n",
      "ep 3: game finished, reward: -1.000000\n",
      "ep 3: game finished, reward: -1.000000\n",
      "ep 3: game finished, reward: -1.000000\n",
      "resetting env. episode 4 reward total was -19.000000. running mean: -20.940598\n",
      "ep 4: game finished, reward: -1.000000\n",
      "ep 4: game finished, reward: -1.000000\n",
      "ep 4: game finished, reward: -1.000000\n",
      "ep 4: game finished, reward: -1.000000\n",
      "ep 4: game finished, reward: -1.000000\n",
      "ep 4: game finished, reward: -1.000000\n",
      "ep 4: game finished, reward: -1.000000\n",
      "ep 4: game finished, reward: -1.000000\n",
      "ep 4: game finished, reward: -1.000000\n",
      "ep 4: game finished, reward: -1.000000\n",
      "ep 4: game finished, reward: -1.000000\n",
      "ep 4: game finished, reward: -1.000000\n",
      "ep 4: game finished, reward: -1.000000\n",
      "ep 4: game finished, reward: -1.000000\n",
      "ep 4: game finished, reward: -1.000000\n",
      "ep 4: game finished, reward: -1.000000\n",
      "ep 4: game finished, reward: -1.000000\n",
      "ep 4: game finished, reward: -1.000000\n",
      "ep 4: game finished, reward: -1.000000\n",
      "ep 4: game finished, reward: -1.000000\n",
      "ep 4: game finished, reward: -1.000000\n",
      "resetting env. episode 5 reward total was -21.000000. running mean: -20.941192\n",
      "ep 5: game finished, reward: -1.000000\n",
      "ep 5: game finished, reward: -1.000000\n",
      "ep 5: game finished, reward: -1.000000\n",
      "ep 5: game finished, reward: -1.000000\n",
      "ep 5: game finished, reward: -1.000000\n",
      "ep 5: game finished, reward: -1.000000\n",
      "ep 5: game finished, reward: -1.000000\n",
      "ep 5: game finished, reward: -1.000000\n",
      "ep 5: game finished, reward: -1.000000\n",
      "ep 5: game finished, reward: -1.000000\n",
      "ep 5: game finished, reward: -1.000000\n",
      "ep 5: game finished, reward: -1.000000\n",
      "ep 5: game finished, reward: -1.000000\n",
      "ep 5: game finished, reward: -1.000000\n",
      "ep 5: game finished, reward: -1.000000\n",
      "ep 5: game finished, reward: -1.000000\n",
      "ep 5: game finished, reward: -1.000000\n",
      "ep 5: game finished, reward: -1.000000\n",
      "ep 5: game finished, reward: -1.000000\n",
      "ep 5: game finished, reward: -1.000000\n",
      "ep 5: game finished, reward: -1.000000\n",
      "resetting env. episode 6 reward total was -21.000000. running mean: -20.941780\n",
      "ep 6: game finished, reward: -1.000000\n",
      "ep 6: game finished, reward: -1.000000\n",
      "ep 6: game finished, reward: -1.000000\n",
      "ep 6: game finished, reward: -1.000000\n",
      "ep 6: game finished, reward: -1.000000\n",
      "ep 6: game finished, reward: -1.000000\n",
      "ep 6: game finished, reward: -1.000000\n",
      "ep 6: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 6: game finished, reward: -1.000000\n",
      "ep 6: game finished, reward: -1.000000\n",
      "ep 6: game finished, reward: -1.000000\n",
      "ep 6: game finished, reward: -1.000000\n",
      "ep 6: game finished, reward: -1.000000\n",
      "ep 6: game finished, reward: -1.000000\n",
      "ep 6: game finished, reward: -1.000000\n",
      "ep 6: game finished, reward: -1.000000\n",
      "ep 6: game finished, reward: -1.000000\n",
      "ep 6: game finished, reward: -1.000000\n",
      "ep 6: game finished, reward: -1.000000\n",
      "ep 6: game finished, reward: -1.000000\n",
      "ep 6: game finished, reward: -1.000000\n",
      "ep 6: game finished, reward: -1.000000\n",
      "ep 6: game finished, reward: 1.000000 !!!!!!!!\n",
      "resetting env. episode 7 reward total was -19.000000. running mean: -20.922362\n",
      "ep 7: game finished, reward: -1.000000\n",
      "ep 7: game finished, reward: -1.000000\n",
      "ep 7: game finished, reward: -1.000000\n",
      "ep 7: game finished, reward: -1.000000\n",
      "ep 7: game finished, reward: -1.000000\n",
      "ep 7: game finished, reward: -1.000000\n",
      "ep 7: game finished, reward: -1.000000\n",
      "ep 7: game finished, reward: -1.000000\n",
      "ep 7: game finished, reward: -1.000000\n",
      "ep 7: game finished, reward: -1.000000\n",
      "ep 7: game finished, reward: -1.000000\n",
      "ep 7: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 7: game finished, reward: -1.000000\n",
      "ep 7: game finished, reward: -1.000000\n",
      "ep 7: game finished, reward: -1.000000\n",
      "ep 7: game finished, reward: -1.000000\n",
      "ep 7: game finished, reward: -1.000000\n",
      "ep 7: game finished, reward: -1.000000\n",
      "ep 7: game finished, reward: -1.000000\n",
      "ep 7: game finished, reward: -1.000000\n",
      "ep 7: game finished, reward: -1.000000\n",
      "ep 7: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 7: game finished, reward: -1.000000\n",
      "resetting env. episode 8 reward total was -19.000000. running mean: -20.903139\n",
      "ep 8: game finished, reward: -1.000000\n",
      "ep 8: game finished, reward: -1.000000\n",
      "ep 8: game finished, reward: -1.000000\n",
      "ep 8: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 8: game finished, reward: -1.000000\n",
      "ep 8: game finished, reward: -1.000000\n",
      "ep 8: game finished, reward: -1.000000\n",
      "ep 8: game finished, reward: -1.000000\n",
      "ep 8: game finished, reward: -1.000000\n",
      "ep 8: game finished, reward: -1.000000\n",
      "ep 8: game finished, reward: -1.000000\n",
      "ep 8: game finished, reward: -1.000000\n",
      "ep 8: game finished, reward: -1.000000\n",
      "ep 8: game finished, reward: -1.000000\n",
      "ep 8: game finished, reward: -1.000000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ep 8: game finished, reward: -1.000000\n",
      "ep 8: game finished, reward: -1.000000\n",
      "ep 8: game finished, reward: -1.000000\n",
      "ep 8: game finished, reward: -1.000000\n",
      "ep 8: game finished, reward: -1.000000\n",
      "ep 8: game finished, reward: -1.000000\n",
      "ep 8: game finished, reward: -1.000000\n",
      "resetting env. episode 9 reward total was -20.000000. running mean: -20.894107\n",
      "ep 9: game finished, reward: -1.000000\n",
      "ep 9: game finished, reward: -1.000000\n",
      "ep 9: game finished, reward: -1.000000\n",
      "ep 9: game finished, reward: -1.000000\n",
      "ep 9: game finished, reward: -1.000000\n",
      "ep 9: game finished, reward: -1.000000\n",
      "ep 9: game finished, reward: -1.000000\n",
      "ep 9: game finished, reward: -1.000000\n",
      "ep 9: game finished, reward: -1.000000\n",
      "ep 9: game finished, reward: -1.000000\n",
      "ep 9: game finished, reward: -1.000000\n",
      "ep 9: game finished, reward: -1.000000\n",
      "ep 9: game finished, reward: -1.000000\n",
      "ep 9: game finished, reward: -1.000000\n",
      "ep 9: game finished, reward: -1.000000\n",
      "ep 9: game finished, reward: -1.000000\n",
      "ep 9: game finished, reward: -1.000000\n",
      "ep 9: game finished, reward: -1.000000\n",
      "ep 9: game finished, reward: -1.000000\n",
      "ep 9: game finished, reward: -1.000000\n",
      "ep 9: game finished, reward: -1.000000\n",
      "resetting env. episode 10 reward total was -21.000000. running mean: -20.895166\n",
      "ep 10: game finished, reward: -1.000000\n",
      "ep 10: game finished, reward: -1.000000\n",
      "ep 10: game finished, reward: -1.000000\n",
      "ep 10: game finished, reward: -1.000000\n",
      "ep 10: game finished, reward: -1.000000\n",
      "ep 10: game finished, reward: -1.000000\n",
      "ep 10: game finished, reward: -1.000000\n",
      "ep 10: game finished, reward: -1.000000\n",
      "ep 10: game finished, reward: -1.000000\n",
      "ep 10: game finished, reward: -1.000000\n",
      "ep 10: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 10: game finished, reward: -1.000000\n",
      "ep 10: game finished, reward: -1.000000\n",
      "ep 10: game finished, reward: -1.000000\n",
      "ep 10: game finished, reward: -1.000000\n",
      "ep 10: game finished, reward: -1.000000\n",
      "ep 10: game finished, reward: -1.000000\n",
      "ep 10: game finished, reward: -1.000000\n",
      "ep 10: game finished, reward: -1.000000\n",
      "ep 10: game finished, reward: -1.000000\n",
      "ep 10: game finished, reward: -1.000000\n",
      "ep 10: game finished, reward: -1.000000\n",
      "resetting env. episode 11 reward total was -20.000000. running mean: -20.886215\n",
      "ep 11: game finished, reward: -1.000000\n",
      "ep 11: game finished, reward: -1.000000\n",
      "ep 11: game finished, reward: -1.000000\n",
      "ep 11: game finished, reward: -1.000000\n",
      "ep 11: game finished, reward: -1.000000\n",
      "ep 11: game finished, reward: -1.000000\n",
      "ep 11: game finished, reward: -1.000000\n",
      "ep 11: game finished, reward: -1.000000\n",
      "ep 11: game finished, reward: -1.000000\n",
      "ep 11: game finished, reward: -1.000000\n",
      "ep 11: game finished, reward: -1.000000\n",
      "ep 11: game finished, reward: -1.000000\n",
      "ep 11: game finished, reward: -1.000000\n",
      "ep 11: game finished, reward: -1.000000\n",
      "ep 11: game finished, reward: -1.000000\n",
      "ep 11: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 11: game finished, reward: -1.000000\n",
      "ep 11: game finished, reward: -1.000000\n",
      "ep 11: game finished, reward: -1.000000\n",
      "ep 11: game finished, reward: -1.000000\n",
      "ep 11: game finished, reward: -1.000000\n",
      "ep 11: game finished, reward: -1.000000\n",
      "resetting env. episode 12 reward total was -20.000000. running mean: -20.877352\n",
      "ep 12: game finished, reward: -1.000000\n",
      "ep 12: game finished, reward: -1.000000\n",
      "ep 12: game finished, reward: -1.000000\n",
      "ep 12: game finished, reward: -1.000000\n",
      "ep 12: game finished, reward: -1.000000\n",
      "ep 12: game finished, reward: -1.000000\n",
      "ep 12: game finished, reward: -1.000000\n",
      "ep 12: game finished, reward: -1.000000\n",
      "ep 12: game finished, reward: -1.000000\n",
      "ep 12: game finished, reward: -1.000000\n",
      "ep 12: game finished, reward: -1.000000\n",
      "ep 12: game finished, reward: -1.000000\n",
      "ep 12: game finished, reward: -1.000000\n",
      "ep 12: game finished, reward: -1.000000\n",
      "ep 12: game finished, reward: -1.000000\n",
      "ep 12: game finished, reward: -1.000000\n",
      "ep 12: game finished, reward: -1.000000\n",
      "ep 12: game finished, reward: -1.000000\n",
      "ep 12: game finished, reward: -1.000000\n",
      "ep 12: game finished, reward: -1.000000\n",
      "ep 12: game finished, reward: -1.000000\n",
      "resetting env. episode 13 reward total was -21.000000. running mean: -20.878579\n",
      "ep 13: game finished, reward: -1.000000\n",
      "ep 13: game finished, reward: -1.000000\n",
      "ep 13: game finished, reward: -1.000000\n",
      "ep 13: game finished, reward: -1.000000\n",
      "ep 13: game finished, reward: -1.000000\n",
      "ep 13: game finished, reward: -1.000000\n",
      "ep 13: game finished, reward: -1.000000\n",
      "ep 13: game finished, reward: -1.000000\n",
      "ep 13: game finished, reward: -1.000000\n",
      "ep 13: game finished, reward: -1.000000\n",
      "ep 13: game finished, reward: -1.000000\n",
      "ep 13: game finished, reward: -1.000000\n",
      "ep 13: game finished, reward: -1.000000\n",
      "ep 13: game finished, reward: -1.000000\n",
      "ep 13: game finished, reward: -1.000000\n",
      "ep 13: game finished, reward: -1.000000\n",
      "ep 13: game finished, reward: -1.000000\n",
      "ep 13: game finished, reward: -1.000000\n",
      "ep 13: game finished, reward: -1.000000\n",
      "ep 13: game finished, reward: -1.000000\n",
      "ep 13: game finished, reward: -1.000000\n",
      "resetting env. episode 14 reward total was -21.000000. running mean: -20.879793\n",
      "ep 14: game finished, reward: -1.000000\n",
      "ep 14: game finished, reward: -1.000000\n",
      "ep 14: game finished, reward: -1.000000\n",
      "ep 14: game finished, reward: -1.000000\n",
      "ep 14: game finished, reward: -1.000000\n",
      "ep 14: game finished, reward: -1.000000\n",
      "ep 14: game finished, reward: -1.000000\n",
      "ep 14: game finished, reward: -1.000000\n",
      "ep 14: game finished, reward: -1.000000\n",
      "ep 14: game finished, reward: -1.000000\n",
      "ep 14: game finished, reward: -1.000000\n",
      "ep 14: game finished, reward: -1.000000\n",
      "ep 14: game finished, reward: -1.000000\n",
      "ep 14: game finished, reward: -1.000000\n",
      "ep 14: game finished, reward: -1.000000\n",
      "ep 14: game finished, reward: -1.000000\n",
      "ep 14: game finished, reward: -1.000000\n",
      "ep 14: game finished, reward: -1.000000\n",
      "ep 14: game finished, reward: -1.000000\n",
      "ep 14: game finished, reward: -1.000000\n",
      "ep 14: game finished, reward: -1.000000\n",
      "resetting env. episode 15 reward total was -21.000000. running mean: -20.880995\n",
      "ep 15: game finished, reward: -1.000000\n",
      "ep 15: game finished, reward: -1.000000\n",
      "ep 15: game finished, reward: -1.000000\n",
      "ep 15: game finished, reward: -1.000000\n",
      "ep 15: game finished, reward: -1.000000\n",
      "ep 15: game finished, reward: -1.000000\n",
      "ep 15: game finished, reward: -1.000000\n",
      "ep 15: game finished, reward: -1.000000\n",
      "ep 15: game finished, reward: -1.000000\n",
      "ep 15: game finished, reward: -1.000000\n",
      "ep 15: game finished, reward: -1.000000\n",
      "ep 15: game finished, reward: -1.000000\n",
      "ep 15: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 15: game finished, reward: -1.000000\n",
      "ep 15: game finished, reward: -1.000000\n",
      "ep 15: game finished, reward: -1.000000\n",
      "ep 15: game finished, reward: -1.000000\n",
      "ep 15: game finished, reward: -1.000000\n",
      "ep 15: game finished, reward: -1.000000\n",
      "ep 15: game finished, reward: -1.000000\n",
      "ep 15: game finished, reward: -1.000000\n",
      "ep 15: game finished, reward: -1.000000\n",
      "resetting env. episode 16 reward total was -20.000000. running mean: -20.872185\n",
      "ep 16: game finished, reward: -1.000000\n",
      "ep 16: game finished, reward: -1.000000\n",
      "ep 16: game finished, reward: -1.000000\n",
      "ep 16: game finished, reward: -1.000000\n",
      "ep 16: game finished, reward: -1.000000\n",
      "ep 16: game finished, reward: -1.000000\n",
      "ep 16: game finished, reward: -1.000000\n",
      "ep 16: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 16: game finished, reward: -1.000000\n",
      "ep 16: game finished, reward: -1.000000\n",
      "ep 16: game finished, reward: -1.000000\n",
      "ep 16: game finished, reward: -1.000000\n",
      "ep 16: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 16: game finished, reward: -1.000000\n",
      "ep 16: game finished, reward: -1.000000\n",
      "ep 16: game finished, reward: -1.000000\n",
      "ep 16: game finished, reward: -1.000000\n",
      "ep 16: game finished, reward: -1.000000\n",
      "ep 16: game finished, reward: -1.000000\n",
      "ep 16: game finished, reward: -1.000000\n",
      "ep 16: game finished, reward: -1.000000\n",
      "ep 16: game finished, reward: -1.000000\n",
      "ep 16: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 16: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 16: game finished, reward: -1.000000\n",
      "resetting env. episode 17 reward total was -17.000000. running mean: -20.833463\n",
      "ep 17: game finished, reward: -1.000000\n",
      "ep 17: game finished, reward: -1.000000\n",
      "ep 17: game finished, reward: -1.000000\n",
      "ep 17: game finished, reward: -1.000000\n",
      "ep 17: game finished, reward: -1.000000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ep 17: game finished, reward: -1.000000\n",
      "ep 17: game finished, reward: -1.000000\n",
      "ep 17: game finished, reward: -1.000000\n",
      "ep 17: game finished, reward: -1.000000\n",
      "ep 17: game finished, reward: -1.000000\n",
      "ep 17: game finished, reward: -1.000000\n",
      "ep 17: game finished, reward: -1.000000\n",
      "ep 17: game finished, reward: -1.000000\n",
      "ep 17: game finished, reward: -1.000000\n",
      "ep 17: game finished, reward: -1.000000\n",
      "ep 17: game finished, reward: -1.000000\n",
      "ep 17: game finished, reward: -1.000000\n",
      "ep 17: game finished, reward: -1.000000\n",
      "ep 17: game finished, reward: -1.000000\n",
      "ep 17: game finished, reward: -1.000000\n",
      "ep 17: game finished, reward: -1.000000\n",
      "resetting env. episode 18 reward total was -21.000000. running mean: -20.835129\n",
      "ep 18: game finished, reward: -1.000000\n",
      "ep 18: game finished, reward: -1.000000\n",
      "ep 18: game finished, reward: -1.000000\n",
      "ep 18: game finished, reward: -1.000000\n",
      "ep 18: game finished, reward: -1.000000\n",
      "ep 18: game finished, reward: -1.000000\n",
      "ep 18: game finished, reward: -1.000000\n",
      "ep 18: game finished, reward: -1.000000\n",
      "ep 18: game finished, reward: -1.000000\n",
      "ep 18: game finished, reward: -1.000000\n",
      "ep 18: game finished, reward: -1.000000\n",
      "ep 18: game finished, reward: -1.000000\n",
      "ep 18: game finished, reward: -1.000000\n",
      "ep 18: game finished, reward: -1.000000\n",
      "ep 18: game finished, reward: -1.000000\n",
      "ep 18: game finished, reward: -1.000000\n",
      "ep 18: game finished, reward: -1.000000\n",
      "ep 18: game finished, reward: -1.000000\n",
      "ep 18: game finished, reward: -1.000000\n",
      "ep 18: game finished, reward: -1.000000\n",
      "ep 18: game finished, reward: -1.000000\n",
      "resetting env. episode 19 reward total was -21.000000. running mean: -20.836777\n",
      "ep 19: game finished, reward: -1.000000\n",
      "ep 19: game finished, reward: -1.000000\n",
      "ep 19: game finished, reward: -1.000000\n",
      "ep 19: game finished, reward: -1.000000\n",
      "ep 19: game finished, reward: -1.000000\n",
      "ep 19: game finished, reward: -1.000000\n",
      "ep 19: game finished, reward: -1.000000\n",
      "ep 19: game finished, reward: -1.000000\n",
      "ep 19: game finished, reward: -1.000000\n",
      "ep 19: game finished, reward: -1.000000\n",
      "ep 19: game finished, reward: -1.000000\n",
      "ep 19: game finished, reward: -1.000000\n",
      "ep 19: game finished, reward: -1.000000\n",
      "ep 19: game finished, reward: -1.000000\n",
      "ep 19: game finished, reward: -1.000000\n",
      "ep 19: game finished, reward: -1.000000\n",
      "ep 19: game finished, reward: -1.000000\n",
      "ep 19: game finished, reward: -1.000000\n",
      "ep 19: game finished, reward: -1.000000\n",
      "ep 19: game finished, reward: -1.000000\n",
      "ep 19: game finished, reward: -1.000000\n",
      "ep 19: game finished, reward: 1.000000 !!!!!!!!\n",
      "resetting env. episode 20 reward total was -20.000000. running mean: -20.828410\n",
      "ep 20: game finished, reward: -1.000000\n",
      "ep 20: game finished, reward: -1.000000\n",
      "ep 20: game finished, reward: -1.000000\n",
      "ep 20: game finished, reward: -1.000000\n",
      "ep 20: game finished, reward: -1.000000\n",
      "ep 20: game finished, reward: -1.000000\n",
      "ep 20: game finished, reward: -1.000000\n",
      "ep 20: game finished, reward: -1.000000\n",
      "ep 20: game finished, reward: -1.000000\n",
      "ep 20: game finished, reward: -1.000000\n",
      "ep 20: game finished, reward: -1.000000\n",
      "ep 20: game finished, reward: -1.000000\n",
      "ep 20: game finished, reward: -1.000000\n",
      "ep 20: game finished, reward: -1.000000\n",
      "ep 20: game finished, reward: -1.000000\n",
      "ep 20: game finished, reward: -1.000000\n",
      "ep 20: game finished, reward: -1.000000\n",
      "ep 20: game finished, reward: -1.000000\n",
      "ep 20: game finished, reward: -1.000000\n",
      "ep 20: game finished, reward: -1.000000\n",
      "ep 20: game finished, reward: -1.000000\n",
      "resetting env. episode 21 reward total was -21.000000. running mean: -20.830126\n",
      "ep 21: game finished, reward: -1.000000\n",
      "ep 21: game finished, reward: -1.000000\n",
      "ep 21: game finished, reward: -1.000000\n",
      "ep 21: game finished, reward: -1.000000\n",
      "ep 21: game finished, reward: -1.000000\n",
      "ep 21: game finished, reward: -1.000000\n",
      "ep 21: game finished, reward: -1.000000\n",
      "ep 21: game finished, reward: -1.000000\n",
      "ep 21: game finished, reward: -1.000000\n",
      "ep 21: game finished, reward: -1.000000\n",
      "ep 21: game finished, reward: -1.000000\n",
      "ep 21: game finished, reward: -1.000000\n",
      "ep 21: game finished, reward: -1.000000\n",
      "ep 21: game finished, reward: -1.000000\n",
      "ep 21: game finished, reward: -1.000000\n",
      "ep 21: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 21: game finished, reward: -1.000000\n",
      "ep 21: game finished, reward: -1.000000\n",
      "ep 21: game finished, reward: -1.000000\n",
      "ep 21: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 21: game finished, reward: -1.000000\n",
      "ep 21: game finished, reward: -1.000000\n",
      "ep 21: game finished, reward: -1.000000\n",
      "resetting env. episode 22 reward total was -19.000000. running mean: -20.811824\n",
      "ep 22: game finished, reward: -1.000000\n",
      "ep 22: game finished, reward: -1.000000\n",
      "ep 22: game finished, reward: -1.000000\n",
      "ep 22: game finished, reward: -1.000000\n",
      "ep 22: game finished, reward: -1.000000\n",
      "ep 22: game finished, reward: -1.000000\n",
      "ep 22: game finished, reward: -1.000000\n",
      "ep 22: game finished, reward: -1.000000\n",
      "ep 22: game finished, reward: -1.000000\n",
      "ep 22: game finished, reward: -1.000000\n",
      "ep 22: game finished, reward: -1.000000\n",
      "ep 22: game finished, reward: -1.000000\n",
      "ep 22: game finished, reward: -1.000000\n",
      "ep 22: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 22: game finished, reward: -1.000000\n",
      "ep 22: game finished, reward: -1.000000\n",
      "ep 22: game finished, reward: -1.000000\n",
      "ep 22: game finished, reward: -1.000000\n",
      "ep 22: game finished, reward: -1.000000\n",
      "ep 22: game finished, reward: -1.000000\n",
      "ep 22: game finished, reward: -1.000000\n",
      "ep 22: game finished, reward: -1.000000\n",
      "resetting env. episode 23 reward total was -20.000000. running mean: -20.803706\n",
      "ep 23: game finished, reward: -1.000000\n",
      "ep 23: game finished, reward: -1.000000\n",
      "ep 23: game finished, reward: -1.000000\n",
      "ep 23: game finished, reward: -1.000000\n",
      "ep 23: game finished, reward: -1.000000\n",
      "ep 23: game finished, reward: -1.000000\n",
      "ep 23: game finished, reward: -1.000000\n",
      "ep 23: game finished, reward: -1.000000\n",
      "ep 23: game finished, reward: -1.000000\n",
      "ep 23: game finished, reward: -1.000000\n",
      "ep 23: game finished, reward: -1.000000\n",
      "ep 23: game finished, reward: -1.000000\n",
      "ep 23: game finished, reward: -1.000000\n",
      "ep 23: game finished, reward: -1.000000\n",
      "ep 23: game finished, reward: -1.000000\n",
      "ep 23: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 23: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 23: game finished, reward: -1.000000\n",
      "ep 23: game finished, reward: -1.000000\n",
      "ep 23: game finished, reward: -1.000000\n",
      "ep 23: game finished, reward: -1.000000\n",
      "ep 23: game finished, reward: -1.000000\n",
      "ep 23: game finished, reward: -1.000000\n",
      "resetting env. episode 24 reward total was -19.000000. running mean: -20.785669\n",
      "ep 24: game finished, reward: -1.000000\n",
      "ep 24: game finished, reward: -1.000000\n",
      "ep 24: game finished, reward: -1.000000\n",
      "ep 24: game finished, reward: -1.000000\n",
      "ep 24: game finished, reward: -1.000000\n",
      "ep 24: game finished, reward: -1.000000\n",
      "ep 24: game finished, reward: -1.000000\n",
      "ep 24: game finished, reward: -1.000000\n",
      "ep 24: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 24: game finished, reward: -1.000000\n",
      "ep 24: game finished, reward: -1.000000\n",
      "ep 24: game finished, reward: -1.000000\n",
      "ep 24: game finished, reward: -1.000000\n",
      "ep 24: game finished, reward: -1.000000\n",
      "ep 24: game finished, reward: -1.000000\n",
      "ep 24: game finished, reward: -1.000000\n",
      "ep 24: game finished, reward: -1.000000\n",
      "ep 24: game finished, reward: -1.000000\n",
      "ep 24: game finished, reward: -1.000000\n",
      "ep 24: game finished, reward: -1.000000\n",
      "ep 24: game finished, reward: -1.000000\n",
      "ep 24: game finished, reward: -1.000000\n",
      "resetting env. episode 25 reward total was -20.000000. running mean: -20.777812\n",
      "ep 25: game finished, reward: -1.000000\n",
      "ep 25: game finished, reward: -1.000000\n",
      "ep 25: game finished, reward: -1.000000\n",
      "ep 25: game finished, reward: -1.000000\n",
      "ep 25: game finished, reward: -1.000000\n",
      "ep 25: game finished, reward: -1.000000\n",
      "ep 25: game finished, reward: -1.000000\n",
      "ep 25: game finished, reward: -1.000000\n",
      "ep 25: game finished, reward: -1.000000\n",
      "ep 25: game finished, reward: -1.000000\n",
      "ep 25: game finished, reward: -1.000000\n",
      "ep 25: game finished, reward: -1.000000\n",
      "ep 25: game finished, reward: -1.000000\n",
      "ep 25: game finished, reward: -1.000000\n",
      "ep 25: game finished, reward: -1.000000\n",
      "ep 25: game finished, reward: -1.000000\n",
      "ep 25: game finished, reward: -1.000000\n",
      "ep 25: game finished, reward: -1.000000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ep 25: game finished, reward: -1.000000\n",
      "ep 25: game finished, reward: -1.000000\n",
      "ep 25: game finished, reward: -1.000000\n",
      "resetting env. episode 26 reward total was -21.000000. running mean: -20.780034\n",
      "ep 26: game finished, reward: -1.000000\n",
      "ep 26: game finished, reward: -1.000000\n",
      "ep 26: game finished, reward: -1.000000\n",
      "ep 26: game finished, reward: -1.000000\n",
      "ep 26: game finished, reward: -1.000000\n",
      "ep 26: game finished, reward: -1.000000\n",
      "ep 26: game finished, reward: -1.000000\n",
      "ep 26: game finished, reward: -1.000000\n",
      "ep 26: game finished, reward: -1.000000\n",
      "ep 26: game finished, reward: -1.000000\n",
      "ep 26: game finished, reward: -1.000000\n",
      "ep 26: game finished, reward: -1.000000\n",
      "ep 26: game finished, reward: -1.000000\n",
      "ep 26: game finished, reward: -1.000000\n",
      "ep 26: game finished, reward: -1.000000\n",
      "ep 26: game finished, reward: -1.000000\n",
      "ep 26: game finished, reward: -1.000000\n",
      "ep 26: game finished, reward: -1.000000\n",
      "ep 26: game finished, reward: -1.000000\n",
      "ep 26: game finished, reward: -1.000000\n",
      "ep 26: game finished, reward: -1.000000\n",
      "resetting env. episode 27 reward total was -21.000000. running mean: -20.782234\n",
      "ep 27: game finished, reward: -1.000000\n",
      "ep 27: game finished, reward: -1.000000\n",
      "ep 27: game finished, reward: -1.000000\n",
      "ep 27: game finished, reward: -1.000000\n",
      "ep 27: game finished, reward: -1.000000\n",
      "ep 27: game finished, reward: -1.000000\n",
      "ep 27: game finished, reward: -1.000000\n",
      "ep 27: game finished, reward: -1.000000\n",
      "ep 27: game finished, reward: -1.000000\n",
      "ep 27: game finished, reward: -1.000000\n",
      "ep 27: game finished, reward: -1.000000\n",
      "ep 27: game finished, reward: -1.000000\n",
      "ep 27: game finished, reward: -1.000000\n",
      "ep 27: game finished, reward: -1.000000\n",
      "ep 27: game finished, reward: -1.000000\n",
      "ep 27: game finished, reward: -1.000000\n",
      "ep 27: game finished, reward: -1.000000\n",
      "ep 27: game finished, reward: -1.000000\n",
      "ep 27: game finished, reward: -1.000000\n",
      "ep 27: game finished, reward: -1.000000\n",
      "ep 27: game finished, reward: -1.000000\n",
      "ep 27: game finished, reward: 1.000000 !!!!!!!!\n",
      "resetting env. episode 28 reward total was -20.000000. running mean: -20.774412\n",
      "ep 28: game finished, reward: -1.000000\n",
      "ep 28: game finished, reward: -1.000000\n",
      "ep 28: game finished, reward: -1.000000\n",
      "ep 28: game finished, reward: -1.000000\n",
      "ep 28: game finished, reward: -1.000000\n",
      "ep 28: game finished, reward: -1.000000\n",
      "ep 28: game finished, reward: -1.000000\n",
      "ep 28: game finished, reward: -1.000000\n",
      "ep 28: game finished, reward: -1.000000\n",
      "ep 28: game finished, reward: -1.000000\n",
      "ep 28: game finished, reward: -1.000000\n",
      "ep 28: game finished, reward: -1.000000\n",
      "ep 28: game finished, reward: -1.000000\n",
      "ep 28: game finished, reward: -1.000000\n",
      "ep 28: game finished, reward: -1.000000\n",
      "ep 28: game finished, reward: -1.000000\n",
      "ep 28: game finished, reward: -1.000000\n",
      "ep 28: game finished, reward: -1.000000\n",
      "ep 28: game finished, reward: -1.000000\n",
      "ep 28: game finished, reward: -1.000000\n",
      "ep 28: game finished, reward: -1.000000\n",
      "resetting env. episode 29 reward total was -21.000000. running mean: -20.776667\n",
      "ep 29: game finished, reward: -1.000000\n",
      "ep 29: game finished, reward: -1.000000\n",
      "ep 29: game finished, reward: -1.000000\n",
      "ep 29: game finished, reward: -1.000000\n",
      "ep 29: game finished, reward: -1.000000\n",
      "ep 29: game finished, reward: -1.000000\n",
      "ep 29: game finished, reward: -1.000000\n",
      "ep 29: game finished, reward: -1.000000\n",
      "ep 29: game finished, reward: -1.000000\n",
      "ep 29: game finished, reward: -1.000000\n",
      "ep 29: game finished, reward: -1.000000\n",
      "ep 29: game finished, reward: -1.000000\n",
      "ep 29: game finished, reward: -1.000000\n",
      "ep 29: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 29: game finished, reward: -1.000000\n",
      "ep 29: game finished, reward: -1.000000\n",
      "ep 29: game finished, reward: -1.000000\n",
      "ep 29: game finished, reward: -1.000000\n",
      "ep 29: game finished, reward: -1.000000\n",
      "ep 29: game finished, reward: -1.000000\n",
      "ep 29: game finished, reward: -1.000000\n",
      "ep 29: game finished, reward: -1.000000\n",
      "resetting env. episode 30 reward total was -20.000000. running mean: -20.768901\n",
      "ep 30: game finished, reward: -1.000000\n",
      "ep 30: game finished, reward: -1.000000\n",
      "ep 30: game finished, reward: -1.000000\n",
      "ep 30: game finished, reward: -1.000000\n",
      "ep 30: game finished, reward: -1.000000\n",
      "ep 30: game finished, reward: -1.000000\n",
      "ep 30: game finished, reward: -1.000000\n",
      "ep 30: game finished, reward: -1.000000\n",
      "ep 30: game finished, reward: -1.000000\n",
      "ep 30: game finished, reward: -1.000000\n",
      "ep 30: game finished, reward: -1.000000\n",
      "ep 30: game finished, reward: -1.000000\n",
      "ep 30: game finished, reward: -1.000000\n",
      "ep 30: game finished, reward: -1.000000\n",
      "ep 30: game finished, reward: -1.000000\n",
      "ep 30: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 30: game finished, reward: -1.000000\n",
      "ep 30: game finished, reward: -1.000000\n",
      "ep 30: game finished, reward: -1.000000\n",
      "ep 30: game finished, reward: -1.000000\n",
      "ep 30: game finished, reward: -1.000000\n",
      "ep 30: game finished, reward: -1.000000\n",
      "resetting env. episode 31 reward total was -20.000000. running mean: -20.761212\n",
      "ep 31: game finished, reward: -1.000000\n",
      "ep 31: game finished, reward: -1.000000\n",
      "ep 31: game finished, reward: -1.000000\n",
      "ep 31: game finished, reward: -1.000000\n",
      "ep 31: game finished, reward: -1.000000\n",
      "ep 31: game finished, reward: -1.000000\n",
      "ep 31: game finished, reward: -1.000000\n",
      "ep 31: game finished, reward: -1.000000\n",
      "ep 31: game finished, reward: -1.000000\n",
      "ep 31: game finished, reward: -1.000000\n",
      "ep 31: game finished, reward: -1.000000\n",
      "ep 31: game finished, reward: -1.000000\n",
      "ep 31: game finished, reward: -1.000000\n",
      "ep 31: game finished, reward: -1.000000\n",
      "ep 31: game finished, reward: -1.000000\n",
      "ep 31: game finished, reward: -1.000000\n",
      "ep 31: game finished, reward: -1.000000\n",
      "ep 31: game finished, reward: -1.000000\n",
      "ep 31: game finished, reward: -1.000000\n",
      "ep 31: game finished, reward: -1.000000\n",
      "ep 31: game finished, reward: -1.000000\n",
      "resetting env. episode 32 reward total was -21.000000. running mean: -20.763600\n",
      "ep 32: game finished, reward: -1.000000\n",
      "ep 32: game finished, reward: -1.000000\n",
      "ep 32: game finished, reward: -1.000000\n",
      "ep 32: game finished, reward: -1.000000\n",
      "ep 32: game finished, reward: -1.000000\n",
      "ep 32: game finished, reward: -1.000000\n",
      "ep 32: game finished, reward: -1.000000\n",
      "ep 32: game finished, reward: -1.000000\n",
      "ep 32: game finished, reward: -1.000000\n",
      "ep 32: game finished, reward: -1.000000\n",
      "ep 32: game finished, reward: -1.000000\n",
      "ep 32: game finished, reward: -1.000000\n",
      "ep 32: game finished, reward: -1.000000\n",
      "ep 32: game finished, reward: -1.000000\n",
      "ep 32: game finished, reward: -1.000000\n",
      "ep 32: game finished, reward: -1.000000\n",
      "ep 32: game finished, reward: -1.000000\n",
      "ep 32: game finished, reward: -1.000000\n",
      "ep 32: game finished, reward: -1.000000\n",
      "ep 32: game finished, reward: -1.000000\n",
      "ep 32: game finished, reward: -1.000000\n",
      "resetting env. episode 33 reward total was -21.000000. running mean: -20.765964\n",
      "ep 33: game finished, reward: -1.000000\n",
      "ep 33: game finished, reward: -1.000000\n",
      "ep 33: game finished, reward: -1.000000\n",
      "ep 33: game finished, reward: -1.000000\n",
      "ep 33: game finished, reward: -1.000000\n",
      "ep 33: game finished, reward: -1.000000\n",
      "ep 33: game finished, reward: -1.000000\n",
      "ep 33: game finished, reward: -1.000000\n",
      "ep 33: game finished, reward: -1.000000\n",
      "ep 33: game finished, reward: -1.000000\n",
      "ep 33: game finished, reward: -1.000000\n",
      "ep 33: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 33: game finished, reward: -1.000000\n",
      "ep 33: game finished, reward: -1.000000\n",
      "ep 33: game finished, reward: -1.000000\n",
      "ep 33: game finished, reward: -1.000000\n",
      "ep 33: game finished, reward: -1.000000\n",
      "ep 33: game finished, reward: -1.000000\n",
      "ep 33: game finished, reward: -1.000000\n",
      "ep 33: game finished, reward: -1.000000\n",
      "ep 33: game finished, reward: -1.000000\n",
      "ep 33: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 33: game finished, reward: -1.000000\n",
      "resetting env. episode 34 reward total was -19.000000. running mean: -20.748304\n",
      "ep 34: game finished, reward: -1.000000\n",
      "ep 34: game finished, reward: -1.000000\n",
      "ep 34: game finished, reward: -1.000000\n",
      "ep 34: game finished, reward: -1.000000\n",
      "ep 34: game finished, reward: -1.000000\n",
      "ep 34: game finished, reward: -1.000000\n",
      "ep 34: game finished, reward: -1.000000\n",
      "ep 34: game finished, reward: -1.000000\n",
      "ep 34: game finished, reward: -1.000000\n",
      "ep 34: game finished, reward: -1.000000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ep 34: game finished, reward: -1.000000\n",
      "ep 34: game finished, reward: -1.000000\n",
      "ep 34: game finished, reward: -1.000000\n",
      "ep 34: game finished, reward: -1.000000\n",
      "ep 34: game finished, reward: -1.000000\n",
      "ep 34: game finished, reward: -1.000000\n",
      "ep 34: game finished, reward: -1.000000\n",
      "ep 34: game finished, reward: -1.000000\n",
      "ep 34: game finished, reward: -1.000000\n",
      "ep 34: game finished, reward: -1.000000\n",
      "ep 34: game finished, reward: -1.000000\n",
      "resetting env. episode 35 reward total was -21.000000. running mean: -20.750821\n",
      "ep 35: game finished, reward: -1.000000\n",
      "ep 35: game finished, reward: -1.000000\n",
      "ep 35: game finished, reward: -1.000000\n",
      "ep 35: game finished, reward: -1.000000\n",
      "ep 35: game finished, reward: -1.000000\n",
      "ep 35: game finished, reward: -1.000000\n",
      "ep 35: game finished, reward: -1.000000\n",
      "ep 35: game finished, reward: -1.000000\n",
      "ep 35: game finished, reward: -1.000000\n",
      "ep 35: game finished, reward: -1.000000\n",
      "ep 35: game finished, reward: -1.000000\n",
      "ep 35: game finished, reward: -1.000000\n",
      "ep 35: game finished, reward: -1.000000\n",
      "ep 35: game finished, reward: -1.000000\n",
      "ep 35: game finished, reward: -1.000000\n",
      "ep 35: game finished, reward: -1.000000\n",
      "ep 35: game finished, reward: -1.000000\n",
      "ep 35: game finished, reward: -1.000000\n",
      "ep 35: game finished, reward: -1.000000\n",
      "ep 35: game finished, reward: -1.000000\n",
      "ep 35: game finished, reward: -1.000000\n",
      "resetting env. episode 36 reward total was -21.000000. running mean: -20.753313\n",
      "ep 36: game finished, reward: -1.000000\n",
      "ep 36: game finished, reward: -1.000000\n",
      "ep 36: game finished, reward: -1.000000\n",
      "ep 36: game finished, reward: -1.000000\n",
      "ep 36: game finished, reward: -1.000000\n",
      "ep 36: game finished, reward: -1.000000\n",
      "ep 36: game finished, reward: -1.000000\n",
      "ep 36: game finished, reward: -1.000000\n",
      "ep 36: game finished, reward: -1.000000\n",
      "ep 36: game finished, reward: -1.000000\n",
      "ep 36: game finished, reward: -1.000000\n",
      "ep 36: game finished, reward: -1.000000\n",
      "ep 36: game finished, reward: -1.000000\n",
      "ep 36: game finished, reward: -1.000000\n",
      "ep 36: game finished, reward: -1.000000\n",
      "ep 36: game finished, reward: -1.000000\n",
      "ep 36: game finished, reward: -1.000000\n",
      "ep 36: game finished, reward: -1.000000\n",
      "ep 36: game finished, reward: -1.000000\n",
      "ep 36: game finished, reward: -1.000000\n",
      "ep 36: game finished, reward: -1.000000\n",
      "resetting env. episode 37 reward total was -21.000000. running mean: -20.755780\n",
      "ep 37: game finished, reward: -1.000000\n",
      "ep 37: game finished, reward: -1.000000\n",
      "ep 37: game finished, reward: -1.000000\n",
      "ep 37: game finished, reward: -1.000000\n",
      "ep 37: game finished, reward: -1.000000\n",
      "ep 37: game finished, reward: -1.000000\n",
      "ep 37: game finished, reward: -1.000000\n",
      "ep 37: game finished, reward: -1.000000\n",
      "ep 37: game finished, reward: -1.000000\n",
      "ep 37: game finished, reward: -1.000000\n",
      "ep 37: game finished, reward: -1.000000\n",
      "ep 37: game finished, reward: -1.000000\n",
      "ep 37: game finished, reward: -1.000000\n",
      "ep 37: game finished, reward: -1.000000\n",
      "ep 37: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 37: game finished, reward: -1.000000\n",
      "ep 37: game finished, reward: -1.000000\n",
      "ep 37: game finished, reward: -1.000000\n",
      "ep 37: game finished, reward: -1.000000\n",
      "ep 37: game finished, reward: -1.000000\n",
      "ep 37: game finished, reward: -1.000000\n",
      "ep 37: game finished, reward: -1.000000\n",
      "resetting env. episode 38 reward total was -20.000000. running mean: -20.748222\n",
      "ep 38: game finished, reward: -1.000000\n",
      "ep 38: game finished, reward: -1.000000\n",
      "ep 38: game finished, reward: -1.000000\n",
      "ep 38: game finished, reward: -1.000000\n",
      "ep 38: game finished, reward: -1.000000\n",
      "ep 38: game finished, reward: -1.000000\n",
      "ep 38: game finished, reward: -1.000000\n",
      "ep 38: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 38: game finished, reward: -1.000000\n",
      "ep 38: game finished, reward: -1.000000\n",
      "ep 38: game finished, reward: -1.000000\n",
      "ep 38: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 38: game finished, reward: -1.000000\n",
      "ep 38: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 38: game finished, reward: -1.000000\n",
      "ep 38: game finished, reward: -1.000000\n",
      "ep 38: game finished, reward: -1.000000\n",
      "ep 38: game finished, reward: -1.000000\n",
      "ep 38: game finished, reward: -1.000000\n",
      "ep 38: game finished, reward: -1.000000\n",
      "ep 38: game finished, reward: -1.000000\n",
      "ep 38: game finished, reward: -1.000000\n",
      "ep 38: game finished, reward: -1.000000\n",
      "ep 38: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 38: game finished, reward: -1.000000\n",
      "resetting env. episode 39 reward total was -17.000000. running mean: -20.710740\n",
      "ep 39: game finished, reward: -1.000000\n",
      "ep 39: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 39: game finished, reward: -1.000000\n",
      "ep 39: game finished, reward: -1.000000\n",
      "ep 39: game finished, reward: -1.000000\n",
      "ep 39: game finished, reward: -1.000000\n",
      "ep 39: game finished, reward: -1.000000\n",
      "ep 39: game finished, reward: -1.000000\n",
      "ep 39: game finished, reward: -1.000000\n",
      "ep 39: game finished, reward: -1.000000\n",
      "ep 39: game finished, reward: -1.000000\n",
      "ep 39: game finished, reward: -1.000000\n",
      "ep 39: game finished, reward: -1.000000\n",
      "ep 39: game finished, reward: -1.000000\n",
      "ep 39: game finished, reward: -1.000000\n",
      "ep 39: game finished, reward: -1.000000\n",
      "ep 39: game finished, reward: -1.000000\n",
      "ep 39: game finished, reward: -1.000000\n",
      "ep 39: game finished, reward: -1.000000\n",
      "ep 39: game finished, reward: -1.000000\n",
      "ep 39: game finished, reward: -1.000000\n",
      "ep 39: game finished, reward: -1.000000\n",
      "resetting env. episode 40 reward total was -20.000000. running mean: -20.703632\n",
      "ep 40: game finished, reward: -1.000000\n",
      "ep 40: game finished, reward: -1.000000\n",
      "ep 40: game finished, reward: -1.000000\n",
      "ep 40: game finished, reward: -1.000000\n",
      "ep 40: game finished, reward: -1.000000\n",
      "ep 40: game finished, reward: -1.000000\n",
      "ep 40: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 40: game finished, reward: -1.000000\n",
      "ep 40: game finished, reward: -1.000000\n",
      "ep 40: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 40: game finished, reward: -1.000000\n",
      "ep 40: game finished, reward: -1.000000\n",
      "ep 40: game finished, reward: -1.000000\n",
      "ep 40: game finished, reward: -1.000000\n",
      "ep 40: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 40: game finished, reward: -1.000000\n",
      "ep 40: game finished, reward: -1.000000\n",
      "ep 40: game finished, reward: -1.000000\n",
      "ep 40: game finished, reward: -1.000000\n",
      "ep 40: game finished, reward: -1.000000\n",
      "ep 40: game finished, reward: -1.000000\n",
      "ep 40: game finished, reward: -1.000000\n",
      "ep 40: game finished, reward: -1.000000\n",
      "ep 40: game finished, reward: -1.000000\n",
      "resetting env. episode 41 reward total was -18.000000. running mean: -20.676596\n",
      "ep 41: game finished, reward: -1.000000\n",
      "ep 41: game finished, reward: -1.000000\n",
      "ep 41: game finished, reward: -1.000000\n",
      "ep 41: game finished, reward: -1.000000\n",
      "ep 41: game finished, reward: -1.000000\n",
      "ep 41: game finished, reward: -1.000000\n",
      "ep 41: game finished, reward: -1.000000\n",
      "ep 41: game finished, reward: -1.000000\n",
      "ep 41: game finished, reward: -1.000000\n",
      "ep 41: game finished, reward: -1.000000\n",
      "ep 41: game finished, reward: -1.000000\n",
      "ep 41: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 41: game finished, reward: -1.000000\n",
      "ep 41: game finished, reward: -1.000000\n",
      "ep 41: game finished, reward: -1.000000\n",
      "ep 41: game finished, reward: -1.000000\n",
      "ep 41: game finished, reward: -1.000000\n",
      "ep 41: game finished, reward: -1.000000\n",
      "ep 41: game finished, reward: -1.000000\n",
      "ep 41: game finished, reward: -1.000000\n",
      "ep 41: game finished, reward: -1.000000\n",
      "ep 41: game finished, reward: -1.000000\n",
      "resetting env. episode 42 reward total was -20.000000. running mean: -20.669830\n",
      "ep 42: game finished, reward: -1.000000\n",
      "ep 42: game finished, reward: -1.000000\n",
      "ep 42: game finished, reward: -1.000000\n",
      "ep 42: game finished, reward: -1.000000\n",
      "ep 42: game finished, reward: -1.000000\n",
      "ep 42: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 42: game finished, reward: -1.000000\n",
      "ep 42: game finished, reward: -1.000000\n",
      "ep 42: game finished, reward: -1.000000\n",
      "ep 42: game finished, reward: -1.000000\n",
      "ep 42: game finished, reward: -1.000000\n",
      "ep 42: game finished, reward: -1.000000\n",
      "ep 42: game finished, reward: -1.000000\n",
      "ep 42: game finished, reward: -1.000000\n",
      "ep 42: game finished, reward: -1.000000\n",
      "ep 42: game finished, reward: -1.000000\n",
      "ep 42: game finished, reward: -1.000000\n",
      "ep 42: game finished, reward: -1.000000\n",
      "ep 42: game finished, reward: 1.000000 !!!!!!!!\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ep 42: game finished, reward: -1.000000\n",
      "ep 42: game finished, reward: -1.000000\n",
      "ep 42: game finished, reward: -1.000000\n",
      "ep 42: game finished, reward: -1.000000\n",
      "resetting env. episode 43 reward total was -19.000000. running mean: -20.653132\n",
      "ep 43: game finished, reward: -1.000000\n",
      "ep 43: game finished, reward: -1.000000\n",
      "ep 43: game finished, reward: -1.000000\n",
      "ep 43: game finished, reward: -1.000000\n",
      "ep 43: game finished, reward: -1.000000\n",
      "ep 43: game finished, reward: -1.000000\n",
      "ep 43: game finished, reward: -1.000000\n",
      "ep 43: game finished, reward: -1.000000\n",
      "ep 43: game finished, reward: -1.000000\n",
      "ep 43: game finished, reward: -1.000000\n",
      "ep 43: game finished, reward: -1.000000\n",
      "ep 43: game finished, reward: -1.000000\n",
      "ep 43: game finished, reward: -1.000000\n",
      "ep 43: game finished, reward: -1.000000\n",
      "ep 43: game finished, reward: -1.000000\n",
      "ep 43: game finished, reward: -1.000000\n",
      "ep 43: game finished, reward: -1.000000\n",
      "ep 43: game finished, reward: -1.000000\n",
      "ep 43: game finished, reward: -1.000000\n",
      "ep 43: game finished, reward: -1.000000\n",
      "ep 43: game finished, reward: -1.000000\n",
      "ep 43: game finished, reward: 1.000000 !!!!!!!!\n",
      "resetting env. episode 44 reward total was -20.000000. running mean: -20.646600\n",
      "ep 44: game finished, reward: -1.000000\n",
      "ep 44: game finished, reward: -1.000000\n",
      "ep 44: game finished, reward: -1.000000\n",
      "ep 44: game finished, reward: -1.000000\n",
      "ep 44: game finished, reward: -1.000000\n",
      "ep 44: game finished, reward: -1.000000\n",
      "ep 44: game finished, reward: -1.000000\n",
      "ep 44: game finished, reward: -1.000000\n",
      "ep 44: game finished, reward: -1.000000\n",
      "ep 44: game finished, reward: -1.000000\n",
      "ep 44: game finished, reward: -1.000000\n",
      "ep 44: game finished, reward: -1.000000\n",
      "ep 44: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 44: game finished, reward: -1.000000\n",
      "ep 44: game finished, reward: -1.000000\n",
      "ep 44: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 44: game finished, reward: -1.000000\n",
      "ep 44: game finished, reward: -1.000000\n",
      "ep 44: game finished, reward: -1.000000\n",
      "ep 44: game finished, reward: -1.000000\n",
      "ep 44: game finished, reward: -1.000000\n",
      "ep 44: game finished, reward: -1.000000\n",
      "ep 44: game finished, reward: -1.000000\n",
      "resetting env. episode 45 reward total was -19.000000. running mean: -20.630134\n",
      "ep 45: game finished, reward: -1.000000\n",
      "ep 45: game finished, reward: -1.000000\n",
      "ep 45: game finished, reward: -1.000000\n",
      "ep 45: game finished, reward: -1.000000\n",
      "ep 45: game finished, reward: -1.000000\n",
      "ep 45: game finished, reward: -1.000000\n",
      "ep 45: game finished, reward: -1.000000\n",
      "ep 45: game finished, reward: -1.000000\n",
      "ep 45: game finished, reward: -1.000000\n",
      "ep 45: game finished, reward: -1.000000\n",
      "ep 45: game finished, reward: -1.000000\n",
      "ep 45: game finished, reward: -1.000000\n",
      "ep 45: game finished, reward: -1.000000\n",
      "ep 45: game finished, reward: -1.000000\n",
      "ep 45: game finished, reward: -1.000000\n",
      "ep 45: game finished, reward: -1.000000\n",
      "ep 45: game finished, reward: -1.000000\n",
      "ep 45: game finished, reward: -1.000000\n",
      "ep 45: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 45: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 45: game finished, reward: -1.000000\n",
      "ep 45: game finished, reward: -1.000000\n",
      "ep 45: game finished, reward: -1.000000\n",
      "resetting env. episode 46 reward total was -19.000000. running mean: -20.613833\n",
      "ep 46: game finished, reward: -1.000000\n",
      "ep 46: game finished, reward: -1.000000\n",
      "ep 46: game finished, reward: -1.000000\n",
      "ep 46: game finished, reward: -1.000000\n",
      "ep 46: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 46: game finished, reward: -1.000000\n",
      "ep 46: game finished, reward: -1.000000\n",
      "ep 46: game finished, reward: -1.000000\n",
      "ep 46: game finished, reward: -1.000000\n",
      "ep 46: game finished, reward: -1.000000\n",
      "ep 46: game finished, reward: -1.000000\n",
      "ep 46: game finished, reward: -1.000000\n",
      "ep 46: game finished, reward: -1.000000\n",
      "ep 46: game finished, reward: -1.000000\n",
      "ep 46: game finished, reward: -1.000000\n",
      "ep 46: game finished, reward: -1.000000\n",
      "ep 46: game finished, reward: -1.000000\n",
      "ep 46: game finished, reward: -1.000000\n",
      "ep 46: game finished, reward: -1.000000\n",
      "ep 46: game finished, reward: -1.000000\n",
      "ep 46: game finished, reward: -1.000000\n",
      "ep 46: game finished, reward: -1.000000\n",
      "resetting env. episode 47 reward total was -20.000000. running mean: -20.607695\n",
      "ep 47: game finished, reward: -1.000000\n",
      "ep 47: game finished, reward: -1.000000\n",
      "ep 47: game finished, reward: -1.000000\n",
      "ep 47: game finished, reward: -1.000000\n",
      "ep 47: game finished, reward: -1.000000\n",
      "ep 47: game finished, reward: -1.000000\n",
      "ep 47: game finished, reward: -1.000000\n",
      "ep 47: game finished, reward: -1.000000\n",
      "ep 47: game finished, reward: -1.000000\n",
      "ep 47: game finished, reward: -1.000000\n",
      "ep 47: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 47: game finished, reward: -1.000000\n",
      "ep 47: game finished, reward: -1.000000\n",
      "ep 47: game finished, reward: -1.000000\n",
      "ep 47: game finished, reward: -1.000000\n",
      "ep 47: game finished, reward: -1.000000\n",
      "ep 47: game finished, reward: -1.000000\n",
      "ep 47: game finished, reward: -1.000000\n",
      "ep 47: game finished, reward: -1.000000\n",
      "ep 47: game finished, reward: -1.000000\n",
      "ep 47: game finished, reward: -1.000000\n",
      "ep 47: game finished, reward: -1.000000\n",
      "resetting env. episode 48 reward total was -20.000000. running mean: -20.601618\n",
      "ep 48: game finished, reward: -1.000000\n",
      "ep 48: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 48: game finished, reward: -1.000000\n",
      "ep 48: game finished, reward: -1.000000\n",
      "ep 48: game finished, reward: -1.000000\n",
      "ep 48: game finished, reward: -1.000000\n",
      "ep 48: game finished, reward: -1.000000\n",
      "ep 48: game finished, reward: -1.000000\n",
      "ep 48: game finished, reward: -1.000000\n",
      "ep 48: game finished, reward: -1.000000\n",
      "ep 48: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 48: game finished, reward: -1.000000\n",
      "ep 48: game finished, reward: -1.000000\n",
      "ep 48: game finished, reward: -1.000000\n",
      "ep 48: game finished, reward: -1.000000\n",
      "ep 48: game finished, reward: -1.000000\n",
      "ep 48: game finished, reward: -1.000000\n",
      "ep 48: game finished, reward: -1.000000\n",
      "ep 48: game finished, reward: -1.000000\n",
      "ep 48: game finished, reward: -1.000000\n",
      "ep 48: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 48: game finished, reward: -1.000000\n",
      "ep 48: game finished, reward: -1.000000\n",
      "ep 48: game finished, reward: -1.000000\n",
      "resetting env. episode 49 reward total was -18.000000. running mean: -20.575601\n",
      "ep 49: game finished, reward: -1.000000\n",
      "ep 49: game finished, reward: -1.000000\n",
      "ep 49: game finished, reward: -1.000000\n",
      "ep 49: game finished, reward: -1.000000\n",
      "ep 49: game finished, reward: -1.000000\n",
      "ep 49: game finished, reward: -1.000000\n",
      "ep 49: game finished, reward: -1.000000\n",
      "ep 49: game finished, reward: -1.000000\n",
      "ep 49: game finished, reward: -1.000000\n",
      "ep 49: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 49: game finished, reward: -1.000000\n",
      "ep 49: game finished, reward: -1.000000\n",
      "ep 49: game finished, reward: -1.000000\n",
      "ep 49: game finished, reward: -1.000000\n",
      "ep 49: game finished, reward: -1.000000\n",
      "ep 49: game finished, reward: -1.000000\n",
      "ep 49: game finished, reward: -1.000000\n",
      "ep 49: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 49: game finished, reward: -1.000000\n",
      "ep 49: game finished, reward: -1.000000\n",
      "ep 49: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 49: game finished, reward: -1.000000\n",
      "ep 49: game finished, reward: -1.000000\n",
      "ep 49: game finished, reward: -1.000000\n",
      "resetting env. episode 50 reward total was -18.000000. running mean: -20.549845\n",
      "ep 50: game finished, reward: -1.000000\n",
      "ep 50: game finished, reward: -1.000000\n",
      "ep 50: game finished, reward: -1.000000\n",
      "ep 50: game finished, reward: -1.000000\n",
      "ep 50: game finished, reward: -1.000000\n",
      "ep 50: game finished, reward: -1.000000\n",
      "ep 50: game finished, reward: -1.000000\n",
      "ep 50: game finished, reward: -1.000000\n",
      "ep 50: game finished, reward: -1.000000\n",
      "ep 50: game finished, reward: -1.000000\n",
      "ep 50: game finished, reward: -1.000000\n",
      "ep 50: game finished, reward: -1.000000\n",
      "ep 50: game finished, reward: -1.000000\n",
      "ep 50: game finished, reward: -1.000000\n",
      "ep 50: game finished, reward: -1.000000\n",
      "ep 50: game finished, reward: -1.000000\n",
      "ep 50: game finished, reward: -1.000000\n",
      "ep 50: game finished, reward: -1.000000\n",
      "ep 50: game finished, reward: -1.000000\n",
      "ep 50: game finished, reward: -1.000000\n",
      "ep 50: game finished, reward: -1.000000\n",
      "resetting env. episode 51 reward total was -21.000000. running mean: -20.554347\n",
      "ep 51: game finished, reward: -1.000000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ep 51: game finished, reward: -1.000000\n",
      "ep 51: game finished, reward: -1.000000\n",
      "ep 51: game finished, reward: -1.000000\n",
      "ep 51: game finished, reward: -1.000000\n",
      "ep 51: game finished, reward: -1.000000\n",
      "ep 51: game finished, reward: -1.000000\n",
      "ep 51: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 51: game finished, reward: -1.000000\n",
      "ep 51: game finished, reward: -1.000000\n",
      "ep 51: game finished, reward: -1.000000\n",
      "ep 51: game finished, reward: -1.000000\n",
      "ep 51: game finished, reward: -1.000000\n",
      "ep 51: game finished, reward: -1.000000\n",
      "ep 51: game finished, reward: -1.000000\n",
      "ep 51: game finished, reward: -1.000000\n",
      "ep 51: game finished, reward: -1.000000\n",
      "ep 51: game finished, reward: -1.000000\n",
      "ep 51: game finished, reward: -1.000000\n",
      "ep 51: game finished, reward: -1.000000\n",
      "ep 51: game finished, reward: -1.000000\n",
      "ep 51: game finished, reward: -1.000000\n",
      "resetting env. episode 52 reward total was -20.000000. running mean: -20.548804\n",
      "ep 52: game finished, reward: -1.000000\n",
      "ep 52: game finished, reward: -1.000000\n",
      "ep 52: game finished, reward: -1.000000\n",
      "ep 52: game finished, reward: -1.000000\n",
      "ep 52: game finished, reward: -1.000000\n",
      "ep 52: game finished, reward: -1.000000\n",
      "ep 52: game finished, reward: -1.000000\n",
      "ep 52: game finished, reward: -1.000000\n",
      "ep 52: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 52: game finished, reward: -1.000000\n",
      "ep 52: game finished, reward: -1.000000\n",
      "ep 52: game finished, reward: -1.000000\n",
      "ep 52: game finished, reward: -1.000000\n",
      "ep 52: game finished, reward: -1.000000\n",
      "ep 52: game finished, reward: -1.000000\n",
      "ep 52: game finished, reward: -1.000000\n",
      "ep 52: game finished, reward: -1.000000\n",
      "ep 52: game finished, reward: -1.000000\n",
      "ep 52: game finished, reward: -1.000000\n",
      "ep 52: game finished, reward: -1.000000\n",
      "ep 52: game finished, reward: -1.000000\n",
      "ep 52: game finished, reward: -1.000000\n",
      "resetting env. episode 53 reward total was -20.000000. running mean: -20.543316\n",
      "ep 53: game finished, reward: -1.000000\n",
      "ep 53: game finished, reward: -1.000000\n",
      "ep 53: game finished, reward: -1.000000\n",
      "ep 53: game finished, reward: -1.000000\n",
      "ep 53: game finished, reward: -1.000000\n",
      "ep 53: game finished, reward: -1.000000\n",
      "ep 53: game finished, reward: -1.000000\n",
      "ep 53: game finished, reward: -1.000000\n",
      "ep 53: game finished, reward: -1.000000\n",
      "ep 53: game finished, reward: -1.000000\n",
      "ep 53: game finished, reward: -1.000000\n",
      "ep 53: game finished, reward: -1.000000\n",
      "ep 53: game finished, reward: -1.000000\n",
      "ep 53: game finished, reward: -1.000000\n",
      "ep 53: game finished, reward: -1.000000\n",
      "ep 53: game finished, reward: -1.000000\n",
      "ep 53: game finished, reward: -1.000000\n",
      "ep 53: game finished, reward: -1.000000\n",
      "ep 53: game finished, reward: -1.000000\n",
      "ep 53: game finished, reward: -1.000000\n",
      "ep 53: game finished, reward: -1.000000\n",
      "resetting env. episode 54 reward total was -21.000000. running mean: -20.547882\n",
      "ep 54: game finished, reward: -1.000000\n",
      "ep 54: game finished, reward: -1.000000\n",
      "ep 54: game finished, reward: -1.000000\n",
      "ep 54: game finished, reward: -1.000000\n",
      "ep 54: game finished, reward: -1.000000\n",
      "ep 54: game finished, reward: -1.000000\n",
      "ep 54: game finished, reward: -1.000000\n",
      "ep 54: game finished, reward: -1.000000\n",
      "ep 54: game finished, reward: -1.000000\n",
      "ep 54: game finished, reward: -1.000000\n",
      "ep 54: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 54: game finished, reward: -1.000000\n",
      "ep 54: game finished, reward: -1.000000\n",
      "ep 54: game finished, reward: -1.000000\n",
      "ep 54: game finished, reward: -1.000000\n",
      "ep 54: game finished, reward: -1.000000\n",
      "ep 54: game finished, reward: -1.000000\n",
      "ep 54: game finished, reward: -1.000000\n",
      "ep 54: game finished, reward: -1.000000\n",
      "ep 54: game finished, reward: -1.000000\n",
      "ep 54: game finished, reward: -1.000000\n",
      "ep 54: game finished, reward: -1.000000\n",
      "resetting env. episode 55 reward total was -20.000000. running mean: -20.542404\n",
      "ep 55: game finished, reward: -1.000000\n",
      "ep 55: game finished, reward: -1.000000\n",
      "ep 55: game finished, reward: -1.000000\n",
      "ep 55: game finished, reward: -1.000000\n",
      "ep 55: game finished, reward: -1.000000\n",
      "ep 55: game finished, reward: -1.000000\n",
      "ep 55: game finished, reward: -1.000000\n",
      "ep 55: game finished, reward: -1.000000\n",
      "ep 55: game finished, reward: -1.000000\n",
      "ep 55: game finished, reward: -1.000000\n",
      "ep 55: game finished, reward: -1.000000\n",
      "ep 55: game finished, reward: -1.000000\n",
      "ep 55: game finished, reward: -1.000000\n",
      "ep 55: game finished, reward: -1.000000\n",
      "ep 55: game finished, reward: -1.000000\n",
      "ep 55: game finished, reward: -1.000000\n",
      "ep 55: game finished, reward: -1.000000\n",
      "ep 55: game finished, reward: -1.000000\n",
      "ep 55: game finished, reward: -1.000000\n",
      "ep 55: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 55: game finished, reward: -1.000000\n",
      "ep 55: game finished, reward: -1.000000\n",
      "resetting env. episode 56 reward total was -20.000000. running mean: -20.536980\n",
      "ep 56: game finished, reward: -1.000000\n",
      "ep 56: game finished, reward: -1.000000\n",
      "ep 56: game finished, reward: -1.000000\n",
      "ep 56: game finished, reward: -1.000000\n",
      "ep 56: game finished, reward: -1.000000\n",
      "ep 56: game finished, reward: -1.000000\n",
      "ep 56: game finished, reward: -1.000000\n",
      "ep 56: game finished, reward: -1.000000\n",
      "ep 56: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 56: game finished, reward: -1.000000\n",
      "ep 56: game finished, reward: -1.000000\n",
      "ep 56: game finished, reward: -1.000000\n",
      "ep 56: game finished, reward: -1.000000\n",
      "ep 56: game finished, reward: -1.000000\n",
      "ep 56: game finished, reward: -1.000000\n",
      "ep 56: game finished, reward: -1.000000\n",
      "ep 56: game finished, reward: -1.000000\n",
      "ep 56: game finished, reward: -1.000000\n",
      "ep 56: game finished, reward: -1.000000\n",
      "ep 56: game finished, reward: -1.000000\n",
      "ep 56: game finished, reward: -1.000000\n",
      "ep 56: game finished, reward: -1.000000\n",
      "resetting env. episode 57 reward total was -20.000000. running mean: -20.531610\n",
      "ep 57: game finished, reward: -1.000000\n",
      "ep 57: game finished, reward: -1.000000\n",
      "ep 57: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 57: game finished, reward: -1.000000\n",
      "ep 57: game finished, reward: -1.000000\n",
      "ep 57: game finished, reward: -1.000000\n",
      "ep 57: game finished, reward: -1.000000\n",
      "ep 57: game finished, reward: -1.000000\n",
      "ep 57: game finished, reward: -1.000000\n",
      "ep 57: game finished, reward: -1.000000\n",
      "ep 57: game finished, reward: -1.000000\n",
      "ep 57: game finished, reward: -1.000000\n",
      "ep 57: game finished, reward: -1.000000\n",
      "ep 57: game finished, reward: -1.000000\n",
      "ep 57: game finished, reward: -1.000000\n",
      "ep 57: game finished, reward: -1.000000\n",
      "ep 57: game finished, reward: -1.000000\n",
      "ep 57: game finished, reward: -1.000000\n",
      "ep 57: game finished, reward: -1.000000\n",
      "ep 57: game finished, reward: -1.000000\n",
      "ep 57: game finished, reward: -1.000000\n",
      "ep 57: game finished, reward: -1.000000\n",
      "resetting env. episode 58 reward total was -20.000000. running mean: -20.526294\n",
      "ep 58: game finished, reward: -1.000000\n",
      "ep 58: game finished, reward: -1.000000\n",
      "ep 58: game finished, reward: -1.000000\n",
      "ep 58: game finished, reward: -1.000000\n",
      "ep 58: game finished, reward: -1.000000\n",
      "ep 58: game finished, reward: -1.000000\n",
      "ep 58: game finished, reward: -1.000000\n",
      "ep 58: game finished, reward: -1.000000\n",
      "ep 58: game finished, reward: -1.000000\n",
      "ep 58: game finished, reward: -1.000000\n",
      "ep 58: game finished, reward: -1.000000\n",
      "ep 58: game finished, reward: -1.000000\n",
      "ep 58: game finished, reward: -1.000000\n",
      "ep 58: game finished, reward: -1.000000\n",
      "ep 58: game finished, reward: -1.000000\n",
      "ep 58: game finished, reward: -1.000000\n",
      "ep 58: game finished, reward: -1.000000\n",
      "ep 58: game finished, reward: -1.000000\n",
      "ep 58: game finished, reward: -1.000000\n",
      "ep 58: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 58: game finished, reward: -1.000000\n",
      "ep 58: game finished, reward: -1.000000\n",
      "resetting env. episode 59 reward total was -20.000000. running mean: -20.521031\n",
      "ep 59: game finished, reward: -1.000000\n",
      "ep 59: game finished, reward: -1.000000\n",
      "ep 59: game finished, reward: -1.000000\n",
      "ep 59: game finished, reward: -1.000000\n",
      "ep 59: game finished, reward: -1.000000\n",
      "ep 59: game finished, reward: -1.000000\n",
      "ep 59: game finished, reward: -1.000000\n",
      "ep 59: game finished, reward: -1.000000\n",
      "ep 59: game finished, reward: -1.000000\n",
      "ep 59: game finished, reward: -1.000000\n",
      "ep 59: game finished, reward: -1.000000\n",
      "ep 59: game finished, reward: -1.000000\n",
      "ep 59: game finished, reward: -1.000000\n",
      "ep 59: game finished, reward: -1.000000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ep 59: game finished, reward: -1.000000\n",
      "ep 59: game finished, reward: -1.000000\n",
      "ep 59: game finished, reward: -1.000000\n",
      "ep 59: game finished, reward: -1.000000\n",
      "ep 59: game finished, reward: -1.000000\n",
      "ep 59: game finished, reward: -1.000000\n",
      "ep 59: game finished, reward: -1.000000\n",
      "resetting env. episode 60 reward total was -21.000000. running mean: -20.525820\n",
      "ep 60: game finished, reward: -1.000000\n",
      "ep 60: game finished, reward: -1.000000\n",
      "ep 60: game finished, reward: -1.000000\n",
      "ep 60: game finished, reward: -1.000000\n",
      "ep 60: game finished, reward: -1.000000\n",
      "ep 60: game finished, reward: -1.000000\n",
      "ep 60: game finished, reward: -1.000000\n",
      "ep 60: game finished, reward: -1.000000\n",
      "ep 60: game finished, reward: -1.000000\n",
      "ep 60: game finished, reward: -1.000000\n",
      "ep 60: game finished, reward: -1.000000\n",
      "ep 60: game finished, reward: -1.000000\n",
      "ep 60: game finished, reward: -1.000000\n",
      "ep 60: game finished, reward: -1.000000\n",
      "ep 60: game finished, reward: -1.000000\n",
      "ep 60: game finished, reward: -1.000000\n",
      "ep 60: game finished, reward: -1.000000\n",
      "ep 60: game finished, reward: -1.000000\n",
      "ep 60: game finished, reward: -1.000000\n",
      "ep 60: game finished, reward: -1.000000\n",
      "ep 60: game finished, reward: -1.000000\n",
      "resetting env. episode 61 reward total was -21.000000. running mean: -20.530562\n",
      "ep 61: game finished, reward: -1.000000\n",
      "ep 61: game finished, reward: -1.000000\n",
      "ep 61: game finished, reward: -1.000000\n",
      "ep 61: game finished, reward: -1.000000\n",
      "ep 61: game finished, reward: -1.000000\n",
      "ep 61: game finished, reward: -1.000000\n",
      "ep 61: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 61: game finished, reward: -1.000000\n",
      "ep 61: game finished, reward: -1.000000\n",
      "ep 61: game finished, reward: -1.000000\n",
      "ep 61: game finished, reward: -1.000000\n",
      "ep 61: game finished, reward: -1.000000\n",
      "ep 61: game finished, reward: -1.000000\n",
      "ep 61: game finished, reward: -1.000000\n",
      "ep 61: game finished, reward: -1.000000\n",
      "ep 61: game finished, reward: -1.000000\n",
      "ep 61: game finished, reward: -1.000000\n",
      "ep 61: game finished, reward: -1.000000\n",
      "ep 61: game finished, reward: -1.000000\n",
      "ep 61: game finished, reward: -1.000000\n",
      "ep 61: game finished, reward: -1.000000\n",
      "ep 61: game finished, reward: -1.000000\n",
      "resetting env. episode 62 reward total was -20.000000. running mean: -20.525257\n",
      "ep 62: game finished, reward: -1.000000\n",
      "ep 62: game finished, reward: -1.000000\n",
      "ep 62: game finished, reward: -1.000000\n",
      "ep 62: game finished, reward: -1.000000\n",
      "ep 62: game finished, reward: -1.000000\n",
      "ep 62: game finished, reward: -1.000000\n",
      "ep 62: game finished, reward: -1.000000\n",
      "ep 62: game finished, reward: -1.000000\n",
      "ep 62: game finished, reward: -1.000000\n",
      "ep 62: game finished, reward: -1.000000\n",
      "ep 62: game finished, reward: -1.000000\n",
      "ep 62: game finished, reward: -1.000000\n",
      "ep 62: game finished, reward: -1.000000\n",
      "ep 62: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 62: game finished, reward: -1.000000\n",
      "ep 62: game finished, reward: -1.000000\n",
      "ep 62: game finished, reward: -1.000000\n",
      "ep 62: game finished, reward: -1.000000\n",
      "ep 62: game finished, reward: -1.000000\n",
      "ep 62: game finished, reward: -1.000000\n",
      "ep 62: game finished, reward: -1.000000\n",
      "ep 62: game finished, reward: -1.000000\n",
      "resetting env. episode 63 reward total was -20.000000. running mean: -20.520004\n",
      "ep 63: game finished, reward: -1.000000\n",
      "ep 63: game finished, reward: -1.000000\n",
      "ep 63: game finished, reward: -1.000000\n",
      "ep 63: game finished, reward: -1.000000\n",
      "ep 63: game finished, reward: -1.000000\n",
      "ep 63: game finished, reward: -1.000000\n",
      "ep 63: game finished, reward: -1.000000\n",
      "ep 63: game finished, reward: -1.000000\n",
      "ep 63: game finished, reward: -1.000000\n",
      "ep 63: game finished, reward: -1.000000\n",
      "ep 63: game finished, reward: -1.000000\n",
      "ep 63: game finished, reward: -1.000000\n",
      "ep 63: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 63: game finished, reward: -1.000000\n",
      "ep 63: game finished, reward: -1.000000\n",
      "ep 63: game finished, reward: -1.000000\n",
      "ep 63: game finished, reward: -1.000000\n",
      "ep 63: game finished, reward: -1.000000\n",
      "ep 63: game finished, reward: -1.000000\n",
      "ep 63: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 63: game finished, reward: -1.000000\n",
      "ep 63: game finished, reward: -1.000000\n",
      "ep 63: game finished, reward: -1.000000\n",
      "resetting env. episode 64 reward total was -19.000000. running mean: -20.504804\n",
      "ep 64: game finished, reward: -1.000000\n",
      "ep 64: game finished, reward: -1.000000\n",
      "ep 64: game finished, reward: -1.000000\n",
      "ep 64: game finished, reward: -1.000000\n",
      "ep 64: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 64: game finished, reward: -1.000000\n",
      "ep 64: game finished, reward: -1.000000\n",
      "ep 64: game finished, reward: -1.000000\n",
      "ep 64: game finished, reward: -1.000000\n",
      "ep 64: game finished, reward: -1.000000\n",
      "ep 64: game finished, reward: -1.000000\n",
      "ep 64: game finished, reward: -1.000000\n",
      "ep 64: game finished, reward: -1.000000\n",
      "ep 64: game finished, reward: -1.000000\n",
      "ep 64: game finished, reward: -1.000000\n",
      "ep 64: game finished, reward: -1.000000\n",
      "ep 64: game finished, reward: -1.000000\n",
      "ep 64: game finished, reward: -1.000000\n",
      "ep 64: game finished, reward: -1.000000\n",
      "ep 64: game finished, reward: -1.000000\n",
      "ep 64: game finished, reward: -1.000000\n",
      "ep 64: game finished, reward: -1.000000\n",
      "resetting env. episode 65 reward total was -20.000000. running mean: -20.499756\n",
      "ep 65: game finished, reward: -1.000000\n",
      "ep 65: game finished, reward: -1.000000\n",
      "ep 65: game finished, reward: -1.000000\n",
      "ep 65: game finished, reward: -1.000000\n",
      "ep 65: game finished, reward: -1.000000\n",
      "ep 65: game finished, reward: -1.000000\n",
      "ep 65: game finished, reward: -1.000000\n",
      "ep 65: game finished, reward: -1.000000\n",
      "ep 65: game finished, reward: -1.000000\n",
      "ep 65: game finished, reward: -1.000000\n",
      "ep 65: game finished, reward: -1.000000\n",
      "ep 65: game finished, reward: -1.000000\n",
      "ep 65: game finished, reward: -1.000000\n",
      "ep 65: game finished, reward: -1.000000\n",
      "ep 65: game finished, reward: -1.000000\n",
      "ep 65: game finished, reward: -1.000000\n",
      "ep 65: game finished, reward: -1.000000\n",
      "ep 65: game finished, reward: -1.000000\n",
      "ep 65: game finished, reward: -1.000000\n",
      "ep 65: game finished, reward: -1.000000\n",
      "ep 65: game finished, reward: -1.000000\n",
      "resetting env. episode 66 reward total was -21.000000. running mean: -20.504758\n",
      "ep 66: game finished, reward: -1.000000\n",
      "ep 66: game finished, reward: -1.000000\n",
      "ep 66: game finished, reward: -1.000000\n",
      "ep 66: game finished, reward: -1.000000\n",
      "ep 66: game finished, reward: -1.000000\n",
      "ep 66: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 66: game finished, reward: -1.000000\n",
      "ep 66: game finished, reward: -1.000000\n",
      "ep 66: game finished, reward: -1.000000\n",
      "ep 66: game finished, reward: -1.000000\n",
      "ep 66: game finished, reward: -1.000000\n",
      "ep 66: game finished, reward: -1.000000\n",
      "ep 66: game finished, reward: -1.000000\n",
      "ep 66: game finished, reward: -1.000000\n",
      "ep 66: game finished, reward: -1.000000\n",
      "ep 66: game finished, reward: -1.000000\n",
      "ep 66: game finished, reward: -1.000000\n",
      "ep 66: game finished, reward: -1.000000\n",
      "ep 66: game finished, reward: -1.000000\n",
      "ep 66: game finished, reward: -1.000000\n",
      "ep 66: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 66: game finished, reward: -1.000000\n",
      "ep 66: game finished, reward: -1.000000\n",
      "resetting env. episode 67 reward total was -19.000000. running mean: -20.489711\n",
      "ep 67: game finished, reward: -1.000000\n",
      "ep 67: game finished, reward: -1.000000\n",
      "ep 67: game finished, reward: -1.000000\n",
      "ep 67: game finished, reward: -1.000000\n",
      "ep 67: game finished, reward: -1.000000\n",
      "ep 67: game finished, reward: -1.000000\n",
      "ep 67: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 67: game finished, reward: -1.000000\n",
      "ep 67: game finished, reward: -1.000000\n",
      "ep 67: game finished, reward: -1.000000\n",
      "ep 67: game finished, reward: -1.000000\n",
      "ep 67: game finished, reward: -1.000000\n",
      "ep 67: game finished, reward: -1.000000\n",
      "ep 67: game finished, reward: -1.000000\n",
      "ep 67: game finished, reward: -1.000000\n",
      "ep 67: game finished, reward: -1.000000\n",
      "ep 67: game finished, reward: -1.000000\n",
      "ep 67: game finished, reward: -1.000000\n",
      "ep 67: game finished, reward: -1.000000\n",
      "ep 67: game finished, reward: -1.000000\n",
      "ep 67: game finished, reward: -1.000000\n",
      "ep 67: game finished, reward: -1.000000\n",
      "resetting env. episode 68 reward total was -20.000000. running mean: -20.484814\n",
      "ep 68: game finished, reward: -1.000000\n",
      "ep 68: game finished, reward: -1.000000\n",
      "ep 68: game finished, reward: -1.000000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ep 68: game finished, reward: -1.000000\n",
      "ep 68: game finished, reward: -1.000000\n",
      "ep 68: game finished, reward: -1.000000\n",
      "ep 68: game finished, reward: -1.000000\n",
      "ep 68: game finished, reward: -1.000000\n",
      "ep 68: game finished, reward: -1.000000\n",
      "ep 68: game finished, reward: -1.000000\n",
      "ep 68: game finished, reward: -1.000000\n",
      "ep 68: game finished, reward: -1.000000\n",
      "ep 68: game finished, reward: -1.000000\n",
      "ep 68: game finished, reward: -1.000000\n",
      "ep 68: game finished, reward: -1.000000\n",
      "ep 68: game finished, reward: -1.000000\n",
      "ep 68: game finished, reward: -1.000000\n",
      "ep 68: game finished, reward: -1.000000\n",
      "ep 68: game finished, reward: -1.000000\n",
      "ep 68: game finished, reward: -1.000000\n",
      "ep 68: game finished, reward: -1.000000\n",
      "resetting env. episode 69 reward total was -21.000000. running mean: -20.489966\n",
      "ep 69: game finished, reward: -1.000000\n",
      "ep 69: game finished, reward: -1.000000\n",
      "ep 69: game finished, reward: -1.000000\n",
      "ep 69: game finished, reward: -1.000000\n",
      "ep 69: game finished, reward: -1.000000\n",
      "ep 69: game finished, reward: -1.000000\n",
      "ep 69: game finished, reward: -1.000000\n",
      "ep 69: game finished, reward: -1.000000\n",
      "ep 69: game finished, reward: -1.000000\n",
      "ep 69: game finished, reward: -1.000000\n",
      "ep 69: game finished, reward: -1.000000\n",
      "ep 69: game finished, reward: -1.000000\n",
      "ep 69: game finished, reward: -1.000000\n",
      "ep 69: game finished, reward: -1.000000\n",
      "ep 69: game finished, reward: -1.000000\n",
      "ep 69: game finished, reward: -1.000000\n",
      "ep 69: game finished, reward: -1.000000\n",
      "ep 69: game finished, reward: -1.000000\n",
      "ep 69: game finished, reward: -1.000000\n",
      "ep 69: game finished, reward: -1.000000\n",
      "ep 69: game finished, reward: -1.000000\n",
      "resetting env. episode 70 reward total was -21.000000. running mean: -20.495066\n",
      "ep 70: game finished, reward: -1.000000\n",
      "ep 70: game finished, reward: -1.000000\n",
      "ep 70: game finished, reward: -1.000000\n",
      "ep 70: game finished, reward: -1.000000\n",
      "ep 70: game finished, reward: -1.000000\n",
      "ep 70: game finished, reward: -1.000000\n",
      "ep 70: game finished, reward: -1.000000\n",
      "ep 70: game finished, reward: -1.000000\n",
      "ep 70: game finished, reward: -1.000000\n",
      "ep 70: game finished, reward: -1.000000\n",
      "ep 70: game finished, reward: -1.000000\n",
      "ep 70: game finished, reward: -1.000000\n",
      "ep 70: game finished, reward: -1.000000\n",
      "ep 70: game finished, reward: -1.000000\n",
      "ep 70: game finished, reward: -1.000000\n",
      "ep 70: game finished, reward: -1.000000\n",
      "ep 70: game finished, reward: -1.000000\n",
      "ep 70: game finished, reward: -1.000000\n",
      "ep 70: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 70: game finished, reward: -1.000000\n",
      "ep 70: game finished, reward: -1.000000\n",
      "ep 70: game finished, reward: -1.000000\n",
      "ep 70: game finished, reward: 1.000000 !!!!!!!!\n",
      "resetting env. episode 71 reward total was -19.000000. running mean: -20.480115\n",
      "ep 71: game finished, reward: -1.000000\n",
      "ep 71: game finished, reward: -1.000000\n",
      "ep 71: game finished, reward: -1.000000\n",
      "ep 71: game finished, reward: -1.000000\n",
      "ep 71: game finished, reward: -1.000000\n",
      "ep 71: game finished, reward: -1.000000\n",
      "ep 71: game finished, reward: -1.000000\n",
      "ep 71: game finished, reward: -1.000000\n",
      "ep 71: game finished, reward: -1.000000\n",
      "ep 71: game finished, reward: -1.000000\n",
      "ep 71: game finished, reward: -1.000000\n",
      "ep 71: game finished, reward: -1.000000\n",
      "ep 71: game finished, reward: -1.000000\n",
      "ep 71: game finished, reward: -1.000000\n",
      "ep 71: game finished, reward: -1.000000\n",
      "ep 71: game finished, reward: -1.000000\n",
      "ep 71: game finished, reward: -1.000000\n",
      "ep 71: game finished, reward: -1.000000\n",
      "ep 71: game finished, reward: -1.000000\n",
      "ep 71: game finished, reward: -1.000000\n",
      "ep 71: game finished, reward: -1.000000\n",
      "resetting env. episode 72 reward total was -21.000000. running mean: -20.485314\n",
      "ep 72: game finished, reward: -1.000000\n",
      "ep 72: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 72: game finished, reward: -1.000000\n",
      "ep 72: game finished, reward: -1.000000\n",
      "ep 72: game finished, reward: -1.000000\n",
      "ep 72: game finished, reward: -1.000000\n",
      "ep 72: game finished, reward: -1.000000\n",
      "ep 72: game finished, reward: -1.000000\n",
      "ep 72: game finished, reward: -1.000000\n",
      "ep 72: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 72: game finished, reward: -1.000000\n",
      "ep 72: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 72: game finished, reward: -1.000000\n",
      "ep 72: game finished, reward: -1.000000\n",
      "ep 72: game finished, reward: -1.000000\n",
      "ep 72: game finished, reward: -1.000000\n",
      "ep 72: game finished, reward: -1.000000\n",
      "ep 72: game finished, reward: -1.000000\n",
      "ep 72: game finished, reward: -1.000000\n",
      "ep 72: game finished, reward: -1.000000\n",
      "ep 72: game finished, reward: -1.000000\n",
      "ep 72: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 72: game finished, reward: -1.000000\n",
      "ep 72: game finished, reward: -1.000000\n",
      "ep 72: game finished, reward: -1.000000\n",
      "resetting env. episode 73 reward total was -17.000000. running mean: -20.450461\n",
      "ep 73: game finished, reward: -1.000000\n",
      "ep 73: game finished, reward: -1.000000\n",
      "ep 73: game finished, reward: -1.000000\n",
      "ep 73: game finished, reward: -1.000000\n",
      "ep 73: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 73: game finished, reward: -1.000000\n",
      "ep 73: game finished, reward: -1.000000\n",
      "ep 73: game finished, reward: -1.000000\n",
      "ep 73: game finished, reward: -1.000000\n",
      "ep 73: game finished, reward: -1.000000\n",
      "ep 73: game finished, reward: -1.000000\n",
      "ep 73: game finished, reward: -1.000000\n",
      "ep 73: game finished, reward: -1.000000\n",
      "ep 73: game finished, reward: -1.000000\n",
      "ep 73: game finished, reward: -1.000000\n",
      "ep 73: game finished, reward: -1.000000\n",
      "ep 73: game finished, reward: -1.000000\n",
      "ep 73: game finished, reward: -1.000000\n",
      "ep 73: game finished, reward: -1.000000\n",
      "ep 73: game finished, reward: -1.000000\n",
      "ep 73: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 73: game finished, reward: -1.000000\n",
      "ep 73: game finished, reward: -1.000000\n",
      "resetting env. episode 74 reward total was -19.000000. running mean: -20.435956\n",
      "ep 74: game finished, reward: -1.000000\n",
      "ep 74: game finished, reward: -1.000000\n",
      "ep 74: game finished, reward: -1.000000\n",
      "ep 74: game finished, reward: -1.000000\n",
      "ep 74: game finished, reward: -1.000000\n",
      "ep 74: game finished, reward: -1.000000\n",
      "ep 74: game finished, reward: -1.000000\n",
      "ep 74: game finished, reward: -1.000000\n",
      "ep 74: game finished, reward: -1.000000\n",
      "ep 74: game finished, reward: -1.000000\n",
      "ep 74: game finished, reward: -1.000000\n",
      "ep 74: game finished, reward: -1.000000\n",
      "ep 74: game finished, reward: -1.000000\n",
      "ep 74: game finished, reward: -1.000000\n",
      "ep 74: game finished, reward: -1.000000\n",
      "ep 74: game finished, reward: -1.000000\n",
      "ep 74: game finished, reward: -1.000000\n",
      "ep 74: game finished, reward: -1.000000\n",
      "ep 74: game finished, reward: -1.000000\n",
      "ep 74: game finished, reward: -1.000000\n",
      "ep 74: game finished, reward: -1.000000\n",
      "resetting env. episode 75 reward total was -21.000000. running mean: -20.441597\n",
      "ep 75: game finished, reward: -1.000000\n",
      "ep 75: game finished, reward: -1.000000\n",
      "ep 75: game finished, reward: -1.000000\n",
      "ep 75: game finished, reward: -1.000000\n",
      "ep 75: game finished, reward: -1.000000\n",
      "ep 75: game finished, reward: -1.000000\n",
      "ep 75: game finished, reward: -1.000000\n",
      "ep 75: game finished, reward: -1.000000\n",
      "ep 75: game finished, reward: -1.000000\n",
      "ep 75: game finished, reward: -1.000000\n",
      "ep 75: game finished, reward: -1.000000\n",
      "ep 75: game finished, reward: -1.000000\n",
      "ep 75: game finished, reward: -1.000000\n",
      "ep 75: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 75: game finished, reward: -1.000000\n",
      "ep 75: game finished, reward: -1.000000\n",
      "ep 75: game finished, reward: -1.000000\n",
      "ep 75: game finished, reward: -1.000000\n",
      "ep 75: game finished, reward: -1.000000\n",
      "ep 75: game finished, reward: -1.000000\n",
      "ep 75: game finished, reward: -1.000000\n",
      "ep 75: game finished, reward: -1.000000\n",
      "resetting env. episode 76 reward total was -20.000000. running mean: -20.437181\n",
      "ep 76: game finished, reward: -1.000000\n",
      "ep 76: game finished, reward: -1.000000\n",
      "ep 76: game finished, reward: -1.000000\n",
      "ep 76: game finished, reward: -1.000000\n",
      "ep 76: game finished, reward: -1.000000\n",
      "ep 76: game finished, reward: -1.000000\n",
      "ep 76: game finished, reward: -1.000000\n",
      "ep 76: game finished, reward: -1.000000\n",
      "ep 76: game finished, reward: -1.000000\n",
      "ep 76: game finished, reward: -1.000000\n",
      "ep 76: game finished, reward: -1.000000\n",
      "ep 76: game finished, reward: -1.000000\n",
      "ep 76: game finished, reward: -1.000000\n",
      "ep 76: game finished, reward: -1.000000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ep 76: game finished, reward: -1.000000\n",
      "ep 76: game finished, reward: -1.000000\n",
      "ep 76: game finished, reward: -1.000000\n",
      "ep 76: game finished, reward: -1.000000\n",
      "ep 76: game finished, reward: -1.000000\n",
      "ep 76: game finished, reward: -1.000000\n",
      "ep 76: game finished, reward: -1.000000\n",
      "resetting env. episode 77 reward total was -21.000000. running mean: -20.442809\n",
      "ep 77: game finished, reward: -1.000000\n",
      "ep 77: game finished, reward: -1.000000\n",
      "ep 77: game finished, reward: -1.000000\n",
      "ep 77: game finished, reward: -1.000000\n",
      "ep 77: game finished, reward: -1.000000\n",
      "ep 77: game finished, reward: -1.000000\n",
      "ep 77: game finished, reward: -1.000000\n",
      "ep 77: game finished, reward: -1.000000\n",
      "ep 77: game finished, reward: -1.000000\n",
      "ep 77: game finished, reward: -1.000000\n",
      "ep 77: game finished, reward: -1.000000\n",
      "ep 77: game finished, reward: -1.000000\n",
      "ep 77: game finished, reward: -1.000000\n",
      "ep 77: game finished, reward: -1.000000\n",
      "ep 77: game finished, reward: -1.000000\n",
      "ep 77: game finished, reward: -1.000000\n",
      "ep 77: game finished, reward: -1.000000\n",
      "ep 77: game finished, reward: -1.000000\n",
      "ep 77: game finished, reward: -1.000000\n",
      "ep 77: game finished, reward: -1.000000\n",
      "ep 77: game finished, reward: -1.000000\n",
      "resetting env. episode 78 reward total was -21.000000. running mean: -20.448381\n",
      "ep 78: game finished, reward: -1.000000\n",
      "ep 78: game finished, reward: -1.000000\n",
      "ep 78: game finished, reward: -1.000000\n",
      "ep 78: game finished, reward: -1.000000\n",
      "ep 78: game finished, reward: -1.000000\n",
      "ep 78: game finished, reward: -1.000000\n",
      "ep 78: game finished, reward: -1.000000\n",
      "ep 78: game finished, reward: -1.000000\n",
      "ep 78: game finished, reward: -1.000000\n",
      "ep 78: game finished, reward: -1.000000\n",
      "ep 78: game finished, reward: -1.000000\n",
      "ep 78: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 78: game finished, reward: -1.000000\n",
      "ep 78: game finished, reward: -1.000000\n",
      "ep 78: game finished, reward: -1.000000\n",
      "ep 78: game finished, reward: -1.000000\n",
      "ep 78: game finished, reward: -1.000000\n",
      "ep 78: game finished, reward: -1.000000\n",
      "ep 78: game finished, reward: -1.000000\n",
      "ep 78: game finished, reward: -1.000000\n",
      "ep 78: game finished, reward: -1.000000\n",
      "ep 78: game finished, reward: -1.000000\n",
      "resetting env. episode 79 reward total was -20.000000. running mean: -20.443897\n",
      "ep 79: game finished, reward: -1.000000\n",
      "ep 79: game finished, reward: -1.000000\n",
      "ep 79: game finished, reward: -1.000000\n",
      "ep 79: game finished, reward: -1.000000\n",
      "ep 79: game finished, reward: -1.000000\n",
      "ep 79: game finished, reward: -1.000000\n",
      "ep 79: game finished, reward: -1.000000\n",
      "ep 79: game finished, reward: -1.000000\n",
      "ep 79: game finished, reward: -1.000000\n",
      "ep 79: game finished, reward: -1.000000\n",
      "ep 79: game finished, reward: -1.000000\n",
      "ep 79: game finished, reward: -1.000000\n",
      "ep 79: game finished, reward: -1.000000\n",
      "ep 79: game finished, reward: -1.000000\n",
      "ep 79: game finished, reward: -1.000000\n",
      "ep 79: game finished, reward: -1.000000\n",
      "ep 79: game finished, reward: -1.000000\n",
      "ep 79: game finished, reward: -1.000000\n",
      "ep 79: game finished, reward: -1.000000\n",
      "ep 79: game finished, reward: -1.000000\n",
      "ep 79: game finished, reward: -1.000000\n",
      "resetting env. episode 80 reward total was -21.000000. running mean: -20.449458\n",
      "ep 80: game finished, reward: -1.000000\n",
      "ep 80: game finished, reward: -1.000000\n",
      "ep 80: game finished, reward: -1.000000\n",
      "ep 80: game finished, reward: -1.000000\n",
      "ep 80: game finished, reward: -1.000000\n",
      "ep 80: game finished, reward: -1.000000\n",
      "ep 80: game finished, reward: -1.000000\n",
      "ep 80: game finished, reward: -1.000000\n",
      "ep 80: game finished, reward: -1.000000\n",
      "ep 80: game finished, reward: -1.000000\n",
      "ep 80: game finished, reward: -1.000000\n",
      "ep 80: game finished, reward: -1.000000\n",
      "ep 80: game finished, reward: -1.000000\n",
      "ep 80: game finished, reward: -1.000000\n",
      "ep 80: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 80: game finished, reward: -1.000000\n",
      "ep 80: game finished, reward: -1.000000\n",
      "ep 80: game finished, reward: -1.000000\n",
      "ep 80: game finished, reward: -1.000000\n",
      "ep 80: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 80: game finished, reward: -1.000000\n",
      "ep 80: game finished, reward: -1.000000\n",
      "ep 80: game finished, reward: -1.000000\n",
      "resetting env. episode 81 reward total was -19.000000. running mean: -20.434964\n",
      "ep 81: game finished, reward: -1.000000\n",
      "ep 81: game finished, reward: -1.000000\n",
      "ep 81: game finished, reward: -1.000000\n",
      "ep 81: game finished, reward: -1.000000\n",
      "ep 81: game finished, reward: -1.000000\n",
      "ep 81: game finished, reward: -1.000000\n",
      "ep 81: game finished, reward: -1.000000\n",
      "ep 81: game finished, reward: -1.000000\n",
      "ep 81: game finished, reward: -1.000000\n",
      "ep 81: game finished, reward: -1.000000\n",
      "ep 81: game finished, reward: -1.000000\n",
      "ep 81: game finished, reward: -1.000000\n",
      "ep 81: game finished, reward: -1.000000\n",
      "ep 81: game finished, reward: -1.000000\n",
      "ep 81: game finished, reward: -1.000000\n",
      "ep 81: game finished, reward: -1.000000\n",
      "ep 81: game finished, reward: -1.000000\n",
      "ep 81: game finished, reward: -1.000000\n",
      "ep 81: game finished, reward: -1.000000\n",
      "ep 81: game finished, reward: -1.000000\n",
      "ep 81: game finished, reward: -1.000000\n",
      "resetting env. episode 82 reward total was -21.000000. running mean: -20.440614\n",
      "ep 82: game finished, reward: -1.000000\n",
      "ep 82: game finished, reward: -1.000000\n",
      "ep 82: game finished, reward: -1.000000\n",
      "ep 82: game finished, reward: -1.000000\n",
      "ep 82: game finished, reward: -1.000000\n",
      "ep 82: game finished, reward: -1.000000\n",
      "ep 82: game finished, reward: -1.000000\n",
      "ep 82: game finished, reward: -1.000000\n",
      "ep 82: game finished, reward: -1.000000\n",
      "ep 82: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 82: game finished, reward: -1.000000\n",
      "ep 82: game finished, reward: -1.000000\n",
      "ep 82: game finished, reward: -1.000000\n",
      "ep 82: game finished, reward: -1.000000\n",
      "ep 82: game finished, reward: -1.000000\n",
      "ep 82: game finished, reward: -1.000000\n",
      "ep 82: game finished, reward: -1.000000\n",
      "ep 82: game finished, reward: -1.000000\n",
      "ep 82: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 82: game finished, reward: -1.000000\n",
      "ep 82: game finished, reward: -1.000000\n",
      "ep 82: game finished, reward: -1.000000\n",
      "ep 82: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 82: game finished, reward: -1.000000\n",
      "resetting env. episode 83 reward total was -18.000000. running mean: -20.416208\n",
      "ep 83: game finished, reward: -1.000000\n",
      "ep 83: game finished, reward: -1.000000\n",
      "ep 83: game finished, reward: -1.000000\n",
      "ep 83: game finished, reward: -1.000000\n",
      "ep 83: game finished, reward: -1.000000\n",
      "ep 83: game finished, reward: -1.000000\n",
      "ep 83: game finished, reward: -1.000000\n",
      "ep 83: game finished, reward: -1.000000\n",
      "ep 83: game finished, reward: -1.000000\n",
      "ep 83: game finished, reward: -1.000000\n",
      "ep 83: game finished, reward: -1.000000\n",
      "ep 83: game finished, reward: -1.000000\n",
      "ep 83: game finished, reward: -1.000000\n",
      "ep 83: game finished, reward: -1.000000\n",
      "ep 83: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 83: game finished, reward: -1.000000\n",
      "ep 83: game finished, reward: -1.000000\n",
      "ep 83: game finished, reward: -1.000000\n",
      "ep 83: game finished, reward: -1.000000\n",
      "ep 83: game finished, reward: -1.000000\n",
      "ep 83: game finished, reward: -1.000000\n",
      "ep 83: game finished, reward: -1.000000\n",
      "resetting env. episode 84 reward total was -20.000000. running mean: -20.412046\n",
      "ep 84: game finished, reward: -1.000000\n",
      "ep 84: game finished, reward: -1.000000\n",
      "ep 84: game finished, reward: -1.000000\n",
      "ep 84: game finished, reward: -1.000000\n",
      "ep 84: game finished, reward: -1.000000\n",
      "ep 84: game finished, reward: -1.000000\n",
      "ep 84: game finished, reward: -1.000000\n",
      "ep 84: game finished, reward: -1.000000\n",
      "ep 84: game finished, reward: -1.000000\n",
      "ep 84: game finished, reward: -1.000000\n",
      "ep 84: game finished, reward: -1.000000\n",
      "ep 84: game finished, reward: -1.000000\n",
      "ep 84: game finished, reward: -1.000000\n",
      "ep 84: game finished, reward: -1.000000\n",
      "ep 84: game finished, reward: -1.000000\n",
      "ep 84: game finished, reward: -1.000000\n",
      "ep 84: game finished, reward: -1.000000\n",
      "ep 84: game finished, reward: -1.000000\n",
      "ep 84: game finished, reward: -1.000000\n",
      "ep 84: game finished, reward: -1.000000\n",
      "ep 84: game finished, reward: -1.000000\n",
      "resetting env. episode 85 reward total was -21.000000. running mean: -20.417925\n",
      "ep 85: game finished, reward: -1.000000\n",
      "ep 85: game finished, reward: -1.000000\n",
      "ep 85: game finished, reward: -1.000000\n",
      "ep 85: game finished, reward: -1.000000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ep 85: game finished, reward: -1.000000\n",
      "ep 85: game finished, reward: -1.000000\n",
      "ep 85: game finished, reward: -1.000000\n",
      "ep 85: game finished, reward: -1.000000\n",
      "ep 85: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 85: game finished, reward: -1.000000\n",
      "ep 85: game finished, reward: -1.000000\n",
      "ep 85: game finished, reward: -1.000000\n",
      "ep 85: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 85: game finished, reward: -1.000000\n",
      "ep 85: game finished, reward: -1.000000\n",
      "ep 85: game finished, reward: -1.000000\n",
      "ep 85: game finished, reward: -1.000000\n",
      "ep 85: game finished, reward: -1.000000\n",
      "ep 85: game finished, reward: -1.000000\n",
      "ep 85: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 85: game finished, reward: -1.000000\n",
      "ep 85: game finished, reward: -1.000000\n",
      "ep 85: game finished, reward: -1.000000\n",
      "ep 85: game finished, reward: -1.000000\n",
      "resetting env. episode 86 reward total was -18.000000. running mean: -20.393746\n",
      "ep 86: game finished, reward: -1.000000\n",
      "ep 86: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 86: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 86: game finished, reward: -1.000000\n",
      "ep 86: game finished, reward: -1.000000\n",
      "ep 86: game finished, reward: -1.000000\n",
      "ep 86: game finished, reward: -1.000000\n",
      "ep 86: game finished, reward: -1.000000\n",
      "ep 86: game finished, reward: -1.000000\n",
      "ep 86: game finished, reward: -1.000000\n",
      "ep 86: game finished, reward: -1.000000\n",
      "ep 86: game finished, reward: -1.000000\n",
      "ep 86: game finished, reward: -1.000000\n",
      "ep 86: game finished, reward: -1.000000\n",
      "ep 86: game finished, reward: -1.000000\n",
      "ep 86: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 86: game finished, reward: -1.000000\n",
      "ep 86: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 86: game finished, reward: -1.000000\n",
      "ep 86: game finished, reward: -1.000000\n",
      "ep 86: game finished, reward: -1.000000\n",
      "ep 86: game finished, reward: -1.000000\n",
      "ep 86: game finished, reward: -1.000000\n",
      "ep 86: game finished, reward: -1.000000\n",
      "ep 86: game finished, reward: -1.000000\n",
      "resetting env. episode 87 reward total was -17.000000. running mean: -20.359808\n",
      "ep 87: game finished, reward: -1.000000\n",
      "ep 87: game finished, reward: -1.000000\n",
      "ep 87: game finished, reward: -1.000000\n",
      "ep 87: game finished, reward: -1.000000\n",
      "ep 87: game finished, reward: -1.000000\n",
      "ep 87: game finished, reward: -1.000000\n",
      "ep 87: game finished, reward: -1.000000\n",
      "ep 87: game finished, reward: -1.000000\n",
      "ep 87: game finished, reward: -1.000000\n",
      "ep 87: game finished, reward: -1.000000\n",
      "ep 87: game finished, reward: -1.000000\n",
      "ep 87: game finished, reward: -1.000000\n",
      "ep 87: game finished, reward: -1.000000\n",
      "ep 87: game finished, reward: -1.000000\n",
      "ep 87: game finished, reward: -1.000000\n",
      "ep 87: game finished, reward: -1.000000\n",
      "ep 87: game finished, reward: -1.000000\n",
      "ep 87: game finished, reward: -1.000000\n",
      "ep 87: game finished, reward: -1.000000\n",
      "ep 87: game finished, reward: -1.000000\n",
      "ep 87: game finished, reward: -1.000000\n",
      "resetting env. episode 88 reward total was -21.000000. running mean: -20.366210\n",
      "ep 88: game finished, reward: -1.000000\n",
      "ep 88: game finished, reward: -1.000000\n",
      "ep 88: game finished, reward: -1.000000\n",
      "ep 88: game finished, reward: -1.000000\n",
      "ep 88: game finished, reward: -1.000000\n",
      "ep 88: game finished, reward: -1.000000\n",
      "ep 88: game finished, reward: -1.000000\n",
      "ep 88: game finished, reward: -1.000000\n",
      "ep 88: game finished, reward: -1.000000\n",
      "ep 88: game finished, reward: -1.000000\n",
      "ep 88: game finished, reward: -1.000000\n",
      "ep 88: game finished, reward: -1.000000\n",
      "ep 88: game finished, reward: -1.000000\n",
      "ep 88: game finished, reward: -1.000000\n",
      "ep 88: game finished, reward: -1.000000\n",
      "ep 88: game finished, reward: -1.000000\n",
      "ep 88: game finished, reward: -1.000000\n",
      "ep 88: game finished, reward: -1.000000\n",
      "ep 88: game finished, reward: -1.000000\n",
      "ep 88: game finished, reward: -1.000000\n",
      "ep 88: game finished, reward: -1.000000\n",
      "resetting env. episode 89 reward total was -21.000000. running mean: -20.372548\n",
      "ep 89: game finished, reward: -1.000000\n",
      "ep 89: game finished, reward: -1.000000\n",
      "ep 89: game finished, reward: -1.000000\n",
      "ep 89: game finished, reward: -1.000000\n",
      "ep 89: game finished, reward: -1.000000\n",
      "ep 89: game finished, reward: -1.000000\n",
      "ep 89: game finished, reward: -1.000000\n",
      "ep 89: game finished, reward: -1.000000\n",
      "ep 89: game finished, reward: -1.000000\n",
      "ep 89: game finished, reward: -1.000000\n",
      "ep 89: game finished, reward: -1.000000\n",
      "ep 89: game finished, reward: -1.000000\n",
      "ep 89: game finished, reward: -1.000000\n",
      "ep 89: game finished, reward: -1.000000\n",
      "ep 89: game finished, reward: -1.000000\n",
      "ep 89: game finished, reward: -1.000000\n",
      "ep 89: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 89: game finished, reward: -1.000000\n",
      "ep 89: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 89: game finished, reward: -1.000000\n",
      "ep 89: game finished, reward: -1.000000\n",
      "ep 89: game finished, reward: -1.000000\n",
      "ep 89: game finished, reward: -1.000000\n",
      "resetting env. episode 90 reward total was -19.000000. running mean: -20.358823\n",
      "ep 90: game finished, reward: -1.000000\n",
      "ep 90: game finished, reward: -1.000000\n",
      "ep 90: game finished, reward: -1.000000\n",
      "ep 90: game finished, reward: -1.000000\n",
      "ep 90: game finished, reward: -1.000000\n",
      "ep 90: game finished, reward: -1.000000\n",
      "ep 90: game finished, reward: -1.000000\n",
      "ep 90: game finished, reward: -1.000000\n",
      "ep 90: game finished, reward: -1.000000\n",
      "ep 90: game finished, reward: -1.000000\n",
      "ep 90: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 90: game finished, reward: -1.000000\n",
      "ep 90: game finished, reward: -1.000000\n",
      "ep 90: game finished, reward: -1.000000\n",
      "ep 90: game finished, reward: -1.000000\n",
      "ep 90: game finished, reward: -1.000000\n",
      "ep 90: game finished, reward: -1.000000\n",
      "ep 90: game finished, reward: -1.000000\n",
      "ep 90: game finished, reward: -1.000000\n",
      "ep 90: game finished, reward: -1.000000\n",
      "ep 90: game finished, reward: -1.000000\n",
      "ep 90: game finished, reward: -1.000000\n",
      "resetting env. episode 91 reward total was -20.000000. running mean: -20.355235\n",
      "ep 91: game finished, reward: -1.000000\n",
      "ep 91: game finished, reward: -1.000000\n",
      "ep 91: game finished, reward: -1.000000\n",
      "ep 91: game finished, reward: -1.000000\n",
      "ep 91: game finished, reward: -1.000000\n",
      "ep 91: game finished, reward: -1.000000\n",
      "ep 91: game finished, reward: -1.000000\n",
      "ep 91: game finished, reward: -1.000000\n",
      "ep 91: game finished, reward: -1.000000\n",
      "ep 91: game finished, reward: -1.000000\n",
      "ep 91: game finished, reward: -1.000000\n",
      "ep 91: game finished, reward: -1.000000\n",
      "ep 91: game finished, reward: -1.000000\n",
      "ep 91: game finished, reward: -1.000000\n",
      "ep 91: game finished, reward: -1.000000\n",
      "ep 91: game finished, reward: -1.000000\n",
      "ep 91: game finished, reward: -1.000000\n",
      "ep 91: game finished, reward: -1.000000\n",
      "ep 91: game finished, reward: -1.000000\n",
      "ep 91: game finished, reward: -1.000000\n",
      "ep 91: game finished, reward: -1.000000\n",
      "resetting env. episode 92 reward total was -21.000000. running mean: -20.361682\n",
      "ep 92: game finished, reward: -1.000000\n",
      "ep 92: game finished, reward: -1.000000\n",
      "ep 92: game finished, reward: -1.000000\n",
      "ep 92: game finished, reward: -1.000000\n",
      "ep 92: game finished, reward: -1.000000\n",
      "ep 92: game finished, reward: -1.000000\n",
      "ep 92: game finished, reward: -1.000000\n",
      "ep 92: game finished, reward: -1.000000\n",
      "ep 92: game finished, reward: -1.000000\n",
      "ep 92: game finished, reward: -1.000000\n",
      "ep 92: game finished, reward: -1.000000\n",
      "ep 92: game finished, reward: -1.000000\n",
      "ep 92: game finished, reward: -1.000000\n",
      "ep 92: game finished, reward: -1.000000\n",
      "ep 92: game finished, reward: -1.000000\n",
      "ep 92: game finished, reward: -1.000000\n",
      "ep 92: game finished, reward: -1.000000\n",
      "ep 92: game finished, reward: -1.000000\n",
      "ep 92: game finished, reward: -1.000000\n",
      "ep 92: game finished, reward: -1.000000\n",
      "ep 92: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 92: game finished, reward: -1.000000\n",
      "resetting env. episode 93 reward total was -20.000000. running mean: -20.358065\n",
      "ep 93: game finished, reward: -1.000000\n",
      "ep 93: game finished, reward: -1.000000\n",
      "ep 93: game finished, reward: -1.000000\n",
      "ep 93: game finished, reward: -1.000000\n",
      "ep 93: game finished, reward: -1.000000\n",
      "ep 93: game finished, reward: -1.000000\n",
      "ep 93: game finished, reward: -1.000000\n",
      "ep 93: game finished, reward: -1.000000\n",
      "ep 93: game finished, reward: -1.000000\n",
      "ep 93: game finished, reward: -1.000000\n",
      "ep 93: game finished, reward: -1.000000\n",
      "ep 93: game finished, reward: -1.000000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ep 93: game finished, reward: -1.000000\n",
      "ep 93: game finished, reward: -1.000000\n",
      "ep 93: game finished, reward: -1.000000\n",
      "ep 93: game finished, reward: -1.000000\n",
      "ep 93: game finished, reward: -1.000000\n",
      "ep 93: game finished, reward: -1.000000\n",
      "ep 93: game finished, reward: -1.000000\n",
      "ep 93: game finished, reward: -1.000000\n",
      "ep 93: game finished, reward: -1.000000\n",
      "resetting env. episode 94 reward total was -21.000000. running mean: -20.364485\n",
      "ep 94: game finished, reward: -1.000000\n",
      "ep 94: game finished, reward: -1.000000\n",
      "ep 94: game finished, reward: -1.000000\n",
      "ep 94: game finished, reward: -1.000000\n",
      "ep 94: game finished, reward: -1.000000\n",
      "ep 94: game finished, reward: -1.000000\n",
      "ep 94: game finished, reward: -1.000000\n",
      "ep 94: game finished, reward: -1.000000\n",
      "ep 94: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 94: game finished, reward: -1.000000\n",
      "ep 94: game finished, reward: -1.000000\n",
      "ep 94: game finished, reward: -1.000000\n",
      "ep 94: game finished, reward: -1.000000\n",
      "ep 94: game finished, reward: -1.000000\n",
      "ep 94: game finished, reward: -1.000000\n",
      "ep 94: game finished, reward: -1.000000\n",
      "ep 94: game finished, reward: -1.000000\n",
      "ep 94: game finished, reward: -1.000000\n",
      "ep 94: game finished, reward: -1.000000\n",
      "ep 94: game finished, reward: -1.000000\n",
      "ep 94: game finished, reward: -1.000000\n",
      "ep 94: game finished, reward: -1.000000\n",
      "resetting env. episode 95 reward total was -20.000000. running mean: -20.360840\n",
      "ep 95: game finished, reward: -1.000000\n",
      "ep 95: game finished, reward: -1.000000\n",
      "ep 95: game finished, reward: -1.000000\n",
      "ep 95: game finished, reward: -1.000000\n",
      "ep 95: game finished, reward: -1.000000\n",
      "ep 95: game finished, reward: -1.000000\n",
      "ep 95: game finished, reward: -1.000000\n",
      "ep 95: game finished, reward: -1.000000\n",
      "ep 95: game finished, reward: -1.000000\n",
      "ep 95: game finished, reward: -1.000000\n",
      "ep 95: game finished, reward: -1.000000\n",
      "ep 95: game finished, reward: -1.000000\n",
      "ep 95: game finished, reward: -1.000000\n",
      "ep 95: game finished, reward: -1.000000\n",
      "ep 95: game finished, reward: -1.000000\n",
      "ep 95: game finished, reward: -1.000000\n",
      "ep 95: game finished, reward: -1.000000\n",
      "ep 95: game finished, reward: -1.000000\n",
      "ep 95: game finished, reward: -1.000000\n",
      "ep 95: game finished, reward: -1.000000\n",
      "ep 95: game finished, reward: -1.000000\n",
      "resetting env. episode 96 reward total was -21.000000. running mean: -20.367232\n",
      "ep 96: game finished, reward: -1.000000\n",
      "ep 96: game finished, reward: -1.000000\n",
      "ep 96: game finished, reward: -1.000000\n",
      "ep 96: game finished, reward: -1.000000\n",
      "ep 96: game finished, reward: -1.000000\n",
      "ep 96: game finished, reward: -1.000000\n",
      "ep 96: game finished, reward: -1.000000\n",
      "ep 96: game finished, reward: -1.000000\n",
      "ep 96: game finished, reward: -1.000000\n",
      "ep 96: game finished, reward: -1.000000\n",
      "ep 96: game finished, reward: -1.000000\n",
      "ep 96: game finished, reward: -1.000000\n",
      "ep 96: game finished, reward: -1.000000\n",
      "ep 96: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 96: game finished, reward: -1.000000\n",
      "ep 96: game finished, reward: -1.000000\n",
      "ep 96: game finished, reward: -1.000000\n",
      "ep 96: game finished, reward: -1.000000\n",
      "ep 96: game finished, reward: -1.000000\n",
      "ep 96: game finished, reward: -1.000000\n",
      "ep 96: game finished, reward: -1.000000\n",
      "ep 96: game finished, reward: -1.000000\n",
      "resetting env. episode 97 reward total was -20.000000. running mean: -20.363559\n",
      "ep 97: game finished, reward: -1.000000\n",
      "ep 97: game finished, reward: -1.000000\n",
      "ep 97: game finished, reward: -1.000000\n",
      "ep 97: game finished, reward: -1.000000\n",
      "ep 97: game finished, reward: -1.000000\n",
      "ep 97: game finished, reward: -1.000000\n",
      "ep 97: game finished, reward: -1.000000\n",
      "ep 97: game finished, reward: -1.000000\n",
      "ep 97: game finished, reward: -1.000000\n",
      "ep 97: game finished, reward: -1.000000\n",
      "ep 97: game finished, reward: -1.000000\n",
      "ep 97: game finished, reward: -1.000000\n",
      "ep 97: game finished, reward: -1.000000\n",
      "ep 97: game finished, reward: -1.000000\n",
      "ep 97: game finished, reward: -1.000000\n",
      "ep 97: game finished, reward: -1.000000\n",
      "ep 97: game finished, reward: -1.000000\n",
      "ep 97: game finished, reward: -1.000000\n",
      "ep 97: game finished, reward: -1.000000\n",
      "ep 97: game finished, reward: -1.000000\n",
      "ep 97: game finished, reward: -1.000000\n",
      "resetting env. episode 98 reward total was -21.000000. running mean: -20.369924\n",
      "ep 98: game finished, reward: -1.000000\n",
      "ep 98: game finished, reward: -1.000000\n",
      "ep 98: game finished, reward: -1.000000\n",
      "ep 98: game finished, reward: -1.000000\n",
      "ep 98: game finished, reward: -1.000000\n",
      "ep 98: game finished, reward: -1.000000\n",
      "ep 98: game finished, reward: -1.000000\n",
      "ep 98: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 98: game finished, reward: -1.000000\n",
      "ep 98: game finished, reward: -1.000000\n",
      "ep 98: game finished, reward: -1.000000\n",
      "ep 98: game finished, reward: -1.000000\n",
      "ep 98: game finished, reward: -1.000000\n",
      "ep 98: game finished, reward: -1.000000\n",
      "ep 98: game finished, reward: -1.000000\n",
      "ep 98: game finished, reward: -1.000000\n",
      "ep 98: game finished, reward: -1.000000\n",
      "ep 98: game finished, reward: -1.000000\n",
      "ep 98: game finished, reward: -1.000000\n",
      "ep 98: game finished, reward: -1.000000\n",
      "ep 98: game finished, reward: -1.000000\n",
      "ep 98: game finished, reward: -1.000000\n",
      "resetting env. episode 99 reward total was -20.000000. running mean: -20.366224\n",
      "ep 99: game finished, reward: -1.000000\n",
      "ep 99: game finished, reward: -1.000000\n",
      "ep 99: game finished, reward: -1.000000\n",
      "ep 99: game finished, reward: -1.000000\n",
      "ep 99: game finished, reward: -1.000000\n",
      "ep 99: game finished, reward: -1.000000\n",
      "ep 99: game finished, reward: -1.000000\n",
      "ep 99: game finished, reward: -1.000000\n",
      "ep 99: game finished, reward: -1.000000\n",
      "ep 99: game finished, reward: -1.000000\n",
      "ep 99: game finished, reward: -1.000000\n",
      "ep 99: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 99: game finished, reward: -1.000000\n",
      "ep 99: game finished, reward: -1.000000\n",
      "ep 99: game finished, reward: -1.000000\n",
      "ep 99: game finished, reward: -1.000000\n",
      "ep 99: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 99: game finished, reward: -1.000000\n",
      "ep 99: game finished, reward: -1.000000\n",
      "ep 99: game finished, reward: -1.000000\n",
      "ep 99: game finished, reward: -1.000000\n",
      "ep 99: game finished, reward: -1.000000\n",
      "ep 99: game finished, reward: -1.000000\n",
      "resetting env. episode 100 reward total was -19.000000. running mean: -20.352562\n",
      "ep 100: game finished, reward: -1.000000\n",
      "ep 100: game finished, reward: -1.000000\n",
      "ep 100: game finished, reward: -1.000000\n",
      "ep 100: game finished, reward: -1.000000\n",
      "ep 100: game finished, reward: -1.000000\n",
      "ep 100: game finished, reward: -1.000000\n",
      "ep 100: game finished, reward: -1.000000\n",
      "ep 100: game finished, reward: -1.000000\n",
      "ep 100: game finished, reward: -1.000000\n",
      "ep 100: game finished, reward: -1.000000\n",
      "ep 100: game finished, reward: -1.000000\n",
      "ep 100: game finished, reward: -1.000000\n",
      "ep 100: game finished, reward: -1.000000\n",
      "ep 100: game finished, reward: -1.000000\n",
      "ep 100: game finished, reward: -1.000000\n",
      "ep 100: game finished, reward: -1.000000\n",
      "ep 100: game finished, reward: -1.000000\n",
      "ep 100: game finished, reward: -1.000000\n",
      "ep 100: game finished, reward: -1.000000\n",
      "ep 100: game finished, reward: -1.000000\n",
      "ep 100: game finished, reward: -1.000000\n",
      "resetting env. episode 101 reward total was -21.000000. running mean: -20.359037\n",
      "ep 101: game finished, reward: -1.000000\n",
      "ep 101: game finished, reward: -1.000000\n",
      "ep 101: game finished, reward: -1.000000\n",
      "ep 101: game finished, reward: -1.000000\n",
      "ep 101: game finished, reward: -1.000000\n",
      "ep 101: game finished, reward: -1.000000\n",
      "ep 101: game finished, reward: -1.000000\n",
      "ep 101: game finished, reward: -1.000000\n",
      "ep 101: game finished, reward: -1.000000\n",
      "ep 101: game finished, reward: -1.000000\n",
      "ep 101: game finished, reward: -1.000000\n",
      "ep 101: game finished, reward: -1.000000\n",
      "ep 101: game finished, reward: -1.000000\n",
      "ep 101: game finished, reward: -1.000000\n",
      "ep 101: game finished, reward: -1.000000\n",
      "ep 101: game finished, reward: -1.000000\n",
      "ep 101: game finished, reward: -1.000000\n",
      "ep 101: game finished, reward: -1.000000\n",
      "ep 101: game finished, reward: -1.000000\n",
      "ep 101: game finished, reward: -1.000000\n",
      "ep 101: game finished, reward: -1.000000\n",
      "resetting env. episode 102 reward total was -21.000000. running mean: -20.365446\n",
      "ep 102: game finished, reward: -1.000000\n",
      "ep 102: game finished, reward: -1.000000\n",
      "ep 102: game finished, reward: -1.000000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ep 102: game finished, reward: -1.000000\n",
      "ep 102: game finished, reward: -1.000000\n",
      "ep 102: game finished, reward: -1.000000\n",
      "ep 102: game finished, reward: -1.000000\n",
      "ep 102: game finished, reward: -1.000000\n",
      "ep 102: game finished, reward: -1.000000\n",
      "ep 102: game finished, reward: -1.000000\n",
      "ep 102: game finished, reward: -1.000000\n",
      "ep 102: game finished, reward: -1.000000\n",
      "ep 102: game finished, reward: -1.000000\n",
      "ep 102: game finished, reward: -1.000000\n",
      "ep 102: game finished, reward: -1.000000\n",
      "ep 102: game finished, reward: -1.000000\n",
      "ep 102: game finished, reward: -1.000000\n",
      "ep 102: game finished, reward: -1.000000\n",
      "ep 102: game finished, reward: -1.000000\n",
      "ep 102: game finished, reward: -1.000000\n",
      "ep 102: game finished, reward: -1.000000\n",
      "resetting env. episode 103 reward total was -21.000000. running mean: -20.371792\n",
      "ep 103: game finished, reward: -1.000000\n",
      "ep 103: game finished, reward: -1.000000\n",
      "ep 103: game finished, reward: -1.000000\n",
      "ep 103: game finished, reward: -1.000000\n",
      "ep 103: game finished, reward: -1.000000\n",
      "ep 103: game finished, reward: -1.000000\n",
      "ep 103: game finished, reward: -1.000000\n",
      "ep 103: game finished, reward: -1.000000\n",
      "ep 103: game finished, reward: -1.000000\n",
      "ep 103: game finished, reward: -1.000000\n",
      "ep 103: game finished, reward: -1.000000\n",
      "ep 103: game finished, reward: -1.000000\n",
      "ep 103: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 103: game finished, reward: -1.000000\n",
      "ep 103: game finished, reward: -1.000000\n",
      "ep 103: game finished, reward: -1.000000\n",
      "ep 103: game finished, reward: -1.000000\n",
      "ep 103: game finished, reward: -1.000000\n",
      "ep 103: game finished, reward: -1.000000\n",
      "ep 103: game finished, reward: -1.000000\n",
      "ep 103: game finished, reward: -1.000000\n",
      "ep 103: game finished, reward: -1.000000\n",
      "resetting env. episode 104 reward total was -20.000000. running mean: -20.368074\n",
      "ep 104: game finished, reward: -1.000000\n",
      "ep 104: game finished, reward: -1.000000\n",
      "ep 104: game finished, reward: -1.000000\n",
      "ep 104: game finished, reward: -1.000000\n",
      "ep 104: game finished, reward: -1.000000\n",
      "ep 104: game finished, reward: -1.000000\n",
      "ep 104: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 104: game finished, reward: -1.000000\n",
      "ep 104: game finished, reward: -1.000000\n",
      "ep 104: game finished, reward: -1.000000\n",
      "ep 104: game finished, reward: -1.000000\n",
      "ep 104: game finished, reward: -1.000000\n",
      "ep 104: game finished, reward: -1.000000\n",
      "ep 104: game finished, reward: -1.000000\n",
      "ep 104: game finished, reward: -1.000000\n",
      "ep 104: game finished, reward: -1.000000\n",
      "ep 104: game finished, reward: -1.000000\n",
      "ep 104: game finished, reward: -1.000000\n",
      "ep 104: game finished, reward: -1.000000\n",
      "ep 104: game finished, reward: -1.000000\n",
      "ep 104: game finished, reward: -1.000000\n",
      "ep 104: game finished, reward: -1.000000\n",
      "resetting env. episode 105 reward total was -20.000000. running mean: -20.364393\n",
      "ep 105: game finished, reward: -1.000000\n",
      "ep 105: game finished, reward: -1.000000\n",
      "ep 105: game finished, reward: -1.000000\n",
      "ep 105: game finished, reward: -1.000000\n",
      "ep 105: game finished, reward: -1.000000\n",
      "ep 105: game finished, reward: -1.000000\n",
      "ep 105: game finished, reward: -1.000000\n",
      "ep 105: game finished, reward: -1.000000\n",
      "ep 105: game finished, reward: -1.000000\n",
      "ep 105: game finished, reward: -1.000000\n",
      "ep 105: game finished, reward: -1.000000\n",
      "ep 105: game finished, reward: -1.000000\n",
      "ep 105: game finished, reward: -1.000000\n",
      "ep 105: game finished, reward: -1.000000\n",
      "ep 105: game finished, reward: -1.000000\n",
      "ep 105: game finished, reward: -1.000000\n",
      "ep 105: game finished, reward: -1.000000\n",
      "ep 105: game finished, reward: -1.000000\n",
      "ep 105: game finished, reward: -1.000000\n",
      "ep 105: game finished, reward: -1.000000\n",
      "ep 105: game finished, reward: -1.000000\n",
      "resetting env. episode 106 reward total was -21.000000. running mean: -20.370749\n",
      "ep 106: game finished, reward: -1.000000\n",
      "ep 106: game finished, reward: -1.000000\n",
      "ep 106: game finished, reward: -1.000000\n",
      "ep 106: game finished, reward: -1.000000\n",
      "ep 106: game finished, reward: -1.000000\n",
      "ep 106: game finished, reward: -1.000000\n",
      "ep 106: game finished, reward: -1.000000\n",
      "ep 106: game finished, reward: -1.000000\n",
      "ep 106: game finished, reward: -1.000000\n",
      "ep 106: game finished, reward: -1.000000\n",
      "ep 106: game finished, reward: -1.000000\n",
      "ep 106: game finished, reward: -1.000000\n",
      "ep 106: game finished, reward: -1.000000\n",
      "ep 106: game finished, reward: -1.000000\n",
      "ep 106: game finished, reward: -1.000000\n",
      "ep 106: game finished, reward: -1.000000\n",
      "ep 106: game finished, reward: -1.000000\n",
      "ep 106: game finished, reward: -1.000000\n",
      "ep 106: game finished, reward: -1.000000\n",
      "ep 106: game finished, reward: -1.000000\n",
      "ep 106: game finished, reward: -1.000000\n",
      "resetting env. episode 107 reward total was -21.000000. running mean: -20.377042\n",
      "ep 107: game finished, reward: -1.000000\n",
      "ep 107: game finished, reward: -1.000000\n",
      "ep 107: game finished, reward: -1.000000\n",
      "ep 107: game finished, reward: -1.000000\n",
      "ep 107: game finished, reward: -1.000000\n",
      "ep 107: game finished, reward: -1.000000\n",
      "ep 107: game finished, reward: -1.000000\n",
      "ep 107: game finished, reward: -1.000000\n",
      "ep 107: game finished, reward: -1.000000\n",
      "ep 107: game finished, reward: -1.000000\n",
      "ep 107: game finished, reward: -1.000000\n",
      "ep 107: game finished, reward: -1.000000\n",
      "ep 107: game finished, reward: -1.000000\n",
      "ep 107: game finished, reward: -1.000000\n",
      "ep 107: game finished, reward: -1.000000\n",
      "ep 107: game finished, reward: -1.000000\n",
      "ep 107: game finished, reward: -1.000000\n",
      "ep 107: game finished, reward: -1.000000\n",
      "ep 107: game finished, reward: -1.000000\n",
      "ep 107: game finished, reward: -1.000000\n",
      "ep 107: game finished, reward: -1.000000\n",
      "resetting env. episode 108 reward total was -21.000000. running mean: -20.383271\n",
      "ep 108: game finished, reward: -1.000000\n",
      "ep 108: game finished, reward: -1.000000\n",
      "ep 108: game finished, reward: -1.000000\n",
      "ep 108: game finished, reward: -1.000000\n",
      "ep 108: game finished, reward: -1.000000\n",
      "ep 108: game finished, reward: -1.000000\n",
      "ep 108: game finished, reward: -1.000000\n",
      "ep 108: game finished, reward: -1.000000\n",
      "ep 108: game finished, reward: -1.000000\n",
      "ep 108: game finished, reward: -1.000000\n",
      "ep 108: game finished, reward: -1.000000\n",
      "ep 108: game finished, reward: -1.000000\n",
      "ep 108: game finished, reward: -1.000000\n",
      "ep 108: game finished, reward: -1.000000\n",
      "ep 108: game finished, reward: -1.000000\n",
      "ep 108: game finished, reward: -1.000000\n",
      "ep 108: game finished, reward: -1.000000\n",
      "ep 108: game finished, reward: -1.000000\n",
      "ep 108: game finished, reward: -1.000000\n",
      "ep 108: game finished, reward: -1.000000\n",
      "ep 108: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 108: game finished, reward: -1.000000\n",
      "resetting env. episode 109 reward total was -20.000000. running mean: -20.379438\n",
      "ep 109: game finished, reward: -1.000000\n",
      "ep 109: game finished, reward: -1.000000\n",
      "ep 109: game finished, reward: -1.000000\n",
      "ep 109: game finished, reward: -1.000000\n",
      "ep 109: game finished, reward: -1.000000\n",
      "ep 109: game finished, reward: -1.000000\n",
      "ep 109: game finished, reward: -1.000000\n",
      "ep 109: game finished, reward: -1.000000\n",
      "ep 109: game finished, reward: -1.000000\n",
      "ep 109: game finished, reward: -1.000000\n",
      "ep 109: game finished, reward: -1.000000\n",
      "ep 109: game finished, reward: -1.000000\n",
      "ep 109: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 109: game finished, reward: -1.000000\n",
      "ep 109: game finished, reward: -1.000000\n",
      "ep 109: game finished, reward: -1.000000\n",
      "ep 109: game finished, reward: -1.000000\n",
      "ep 109: game finished, reward: -1.000000\n",
      "ep 109: game finished, reward: -1.000000\n",
      "ep 109: game finished, reward: -1.000000\n",
      "ep 109: game finished, reward: -1.000000\n",
      "ep 109: game finished, reward: -1.000000\n",
      "resetting env. episode 110 reward total was -20.000000. running mean: -20.375644\n",
      "ep 110: game finished, reward: -1.000000\n",
      "ep 110: game finished, reward: -1.000000\n",
      "ep 110: game finished, reward: -1.000000\n",
      "ep 110: game finished, reward: -1.000000\n",
      "ep 110: game finished, reward: -1.000000\n",
      "ep 110: game finished, reward: -1.000000\n",
      "ep 110: game finished, reward: -1.000000\n",
      "ep 110: game finished, reward: -1.000000\n",
      "ep 110: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 110: game finished, reward: -1.000000\n",
      "ep 110: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 110: game finished, reward: -1.000000\n",
      "ep 110: game finished, reward: -1.000000\n",
      "ep 110: game finished, reward: -1.000000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ep 110: game finished, reward: -1.000000\n",
      "ep 110: game finished, reward: -1.000000\n",
      "ep 110: game finished, reward: -1.000000\n",
      "ep 110: game finished, reward: -1.000000\n",
      "ep 110: game finished, reward: -1.000000\n",
      "ep 110: game finished, reward: -1.000000\n",
      "ep 110: game finished, reward: -1.000000\n",
      "ep 110: game finished, reward: -1.000000\n",
      "ep 110: game finished, reward: -1.000000\n",
      "resetting env. episode 111 reward total was -19.000000. running mean: -20.361888\n",
      "ep 111: game finished, reward: -1.000000\n",
      "ep 111: game finished, reward: -1.000000\n",
      "ep 111: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 111: game finished, reward: -1.000000\n",
      "ep 111: game finished, reward: -1.000000\n",
      "ep 111: game finished, reward: -1.000000\n",
      "ep 111: game finished, reward: -1.000000\n",
      "ep 111: game finished, reward: -1.000000\n",
      "ep 111: game finished, reward: -1.000000\n",
      "ep 111: game finished, reward: -1.000000\n",
      "ep 111: game finished, reward: -1.000000\n",
      "ep 111: game finished, reward: -1.000000\n",
      "ep 111: game finished, reward: -1.000000\n",
      "ep 111: game finished, reward: -1.000000\n",
      "ep 111: game finished, reward: -1.000000\n",
      "ep 111: game finished, reward: -1.000000\n",
      "ep 111: game finished, reward: -1.000000\n",
      "ep 111: game finished, reward: -1.000000\n",
      "ep 111: game finished, reward: -1.000000\n",
      "ep 111: game finished, reward: -1.000000\n",
      "ep 111: game finished, reward: -1.000000\n",
      "ep 111: game finished, reward: -1.000000\n",
      "resetting env. episode 112 reward total was -20.000000. running mean: -20.358269\n",
      "ep 112: game finished, reward: -1.000000\n",
      "ep 112: game finished, reward: -1.000000\n",
      "ep 112: game finished, reward: -1.000000\n",
      "ep 112: game finished, reward: -1.000000\n",
      "ep 112: game finished, reward: -1.000000\n",
      "ep 112: game finished, reward: -1.000000\n",
      "ep 112: game finished, reward: -1.000000\n",
      "ep 112: game finished, reward: -1.000000\n",
      "ep 112: game finished, reward: -1.000000\n",
      "ep 112: game finished, reward: -1.000000\n",
      "ep 112: game finished, reward: -1.000000\n",
      "ep 112: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 112: game finished, reward: -1.000000\n",
      "ep 112: game finished, reward: -1.000000\n",
      "ep 112: game finished, reward: -1.000000\n",
      "ep 112: game finished, reward: -1.000000\n",
      "ep 112: game finished, reward: -1.000000\n",
      "ep 112: game finished, reward: -1.000000\n",
      "ep 112: game finished, reward: -1.000000\n",
      "ep 112: game finished, reward: -1.000000\n",
      "ep 112: game finished, reward: -1.000000\n",
      "ep 112: game finished, reward: -1.000000\n",
      "resetting env. episode 113 reward total was -20.000000. running mean: -20.354686\n",
      "ep 113: game finished, reward: -1.000000\n",
      "ep 113: game finished, reward: -1.000000\n",
      "ep 113: game finished, reward: -1.000000\n",
      "ep 113: game finished, reward: -1.000000\n",
      "ep 113: game finished, reward: -1.000000\n",
      "ep 113: game finished, reward: -1.000000\n",
      "ep 113: game finished, reward: -1.000000\n",
      "ep 113: game finished, reward: -1.000000\n",
      "ep 113: game finished, reward: 1.000000 !!!!!!!!\n",
      "ep 113: game finished, reward: -1.000000\n",
      "ep 113: game finished, reward: -1.000000\n",
      "ep 113: game finished, reward: -1.000000\n",
      "ep 113: game finished, reward: -1.000000\n",
      "ep 113: game finished, reward: -1.000000\n",
      "ep 113: game finished, reward: -1.000000\n",
      "ep 113: game finished, reward: -1.000000\n",
      "ep 113: game finished, reward: -1.000000\n",
      "ep 113: game finished, reward: -1.000000\n",
      "ep 113: game finished, reward: -1.000000\n",
      "ep 113: game finished, reward: -1.000000\n",
      "ep 113: game finished, reward: -1.000000\n",
      "ep 113: game finished, reward: -1.000000\n",
      "resetting env. episode 114 reward total was -20.000000. running mean: -20.351139\n",
      "ep 114: game finished, reward: -1.000000\n",
      "ep 114: game finished, reward: -1.000000\n",
      "ep 114: game finished, reward: -1.000000\n",
      "ep 114: game finished, reward: -1.000000\n",
      "ep 114: game finished, reward: -1.000000\n",
      "ep 114: game finished, reward: -1.000000\n",
      "ep 114: game finished, reward: -1.000000\n",
      "ep 114: game finished, reward: -1.000000\n",
      "ep 114: game finished, reward: -1.000000\n",
      "ep 114: game finished, reward: -1.000000\n",
      "ep 114: game finished, reward: -1.000000\n",
      "ep 114: game finished, reward: -1.000000\n",
      "ep 114: game finished, reward: -1.000000\n",
      "ep 114: game finished, reward: -1.000000\n",
      "ep 114: game finished, reward: -1.000000\n"
     ]
    }
   ],
   "source": [
    "observation = env.reset()\n",
    "while True:\n",
    "    if render: env.render()\n",
    "\n",
    "    # preprocess the observation, set input to network to be difference image\n",
    "    cur_x = prepro(observation)\n",
    "    x = cur_x - prev_x if prev_x is not None else np.zeros(D)\n",
    "    prev_x = cur_x\n",
    "\n",
    "    # forward the policy network and sample an action from the returned probability\n",
    "    aprob, h = policy_forward(x)\n",
    "    action = 2 if np.random.uniform() < aprob else 3 # roll the dice!\n",
    "\n",
    "    # record various intermediates (needed later for backprop)\n",
    "    xs.append(x) # observation\n",
    "    hs.append(h) # hidden state\n",
    "    y = 1 if action == 2 else 0 # a \"fake label\"\n",
    "    dlogps.append(y - aprob) # grad that encourages the action that was taken to be taken (see http://cs231n.github.io/neural-networks-2/#losses if confused)\n",
    "\n",
    "    # step the environment and get new measurements\n",
    "    observation, reward, done, info = env.step(action)\n",
    "    reward_sum += reward\n",
    "\n",
    "    drs.append(reward) # record reward (has to be done after we call step() to get reward for previous action)\n",
    "\n",
    "    if done: # an episode finished\n",
    "        episode_number += 1\n",
    "\n",
    "        # stack together all inputs, hidden states, action gradients, and rewards for this episode\n",
    "        epx = np.vstack(xs)\n",
    "        eph = np.vstack(hs)\n",
    "        epdlogp = np.vstack(dlogps)\n",
    "        epr = np.vstack(drs)\n",
    "        xs,hs,dlogps,drs = [],[],[],[] # reset array memory\n",
    "\n",
    "        # compute the discounted reward backwards through time\n",
    "        discounted_epr = discount_rewards(epr)\n",
    "        # standardize the rewards to be unit normal (helps control the gradient estimator variance)\n",
    "        discounted_epr -= np.mean(discounted_epr)\n",
    "        discounted_epr /= np.std(discounted_epr)\n",
    "\n",
    "        epdlogp *= discounted_epr # modulate the gradient with advantage (PG magic happens right here.)\n",
    "        grad = policy_backward(eph, epdlogp)\n",
    "        for k in model: grad_buffer[k] += grad[k] # accumulate grad over batch\n",
    "\n",
    "        # perform rmsprop parameter update every batch_size episodes\n",
    "        if episode_number % batch_size == 0:\n",
    "            for k,v in model.iteritems():\n",
    "                g = grad_buffer[k] # gradient\n",
    "                rmsprop_cache[k] = decay_rate * rmsprop_cache[k] + (1 - decay_rate) * g**2\n",
    "                model[k] += learning_rate * g / (np.sqrt(rmsprop_cache[k]) + 1e-5)\n",
    "                grad_buffer[k] = np.zeros_like(v) # reset batch gradient buffer\n",
    "\n",
    "        # boring book-keeping\n",
    "        running_reward = reward_sum if running_reward is None else running_reward * 0.99 + reward_sum * 0.01\n",
    "#         if episode_number % 10 == 0: \n",
    "        print 'resetting env. episode %s reward total was %f. running mean: %f' % (episode_number, reward_sum, running_reward)\n",
    "        if episode_number % 100 == 0: pickle.dump(model, open('save.p', 'wb'))\n",
    "        reward_sum = 0\n",
    "        observation = env.reset() # reset env\n",
    "        prev_x = None\n",
    "\n",
    "    if reward != 0: # Pong has either +1 or -1 reward exactly when game ends.\n",
    "        print ('ep %d: game finished, reward: %f' % (episode_number, reward)) + ('' if reward == -1 else ' !!!!!!!!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-12T23:26:12.816089Z",
     "start_time": "2017-07-13T07:26:07.602092+08:00"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ipython2",
   "language": "python",
   "name": "ipython2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}