McCulloughRT/Architectural_Classifier-Logistic_Regression.ipynb

## Architectural_Classifier-Logistic_Regression.ipynb
{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building an Architectural Classifier - Notebook 1 - Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal in this notebook is to take our pre-processed dataset of interior architectural imagery (containing images of kitchens, bathrooms, bedrooms, living rooms, etc...) and build a machine learning model that can accurately classify when it is looking at an image of a kitchen. \n",
    "#### Model:\n",
    "Starting with the simplest type of model first, we'll use logistic regression in this notebook. It is not likely to do well on images of this size and scene complexity (spoiler alert: it doesn't), but it will at least provide a useful baseline. \n",
    "\n",
    "As logistic regression can be formulated as a single neuron neural network I will be using TensorFlow and leverage this as a jumping off point for exploring deep neural networks and convolutions in the next notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import math\n",
    "import time\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load in the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x: (4686, 85, 85, 3) | y: (4686, 2)\n"
     ]
    }
   ],
   "source": [
    "x = np.load('./all_X_shuffled_85_x4686.npy')\n",
    "y = np.load('./all_Y_shuffled_2_x4686.npy')\n",
    "\n",
    "print('x: %s | y: %s' % (x.shape, y.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split the data up into train / test / validation (80% / 10% / 10%)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def split(x, y, test=0.1, train=0.8, validation=0.1):\n",
    "    assert(len(x) == len(y))\n",
    "    test_size = int(len(x) * test)\n",
    "    train_size = int(len(x) * train)\n",
    "    valid_size = int(len(x) * validation)\n",
    "    \n",
    "    x_train = np.array(x[:train_size])\n",
    "    y_train = np.array(y[:train_size])\n",
    "    x_val = np.array(x[train_size:train_size + valid_size])\n",
    "    y_val = np.array(y[train_size:train_size + valid_size])\n",
    "    x_test = np.array(x[train_size + valid_size:])\n",
    "    y_test = np.array(y[train_size + valid_size:])\n",
    "    \n",
    "    return (x_train, y_train, x_val, y_val, x_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train:  (3748, 85, 85, 3)\n",
      "y_train:  (3748, 2)\n",
      "x_val:    (468, 85, 85, 3)\n",
      "y_val:    (468, 2)\n",
      "x_test:   (470, 85, 85, 3)\n",
      "y_test:   (470, 2)\n"
     ]
    }
   ],
   "source": [
    "x_train, y_train, x_val, y_val, x_test, y_test = split(x,y)\n",
    "\n",
    "print('x_train: ', x_train.shape)\n",
    "print('y_train: ', y_train.shape)\n",
    "print('x_val:   ', x_val.shape)\n",
    "print('y_val:   ', y_val.shape)\n",
    "print('x_test:  ', x_test.shape)\n",
    "print('y_test:  ', y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Why is y a 2 dimensional vector for binary classification?\n",
    "\n",
    "While this is a binary classification problem, I already know it will turn into a multi-class problem as more data is obtained. Thus during data-prep I set y to a one-hot vector [0,1] to indicate positive so that I can structure the models for a future multiclass problem (add more values to the y vector)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lets take a look at one of the examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label:  [0 1]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP4AAAD8CAYAAABXXhlaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsvWuwJdd1HrZ2d5/3ue+5834BA4AAAZEgCb5MSqYokqJk\nRVIqkkw5VhRZKVa57EROXBUpqUpkV/mHnUpZVqUSVbH0MOXYpmSRimmJskhRpCiaNIgHCYAA8Rhg\nAMz7vu89955nd+/82Gvt9fV0z8wdgrgAdPeqAm7PPt27997d56xvr8e3jLWWggQJsr8keq0HECRI\nkL2X8MUPEmQfSvjiBwmyDyV88YME2YcSvvhBguxDCV/8IEH2oYQvfpAg+1Be0RffGPNRY8wzxpiz\nxphf+V4NKkiQIK+umO82gMcYExPRs0T0YSK6QEQPEdHPWmuf+t4NL0iQIK+GJK/g2ncR0Vlr7QtE\nRMaYTxHRTxDRdb/4kTF2Nzc0t/B5ZNy/WomCFxOZ6/aDbSaKiYio0e1of3EVCIKrdvlDaUnPM9pI\nFYe3LFXzqupvt+fd6rml843+a7fjqPzUSHe43je8uHDvG3ddcV7ltabi4/IYC30WF6Oi6xutwHU+\nM/bGn1+nn6WVFdrs9W560Sv54h8jovPw7wtE9O6b3ewAfynJypdTv2iRyd1fyn2bzCCCFwC/mp3E\nTeG+g1O+rdmMuT8D15jStY1Wl4iI7vzr7/FtremuP/ZLb/Uqm+X8mY7Rf2ZzOM50DvxjEaV6rn56\n6/JdffF5La6H8GL+i7OqPNNE3Df0HsH6VPxuRlG5Ua+HZxQn/Bf6s+VfS2PgtZVnHMEXVo4Nvlux\nHst4Ym0zMi+D/fB7hOcl0A/3ifOTcw3OWRRR4dcgLo2xsOL8XbA4h8rF5b75s1/6R/+4fE6FvJIv\n/q7EGPNxIvo4kb5cQYIEeW3llXzxLxLRCfj3cW4riLX2E0T0CSKiujGW8qKmL2gO0Ur4yygq6DqQ\nT67PoR+bu75zgzCBtS7o/NkDh4mI6M63vt+3ra9f8Mf9jRWZA96wPG6qgGUWfuZ4HDkqqjwv931D\nWFuxdaj8tDTUwgnXu0UlsKyAwrKFwX5uhitljqaqDTSxICPcTkURXiVoEVty/otrz88fcJXF90PQ\nZg4ITZ4rbvUYweUAzyIYm43d51mufWf87qHGj2WOsZ4XR3npvOIKMXIobC0EEYPwwHOPMHe3iXwl\nVv2HiOhOY8xtxpg6EX2MiD77CvoLEiTIHsl3rfGttakx5u8T0Z+S+3n6bWvtkze98BotYg3sKvlX\nMMpt6XSLP+g30zG2rN4s/ypnkd6vMbtARESn7n+73u9Z/S3sb65xN7gjv77RpbA/hF/3PM/LV/i9\nG/5C24o2JzBsyuHzvGIporL5gVgREYIgRFbSZxEk+U21b8v8sOEZwcMxOWsquLeVTrFzj+50sBl/\nHNN1+pY9d4TvB/cDOkzOM/Aeoc3FcntesXh5YYgyF0UlNgIkZ8v7ebE/FOxLMa8tvBNixzK4jtiP\nR8Sgm+OyrUAQiiCe6EbGTpBXtMe31n6OiD73SvoIEiTI3kuI3AsSZB/Kq27Vv1ZuBES84ajCjVtA\niQXLkhibALYyfIrRoMM9ZZlefOWFZ4iI6Guf/pe+LW4glCtDdG+Mu7lFS4d4zV8eEBERRbYClwPU\nFQ+OKcBfPfawGBetwh/uXaQFwxj+7pc/F72ARtKI4XEOEB23Hh5eV0DOAkT1x4X9mPuTat+pLcP6\nApz1bkoqtRVwu8Xxll2A4pKMM/1KJEnNtcU134a2P2mPag1tY/dyjC7AKrdgJEZAXJO4dFxYxgo3\npbwXeVb1Hl1fgsYPEmQfyp5rfO82qbJj3cTl5E8rnCC/+Ppr6X/8onJPGFE3noyIiGhne9O3tUij\n+DSaBbWt/C1DkLzw81w2UBahA6OJvGq2VQY/lYKHy5Y1pw6tjDqu5z40FQ/EG+Xy8ryMwfPgU+ui\nlHJ0e0nQU4VWNqZqroUO6VqJIwzGcUY7Uwiiqbu2GLR3wSAWF/4SEZmk6a5tqvZOanXuD+6X1GEc\nNZ4Dfl6lya8f4YdrW4jlkQAe6CfiEyI40YpbeDLhhlffnRckSJA3qIQvfpAg+1D2FOpbKkdxFSBz\nLr5dbVKEVg1hxKiXw0Viv0O7WRXczqkcPWcLBqFyJJkcF6Ccj0IrG9jc5+Vxi1+9AP/ysi8dbgxd\nA9TzkXT4uS0Nu7pHWCBrK9pKt/YQ3kI4Ww4LnWUOcqbjkbZNBnwLPS9maN1oa46F+LkjgPIYkeeT\nr/AZit8cr0kcXI8bCttrtVrpOIKtgJXjilj8ApSHa2R7EaOxTcYY44MtR/PJeDFOIYZ3amLHPEHt\nJa51SvezPo7h1iRo/CBB9qHsuXHPq2EfkVeOby8aryqC9dG95DUQusAkVhvdHq7TGGEAX5OOhr5p\nBD+FecrpdKzFiDQTq9ZuldriDO5n0ShTdm2J3RE1ta2wbsp5BRsgzN9UwAOPYDBbUOaE6MbeGCVk\nucSqq3bPMrcmk6Gu2WQ8hGvcWqFLjnJZP3BDJg4RmAg0catNRNXZbkRq1IsTvcb4jD5wudWdpo8a\nYNyrQT98DWpy8dMVYud9xh5m+VV9nsDn8qxRK7vPCy5nOYYhZKDex4MBz1XXrNWY4Z5x3Hy3lnsf\nqzIhqyRo/CBB9qGEL36QIPtQ9hTqG4KkG0mNRMOI9/Er5IlMRURSwUBXYcmrCEITg1gW6ZRzxs+T\n4Q7cT2Gtt78B/q2x8aeGMDoVHypGDyLUKxODqJT92LYCjmMbwn6ZgwU4nrLhrUAMIv7evHw/IiJj\nhCUE05slfRWukL6zMtEIkSZYGUy+4b0b3jubOOPV9vqKb5trHCcionpDt1FJogY6MZ6mqRoOY35K\n9XZbz+NnZJEgBOaQVhh6ZWsmcQFERDHDf9yOjlLd9vmkMgsMK/KswQgYMwNLBO+eJNdENdgSIosU\nr+8E1jmakchFDB/kMUqK8S6TdILGDxJkH0r44gcJsg9l75N0fNbJtbG74DYvhJ9W+G6rQlGBTSU3\n5W2EREbaDCEzn4e+4iqPKIZ78m9lnINlVaAs5rejlyF3UC+Fe/ukoRzzxBlGIuWLrJdFvzn2I3Ac\n55UXLnV9l9lbjKnBP+LCuNzn5Zx52Ueh5yWzhb2AOys31zYRYmt5XBY8JrIVaHXUt4/W+hF7Evrb\nfR1jwiHCsCXwnoCCNR48CpF4AsAaL9ZzuCYzbjtSq+u19bqG7DY5TiAu9FMOByYfD4AeAVP8S0WL\nvK27OY5gZyEeBzxP7pP7OIYA9YMECXId2XuNX5mx4j8kIiqwqWRVpxWzPfga9JHKJ5guWjYSqiLC\nhAr9XIxxBVuK9wGj9pIxVMdPeTYVoMbJUjbKjdRQZSkt9S3zK6bYgvisXGAmFhVb0MSivQEFIZux\nAIoCVbZENqrxSvz02UTHnaPW9pGESK3KKACMnz7iDA1sbKiKIcoOfeQZG7piaGs0nAbutCB5hv38\n6JNPUCtL1BxqTjkG3/5w7DQ+MvTWa3qfpM73obLWLnD8VRz5FkBqGJ8Q+QQgQH9VayapypIctDuF\nf3ONb4z5bWPMkjHm29A2b4z5gjHmOf47t7vbBQkS5PUgu4H6/5KIPnpN268Q0RettXcS0Rf530GC\nBHmDyE2hvrX2K8aY09c0/wQRfYCPP0lEXyaiX97NDa/lb8TwQ6FSRv9zXplpUk6GKUKrcnaJ8Uko\nMBa5BAwxOeZey1/wtfocbQg1pWxcGisaG9UHD6SNtnyeQjk0Al1zwTWT8GsFkNDncKN/ne9dQP9I\nNkRlkcszWNuMz5zAFZktb3vQPplXJTuRQOIyEI6SpHQekYbs1jD8VqA1kmmKL72Q1w+HFXEOccyx\nA5iEI7ENUcXzcDMjIqJspFshKww8CRpOxdiq4044rHgM4eJkdBshhkBMhprw9iqCrU69IQSku8T4\ncv9bOlvlkLX2Mh9fIaJD1zsxFNQIEuT1J6/YuGettaYqU0Q/9wU1phpNe9cxF501PTVNREQzM7P+\n3IU5Zyq4cFmLWnzz8W8REdEEIqYKZjrRjui6qvjxy8VAU+Bwc9NPIBWzHuuvbmTFuANuoUSip9Ag\nyO4hdKlF6F6U6DpIu8xd3xlwpUnfRf0rvHdUanPTYXcOLIp0WTCwsSsxw9JeFYUiMJ3UeuAABiiB\nCaC9MJFEiplQhEYpcUliKrOkU+tpde+GKxs3iaoZaOR5psOxjidyz9DG6OKERCt53hhdKe5O+EZE\nnhcPjY0wh6x4LZGmAseY3huJQRgSkjjJCxOcii5HT7YIY5Rhl12ukd0bd95VY8wRvvERIlr6LvsJ\nEiTIayDf7Rf/s0T083z880T07783wwkSJMheyE2hvjHm35Iz5B0wxlwgol8lon9KRL9vjPlFInqJ\niH5mNzebm5ujn/rpn3Y3ZiNZA1hSTpw4QkREzz33tG979jlXnKfXUyiHlWK8D7Vg/CpH0lEFipYd\nSl2HQM0GJqSXc6YljyIqJOHwX4Bg+LnPa0ceAf4YIbyP9sMxii8dq/kU0JwptSUxG6/ACBT7iisw\nmULZGHcNbnsEzReTi3hLkGkiDUUKo2PeKuHWo9V2fTdasM0YuuMXzutzlTlgLbqosGZSpaZccabW\naPq2pk/YgTgFjINgwswsq4iazNDiWY6hqDJkmuID4/6QH8LyffU0MWAim04UledtKqJGi218j12X\n03ayG6v+z17nox+6pTsFCRLkdSN7GrmXZRmtb267f/Cv1nRb6ayjE0f5qMywg1oHjXuV3i5Pr40n\nVtgfpYbaGLQOsLbEtXLcdezrsqGBTRBGBZsOkVZJzVEDsaGqIuU3LtRQkwO4XwHyuOtjMGQJKqnX\ngIGm49hbemvrvm04xPG4v1Ozep+73++cNdMLmvLa6LgTE4iu669pFN/aFadFd9Y0nj62G66fLqSY\njt14Ly9rWq7nq4MHF+G7EJXRTeQj18oU15Wci6Rrjoa19V7PXQMGuHanS0REvS0d4+VLl/zxwcPO\nUH30MDi1ets8RNDK7NqrTWsOQszG6oKWL9TgkzaYg7xHFcZPH2Ua0nKDBAlyPQlf/CBB9qHsKdSP\nopiaTQftBbk2IM1RDE9ZgahREmBUbKHWc/k+3qhXLPHCf9CA4o5bLTUMNaCSSlx3y4OMMHHGhqEh\nRooVbuGGjYSZ3qoDPvsJJ6QAlpP8j7mWQtDEOuiYgh8aIXyz5W5++PYjvm2q6dqakKraOHiaiIi+\n+mfKNnThCqS38lq0pxTq3vFe1+fUIV0fmUO6rfM/e1Hn9fhXXT3C9au6pUgHDkZH+cC3nTntSpTT\nGOvSMe11BeQlUmMjxhr4HRDSfafiYC8/IyKiSerWd3tzw7dduujg/ASMsmfuOENERKOdLd82HkM8\niU+QKjPiJE3dHvmUZxjEZNAvz68ifAGhfuTnXb4oukXjXtD4QYLsQ9ljjW+owxFLYsipYflRjj5K\nJxBxJZxyaNyLUZ3ypRjhdYPfM+SCMw2nYbpzM75tdn7RH9cZCRhIxUy3nAYeWdUCPhUVyUCQS0Pc\nQhAcP+Gf8glEwNU4Au7gAhA/ZE5L9vuKAmoHdbxJ3fU9d0A151TTXT98dtW3jV9w12frEClGGE/O\nLsCKunU201h0I1oUJnjl+fP++OkX3D1HfTX4zdbYQDvWvi9dcOuX1uZ1rsJDVzDKlgktsAqyUGRb\nzA3g8ZoYjaTwfjC5R3fmgG+7e5YNdEiVzUbd6Sld7yOn9V0QdJig8ZfddGgQjny1XBwD5x0kigzw\nvfVFWtDQKfH7hbRcqeh8a1/loPGDBNmHEr74QYLsQ9lbem1jvK/aJxcA1JXItkmmfnVJEMEEmAKs\njyXCCXn6BP5ghJMYCfW3rsG+6KmZBd/WmlFfa1Jj4xjSHnMaZVovENq58WcIQUsfk0WjlBxDyq98\nXkjp5fGmEFGWAYymLfabz6jRrj7PyUenj/m2dIOTUDbBwLgGWxO/ZYJt1rLrMx/ilkD+wP0S3T50\nxKgF27WE01t7AzWMTZh5aOawwmhfNgjLYIPhLPbbI0yTdnPtTHX1Gm/8RCMwsu2Uk4F8Giwa4Djb\nqVDZB74xsS+HBKxGmWzr9N4p8wIWkpS4SlOE29axzrs/dJ8XGKi4T4z2S4SHz8+DdiVB4wcJsg8l\nfPGDBNmHsudkmxqiWA4/FP88Uk6TT1IBqaCXLoTICuwvwB6L3RGRht8mkKUTN4FMUaz5YB3OhRAS\n4J/QS2N4AdJdy7Ymxqop4n9FKCv3AaibTWR+em0zgjE2OR8fHuU4df20p9Ua31p0kDp5SSE6rSv0\n9tbjTGH9eMnB8f6m+t8p5nxz0hiAuWm1TCcJezsw14XnM6ygNp+GeUlVHPRJ4/rIbEYQ55Fx+HIM\nvAaGz4yR3QdDnhk/F15+gdxI0OmpsrVtAnMwHIthYqwqxO8HzsFfAnOVHP0IXxo9rAmzDoQQ+3eg\nEJ8i70zw4wcJEuQmsvf02tdEGBW4woRzDrMrWAo/csUOS22aqljB51YsvEdERY3fhGIOkm6JPtlU\nfLKoQXgZIzD41Wc12m206TTrBGioYzYYJmDwi8VnjfX9qAIFpHqNGH/WVnu+bTJxsRIpgRGwt86f\nVWeuiHErHevnly9xzEICUXFD56efDNQA29vWa6aZUGmYKtro77jPx5AAlU4EyWH8hYylWgvGcXnt\nhe68kNjE71QM2hKNaMLJlxjUwKK9MU1W4gp07TtQ4EPiOzBwT56dgfMkgSgGLkGJ5izQEOK0eRyj\nIt2UXA3XVISN7kKCxg8SZB9K+OIHCbIPZTcMPCeI6HfJMelaIvqEtfbXjTHzRPR7RHSaiF4kop+x\n1q5frx8RD7l97bACTuKTIDzXV9fBQVX1jOGO0k8FZMQ0eWZiqU8p4WejpfwAxNDcAOVy3mR/8ZQO\nYnrxMP9Vn3Q0p0a0ncvupstnX/Bt46EzmE2AXVmg5dKqbgm2+w4y74zAqLSiMHvCxqFWTWF9d8Z1\n2rikY4jZENUbQPJIjP55N58UquZsbrgkliiG+/XdcX9727f1NvWa2sjd80BTtwdb7NPPYSskNOZx\nQfW4frByD4HRTrhI2w2E0e7zhPSaJBIK9HK4K5GGfxeYfIwwFJXDfAtETpg0I23IxuT/VvCY43ut\njKjQNxi6PSU5GA755GLtREk+k6pHtCvZjcZPiegfWmvfTETvIaK/Z4x5M4WiGkGCvGFlN9Rbl4no\nMh/3jDHfIaJj9N0W1fBcYvILix9JskvZKFc4scJzV+Q4y0snihGpQIvNxpakpWm3KRjWarw8czOa\n8to88lbXT1t/iZstp4GGw6u+bWnjKX+8eOf7iIho/rAmhRxZdmO8+MzXtZ+u08bnHnvYt62vOrdZ\nBvOLwNiYsIuwQJ83ceOpT3QuogkSWMikgJzY2AhJTG3iarERGOq48u8EPK5tMI5OGg55tIADzzMc\nTbQfGS9Gwk045XW4oQlQqLUb3CcaiC2n4E52FN1EbX7HmuUEn4JUuHsLmprniJTkWKk5tuXoU7I8\noUKlZp430gvK+38dDZ0N3DqmViFhVhckqGOcMLtPXJdoxOr+rpVb2uNzRZ23EdGDdAtFNYIECfL6\nkl1/8Y0xXSL6NBH9A2shJ5WIrNuQVP52GWM+box52Bjz8HZ/p+qUIEGC7LHsyo9vXAmQTxPRv7bW\nfoabrxpjjlhrL9+oqAZW0jl97Jj10XlGEiAwcs/9zQvMKeXfk0JuvkTugRFIosKQklrKIBciuOR+\nANUMHLdartpPd/6ob0uNg969gS8eTNY4iB63NcKtC/OqdwfSoW+bPXza/T2uhqo653dPT+uW4KH1\n33H366mfHg1MEl2WAG4XJqPVFf2hFTCeNyD/2ygc13LMuo7CNmNgbUdcsWZrUw2QA4DwA44xGI10\n7ceZN4PBvQUmw2Skpt0Eym4naDhr8FjhnaFytGcq8RIQS7CdKs5e3XLPcHV1Tce949b3cBu2h0y/\nHSVQGw+iJvtjZmiC6kvEx505NRi3pp3R9/CiJoPV685InMAWBOc1GbkxTsD6W2/z8zBoOJSx3ZqD\n7qZnGzea3yKi71hr/zl8FIpqBAnyBpXdaPz3EdHPEdETxphvcdv/St9lUQ351a8qACAGKDRUVRS+\nLbgsBDmk8IsuWhANNVKYAo08faY4fumyMtVkV5b98YlDjh2mcUA1eVRzWmIy0msm5nYiIlq5qtTL\n/Z1N7fPEc0REVMNIsab7Re/ONKHNaf+j97zVt9X+wo0h3wSND/PyNSEgDfa5F929z72sYzgy4zTR\nbXcBw1ADUYLTAXmmr8TSVbc+Fvj+hkyL3dtEZKCuRMlHGEFk33jk2rKKAn9YjEKKSBR45rAQhkRS\nVsS3Z7lq5ZWrV4iI6BtPKzPQk8AStLLinuGgr891oePQxEffeca3zc66ts0Nnf+zF/W5v/0ul/Z8\naR3W+YiL/HzsQfVst7pO49/75pO+LY/deSfvvMu3zS2omUzeXYvaXeZfoNyTvJRbi9XfjVX/q3R9\nW2EoqhEkyBtQQuRekCD7UPY2Sccan0YYeb96uVYZMtD4emFIiw1d5hyxNIE6aOocRWhZNhI+/uxZ\nIiL67P/2f2jfcNp/+aF3EBHRf/PfvleHaJ2vPukAm0zm4HpMynizuPAOPT54PxER9bcf8W0bO+eI\niGgM1VxmD7gy4b2eDqK/4yCzQGwiogjSe4lpvrORtj36oou4W1tT416n6bYMppDMot3IP3JY5/We\ng/CjsUJiSRDKRsCShIlW/BhqQAFe58cwSsFo63OPMFU5Kvy99jhhw2uOBmHerm1eedG3Pf/Ms0RE\n9LXHrvi2tW3djlRlsNam3DNs1DQmYXvbzWEE7877336nP+4PXZ/HD835tjEnH73rbad828Xzbpv2\nMmw3Di64CNHPPfaYb1u8483++K+96z43no4aY+t1SQqCCkkcfeq/J+WpVUrQ+EGC7EMJX/wgQfah\n7C3UNwo1TQVHOIkFE8z21SGNpvR5CkkanvSywHjJFmOIEagzTHpTV5NrfvgjP+iP3/m+e4iIqFtX\nIsflc84av50qbNvc+AoRER09+hbf9pYfvN8fJzUH61qtt/u273z9USIieuaZr/i2A4edpbc7r/BO\nAPUESkcb3WX4NYsgRLbTdeOd7U77trmuVKmBnPAC1hevh7ZMmOBTWICINI8e/fQYLxHFzIhT1/iE\nGnsrsCClJCmhl0XYZjACNi5w0bu/raa2Nbiw6XAT+ArYY4BeBCSt9IliWPlI6jcU7u3aFlraOALv\n0ea226bNzev2YNBzD6ee6EOa56KjOwMdT49Dcq9c0ViCv3z0M/74yrILi/nIhz/g22YWJCy3zD3g\nk3VodxI0fpAg+1BeOwYeU6GVqazx5Tcsh19+WyglgmdJL5KqWL4/BKFRHDlNde/tGpn3tz/+c/64\n2XHGsUFffbJbidOm2+samTVcdf77ndZl37ax9aD20+a4gr6y+wxWHEKp9VVTbzzn2i72FE1srTvN\nOIQqNAZLYnN9v2mo1XZm3k2yNaMa30ychikYy0BzZEZSPnWBNgZOo/W31QCZsvZPIUvHJmigc+Op\nAUtQzm9ZH7JtLXP7zaL2Ssp01jHk7bbZ1z4HsQ82cwhsZ0Mr8jQ6PG+jSVMoPjQEXz2+DRog1zbd\nvE8ABXgGUYXJlIvEfPKyGlElXmIEwSgTTlK6vKKpzHPT7rwcDNA9SDT68pf+koiI7nvzPb7ttlN3\nuGnB2+5Th/2/dydB4wcJsg8lfPGDBNmHsudQX5F+0cjH/yAiYB/BC0AqGYkxb0H6RodtJJAoKZ13\naVlDbS9eVd/vW999nIiI0mUN400zB5m3d5ReesyklpkB9hpIAKnVXcju+LLCyNGm2z50DLD2cP74\n8y/q/S5d4VLOI/VDGyBtjNvueNWAX33s7lNb1zHmXAHowBwkLmFuiffjA3MOxwhsjuEajheYjBTy\njgo2VPePGlhlJTZgAMU36/ys80WF6ILwY7DuxTV9P6bY6Nnp6ueDLddnvakJUJJchTELBQ4HHm+R\n59WdW4vVUDfHz2MLYgAOzujnD37DxWI8/ZJu8U4edQlWH3nv3b6tx0lD8zO6HasxUWeaV29hZcuF\nhk4JVcb32kP9QLYZJEiQm8ne1s4jo7/CRvjM8LenzJIiabmm4OIr+ADd/1FLiMuqACbKHH9S6GFl\nSxNgPvPbv+mPm9HfICKieQ3CopwZZvpwzaEZl3zRHKrRaeusGmryVWes2wHXTT5xGjoq0Dk7g5eB\nIMSME0lySDttJjrXBquyaKJaKWGtlQOVtqR/ojYtlJFmTR+DsXWeDYcdUi2XMtV2H3jmBsABLRyJ\nzRmNZmslThv3+prMMhn2ec5oGORozuu483xqdYFrXairdTxNTn/GuQK7YKUBTEpUF6Liuu44Bxce\n9ilvJAyR6iYqnScMPWOsJ+jHUk5Lx/YY062lTmTBuCffjyrD+PUlaPwgQfahhC9+kCD7UPYU6rsc\nHYH4FZ5HxjpZgSSzDPULljz+7cKqObGn5C7TFecYZWYdAMwB3j32LSXJPPZHzrh31wPHfdvS0gUi\nIqolakzqtJxvH/PEJxsK9be2mTYaoLfAVkxSiRN3/VRTgWmXjycTYAYCeukWs/YgU5EkyGQRkESO\nxY+P0LkczoZJU3JuAkYy4ZLMAJcOcthmMeSeXgBi0TPOF715+UXftnnZrWNcw/txH1CLrhBdKGSs\nsBWS4wh0WK3hDGcxzAU3kVVgWOIFDDyPVa5OdPwoxF9Aws67H3A8DKdOHPRthw6wYRH6nmXD40XY\nHtbr7p0olHeHY5lNIaaBZ4Hbg+gGMSs3kqDxgwTZh7L37rwbGPc0mK8cmVT8jUKDCMdlw8dZLIaO\nstujwO5TET04hhMuLjsD1OUvPOHbjh902vbkAXVDKXpRjY5037kU5oA5eNsPaMvIJKXzqGJNivqk\nTNNcEyMhuBcto4A4QSMoUEDnUoREr8kmUuBCO89Sd7wzhDh4AGBHjjqNd+/9qvHvfbfTjMuXtFjJ\nI3/mtF/Q5YeaAAAgAElEQVTDYu4AazRk3Skw8EgMvt4vU6ji2xJ2d6LRDUUCH4teYzf/eqIG2uMH\nHZpq4rcELlrdcobXU8f0XVhhPr9uWxFhnLvnsTCvOR+1mrBEgZaH8ci8awZct1WVoT38uTUGnt1w\n7jWNMd8wxjxmjHnSGPOPuf02Y8yDxpizxpjfMwa9wkGCBHk9y26g/oiIPmitfSsR3U9EHzXGvIeI\n/hkR/Zq19g4iWieiX3z1hhkkSJDvpeyGc88SkWQX1Pg/S0QfJKK/xe2fJKJ/RES/caO+DBn1lwoM\nj4pnVFzkxnEdJOPpteFaMTxFYNLJZUsA10o6qQGCSYRRk9RBuYsXL/q2zZ6Dawe74KduSslnpEqG\n+/hIQlv6vABreS0ymMtYUkwB38apHjek1DfU90s4JdZABZx0MCnNL0eaatlmYLAjzwepq4WBpwFU\n4R3A1DMtB48nm2rcPP+E42gdDTXZhwPXKEpx/rIoOoYClTbHDqRgYMs48g3nItvH6Hrwt6JZYgMG\nI12znQEnAEFC0vFDCtdTTq1dh2djeKu01de25SUXpakVnoi6Xbd+kwpmKCJ93wvEoxVqWneZWfnD\nG8iujHvGmJgZdpeI6AtE9DwRbVitbniBCHinitf6ghq9ne2qU4IECbLHsqsvvrU2s9beT0THiehd\nRHT3TS7Baz9hrX3AWvvAVKd78wuCBAnyqsstWfWttRvGmC8R0XuJaNYYk7DWP05EF298tZPIeozL\nf8slhk0BJ5fbMGTTEzTGGErJ8QBV/PzYJAwtyPeOMQQMn5DJZTIZl87ToVX/jnoiRDC9xxWJROLb\nHwDDzhZbzzH8dgQblhGXsK7HMAeu9hLDOk04t35pdcO3zUKorTDqNBr6SsiWKQUomzGpaVyHijxg\nwV7j+IUI+OeXV12obhOuMRxDgfn2Er4cX4dsUwzqWYoeBV4f9KLom0RVUkHFShFvw+rwjTh4wm3n\nzl9R//tjz1zwx+/8PuetuHxVK8odOegs/E89q+ctzLs4gNuOaSWdzZ57bsWQdRV53w0Qq0YSsltR\nSUi3cN+jkF1jzKIxZpaPW0T0YSL6DhF9iYh+ik8LlXSCBHkDyW40/hEi+qQxJib3Q/H71to/MsY8\nRUSfMsb8EyL6JrkyWzcUQ/ALJxF8Famz+HvkjYDgPy2k9UTi+0XkUHbUSj/FRAj5tcS6e4UyPURE\n1IA02BOLzk89M6XbFv8LfJ10YjEy4Ril2k8GyS7EqGVpR/t5eei0ZJ5DoghotybHCBzu6hjnuH4b\n+oUbXL9tCRKFRg21ufRZ48/N6bwmgjKsJul0W24u3bqy+0TTerx5xRlEly5o31Ozbg71RY2AE598\nXGG8qtVwnTByjw15aNxjBIKgLS9pwWukovjMiOvsrUC9vfV112kypZ7q43coW9NTK26uMfAdrq86\nxLN4Eqri8KP5zhpUe+Lxbg4g3RqG6OnnE0REFT57SXVG+LIL2Y1V/3FypbGvbX+B3H4/SJAgbzAJ\nIbtBguxD2WN6bePDKQVnxRXGDQwRlUjSqhxkd65gnAomH2zy7NFQgpkchLM5bhMKVD48RoVWNYbR\nGPrrKY5NNd6SvHdMJJH4A4TjdUaMP/y+N/m273+nI1jMsOR3BXMMGsRqEg+M1k0pQZ0qtJwMwYrI\nfWK5bbfLIzLAEiO8ABlAYjR+Hu2c4PmBIVP81wVCHI41qPI/w6JUFY3Etcdn5y/Py4lbKL48OkxV\njJ5//Kf/WcfNW68E4hQSCHkWjoNCqXceXArv0YjheAoG2pzXbAwViYpU8pwgBS9xJOsMHOCyev47\ns8tknaDxgwTZh7K3DDyRoVqLDSX8U2Ug4gyLGdywHyyoIZF7Fg2CcoTMMHKTchpwQctjgpARd55q\nt8hKJB2kjkZi/IEknQpDH7YJK4uFSduR+3yuBbxvbebCyxHlwBT8nTFBSHJVUeNzP2BNG8Pa56yV\nsOCGT2UulLfmuRTaYH1MWSt7Y1yGqI2bQOFvMQLpa9BfwdAXd9zJdYw+tBUowBtbqVIk1bcD6c9z\ns86oeWBWE4kOclLNTEefR6uthr4mJz7VwPgr70UK69MbOCPgGqRqr2y4qMCVNeVFXNnUY0GJ/Z5G\nO65eFsMsINCWM9o2m2pg3I0EjR8kyD6U8MUPEmQfyp5C/TTN6OqKS1jwhhMIldqZuAipjU2NhPIw\n8bpReALryhGABdZKkkQa2BJ4yFRm/HFd5sUxEFGtwdFnwO5ialzzDXFrVnAsl4YtRk2h1CZSiIo1\n5mSPgqWoC2xEcgxz9buVGPE20zXD1qIOMNrmZaOlGKoiqK5juCw1mertUc5JNwav8da0Queub9h6\nSPBhPYVqNdBPPnHQO2op3BbGnAJxqMSIwO1aDYXrb7rN+eK/7x5lVjp00MUiACEQcWlFz3xDVNzi\npbylquEcYok1gOhC68adjjVvf8KJP/2xbpNWV7Uc+YUVB/sbUINwIKfCFi5hY7XxefuBbDNIkCDX\nkT3V+EtLV+n/+n9+jYgwxh5+LVnjjXsaTz7iumOFHzJ0n/DfQsSdKR1A9HY5Np4qjIVEqk2QunlS\nd5ojw4hDPg/56gr3Zk2XgcaTiK2tLY1pz7wWhRh7RhvotjKFGK9yKqs3yll0AXKcN1UZQclTjVfq\nC7wdGzoz0Mo5uPbE3YcU6FWuJm8wBDQ1VeNoPqCzzoGhaH3Dxfxv7agWHPad8Wsb6vtt87Q7Hc0N\nePcDWoPuHW+5jYi0pqGbj8xL753x+uW4APgceAFTQAGWv1L4OoobD3NM4sTNoQ6G0RPHtR7jqdOO\nx+/ECS24MsOGR4tok/vc3HaGw2yX6blB4wcJsg8lfPGDBNmHsqdQf5JOaPmKq1PnE1uwkgpDqhpA\nVCEYxFTErAqQFvzvAnWRlYcJGOESW5EajP1IlN78vCaXDJmAEiFVo8EwGe15sD3IOY10Agw0O30H\nKV96acW3LS+7un3jsUbXbQ3d/cZoS4MqLT6Yq5CBytASDExCkV2vKUzGKjVRzO2wHfG12jB1Vtas\nyFrqjxTVY3qzGwdC4ozHmEQ6njZHxaUEJKFIL83wmICgs8YWuFpdDX7THfe8PvLBd/q24yfUsFbn\n5cshPkNWKsXELjHAItKvIILN0Pg3kTThcgxJIeKSZKvbgEYw9PL79fTD3/Btmzuug7WexgPIFmeb\nmYOWl3XreCMJGj9IkH0oe1w7jygWV4Q3NoHWsVIwoMyVF4H7CA1U/ve38Est/UH6pq+xVyZnyEFV\nFyLuRNsAAuk2XFt3WiOlpuachsH6dhiXnbLrxibadyd1v/RHjitjmXh2ri4rp0mN1dNwG7QT1GDr\nbTOX3liNUs0Gu+GmQIOw+zQGwpIkBu0vi1ZhnyzwAnqa8rLhlEhzJ9JCbgGvI6aYSjQjZpiyazcC\nwuYo0TFGsdCPlymnp6bVMDbfdddPQcSdAWSRZmKYRJ4+RpuwJrlnliujACJ9Ny1wu8s7UPBcCgoC\nQ6UYow3eL9WLJowsJ5uqwQcc2bi1rinPV9edwbPHvr4JoMUbSdD4QYLsQwlf/CBB9qHsGuozA8/D\nRHTRWvtjxpjbiOhTRLRARI8Q0c9Za8c36oMIK6NU8KEJdMJ82gpIGCH0qjjSKjbIcccHaLyy8hn6\nyAGucwrrwiGtjZawH38Exps2J3Ng4koG1rh0JDx1CMOYbWdJYVvKPmJwi9O4764dAqPN4ZH6p8Xu\ntjHRi0acWovJI41phtGQdmvBuNdg3jzkuEt5IIX0ZoaoMiciosFEffqZwGjIeR2P3bkRhMXNzLg1\n686o4XSu6+Z14pRufzKrsP/KxatuDLjVY2iNkXkNhvi1hp5Xr+Gr7saRgw9dssZyjE9gIyI+1xzq\nEca85rj1Eu4+3HrK8mBsg8R0pCn0BwbccSp08JiCK1F6emK9LgZx2crSruRWNP4vkePaEwkFNYIE\neYPKbnn1jxPR3yCi3+R/G3IFNf6AT/kkEf3kqzHAIEGCfO9lt1D/XxDR/0xEgssWaJcFNcrCkKQC\n6ptKBpvy1gAZeCAjp3wJiLci2/I2Aa3WNYCjCSexJBD6eWXJxSEsAAFjxD7kekPb0lwhY8pQN96B\nnOnIHY8m53zT2qqDsv0dTdZYWXI+202A8pdqEA/A9xknChmPGBfaOQvbDSEMbTaVCtuA3zhl6IlG\nfYGhGUDiPsPenW31JW/CvcdbTP45raGmPfbf74A1usXJTgdOKuX0sWMO/v/Aj33Itw1HCuG/9ZUv\nu3vAWsjWo47ltjk5qQO8Bm2Ytzz4yUjXcTx2a55ZYGgSGI5cCOC5qY/cucO6Pq9c8vExjJnjEkZQ\nkSc3sq3DcF/wDjAvAs5VwrcNsi3xc60Lv8Uui2fuhl77x4hoyVr7yK56LF/vK+ncIhFokCBBXiXZ\njcZ/HxH9uDHmR4moSUTTRPTrtMuCGtbaTxDRJ4iI6iay4qM3HCuFv0+2SrsX3f5EdM2vlZVIKfAb\nR5I0U2H6q2hrd9XAdOr4YX/caTqNcXHpJd/2wrLT0H3gq0tqTnu/4333+rbZw+pXTtivntSgMEfK\nxTMGqgUHOy4tGTXstnH20ouxapWDiY53RO7z3ljnf4wPDzf0PMu12lDrWPD5SkJSBAZBocfBH+w+\nUzxHgGiiBmjTiUsnHZL2bY857T5ev+rbzi+7Xm+/W4Fil42ks4f1GWxt6ngWji66YWEiitD5YQQo\nG9iawLBTA+OfcOWlMP/h2CEYC2XChWsQ06AnY0UJKfPlxZtacEPqA5oxGDyZGt0CX58wL0WT6tgP\nSWuOwF5e4xTuFnxtJ7k7HuU3LtBxrdz0LGvt/2KtPW6tPU1EHyOiP7fW/tcUCmoECfKGlVfix/9l\nIvqfjDFnye35b1pQI0iQIK8PudXaeV8moi/z8XdRUMOSYUOJhyRokxMjSqHkMUNQNOhF6Ff1R3Cb\ncp69ZYYaLKctJJszUAnmxOnT/ni64yCsHSlEe+z8k0RE9ML6i3rNyweIiOj0KfX3H4YtA6MxSsEI\ntrMjBh8wAnIu+GCocHOL27JYr10yaliTsNGFGTUcNhjiD8DgNdoSWKvr2ATfdk1qtWEILUPmtKnn\nddjYdKKp416pQ7zEIocY5wqj+xPmNYDQ5ytcOnp1R0NSO9OuFqsBws8Y4PH0IhsCAXqnI3efKMZN\noyQA6bW1tq5PLGw1ffjcU3dDKIrsLwH+R5AgFLFhziDZpphHgcg0arqtTpzjNlMqSuk6GgjplnAL\nE+s6RrHAefjacniz4c+wXPqNJETuBQmyD2VvC2oQaXSe+tL0I19VtoI+u8yK7Y75LyaNSOJGsaBC\nXjpPfvXwPAM0xaPEGYTMRI1Ety84Tf7UhRd927OXzhIR0QfaCoASqDorkYgYmdVuO23RnVZXoSQF\ntTt6vwOGowJTpV7eBiNQQ5J9OmpMXOCEldUt5S5Mxq7vqY7SRxeoqw3Ta4MukFTehYZqncbQsSPN\nzup4Zo4qYsq+486dQPLJSt31U6tr38O+u99OX8e4xpTTD37+875tdl7dfSfveYCIipGNV1/8NhER\n2UIUHhewgCg8O9Dn3u+5xJbll8/7ti1OZ80BlaVtd57paD/jFZ1XSzKUM33WA66mHGW6zmM2CLaa\n6uJsxPxcYU3yERhMGUUkkbqIx4k8B51Lzs+rw+9YVJGEViVB4wcJsg8lfPGDBNmHsvdQX9hhJKmg\nWFCNuFElKhudqvLDi/X0xJCHXYuhBqAVG4lG4Es3I4Wwqz3ndz6/fNa3DTJ37hD8tM8tOciIsC0B\nWmT5fc2BqLE77e596LhC2UbbXdOBCMAjZxwF9GBHIXEHfNKnjp3gyejdHn3KhVRsAnNOt+W2FAtT\nYAQE417OEWsWWGnimjuen1djYmdWar7p/CYGDGJH3djqQ8j1HziomwDd9xzvDk6c1BiAGufjX7qk\nrETjofZ95n6GzxDhKSSZOUS4pRyRZ4Fy/OVzGovx8nMuFmO0A0w2PeeL35no85894+aCFZ42t3Tb\nM82lyTeWYH3YuNm7CvES1vU5N6Ulto/NvtkdIGsTMBTJcQKMSRKlN8KtLp83Hm3zvXYXJhc0fpAg\n+1BeA+Me//Eav+qcMn8e2iwiMNCZCjeUtBXYdPJyCKD0vdlTjX/22ef8cfe4c9M9e3HZt+2wNhmM\ntJ+X+s7g9dWvP+Tb3vLO+/xxwm41LGbR4DjyRWTgYY33/NoV3zabOU3/wNHbfNtR1OSZ0zZrUPjW\nslGrU1dt2mTNkYCKaUMxD+G+GwEvYM7WyLSh12y3OOISDJ6pghGyHU4TnQFUdpmppOdUe9f7XCdv\nXe+3suZcfPlEX8u1dT3+2uf/lMcKzz9388ccC4nYzCEOfnNJUUS/5zTwCJ77gAv25dBPe+LQ1ulj\np31b824w0LEBdwLvwjB2yKO/pIsyGLqoy81tMNBuuzZkiUJ2JJlhVoAE/DwGGim4s+nWTNCUqSZI\nL0nQ+EGC7EMJX/wgQfah7CnUt6RVSYyv/1WRJgsRXhFHOxX9kwj1BcIDKwtvBbC0tN8LYN/cD6aq\ngv2Nzl90BqHRlhqOjsy46Lw+RFRtjRys+8yf/IVv+/CP/KA/vvueN/G4tJ/ByEG9x597xrc9t+Eg\n/vqWGot2OLJvxihUTY4e8cdx19FGv3BWE2Auc901INuhmmGiRtgSYH27TscZ/ZAWetiXrZAaGy33\ng3uvhYO3++MXr7LhDIyk6Y7rs1HTfg4dc9uoMwfu923Pn3NzbM2q8TKCZzjkdWm29Hm12CCKNe0y\nZq/Z7MEz2tIthRhmd0a69ehn7ng00Gd08WWXgh2Rjmd2Xq9pc7p2I4FYDM9ZqkbUGqdgE/S9uebm\nigbPRg3TiDkiD1Jwx5xINOxrpamry24bOmDj5giMzjeSoPGDBNmHEr74QYLsQ9lzXv3IQ3NuLPgd\nmdwQkhByqeYCZ8VorZetgy378YtVIa+5LxG1Ow5O3fPWO33b5XW14AsjztHDOp5Th5zf/bkrCrcW\njjin9LlLl33b73zy3/njv/t3PkZExe3K1x96lIiI/uTP/tS3jdjfjb7Y2bZLuOlv61yev6z3Xtly\n0HN9W/P1pziWtAklyLttJqAEaFmHWANJmopxG8WJL3ao4afDoVuzDDjgEyDEfO45jllI1UJ9YMqF\npy4c0piFM3c6P/bKpkLni09fICKiQ2cgtmFG+9npuXXZBoai7YE7vrK65tuWlt36rPd0u4EW8xav\nSzYBBh4O+R1DghQZF8Zbf+Jl39Rs6zajzRWUWo2ub+twSHQCMRJR3W0FLFj1G9bde3ZG++sCrUGf\nt4IxxCzYsfNCDHaAgYm3D6vbHCqdBT9+kCBBriN778eXyDmpJIL+90hYeZDuWjRLmXKbSAOfsgJy\nkIJyN/ZpdpniuTdWbbE11MSNCblf1sPHNN22F7lf3X6qvlQaOAPbwUXVVH/0pa/544P8iz83o5rh\nq0/8ZyIiGo20n4RTUBebqkE/+H3O+HVwas63YXxCjbX2CBI8elw6OgcfcIONRBgJFkeo8d3f4QRZ\nadxxCtx0A9awJtdrp+Y1BXnx2FG+Bth9mK0nn6imXmLDKcEYulNO5bVTSObZUi35jW+5CMoXLqkh\nc6fv7mOgSk/MGj2HZKYMjtcYESaQytth7Z3UMW2b05Ihmm9pWZ+X8OVhRGZUEWkq/0Aad8mPqmFk\nHvrxGa0e66rB7+RCk/vRrmc4oUsKDtXWvkcMPEGCBPmrJ+GLHyTIPpRdQX1jzItE1CNXbiS11j5g\njJknot8jotNE9CIR/Yy1dv2mnUmorsdCxVQaomsTbgqXOYnKv1e2mNlT7qciNnjMVSoff0qTcOpT\nUP6666D7lVUNvzTWHWfgX+73HfRcPKrWmR2Aul/51sNERHR8UaH+zo7bMrQB6rU43PW2Qxqe26g5\niD8cA99AgUlcKJcBek+7rUIM8494G9FuqaEOqaIts950MQFEFg2MkkL9/eJzF3zbE08+7Y+PHHXb\nohMnj/s2qfbS7EASE2PTUV8heHzOxTGcO6fGtDEWNB26NZ1v61zneFsUA1V4xCSpGfi0B/A8pE+o\nzUn1JhNnQm59wqHDMcw/t5CcxKHRGYQG+0pE+M3ita0DKekUMwIdPHDIt436Gr/R23B+flgySjnk\ne25Wn+H0DCek8RAa52/+FSS6NY3/g9ba+621D/C/f4WIvmitvZOIvsj/DhIkyBtAXolx7yeI6AN8\n/ElyXHy/fLOLxGhhq2roieGvwo5nTIXKdh+4vxVejAK/ngcYGCnIBq9IDSj9nhqgjh92Ka8vnH3c\nt02EPhmirEasibJIjWAWtMRS3/0Kz2XaJoalqTZQex9wmv7kgkbCGVPjsYLmwwIgzNOWgUEsZe1n\nAeZIMYb1DRgjGu1YC8agYevMRjQCTrk6p8FG4CrMwP06zcaodKzai2kDKQUWnDhx6jaCNOkua8Gr\nEOFm6hoBNxDOuroip7kp50rtr6uB9vxVpy0RlQEhEAk4mJ02pc8HGzB/XrM2cPe1ujqeg5HTvNNN\nfRfm5l3bsXllRDrAPI5v+9CPaxsn/rQ7Opd0qO/e5qpDPy898aBvu/idb7nzoCT6gN24lqMVazUt\n0HIj2a3Gt0T0eWPMI8aYj3PbIWutOK6vENGh6kuDBAnyepPdavz3W2svGmMOEtEXjDFP44fWWmuM\nqfSd8Q/Fx4mK5ZmCBAny2smuvvjW2ov8d8kY84fkaLWvGmOOWGsvG2OOENHSda6FSjrGCpz3JYML\nJXLEmJSX2grMIph7I3THGOHkjwHq23KO/oSNP2duP+HbHn1WCwKff9lRaXdmFScONjgyDXyymwz1\nL1xSqDYGaD4ec1lmyJ84Ouf83ScXlJXl6IIbB8I/YZHJoaYblnyWiMVaTRNFMitloHWukhyVF8p3\n63gHbKiKgbQy4sQdamjfkppvY4W8WxA1eOSEM+oZoMVOR5w0s6lU2jlXfpmJ1cI24biB6a7Ovw9R\nasOJMAId8G1iwOxPFP5u8/OgjhrdDt4OsJ7fwe0tUEVDLn8Oaztm1iOruxYaXYbXnN+pE7M63pN9\nV+1nO1X4v5U64/GP/rzGOxy98010Izl82s1xKlLDcr7h1g8rCWUcqRfxdqT5xd1VuttN7byOMWZK\njonoI0T0bSL6LLkKOkShkk6QIG8o2Y3GP0REf8jGtYSI/o219j8aYx4iot83xvwiEb1ERD+zmxvm\nEp0nmr/AiJPKgbZVMPVgRVB/ZMtuQVuk5YFPnAzZndcFjrsZiJTqMaV1AyOzmLqZUkUBxxbmeaza\ndn5Lo8vqCdetG6hmOL/jNNTGphpjBrnTbqcbJ33bfNu582pggKwlYKliRBTXdQ4S8x8BB6AUqbAQ\nmba8rdo0Zrad6RnV7jEb92pNpc+Wmg87ENMe1XVsM1zYY7oDiGDV0VRfOqeupoUZx2Qz3VWtu7Lm\nkMNgXbXqlRXVeH12yXXAn7kzcM8oHcF4pA5gU89rTulxo+aOt1cU/cwdYKMlIIzuCadZWzp9evkJ\ncONucmQfxMdLHcGdvvazykUz7E03u4Bqx25eV84+75se/LMvExHREM4To7c86p1tiCi9gdz0i88V\nc95a0b5KRD+0q7sECRLkdSUhci9IkH0oe56kYwqgnQrY2/udK4hzIrwOfOSSSIFI3/vvC23+Yn/U\nY3vQOUi86ECq6mDAhjXlZKR5ZrxJM4Wypw4xNIeSx2ZNj2NOILkAbQnDvjTRxB5rHMy+tKVkm0ub\nDibfdviUjgF86GLUjMDX3GD67fFI7ydRZulADXHrq8rq02rz1qsLSSO567MeadRb0nDzbs5o9FgT\nYhGavOVotaACDJcHf/s7gIA0loQsHeNi3/VpIRJupafPRsybO0uaOi0PtgaxBAeOuXs3j0G58SVI\nt51z78fpMzpXKam9cgW2nrwtmulCKvOdOq8Lz7lxYnLNeN2taR/evcV7nSEvSiBUsEpSqAm5wc9p\nrB3NzLg9B6b39tiw2uzKFi1U0gkSJMh1ZO81vqfaK1fGBXaOUkuFnY674SqnFRV2izCgTOgxM+sM\nTN2mGt2OHFZLzuWei0+6sqwa5tSCo8N+97t/yreduPNeIiKamtHU2Qiq24qxqdlUw1mLXWT5WH+9\nz3/TkXJs9NXttcnVZK8C5baZ0VipaWZvyMZwP+aAszEUCmE01e7q/G6/G6xWvOZRjJF7DjkkTdXo\nNTZeTYY6xgzoniX1tJg67F6z1pRSUw+2nQHPJGoYrNfYmAi5GAe6umYSfYiIwMRld+5Uh+cSAcIC\nHr/J0GGHHsTyDxndtdVTSFPzru+0p2/NfFe/MlP3OPR35awaIC9surVAFDB7wKG6FN4+8bQigUy2\nqWisd9W9c5gHcOiYcwcOntfiIGuc6nzgkHMFIwHIjSRo/CBB9qGEL36QIPtQ9hbqGyIr8Ev881gV\nh/9GucIyOd+aMvwnUv90oXaypKpWVNdBX+mbTzqj3Nvf/Wbflo802ePgJvPd9dUIFtU44mxK4e/R\nY66fg8c1uQbhasS8gVKKmoiI2Pe7uazprZ0pNxesaXdw5jDPE6reQFScwOg818kOuCbciy8qJNxm\nemnk2Rtn2k+fDYGHFtRod/iI21I0wE+dsHGwNaOYuAWxD1LDrdPVbU8scDwCWMs03p56msgbbdsN\nXbvDUwr1mzMOMkcT7WckVYWAXnvE1XWWrlzybTlUA0onbt5DoDuPGmyoa+m9d15m/zskF7XGOtf5\nputzcV7fhRFbjLsQa9Hkbd36sm6PatyWwHs73tQt0+YV9170+xr70OI53H6bGoTvvs8ldo35uSCr\n0I0kaPwgQfahhC9+kCD7UPYW6luF9uJrt4XwXEngqbg0qmgkooihnsGqOZWFA7k8N/zWjfrOor4A\nJJgD9psTEc22XfvRQ8omk7C/eAKltf3vJ8zFFibB2xVINMqZMiUnTS7pMnOObWsud8aJKRNIQsFK\nMvxXhEUAACAASURBVLKFicCKLEd33aG+f2GBwSKMVy/pXL/1yLeJiOie23XbM+3peDAM1M11Z1Up\nvgfrCmFNxEk6GHbBsBfzrM4zU8xGT6Gs4bU4dUIrBU3AW0Gc0PPssxrmvLzpnkMCZJvDttvW7JCG\nzY7AsT7mOtMLC7COvKVMoWrSZMe1dYCqJwEugENN5+FJO2CtnxNeAx3PylW3Pme/8lnfdpbXogOe\njkLlpwEXEB0A1G+7cdQh7LrJXp1RzAxCUYD6QYIEuY7sPb02R4NVOug9CoCWqIwCMPovZ6NOVEnJ\nXSV67ZDTUnNSf24NDEtHWs6wFoOff7jtflnzsUazKXcdJhzheOQYOO44RzeCdNupmSmeC9Ty4zTZ\nEUTcYd/k/eaQhNJ2xwkk7sgCxjW99uKLGhtgmbMvqqmxrdHieaNhUcpIL0GpZihwEbGxrgaJOzp/\nHePCEZe+evbcE77t1O0OLZzqKFJJwZDX58SXIdQ/tKxNByPNnd2a7PDdAAVBZN8Mxyc0a/C8WEOP\nwUg6acq1UBtvU1HEhVWmpQC0JRGEBih/tteckfHPl7VO4uJhh2rufcsDvq0P65hM3PpC/Q5KOPEp\nBuOuFI8xnnMwaPwgQYJcR8IXP0iQfSh7D/VvID5HJyrnG2P0LVbNkaPiL1gFkafexR8Nmc6431ND\nXQy+3XrX9dqGsFJin3XUVEicjR38y4DRpgbhuabC4OJjFmBeOfMDGMzB4a1OHe8HVWE0AhpoqtmX\nGwO7Tc7bERzLECinxdce19SP3511YaBI7T3uu61Ob0XzxOdmlVnm8kVnjNreVOidc/KShRLcW5Jo\nUoO6c2zomoNEoQQgc43X4O3Rvdo3Q/Q00y3TFx9yVYr+/JtazWjhyB3++OQdLsv8ykWdwxpX7GkZ\nfYZt4+Z6+rAyNM2OwM+/5eD4eKiGVztwaxpBwk2cu+e6vqnv2eKcW59Ds7q267G+Z9srYtTDd6eq\nJuTuoP21EjR+kCD7UHZbUGOWiH6TiO4jp2T+DhE9Q99NQQ3p06twNMQJLTREeGXi9kL+OJW6JJcU\nw/m4Z2yUrAiM4OJKs+uqnSap/uJ3fc00oLNmQ04CCSBjdu2N+5pwU4dCGcLPVzA6soaNIJJOar4Z\nUtQhaccZRniNAPGIJgctIJrVgkspZprqBLj5+lDM4vgpFwF26bwmJC1dclGMWGl2navSfufJp3zb\n4gmNWLx4xbkIJ8jnx5GEI0AYE9aC9Sl1XT72iDOWNQB1tdoQuSdJQzAeaWu2FREtJs5IeueCaurT\nd9/vj6dvc+nBZ+5+h2+LWNuOxzruOrnxngH2njvn1dpmGY2hAVJ4HLNUEYjUEexD9d7+5YuuDzDU\nTc1p0pS8z/W6PqOaRHQC5564cWUEpqLYTJXsVuP/OhH9R2vt3eTYeL5DoaBGkCBvWNkN2eYMEf0A\nEf0WEZG1dmyt3SBXUOOTfNoniegnX61BBgkS5Hsru4H6txHRMhH9jjHmrUT0CBH9En2XBTV8RJv/\nqzBKWHZyrOnmo5mAHhn6k1JnFpITqk17QtoJTDUMnZpQlpomeekaYWchUoYeAz7gEdfBGw51y1CH\n6iqRcceAUH2J6gTiBrKaG8eor8aiPm8f0qHCZFwfMcr5ZCWCKjXgf3a7NaLNTYWbGUDUlcvPEhHR\n8+c0aWiHS2YfWlzUuXDMQ29Ho/4a60osGk27uQ4HuhYzTIe91dP7DdjQ9aZTZ3zblStum3HxnNYy\nzHPNdZfnUQODX9dHvkFJbKbunrPzvm11Xf3v5oib1/GjuhWQuIM+bEcirgaUr7/g20aJ9nPsNBsM\nIc4h57JBORqg2dA3GOsW5iobf5sz+p4Md3R7MH/UfZ1aXd1aSAlurBMoW+aUt4K1BhiibyC7gfoJ\nEb2diH7DWvs2Itqha2C9dZvy6xbUMMY8bIx5uKLKVZAgQV4D2Y3Gv0BEF6y1UsTrD8h98b+rghoS\nmy8x+2iUE3rtIi02G+ogjjk1GPNe/hWsMddwq6PpkgcXnMY7dkzTSU+fcsUsFufVhTUZ65I0OFKq\n3SlXmN2BSLrBlrNpDjY1fr3ZAJdUnaPmwL0mv96ovSX1dgyaobfhNCNquQTq9kkUVw6uMonyGwxU\ney2vOq186cXzvu3lFzVtdYFTcNcGL/q2l644QDe06nJrc6RYBK6psdFY/atr7O6bqPZ6mlFEDK/b\nwpzTeMMNRRjbKy79eTAEI9g2uAUHrs9WXZ/rkaNO665c1hTkzSU37hyKg/Tg/VmdcvOZm1MkY8bu\nWT98VguqLLL7MF3TMR4/oIbMIa8z5kmIWxXfVwk0HA+1bfawi/M3Kawj6fOibbcW0w00+Im/G1So\nIAx2KUeZosUbyU01vrX2ChGdN8ZI6Y8fIqKnKBTUCBLkDSu7DeD574noXxtj6kT0AhH9ArkfjVsu\nqBEkSJDXXnZbO+9bRPRAxUe3VlDDGIo5jbbJfu75aUhzPOAMNYcX1ShT4yo3o5FCmK0NIHdk2HP0\ngLKSnDzpYGt3RmFSnVNDCwa2xEG5DJJidkZA58wJNAaj4thPOh4qRM+tM3SN+grbsone26bsY4YE\nGKnVLEk/RESTHvuAoeZdq+tgLfquawluGVz7aARGOy7l3ahrP4tcKYYyvXZpRbdUp+9wgA6JNVfW\nXfprzei4U452TJcVLM5P6dpvpW79Rn2F6MO+W580BZpuns54ov3IbiWfQJls2PbUue5fAwhB56Ry\nDRgqJYknj3S7MRrqs5GU6q1N3Z1aTi5aXb3s2+7kpKH3/7W3+LZuU78y21tua2dgmylbL4T6El03\nGgKTERsTxzv63BIoR371vNuGTS7qFubwovt+NMDYLDEvsZEKVbuzpIXIvSBB9qGEL36QIPtQ9jRJ\nZ3a6Qz/x/W8nIqIDcw62zM8oA8kUM94k4JNOOW89B2tlDfLRxW9Zj7VNwhaHwFqTMz95DeB2vcOE\nhy1oG6sFP+VimSs9hU/iq9+BcNfOlIOb0wc0Ynl6oKGoTe4/hSQMCeNMwW9sueR3kijcpqjPfeh2\nJIkTOOatU0chfGtauO/V89Di7UYTSnDXoBrkidMuB/6OO9TSnZh7iKgYBjrm8Q76kIOf6ZYhte7z\njODzuoO/L51TGP3kN9024tDtd/k203bvQrurcxmsabxAtur6xJDuwYaDws2WbtF+4Ced5X0WvDUv\nbeq78LVtt30YwtZjLN4K2EYcbLnP7US9H/0JhAu3hCkIrPoM8W2hsKVbv3WA7bLNTCD5ygAZa/vI\naSIi2uop+evSSy7Md7Gj91vkgq3yDhb5La4vQeMHCbIPZU81fqfVone+xXG6STRTDNor5l+rFHzb\nSSJsMsCPBn3mrG2GkJYpfs4xMN40OTml3tR+Ooww6l3w04/1l7qfur7HgBzabPzqzquBpcUcebVE\nx2Az1UAmdRrMgNEqZlQzN6caOOu4X+8cEnIyy1xqQLmNRqC6Z9mBVF1WiDkYCaXPSapzOXwSEkAY\nOeWTIVzjzsWEk4w7H/e1bQdScGP2sXdnVWvPLbr1PXT4qI6Rqb1vP6N8hidOuvTew4fer/ODeIjh\nlovi215XZLWz5WIIli7peVOLTp/NzIBmhLUYXnBatH8QECHz2HW7qoG7nCAzII0eNAO9ZsTaf3b2\nmG+T2I8cEo0sv+sZIL7nnnPavwUUO7WWzuFu/p7ccfjtvq3HFZ2WX1AmnxcuOzTSXHJjHI7ge3AD\nCRo/SJB9KOGLHyTIPpQ9hfpRFFGz5WBfwgajuFDsknPPIY9cQluLBJvl3HMkspSsgXah4ky5RHFu\n3edxDASTXTB+8TjyXI1AAj2nZzTWYGrexQ3U61AUswlMNlNsgGzofSL2/Q5I4V0WO9iXQBWWlBNO\nMLQX6Yi09DKuiftrgMko4fNqELMwhrDajMNbI2Abkr5rFeSnk5auSXtGjZF19nPXm/oMJQbh0FE9\n74M/7Nav0dEYAMoljBuStIBYc8L+6xmgQ9/ZcGve31E//cMPPUlERGvbEFcB4dLElXGW+t/SeXFS\nVQPW7Ny263sDEqm6wGdwx91uDgbCaqWMeA7bo1UOIa53NVz81PcdJCKi7RU1eKJ/vmbk/dF35uDJ\nO4mI6Ohtd/q29WVHmPryd9ycTU1Zh24kQeMHCbIPZU81fhLHtDjHv5JSE8+qsaTOaakxJKEIR1wE\nWjCbQNQca/zRCBhquAwy5K149hJMCoo5xbbR0GWoVRgRY/iVTzjiEBOAmp0W96Nt7TZowbowx+jv\nrNBho2uOLKf8ArpJIrcmSAGOCTk+OSlCl5o7F42AlAuTTzVyqPOaI3ffhNNykc7bCp05RKvVW/q8\naoyIcqANTycyNlxnYU4C9MJK2QLFtYFrGrHTgjEmLDEqOXOfjiGZdu/YE09rOu25K+oWm5EIQUhi\nyiQZCBDGy+vOYGaYnYiI6Pb77vPHs/POGBnBMzSJc0kik08/E55uXZO5Bec2nZpWt+8EEr+22OVo\nY207cswhiwRYm+bY7deddaiz2flD2o0EjR8kyD6U8MUPEmQfyp7TawtUlLLGDfRjMkxEQ5zkqCPc\nTA3QGZPQQgMhJvtkk0T7bncZrgO0zNj334BIqChRw5FlI2Ib6puJcStJINpP2FSg4g7CbEnsMRhV\nxVC/XsfEnT6fB6cxrIddAqVglJP6fwj1c6kTiBCUZAwYAwBbCoHPsD4xzwFjLcZcQWgIxjSkPhdo\nXqgk5LdrMEaG82iz9LTpMAZcHzGOmi5wM/Ba1MEoO73oDGf33K8EmxkYUdOJG+NwPIbPpbaibhkT\n3s7UIr22Dmw7TTZk1pqwZhz5V4cae91p55O3TUjcYj//cFNjEnLYwmZZeTyyPhNISJK1NRFvdULk\nXpAgQa4ne6rx0yyn9S0XGdZsuF/ODIxW8sMZAzV17Kn5gJsuxSg1+fVDGmL+pa6DS4ln2mxDlB5r\n4gi0c16wCPIYwC0oWhsNcMRuvwlo4oLG4+MYYt5FQZsaVGIlPs4x5df9HUOtvtFAUzmloxiMn8Ir\nmGc6FzEwki0bGN0YOU8AIgRriVBOazRfrS7123TcWCgj4rVqQD+eUQnGk3AarIVXcDLhWHyILsSC\nGoZzJ2KojFtjFFB4ZxgxYuGVqKaoLWL3rQUD5YQjG5HGXUBkggZReM/ErVyoDMzQrMDKw88zasB7\nJAisrVSVhToZPPYRPHfJCYhgTaQgi5yPKcI3kqDxgwTZhxK++EGC7EO5KdRnrr3fg6bbieh/J6Lf\npVuspLO1PaA/+0+PERHRe9/vqpicPH7af15vsc86U9/lhIkXDfj7MUorZjjaaGDSjDPqtafVmCL0\nyRaSJ1KOVjNgsEE4LtVuEBKnzAhTgJZigIG0ykmGMPraA6KMIw0T2BIIm06WQz887zQDuIk+dE7r\nzLG+kIe4cD/eHiGUr0OKsmxd4sK2J+M2xaBitLQQPTkZ6VZAKdLR4md4XtDGSUpSVpuIqM7wOJvo\n88eIRNlm4DYr5sSXGOIqanW3Zn1gbcoNEJRKhCWkf+fGzSGHVF0rOBoMpxhVqUZbLFteOvCl3BGG\nG+7bEG69wGjHa9Vo6JZKkqWwIpPcesjPwBbqTF1fdkO2+Yy19n5r7f1E9A4i6hPRH1KopBMkyBtW\nbhXq/xARPW+tfYlCJZ0gQd6wcqtW/Y8R0b/l41uupLMzGNDXH3PJBBmHg/7kglpb63UXApmRhsha\nhugZhOTOHNP852bbQa8oU7gZM4zOIfxScvwtwGixQKMFFtlmIgmXRQJDD9fR0uugF0LeYaaW9zbn\n/TcBosWM0VLoO8sZbtpy8kwBwkHVINkWZMAhL373BpTqlriBCOYXRwojYw5VNpH2Q9x3UoCj5e1R\n0qnwT/d75WvAEyB+fAtbGIHCxmj8xQSs2g3e1mAMhYfPGEyQcz8WPCYAmWvMyVAVa5Bk4AkSjwnw\nKGBIs3ozYCsUX1+X5oRry+dhXAnwPsi8ImSWiqWGAoZDc3i6JAXhOtxAdq3xmVr7x4no31372W4r\n6SDBRpAgQV47uRWN/yNE9Ki1Vgql3XolnSSx66wJvvb440REdOKk1i97/wddimbSgMQN/kXM0cAy\nPeePc/bpppn650UX1WLQsDVJisEEFwkSwKQfTIbhemR1NQw1WFtEsHQ+wg185BM0IvIv+QhQQiI+\nWajBN2H2mzyHyEQpeY2/q4XgLDYCQa0+MaIhChBKbjRKRoToRiLpyka5BCIp81SquYBhDLSM0Kcb\nQBNSBagOMRQp+/SjWLV7ysa4MXAFIgOPlScblRGahRiBNa4atDPWZ3kAjMgSi4ERiVJufQIa3TDK\nwpToPIPaehz7gCgyj8oIJBWjHNxPkAMmHGEimowRabqFdh2mSpNcxsjP0hZejuvKrezxf5YU5hOF\nSjpBgrxhZVdffGNMh4g+TESfgeZ/SkQfNsY8R0Qf4n8HCRLkDSC7raSzQ0QL17St0i1W0jEUUU2q\n13Dlj09/4c/952cvud3C3W9SyuUTp08SEdHpU1qsEJMmKHdGNIRbYkSyADdTTtgxANsljzoCCJ72\ntfBlZl0+dqEsMR8jtJTE/wbQdBdYdBjqjSdq8JswXbgpcISKv7dsLkEEFwHWlxiDDEhChZIcdh7U\naDiYHaHvGkKMbSqxnwgV2SBaqGzqOkUCzhQLmkroK8QxSPKJtUC5LesDYxR0jPntzTYYKHneeG85\nQth+4Ih7Z+aRRwDiEzKedwpJOhIGbBDWy1zg3bJA6ipGX0wQkzlgYVMra472R0/DrZIWQr4lsQm2\nB8oQ4dvqNWc4znlbi0bXG0mI3AsSZB/KnibpmCiiqMHlkVP3639lTW2CF//i80RE9J8e+kvftjDv\nDHlnTigKuPNurWV2+oyjbD5+/IhvOzDvGFhabUjcYS61Jhj8LGu0CST99EfgrmG3ogHtPWJ2H9QM\nsXEGywa61CD5Rn6962BYS1lXId21GJEMJL2I97AOhRdiSK0VrYXFIRK+dx2Ti/hvQVuCxttZc89j\nZ0ddkmsrjrr6yrI+o6VV17a2qoUutjeh3uDAMdnEoMtanDS1AFx5bS5VPTWrQPLIUX6GgOjaM8pQ\nIxGZSLWecN/IZ9gQXkdMeQbK8tjrOzDKZVKDDqLiOMkrttpPPsJEK1lV1NvihgNWolgMp5A05I22\nKhFESEqiFaZyizKPa5gAJsZEPifanS4PGj9IkH0o4YsfJMg+lL1n4JEy02wYwci1CRtRooFCx+El\nl5Bz9bLWL3v4m1/3x52Og3WLBzRw8PSJ00REdPtduj246+7vIyKi22476dsWFh2MzLEKT65Qt14X\ng49Cwj6Xf7aYhMFllzHArzmv0YUHeDydaU0kmYxcP6O+Vmnx/nsseT0SEkiEd8A2xEYkTL6RiDQ0\nEo7Hrs/NDa168+Tjz/pjIaY8v6J5VmtcuebyJV37tVVXzWVrfVXHDVuGphilQKXMtBjqT+n2Z3rK\nwfmTJ3Wdxn23zss9qGmXQvwGk0y2GhrtOT/v1vTQQd0yzDCh6+ETSpLZndMtw+JBV71ndkHbEq7v\nh5pQkqbiCv87ThK3HlEF05FY/OI6GN6kahTEGpgcoX452Um+O0WKJv7Dz3x3cXtB4wcJsi9lzzW+\naCFxdyG99IQ1b15wKQlTj8oYXUk9p6F6O4oSXn7ZVWL9xkOKDGa7TkscP6L1237wR36UiIiOAX1y\nlOgvepOLQsRj1ZJ1NhLOTGsUWmvRFUrwvGdElJmKwhQd1TC1rjNatrXGgocMFufHNM0W6JpziF+X\nc2PISxgzShhuqQHu3POu0uq//9yXfduj337eH5spN7b2gg5oxDX/1gba9/rQHW+PoS7fBIyjrHkw\nLn2ZEcGL24qmOmvu+V8c6PPvMcX1NtJep+hKdPesQ67CsSmXJ7Awq+vdmnaGwy7kEFxc0Xt3Zxwn\n3wl4F8686Q4iIrrtjCLCE8fd8dwCsveorhRtjEZSw++uMRgBKm4/oAXnWoUF9xsgtAnnMCAjlKCN\nGvQjKcNmlxF7/rJbOjtIkCB/JSR88YME2Yeyt1DfEBmGaZKkgvTJkiCD7C7+t6nMTC1X8cdgbBOC\nSkhz3NhyEXnjgbL3HPymg7dZrm2jkUJqMWQ9//gTvq3N9c1OHFFI+I4PuC3DPe/7ft82u6CJRN0p\njj6DxA1Jo0So56O0AMrWuEpN1NH+UHzyDkDC7SWXpPL7/+/nfNsffNqlUlxaU/iPy2xzV7Y5BUOm\nMBSlEJ9g2RiVQRJShtsMfoY5+sMlFyrVti0u53xlQw2ZPuEG1wnMVXkFmr267bYFrVWI3Jt3EP/U\nQV2Ti1fVGDm64Nbn289ouenmgw8SEdFsV7djJ485I+B9997h2x541zv88e13nnbXtpFK3R1jxSKB\n6/kYo/7c38Lzh+MG91OYP/eD2+MaH0vSkwn02kGCBLmeGLvLxP3vhTQbLXvy6Gki0mIWY+BX6/ed\nNsK0zJYgAzCqINeB/MDFRl1F4vWAsHRP09yE37pjh1z9snvfdrdvqwMl98aGMxg+9qhWVe3v9Lk/\n7bwz5bTxEUj9fNObtZiDFHY4dae6Fw8fd26s6XnVME2O9Y8hNt7XqsM20JxCEd0DQ97//Wv/nIiI\nPvm7mkw54LpsCablAgGJ1MmbTMopwYhUMh7PhLAYRZmUpOD2stKGNfE4NRazgCXvwIBBCxSYpBsj\nA7p8Xv0Wl+9HpFx5NcwJ4GjHDrjm2nzcgEi5BTDq3nXa5Y+8693v9G3veq87PnZC081rLddPwWjN\nY8jAMIj+YDF+F4x7QskOuSM1w/UWGS2890c/Qo88/thN1X7Q+EGC7EMJX/wgQfah7CnUbzVa9swx\nZyjZYd/4BNhmRkybvQ2Re00GLR2onhIjnbHwohVoqvk82BMkjPtrSE3NUPf204d928KC+n6HzAhz\n4cJV37a27uIGcNVyhpERQLkEDXQM29rgVz5wyPmQF46d0nsfccd333Ovbzt1u4sxOAxJSLNz6leW\nVOC/+PwXfNs//Ad/n4iIVjbUaCmpzOiHxlLXufAUQsJSJglEaMjLpeIMJBLBNiSTlYF+Uiu8gSCS\nVFJYSIH6uJWBS8SOacsGv8JrLG24JcBb28JpBcEtlbwrTUiN7QB3X8eXq9Z1PH3MPdcPfeAHfNtH\nf/y/ICKiM3e8ybfFnqYcKvNAIpHU8kPDqTD+IL26RPvJ1uqv/8SP06NPPB6gfpAgQcoSvvhBguxD\n2ZUf3xjzPxLRf0cOMT1BRL9AREeI6FPkmHkeIaKfsxbK3VRIFEXU4LzpbQmDLeQblxMNMsZlhSIs\nAAUjhlloMRYoV2hjRDUBEDTiHPbnzyuUNzW16nY4kWT+gPrQd9jj0IfClUIEGuVYhQbCL/k+29vq\nS454D3Npc9m3vfS5zxIR0cyUJpxMtZ3Vf25h3rcdPnrcHx9h6/GTT2uswcqai1nA+eeV8RDga+et\nSRPKlsdcxPPI4UXfNhxglRsnhxZ1bLKlAiJp2uYt3OaWJgBJPxMoDb297dZ0AHEDGVBb595joH1H\nEuZc8HeXhliE9RVAuGobMeYtymisY+hNoICoT8jRDpe3nXfl6RfP+bYvf82Fjv/tv/kx3/bBj36I\niIiaTSgTDzn8kWd6qmBjwglKEVfxUOwycvemGt8Yc4yI/gciesBaex+5Dc3HiOifEdGvWWvvIKJ1\nIvrF3d0ySJAgr7XsNnIvIaKWcZkHbSK6TEQfJKK/xZ9/koj+ERH9xo27MZSwQc1W+Httxc+Q932i\n9i6e4HqGPFB/JhqBfPqito3413RpWX3gF7a+44+PH3bJHEcOqAaOWAMfmNe203fdSUREk6FqgwvP\nnfXHvTWn6VEzHDjkNHVvrBove+FlIiLqQ3ThkBltllc1NfbZs0/5Y5nNFiCQVJJCwOApvHGTvJw8\nQkS0MO+Sc+66W+MPItbb81MaaxAzFXm3qwbGdls1VX9rxbV1lCtP7jPcUT5DKYU9Pado4dwLzxER\nUROi565eUTQmsQhDQAS9LWcI3h5qYs+2nDcpR8oREY3YiIaG7SoTd2UbNEpqeaFvjnPYgbp9699y\ncSBnX3rRtz39vHuGv/ALqi9np3XeEi9hwLAoyV5IgS5HEqewW2P9bmrnXSSi/5OIXib3hd8kB+03\nrFagvEBEx6quLxTUwLz3IEGCvGayG6g/R65O3m1EdJSIOkT00d3ewFr7CWvtA9baBxIoHhAkSJDX\nTnYD9T9EROestctERMaYzxDR+4ho1hiTsNY/TkQXb9aRMUoG6CsQI6GJwPYCfXSZ9hnZp62ROnna\nJlDIACQyUaaDYBkzueUOIJF1YH9Z5fzwp1664NvEPf/973m3b/vof+V2POlYof4T3/iGP/6L/+DK\nEeSQuDHNJJPjDY1ZyHmvMwZ/rpHwVTSCQraK+IFTvEbOg4WS8s87BXJPgIxrvPV65pu+barr4g6u\nJpf1PJ5DDRJKOlCiWsqVz86oQbQhiUZQy67RYFpoo0larZbrZ2FOt1GtArGoW1+f9EREIw6hHsD7\nMZi4NV1dXtMxTmufT59127ARbHUurbhzxxkm11QY1kotRfEJYgC5J7zlOLe84tt+41/9GyIi2hno\nO/P3/u4v+eNmnY2kUI+x3nJrhmHXPh5gXGHRvIHsxp33MhG9xxjTNm6T+kNE9BQRfYmIforPCZV0\nggR5A8lNNb619kFjzB8Q0aPkvDTfJFcL74+J6FPGmH/Cbb9189sZiiNxv6lZwn/KTXHxEvcHf6Ig\niSPnY1vxS10I5vJ0ZejvcMedQjKPfp4yYhgh4w3/ffJJNQI+9IUvEhHRbfec8W3Ts0ol3Zl1hrDe\nsrrzBj2nlXYguUaMTiOkYRZeN9DeMcxBbKNYGVjq+uGSiUFsBKgjqTCIInLY6rmxFRJF+HoDGq2Q\nEs3nYoqp3KUOUW9SKGMekpRSNsr1Jjqu/qa6AKUGYXsDIgU51TWKoVZfw82/DjTd3bo+5OMHju28\nhQAABc9JREFU2EUK1N3npx26We2pYVWMd8vryos4nOi7kHuEqmLKANUfp/BGrvbdXP/Vp1Vftlsa\n2fmTP/03iYioUYO09Z57f1oddQE2mMlHxpJV+TIrZLeVdH6ViH71muYXiOhdu7pLkCBBXlcSIveC\nBNmHsseVdIzPTbbeB4qVa9j/DJAoZitgzUA0F6R/C9ML+i8F1iMkrmImEeNXPdHPuvC5lB7G4Cmx\noWxBFNrnPvv7RER06Ktq0BIyRSKirQ0HFSdAovnME48QEdE2EFUO2Ag0BqhvpQoPjKsG+55Ycupx\nrlLyGoxTAvHRzVuAqLzFQYpnWVMs1awdYI4+xuk5SQFyik96CNwLPaYVX1m5UrrfZfDdpxM0dLox\n1qBWndy7Drn1MSfP4LZlYQQsQUzm2WxoP4cWXBLUbFeh/NSMMzY++7xG4S339Lmvbm+XxuijC28S\nQSfLuLKtW4tPffb/88dvecAx/bz9ATUiS209LLfdH7jtj7xbu4X6QeMHCbIPZW8596yllN1KUicO\njVLiC8HirC0+bsFPVAbaZsJawKLG80oQjVccV12w+JXjvAvGRomDhs+nuE9Maa0xUhkOIXqup5Tc\nY47ispHOdcSVc4dAHy13yZDxRow2MMIRpMTW+KpmoVaf5Ddgaiwbogq1GDBWn+cF2sTXesNrxHAK\nmqXo9vJkctq3/MUYe34eaDjMeXCTHNEdaHyeQwruTKk9uIPuziG7wqCfHWC6SZnvb31Ltfvb3v9e\nN4a+auCYqbvrl9VTfboLKdzsSlyHeoOXVl2dwUJUoCxJFQqANXnpqtYo/JM//g9ERPS+96gZbY6L\nxhgwfwsaTfn71KghNry+BI0fJMg+lPDFDxJkH8qeMvAYY5aJaIeIVm527htEDlCYy+tV/irN51bm\ncspau3izk/b0i09EZIx52Fr7wJ7e9FWSMJfXr/xVms+rMZcA9YME2YcSvvhBguxDeS2++J94De75\nakmYy+tX/irN53s+lz3f4wcJEuS1lwD1gwTZh7KnX3xjzEeNMc8YY84aY35lL+/9SsUYc8IY8yVj\n/v/2zuclqigMw89LlkFBZgsZMNBACDdlq6QWIUQk0T76E1pYBJG06h+IWkSbokVERBYlsyjK9vaD\nIiw1iwINzFq1LfhanDMygsIMOvfew/0euMycc2e43+Gd73LvN+e+R58kfZQ0Evs7JT2XNBdfV1/W\ntoBI2iTpnaRqbPdKmoz63JfU2DSwnJHUIWlM0oykaUmDqeoi6Vz8fU1Juidpayt0ySzxFVb6uw4c\nB/qBU5L6szr+BvAPOG9m/cBB4EyM/yIwYWZ9wERsp8IIMF3XTtU5+Rrw1Mz2AvsIY0pOl0wdrc0s\nkw0YBJ7VtUeB0ayO34LxPAGOArNAJfZVgNm8Y2sw/m5CQgwBVcKU+t9A22p6FXUDdgDfiPWquv7k\ndCEY1s4DnYTnaKrAsVbokuWlfm1QNdZ05i06knqAAWAS6DKzmindItCVU1jNchW4AMtP/OyiQefk\ngtEL/AJux9uWm5K2kaAutk5H62bw4l6TSNoOPATOmtmf+n0WTsmF/5tE0glgycze5h3LBtAGHABu\nmNkAYUr4isv6hHRZl6N1M2SZ+D+A3XXthpx5i4SkzYSkv2tmj2L3T0mVuL8CLK31/QJxCDgp6Tth\nGbQhwn1yh7Rsa5uKPgvAgplNxvYY4USQoi7LjtZm9hdY4WgdP7MhumSZ+K+Bvlih3EIoWoxnePx1\nER2GbwHTZnalbtc4wWUYEnEbNrNRM+s2sx6CDi/N7DQJOieb2SIwL6m2BnXNBTo5XcjS0Trj4sUw\n8Bn4ClzKu5jSZOyHCZeLH4D3cRsm3BtPAHPAC6Az71ibHNcRoBrf7wFeAV+AB0B73vE1OIb9wJuo\nzWNgZ6q6AJeBGWAKuAO0t0IXn7nnOCXEi3uOU0I88R2nhHjiO04J8cR3nBLiie84JcQT33FKiCe+\n45QQT3zHKSH/AXFbZZrley6QAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fd719fa2780>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(x_train[1])\n",
    "print('Label: ', y_train[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Balancing\n",
    "The numpy files we just imported were built in a seperate notebook, and they have already been shuffled randomly and balanced across classes, but lets check just to be sure, and to establish a baseline error:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training balance:    0.482924226254\n",
      "Validation balance:  0.480769230769\n",
      "Testing balance:     0.436170212766\n"
     ]
    }
   ],
   "source": [
    "print('Training balance:   ', np.sum(y_train, axis=0)[1] / len(y_train))\n",
    "print('Validation balance: ', np.sum(y_val, axis=0)[1] / len(y_val))\n",
    "print('Testing balance:    ', np.sum(y_test, axis=0)[1] / len(y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Define a model, in this case we'll start with logistic (softmax) regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First a placeholder variable for our inputs to TF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x_input = tf.placeholder(tf.float32, [None, 85, 85, 3], name='x_input')\n",
    "y_input = tf.placeholder(tf.float32, [None, 2], name='y_input')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now variables to hold the weights and biases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "W = tf.get_variable('weights', [85 * 85 * 3, 2], initializer = tf.contrib.layers.xavier_initializer())\n",
    "b = tf.get_variable('biases', [2], initializer = tf.zeros_initializer())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the model itself, super simple: flatten the input, take the dot product of the inputs and weights, and add the bias. A softmax activation is added in the cross entropy calculation.\n",
    "$$ logits = x \\boldsymbol{\\cdot} W + b $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "flat_input = tf.reshape(x_input, [-1, 85 * 85 * 3])\n",
    "logits = tf.matmul(flat_input, W) + b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we need to define a metric to optimize, in this case softmax cross-entropy:\n",
    "The basic formula for cross entropy is the average of the product of truths and log-predictions for all examples. However this calculation is numerically unstable and tensorflow provides an abstracted function to a more stable version with 'softmax_cross_entropy_with_logits':\n",
    "$$xentropy = -\\sum_i{y'_i * \\log{(y_i)}}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# normal cost function\n",
    "cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits = logits, labels = y_input))\n",
    "# l2 norm regularization of the weights\n",
    "regularizer = tf.nn.l2_loss(W)\n",
    "# cost function with regularization\n",
    "beta = 0.1\n",
    "reg_xentropy = tf.reduce_mean(cross_entropy + beta * regularizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pick an optimizer to backprop with"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# The argument to Adam here is the learning rate and it can (and should) be experimented with\n",
    "training_step = tf.train.AdamOptimizer(1e-5).minimize(reg_xentropy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Last, I'll set up an accuracy metric to validate on and add a summary so we can watch it train on tensorboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor 'accuracy:0' shape=() dtype=string>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions = tf.argmax(logits, axis=1)\n",
    "truths = tf.argmax(y_input, axis=1)\n",
    "correct_predictions = tf.equal(predictions, truths)\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_predictions, tf.float32))\n",
    "tf.summary.scalar('accuracy', accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finally lets start a session and begin training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sess = tf.Session()\n",
    "init = tf.global_variables_initializer()\n",
    "sess.run(init)\n",
    "merged = tf.summary.merge_all()\n",
    "train_writer = tf.summary.FileWriter('logs/reg_softmax/train', sess.graph)\n",
    "valid_writer = tf.summary.FileWriter('logs/reg_softmax/valid', sess.graph)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 10 | Time per epoch: 0.634 | Train Accuracy: 0.486393 | Validation Accuracy: 0.482906\n",
      "Epoch 20 | Time per epoch: 0.6 | Train Accuracy: 0.487727 | Validation Accuracy: 0.480769\n",
      "Epoch 30 | Time per epoch: 0.589 | Train Accuracy: 0.494397 | Validation Accuracy: 0.485043\n",
      "Epoch 40 | Time per epoch: 0.583 | Train Accuracy: 0.492529 | Validation Accuracy: 0.465812\n",
      "Epoch 50 | Time per epoch: 0.579 | Train Accuracy: 0.497065 | Validation Accuracy: 0.478633\n",
      "\n",
      "... Edited for length in this gist - See tensorboard below for full training data...\n",
      "\n",
      "Epoch 4950 | Time per epoch: 0.565 | Train Accuracy: 0.747332 | Validation Accuracy: 0.536325\n",
      "Epoch 4960 | Time per epoch: 0.565 | Train Accuracy: 0.747866 | Validation Accuracy: 0.536325\n",
      "Epoch 4970 | Time per epoch: 0.565 | Train Accuracy: 0.747866 | Validation Accuracy: 0.534188\n",
      "Epoch 4980 | Time per epoch: 0.565 | Train Accuracy: 0.748132 | Validation Accuracy: 0.534188\n",
      "Epoch 4990 | Time per epoch: 0.565 | Train Accuracy: 0.748399 | Validation Accuracy: 0.534188\n",
      "Time Taken:  2822.636345386505\n"
     ]
    }
   ],
   "source": [
    "num_epochs = 5000\n",
    "\n",
    "start_time = time.time()\n",
    "for epoch in range(num_epochs):\n",
    "    if epoch % 10 == 0:\n",
    "        # every 10th epoch write out accuracy on training and validation\n",
    "        summary_train, train_acc = sess.run([merged, accuracy], {x_input: x_train, y_input: y_train})\n",
    "        summary_valid, valid_acc = sess.run([merged, accuracy], {x_input: x_val, y_input: y_val})\n",
    "        train_writer.add_summary(summary_train, epoch)\n",
    "        valid_writer.add_summary(summary_valid, epoch)\n",
    "        if epoch != 0: \n",
    "            time_taken = round((time.time() - start_time) / epoch, 3)\n",
    "            print('Epoch %s | Time per epoch: %s | Train Accuracy: %s | Validation Accuracy: %s' % (epoch, time_taken, train_acc, valid_acc))\n",
    "    sess.run([training_step], {x_input: x_train, y_input: y_train})\n",
    "\n",
    "train_writer.close()\n",
    "print('Time Taken: ', time.time() - start_time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Not too good... :(\n",
    "[You can view the tensorboard summary here.](https://raw.githubusercontent.com/McCulloughRT/machine-learning/master/lr_training.png) Purple is the validation accuracy and blue training.\n",
    "With more epochs it looks like accuracy might continue to improve a bit more, but after experiementing with length of training and learing rate I did not find significant improvement on this.\n",
    "#### Lets check the models accuracy against the training set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.57021272"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sess.run(accuracy, {x_input: x_test, y_input: y_test})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sess.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Somewhat expectedly, logistic regression isn't great with image classification. In the next notebook we'll try a deep neural network."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}