Stock Market Price Prediction TensorFlow

price.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stock Market Price Regression\n",
    "\n",
    "This is a quick notebook exploring the [Kaggle dataset for the New York Stock Exchange](https://www.kaggle.com/dgawlik/nyse/data), and predicting the future prices. This was made to review the effectiveness of predicting prices of financial markets using neural networks [for an article that I wrote](). The effectiveness of these techniques are reviewed and other avenues are indroduced and explored in the aforementioned article.\n",
    "\n",
    "### Imports\n",
    "\n",
    "Note the version of TensorFlow (and the other libs). In particular, the folks at Google change their API in breaking manners frequently. If you, the reader in the future, have a different version and your TensorFlow code doesn't work, studying the API changes may help!\n",
    "\n",
    "Also note the seeding. You can seed the PRNGs in TensorFlow and Numpy etc to produce predictable and deterministic results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pandas version:       0.20.3\n",
      "NumPy version:        1.13.3\n",
      "SciKit Learn version: 0.18.1\n",
      "TensorFlow version:   1.4.0-rc1\n",
      "MatPlotLib version:   2.0.2\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import sklearn\n",
    "from sklearn.model_selection import KFold\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "from dateutil.parser import parse\n",
    "from datetime import datetime, timedelta\n",
    "from collections import deque\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "print(\"Pandas version:      \", pd.__version__)\n",
    "print(\"NumPy version:       \", np.__version__)\n",
    "print(\"SciKit Learn version:\", sklearn.__version__)\n",
    "print(\"TensorFlow version:  \", tf.__version__)\n",
    "print(\"MatPlotLib version:  \", matplotlib.__version__)\n",
    "\n",
    "seed = 8\n",
    "tf.set_random_seed(seed)\n",
    "np.random.seed(seed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exploring The Data\n",
    "\n",
    "We wil be exploring __prices.csv__, which is raw, as-is daily, prices. Most of data spans from 2010 to the end 2016, for companies new on stock market date range is shorter. There have been approx. 140 stock splits in that time, this set doesn't account for that.\n",
    "\n",
    "We can load the dataset into a pandas dataframe very easily. Below we can print out basic stats on the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>open</th>\n",
       "      <th>close</th>\n",
       "      <th>low</th>\n",
       "      <th>high</th>\n",
       "      <th>volume</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>851264.000000</td>\n",
       "      <td>851264.000000</td>\n",
       "      <td>851264.000000</td>\n",
       "      <td>851264.000000</td>\n",
       "      <td>8.512640e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>70.836986</td>\n",
       "      <td>70.857109</td>\n",
       "      <td>70.118414</td>\n",
       "      <td>71.543476</td>\n",
       "      <td>5.415113e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>83.695876</td>\n",
       "      <td>83.689686</td>\n",
       "      <td>82.877294</td>\n",
       "      <td>84.465504</td>\n",
       "      <td>1.249468e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.850000</td>\n",
       "      <td>0.860000</td>\n",
       "      <td>0.830000</td>\n",
       "      <td>0.880000</td>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>33.840000</td>\n",
       "      <td>33.849998</td>\n",
       "      <td>33.480000</td>\n",
       "      <td>34.189999</td>\n",
       "      <td>1.221500e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>52.770000</td>\n",
       "      <td>52.799999</td>\n",
       "      <td>52.230000</td>\n",
       "      <td>53.310001</td>\n",
       "      <td>2.476250e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>79.879997</td>\n",
       "      <td>79.889999</td>\n",
       "      <td>79.110001</td>\n",
       "      <td>80.610001</td>\n",
       "      <td>5.222500e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1584.439941</td>\n",
       "      <td>1578.130005</td>\n",
       "      <td>1549.939941</td>\n",
       "      <td>1600.930054</td>\n",
       "      <td>8.596434e+08</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                open          close            low           high  \\\n",
       "count  851264.000000  851264.000000  851264.000000  851264.000000   \n",
       "mean       70.836986      70.857109      70.118414      71.543476   \n",
       "std        83.695876      83.689686      82.877294      84.465504   \n",
       "min         0.850000       0.860000       0.830000       0.880000   \n",
       "25%        33.840000      33.849998      33.480000      34.189999   \n",
       "50%        52.770000      52.799999      52.230000      53.310001   \n",
       "75%        79.879997      79.889999      79.110001      80.610001   \n",
       "max      1584.439941    1578.130005    1549.939941    1600.930054   \n",
       "\n",
       "             volume  \n",
       "count  8.512640e+05  \n",
       "mean   5.415113e+06  \n",
       "std    1.249468e+07  \n",
       "min    0.000000e+00  \n",
       "25%    1.221500e+06  \n",
       "50%    2.476250e+06  \n",
       "75%    5.222500e+06  \n",
       "max    8.596434e+08  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataframe = pd.read_csv('prices.csv')\n",
    "dataframe.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also print out the data, in this case the tail-end of the csv file, or in other words, the last 10 rows!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>symbol</th>\n",
       "      <th>open</th>\n",
       "      <th>close</th>\n",
       "      <th>low</th>\n",
       "      <th>high</th>\n",
       "      <th>volume</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>851254</th>\n",
       "      <td>2016-12-30</td>\n",
       "      <td>XRAY</td>\n",
       "      <td>58.290001</td>\n",
       "      <td>57.730000</td>\n",
       "      <td>57.540001</td>\n",
       "      <td>58.360001</td>\n",
       "      <td>949200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>851255</th>\n",
       "      <td>2016-12-30</td>\n",
       "      <td>XRX</td>\n",
       "      <td>8.720000</td>\n",
       "      <td>8.730000</td>\n",
       "      <td>8.700000</td>\n",
       "      <td>8.800000</td>\n",
       "      <td>11250400.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>851256</th>\n",
       "      <td>2016-12-30</td>\n",
       "      <td>XYL</td>\n",
       "      <td>49.980000</td>\n",
       "      <td>49.520000</td>\n",
       "      <td>49.360001</td>\n",
       "      <td>50.000000</td>\n",
       "      <td>646200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>851257</th>\n",
       "      <td>2016-12-30</td>\n",
       "      <td>YHOO</td>\n",
       "      <td>38.720001</td>\n",
       "      <td>38.669998</td>\n",
       "      <td>38.430000</td>\n",
       "      <td>39.000000</td>\n",
       "      <td>6431600.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>851258</th>\n",
       "      <td>2016-12-30</td>\n",
       "      <td>YUM</td>\n",
       "      <td>63.930000</td>\n",
       "      <td>63.330002</td>\n",
       "      <td>63.160000</td>\n",
       "      <td>63.939999</td>\n",
       "      <td>1887100.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>851259</th>\n",
       "      <td>2016-12-30</td>\n",
       "      <td>ZBH</td>\n",
       "      <td>103.309998</td>\n",
       "      <td>103.199997</td>\n",
       "      <td>102.849998</td>\n",
       "      <td>103.930000</td>\n",
       "      <td>973800.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>851260</th>\n",
       "      <td>2016-12-30</td>\n",
       "      <td>ZION</td>\n",
       "      <td>43.070000</td>\n",
       "      <td>43.040001</td>\n",
       "      <td>42.689999</td>\n",
       "      <td>43.310001</td>\n",
       "      <td>1938100.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>851261</th>\n",
       "      <td>2016-12-30</td>\n",
       "      <td>ZTS</td>\n",
       "      <td>53.639999</td>\n",
       "      <td>53.529999</td>\n",
       "      <td>53.270000</td>\n",
       "      <td>53.740002</td>\n",
       "      <td>1701200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>851262</th>\n",
       "      <td>2016-12-30 00:00:00</td>\n",
       "      <td>AIV</td>\n",
       "      <td>44.730000</td>\n",
       "      <td>45.450001</td>\n",
       "      <td>44.410000</td>\n",
       "      <td>45.590000</td>\n",
       "      <td>1380900.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>851263</th>\n",
       "      <td>2016-12-30 00:00:00</td>\n",
       "      <td>FTV</td>\n",
       "      <td>54.200001</td>\n",
       "      <td>53.630001</td>\n",
       "      <td>53.389999</td>\n",
       "      <td>54.480000</td>\n",
       "      <td>705100.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       date symbol        open       close         low  \\\n",
       "851254           2016-12-30   XRAY   58.290001   57.730000   57.540001   \n",
       "851255           2016-12-30    XRX    8.720000    8.730000    8.700000   \n",
       "851256           2016-12-30    XYL   49.980000   49.520000   49.360001   \n",
       "851257           2016-12-30   YHOO   38.720001   38.669998   38.430000   \n",
       "851258           2016-12-30    YUM   63.930000   63.330002   63.160000   \n",
       "851259           2016-12-30    ZBH  103.309998  103.199997  102.849998   \n",
       "851260           2016-12-30   ZION   43.070000   43.040001   42.689999   \n",
       "851261           2016-12-30    ZTS   53.639999   53.529999   53.270000   \n",
       "851262  2016-12-30 00:00:00    AIV   44.730000   45.450001   44.410000   \n",
       "851263  2016-12-30 00:00:00    FTV   54.200001   53.630001   53.389999   \n",
       "\n",
       "              high      volume  \n",
       "851254   58.360001    949200.0  \n",
       "851255    8.800000  11250400.0  \n",
       "851256   50.000000    646200.0  \n",
       "851257   39.000000   6431600.0  \n",
       "851258   63.939999   1887100.0  \n",
       "851259  103.930000    973800.0  \n",
       "851260   43.310001   1938100.0  \n",
       "851261   53.740002   1701200.0  \n",
       "851262   45.590000   1380900.0  \n",
       "851263   54.480000    705100.0  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataframe.tail(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting The Data We Need\n",
    "\n",
    "We don't care about much of the data, as we are just predicting the successive price for each day. We can access multiple columns by passing a list of column names to the dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>open</th>\n",
       "      <th>close</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>123.430000</td>\n",
       "      <td>125.839996</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>125.239998</td>\n",
       "      <td>119.980003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>116.379997</td>\n",
       "      <td>114.949997</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>115.480003</td>\n",
       "      <td>116.620003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>117.010002</td>\n",
       "      <td>114.970001</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         open       close\n",
       "0  123.430000  125.839996\n",
       "1  125.239998  119.980003\n",
       "2  116.379997  114.949997\n",
       "3  115.480003  116.620003\n",
       "4  117.010002  114.970001"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "desired_columns = ['open', 'close']\n",
    "basic_mlp_data = dataframe[desired_columns]\n",
    "basic_mlp_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building A Simple Neural Network\n",
    "\n",
    "The most basic way of tackling this problem with neural networks would be to input a given days open value and predict the close. We can define the simple MLP like so in TensorFlow, and I have commented it as much as possible to make it as transparent as possible:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Call this in IPython notebooks before any elements are added to\n",
    "# the default graph otherwise if you rerun cells you can get \n",
    "# annoying errors.\n",
    "tf.reset_default_graph()\n",
    "\n",
    "# Define the Neural Network topology with the 'net_hidden_sizes'\n",
    "# and how much we should regularise it and how quickly it should\n",
    "# learn. Also the type of non linearity we should use.\n",
    "amount_epochs = 50\n",
    "learning_rate = 0.001\n",
    "batch_size = 128\n",
    "net_hidden_sizes = [128, 64, 8]\n",
    "l2_strength = 0.01\n",
    "non_linearity = tf.nn.relu\n",
    "dropout_amount = 0.7\n",
    "\n",
    "# The input to the graph - the targets (close) and the inputs\n",
    "# (open). Also a placeholder to pass a variable dropout rate. \n",
    "net_input = tf.placeholder(tf.float32, shape=[None, 1])\n",
    "net_target = tf.placeholder(tf.float32, shape=[None, 1])\n",
    "dropout_prob = tf.placeholder(tf.float32)\n",
    "\n",
    "# L2 regularisation to penalise the weights from growing too\n",
    "# large. Useful to prevent overfitting.\n",
    "regulariser = tf.contrib.layers.l2_regularizer(scale=l2_strength)\n",
    "\n",
    "# Build the network from the list of dimensions. Apply l2 and\n",
    "# dropout regularisation to the layers.\n",
    "net = net_input\n",
    "for size in net_hidden_sizes:\n",
    "    net = tf.layers.dense(inputs=net, \n",
    "                          units=size, \n",
    "                          activation=non_linearity, \n",
    "                          kernel_regularizer=regulariser)\n",
    "    net = tf.layers.dropout(inputs=net,\n",
    "                            rate=dropout_prob)\n",
    "\n",
    "# The models prediction has a linear output. \n",
    "net_output = tf.layers.dense(inputs=net,\n",
    "                             units=1, \n",
    "                             activation=None, \n",
    "                             kernel_regularizer=regulariser)    \n",
    "\n",
    "# The main loss for penalising the network on how well it does.\n",
    "loss = tf.losses.mean_squared_error(labels=net_target, \n",
    "                                    predictions=net_output)\n",
    "\n",
    "# TensorFlows manner of applying l2 to the loss.\n",
    "l2_variables = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)\n",
    "l2_loss = tf.contrib.layers.apply_regularization(regulariser, \n",
    "                                                 l2_variables)\n",
    "total_loss = loss + l2_loss\n",
    "\n",
    "# Train and initialisation TensorFlow operations to be ran\n",
    "# in the session.\n",
    "train_op = tf.train.AdamOptimizer(learning_rate).minimize(total_loss)\n",
    "init_op = tf.global_variables_initializer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training The Network\n",
    "\n",
    "So here we use k-fold cross validation to train the neural network on multiple 'versions' (or folds) of the dataset to see how well the model can generalise to new data. At the end of each fold, we print the mean of the test errors, plus the standard deviation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Fold iteration:   0 \n",
      "Error mean:       2.33488 \n",
      "Error deviation:  2.63662 \n",
      "\n",
      "\n",
      "Fold iteration:   1 \n",
      "Error mean:       2.20945 \n",
      "Error deviation:  2.25911 \n",
      "\n",
      "\n",
      "Fold iteration:   2 \n",
      "Error mean:       2.31514 \n",
      "Error deviation:  2.63553 \n",
      "\n",
      "\n",
      "Fold iteration:   3 \n",
      "Error mean:       2.06673 \n",
      "Error deviation:  2.33269 \n",
      "\n",
      "\n",
      "Fold iteration:   4 \n",
      "Error mean:       6.05221 \n",
      "Error deviation:  3.73759 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    \n",
    "    amount_folds = 5\n",
    "    k_folds = KFold(n_splits=amount_folds)\n",
    "    data = basic_mlp_data.as_matrix()\n",
    "    fold_errors = []\n",
    "    fold_iteration = 0\n",
    "    \n",
    "    # Cross validate the dataset, using K-Fold.\n",
    "    for train_indices, test_indices in k_folds.split(data):\n",
    "\n",
    "        # Each new fold, reinitialise the network.\n",
    "        sess.run(init_op)\n",
    "        \n",
    "        # Training phase.\n",
    "        for epoch in range(amount_epochs):\n",
    "            \n",
    "            # Each new epoch, reshuffle the train set.\n",
    "            random_train_indices = np.random.permutation(train_indices)\n",
    "            train_set = data[random_train_indices]\n",
    "            \n",
    "            # Loop over the train set and optimise the network.\n",
    "            for begin in range(0, len(train_set), batch_size):\n",
    "                end = begin + batch_size\n",
    "                batch_x = train_set[begin:end].T[0].reshape((-1, 1))\n",
    "                batch_y = train_set[begin:end].T[1].reshape((-1, 1))\n",
    "                \n",
    "                sess.run(train_op, feed_dict={\n",
    "                    net_input: batch_x,\n",
    "                    net_target: batch_y,\n",
    "                    dropout_prob: dropout_amount\n",
    "                })\n",
    "        \n",
    "        # Testing phase.\n",
    "        test_set = data[test_indices]\n",
    "        \n",
    "        # Collate the error over the test set.\n",
    "        all_error = []\n",
    "        for begin in range(0, len(test_set), batch_size):\n",
    "            end = begin + batch_size \n",
    "            batch_x = train_set[begin:end].T[0].reshape((-1, 1))\n",
    "            batch_y = train_set[begin:end].T[1].reshape((-1, 1))\n",
    "            \n",
    "            error = sess.run(loss, feed_dict={\n",
    "                net_input: batch_x,\n",
    "                net_target: batch_y,\n",
    "                dropout_prob: 1.0\n",
    "            }) \n",
    "            all_error.append(error)\n",
    "        \n",
    "        all_error = np.array(all_error).reshape((-1))\n",
    "        fold_errors.append(all_error)\n",
    "        \n",
    "        print(\"\\nFold iteration:  \", fold_iteration,\n",
    "              \"\\nError mean:      \", np.mean(all_error),\n",
    "              \"\\nError deviation: \", np.std(all_error),\n",
    "              \"\\n\")\n",
    "        fold_iteration += 1      \n",
    "        \n",
    "    fold_errors = np.array(fold_errors).reshape((amount_folds, -1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating The Simple MLP Approach\n",
    "\n",
    "Reviewing the performance of the model on the different folds leaves a lot to be desired. Even with the tweaking of the available parameters there is little predictive information in the in the open data for the target close data. Looking at the histograms graphed below, we can see there is a great amount of variance and uncertainty in the predictions, as well as some incredbily bad predictions! With this kind of data, roughly the best the model can do is return roughly the same number passed in and hope for the best.\n",
    "\n",
    "Of course at this stage with this particular dataset, it could be worth trying more simple polynomial or linear models to see if they return better results. However next we will try a recurrent neural network to try to fit the data a bit better and get information from previous timesteps incorporated into the predictions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fae62dd2048>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlkAAAJOCAYAAACEKxJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3X+0ZWV95/n3J4CQARSx7NvVQCzSkptlslqiNUiWTvoC\nTRpJJtBptalxKdpkatY0dMyYXordmbbsjj2Y6cQfMW13RRiLJAZpowsWTSehgbuMPS1KRUQQCSWB\ngVoFFRSQix0j5jt/nH3lcL1V99x7z3N+3Pt+rXXW2fvZz9nPd19OPXzPs5+9d6oKSZIkDdcPjDsA\nSZKkjcgkS5IkqQGTLEmSpAZMsiRJkhowyZIkSWrAJEuSJKkBkywNLMlskjuSPJXkF1eo+5Yknz3M\n9vkkvzD8KCVpfezrNCwmWVqNdwC3VtXxVfWhlg0l+T+SPJLkm0muSnJ0y/Ykqc9I+rokP57kj5I8\nlsSbVm5AJllajZcAd7duJMnfBy4Hzuna/GHgPa3blaTOSPo64DvAtcAlI2hLY2CSpYEkuQU4C/hw\nkoUkP5LkBUmuTvIXSR5M8itJlv1OJTk3yVeTPJnkw0AO09zFwJVVdXdVPQ78a+Atwz4mSVpqlH1d\nVd1bVVcymoROY2CSpYFU1dnAnwCXVdVxVfVnwG8CL6A30vR3gTcDb1362SRbgE8BvwJsAb4GvPow\nzf0Y8KW+9S8BM0leNIRDkaRDGnFfpw3OJEtrkuQI4CLgXVX1VFU9APw68KZlqp8P3F1Vn6yq7wAf\nAB45zO6PA57sW19cPn7dgUvSKjTu67TBmWRprbYARwEP9pU9CJy0TN2/BTy0uFK9p5I/tEy9RQvA\n8/vWF5efWlOkkrR2Lfs6bXAmWVqrx+hN2nxJX9kPAfuXqXsAOGVxJUn615dxN/DyvvWXA49W1dfX\nHK0krU3Lvk4bnEmW1qSqvkvvqpj3Jjk+yUuAtwO/u0z1/wT8WJKfT3Ik8IvA3zzM7q8GLknysiQn\n0Jvf8LGhHoAkDaBlX5eeY4DndevHeLuajcUkS+vxT4GngfuBzwIfB65aWqmqHgNeD1wBfB04Dfiv\nh9ppVf0h8GvArcD/R29o/t1Djl2SBtWkr6M3Ovbfefbqwv8O3Du0qDV26Z0yliRJ0jA5kiVJktSA\nSZYkSVIDJlmSJEkNmGRJkiQ1cOS4AwDYsmVLbdu2bcV6Tz/9NMcee2z7gIZoGmMG4x61jRL33r17\nH6uqF48xpIm3Un83rd+F1djox7jRjw88xoH7uqoa++uVr3xlDeLWW28dqN4kmcaYq4x71DZK3MDt\nNQF9ynpewAPAl4E7Fo8HOBG4Cbive39hVx7gQ8A+4E7gFSvtf6X+blq/C6ux0Y9xox9flcc4aF/n\n6UJJ+n5nVdXpVbW9W78cuLmqTgNu7tYBXkvvXkinATuBj4w8UkkTyyRLklZ2AbCnW94DXNhXfnX3\n4/ZzwAlJto4jQEmTZyLmZEnSBCngj5MU8B+qajcwU1UHuu2PADPd8kk89wHAD3dlB/rKSLKT3kgX\nMzMzzM/PH7LxhYWFw27fCDb6MW704wOPcVAmWZL0XK+pqv1J/gZwU5Kv9m+squoSsIF1idpugO3b\nt9fc3Nwh687Pz3O47RvBRj/GjX584DEOytOFktSnqvZ37weBTwNnAI8ungbs3g921fcDp/R9/OSu\nTJKmbyRr165dA5VJ0molORb4gap6qlv+aeBfAdcDF9N78O/FwHXdR64HLktyDfAq4Mm+04rrtrRv\ns6+TpsvUJVmS1NAM8Okk0OsfP15Vf5jkC8C1SS4BHgTe0NW/ETif3i0cvgW8dfQhS5pUJlmS1Kmq\n+4GXL1P+deCcZcoLuHQEoUmaQs7JkiRJasAkS5IkqQGTLEmSpAY2xJwsr8CRJEmTxpEsSZKkBkyy\nJEmSGjDJkiRJasAkS5IkqQGTLEmSpAZMsiRJkhowyZIkSWrAJEuSJKkBkyxJkqQGTLIkSZIaMMmS\nJElqwCRLkiSpgYGTrCRHJPlikhu69VOT3JZkX5JPJHleV350t76v276tTeiSJEmTazUjWW8D7ulb\nfx/w/qp6KfA4cElXfgnweFf+/q6eJEnSpjJQkpXkZOBngI926wHOBj7ZVdkDXNgtX9Ct020/p6sv\nSZK0aRw5YL0PAO8Aju/WXwQ8UVXPdOsPAyd1yycBDwFU1TNJnuzqP9a/wyQ7gZ0AMzMzzM/PrxjE\nwsICs7OzK9YbZF+jsrCwMFHxDMq4R8u4NYhdu3Yddl3SZFkxyUrys8DBqtqbZG5YDVfVbmA3wPbt\n22tubuVdz8/Pc/vtt69Yb8eOHesNb2jm5+cZ5NgmjXGPlnFPjiRHALcD+6vqZ5OcClxD78fiXuBN\nVfVXSY4GrgZeCXwd+EdV9cCYwpY0gQY5Xfhq4OeSPECvozkb+CBwQpLFJO1kYH+3vB84BaDb/gJ6\nHZAkTQPnn0oaihWTrKp6V1WdXFXbgIuAW6rqjcCtwOu6ahcD13XL13frdNtvqaoaatSS1IDzTyUN\n06BzspbzTuCaJL8KfBG4siu/EvidJPuAb9BLzCRpGgx9/imsbg5q/zy3leagTut8uI0+l2+jHx94\njINaVZJVVfPAfLd8P3DGMnX+Enj9uqKSpBFrNf8UVjcHtX+e20oT2ydp/ulqbMS5fP02+vGBxzio\n9YxkSdJGsjj/9HzgGOD59M0/7Uazlpt/+rDzTyUtx8fqSBLOP5U0fCZZknR47wTe3s0zfRHPnX/6\noq787cDlY4pP0oTydKEkLeH8U0nD4EiWJElSAyZZkiRJDZhkSZIkNWCSJUmS1IBJliRJUgMmWZIk\nSQ2YZEmSJDVgkiVJktSASZYkSVIDJlmSJEkNmGRJkiQ1YJIlSZLUgEmWJElSAyZZkiRJDZhkSZIk\nNWCSJUmS1IBJliRJUgMrJllJjkny+SRfSnJ3kvd05acmuS3JviSfSPK8rvzobn1ft31b20OQJEma\nPIOMZH0bOLuqXg6cDpyX5EzgfcD7q+qlwOPAJV39S4DHu/L3d/UkSZI2lRWTrOpZ6FaP6l4FnA18\nsivfA1zYLV/QrdNtPydJhhaxJEnSFDhykEpJjgD2Ai8Ffgv4GvBEVT3TVXkYOKlbPgl4CKCqnkny\nJPAi4LEl+9wJ7ASYmZlhfn5+xTgWFhaYnZ1dsd4g+xqVhYWFiYpnUMY9WsYtSRvPQElWVX0XOD3J\nCcCngR9db8NVtRvYDbB9+/aam5tb8TPz8/PcfvvtK9bbsWPHesMbmvn5eQY5tklj3KNl3JMhyTHA\nZ4Cj6fWPn6yqdyc5FbiG3g/GvcCbquqvkhwNXA28Evg68I+q6oGxBC9p4qzq6sKqegK4FfhJ4IQk\ni0naycD+bnk/cApAt/0F9DofSZp0zkGVNDSDXF344m4EiyQ/CJwL3EMv2XpdV+1i4Lpu+fpunW77\nLVVVwwxaklpwDqqkYRrkdOFWYE83L+sHgGur6oYkXwGuSfKrwBeBK7v6VwK/k2Qf8A3gogZxS1IT\n456D2j/PbaU5qNM6H26jz+Xb6McHHuOgVkyyqupO4CeWKb8fOGOZ8r8EXr+uqCRpTMY9B7V/ntuu\nXbsOu99Jmn+6GhttLt9SG/34wGMclHd8l6RlOAdV0nqZZElSxzmokoZpoFs4SNIm4RxUSUNjkiVJ\nHeegShomTxdKkiQ1YJIlSZLUgEmWJElSAyZZkiRJDZhkSZIkNWCSJUmS1IBJliRJUgMmWZIkSQ2Y\nZEmSJDVgkiVJktSASZYkSVIDJlmSJEkNmGRJkiQ1YJIlSZLUgEmWJElSAyZZkiRJDZhkSZIkNWCS\nJUmS1MCKSVaSU5LcmuQrSe5O8rau/MQkNyW5r3t/YVeeJB9Ksi/JnUle0fogJEmSJs0gI1nPAL9c\nVS8DzgQuTfIy4HLg5qo6Dbi5Wwd4LXBa99oJfGToUUuSJE24I1eqUFUHgAPd8lNJ7gFOAi4A5rpq\ne4B54J1d+dVVVcDnkpyQZGu3n5HYtWvXYdclaakkpwBXAzNAAbur6oNJTgQ+AWwDHgDeUFWPJwnw\nQeB84FvAW6rqT8cRu6TJtGKS1S/JNuAngNuAmb7E6RF6HRP0ErCH+j72cFf2nCQryU56I13MzMww\nPz+/YvsLCwvMzs6uJmSAgfbdysLCwljbXyvjHi3jngiLo/Z/muR4YG+Sm4C30Bu1vyLJ5fRG7d/J\nc0ftX0Vv1P5VY4lc0kQaOMlKchzwB8AvVdU3ez/ieqqqktRqGq6q3cBugO3bt9fc3NyKn5mfn+f2\n229fTTMA7NixY9WfGZb5+XkGObZJY9yjZdzjN42j9pIm20BJVpKj6CVYv1dVn+qKH13sUJJsBQ52\n5fuBU/o+fnJXJklTYZij9t3+Bh657x8dXGnkflpHETfYCOj32ejHBx7joFZMsrp5B1cC91TVb/Rt\nuh64GLiie7+ur/yyJNfQGzp/0l92kqbFsEftu88NPHLfPzq40nzScY7Sr8dGGgFdzkY/PvAYBzXI\nSNargTcBX05yR1f2z+klV9cmuQR4EHhDt+1GehNB99GbDPrWdUUoSSPiqL2kYRrk6sLPAjnE5nOW\nqV/ApeuMS5JGylF7ScO2qqsLJWkDc9Re0lCZZEkSjtpLGj6fXShJktSASZYkSVIDJlmSJEkNOCdL\nkqbUcvfR8lmt0uRwJEuSJKkBkyxJkqQGTLIkSZIaMMmSJElqwCRLkiSpAZMsSZKkBkyyJEmSGjDJ\nkiRJasAkS5IkqQGTLEmSpAZMsiRJkhowyZIkSWrAJEuSJKkBkyxJkqQGTLIkSZIaMMmSJElqYMUk\nK8lVSQ4muauv7MQkNyW5r3t/YVeeJB9Ksi/JnUle0TJ4SZKkSTXISNbHgPOWlF0O3FxVpwE3d+sA\nrwVO6147gY8MJ0xJas8flZKGacUkq6o+A3xjSfEFwJ5ueQ9wYV/51dXzOeCEJFuHFawkNfYx/FEp\naUiOXOPnZqrqQLf8CDDTLZ8EPNRX7+Gu7ABLJNlJr2NiZmaG+fn5FRtdWFhgdnZ21cEOsu9WFhYW\nxtr+Whn3aBn3ZKiqzyTZtqT4AmCuW94DzAPvpO9HJfC5JCck2drXN0ra5NaaZH1PVVWSWsPndgO7\nAbZv315zc3MrfmZ+fp7bb7991THu2LFj1Z8Zlvn5eQY5tklj3KNl3BNtpD8q+xPXaftROaiNlpwv\ntdGPDzzGQa01yXp08RdbdzrwYFe+Hzilr97JXZkkTb1R/KjsT1x37dq16hjH+aNyUBs9Od/oxwce\n46DWmmRdD1wMXNG9X9dXflmSa4BXAU86dC5pyk3Vj8qlidlaEjVJwzHILRx+H/hvwGySh5NcQi+5\nOjfJfcDf69YBbgTuB/YBvw38kyZRS9LoLP6ohO//Ufnm7irDM/FHpaQlVhzJqqpDjT2fs0zdAi5d\nb1DDttwvOX/dSVqq+1E5B2xJ8jDwbno/Iq/tfmA+CLyhq34jcD69H5XfAt468oAlTbR1T3yXpI1i\nI/yolDQ5fKyOJElSAyZZkiRJDZhkSZIkNWCSJUmS1IBJliRJUgMmWZIkSQ2YZEmSJDVgkiVJktSA\nSZYkSVID3vFdkjYwHysmjc+mTbJ8Ur0kSWrJ04WSJEkNbNqRLEnarBzJl0bDkSxJkqQGTLIkSZIa\nMMmSJElqwDlZHS9zliRJw+RIliRJUgOOZB2GV+BI2gzs66Q2TLJWwVOKkjYrEzFp9ZolWUnOAz4I\nHAF8tKquaNWWJI3LRuzrTKCk4WiSZCU5Avgt4FzgYeALSa6vqq+0aG+SHThwwF+A0gZlX7d6u3bt\nYnZ29nv9oP2hNrJWI1lnAPuq6n6AJNcAFwAbruNZSwJl0iVtGJumr1tqWP3WWvrMQerYr2oSpKqG\nv9PkdcB5VfUL3fqbgFdV1WV9dXYCO7vVWeDeAXa9BXhsyOG2No0xg3GP2kaJ+yVV9eJxBTNqg/R1\nXflq+rtp/S6sxkY/xo1+fOAxDtTXjW3ie1XtBnav5jNJbq+q7Y1CamIaYwbjHjXj3thW099thr/p\nRj/GjX584DEOqtV9svYDp/Stn9yVSdJGYl8n6ZBaJVlfAE5LcmqS5wEXAdc3akuSxsW+TtIhNTld\nWFXPJLkM+CN6lzVfVVV3D2HXqzq9OCGmMWYw7lEz7inUqK/bDH/TjX6MG/34wGMcSJOJ75IkSZud\nzy6UJElqwCRLkiSpgalIspKcl+TeJPuSXD7ueA4lyVVJDia5q6/sxCQ3Jbmve3/hOGNcTpJTktya\n5CtJ7k7ytq58omNPckySzyf5Uhf3e7ryU5Pc1n1fPtFNSJ4oSY5I8sUkN3Tr0xDzA0m+nOSOJLd3\nZRP9HZkm09LPrca09omrMa3956CmuZ9drRb98sQnWX2PrXgt8DJgR5KXjTeqQ/oYcN6SssuBm6vq\nNODmbn3SPAP8clW9DDgTuLT7G0967N8Gzq6qlwOnA+clORN4H/D+qnop8DhwyRhjPJS3Aff0rU9D\nzABnVdXpffeOmfTvyFSYsn5uNT7GdPaJqzGt/eegprmfXa2h98sTn2TR99iKqvorYPGxFROnqj4D\nfGNJ8QXAnm55D3DhSIMaQFUdqKo/7ZafovclO4kJj716FrrVo7pXAWcDn+zKJy7uJCcDPwN8tFsP\nEx7zYUz0d2SKTE0/txrT2ieuxrT2n4Oa1n52tVr1y9OQZJ0EPNS3/nBXNi1mqupAt/wIMDPOYFaS\nZBvwE8BtTEHs3fDuHcBB4Cbga8ATVfVMV2USvy8fAN4B/HW3/iImP2bodax/nGRveo+JgSn4jkyJ\nae/nVmPDfmemrf8c1JT2s6vVpF+ehiRrw6je/TIm9p4ZSY4D/gD4par6Zv+2SY29qr5bVafTu9P2\nGcCPjjmkw0rys8DBqto77ljW4DVV9Qp6p7QuTfJT/Rsn9TuiybWRvjPT2H8Oatr62dVq2S+P7dmF\nqzDtj614NMnWqjqQZCu9XwITJ8lR9DqI36uqT3XFUxE7QFU9keRW4CeBE5Ic2f0CmbTvy6uBn0ty\nPnAM8Hzgg0x2zABU1f7u/WCST9PrbKfmOzLhpr2fW40N952Z9v5zUFPUz65Ws355Gkaypv2xFdcD\nF3fLFwPXjTGWZXXnnq8E7qmq3+jbNNGxJ3lxkhO65R8EzqU3H+JW4HVdtYmKu6reVVUnV9U2et/l\nW6rqjUxwzABJjk1y/OIy8NPAXUz4d2SKTHs/txob6jszrf3noKaxn12tpv1yVU38Czgf+DN654H/\nxbjjOUycvw8cAL5D7/ztJfTO694M3Af8F+DEcce5TNyvoTeUfSdwR/c6f9JjB/4O8MUu7ruAf9mV\n/zDweWAf8B+Bo8cd6yHinwNumIaYu/i+1L3uXvx3OOnfkWl6TUs/t8pjmso+cZXHOJX95yqOb6r7\n2TUc71D7ZR+rI0mS1MA0nC6UJEmaOiZZkiRJDZhkSZIkNWCSJUmS1IBJliRJUgMmWZIkSQ2YZEmS\nJDVgkiVJktSASZYkSVIDJlmSJEkNmGRJkiQ1YJIlSZLUgEmWJElSAyZZGliS2SR3JHkqyS+uUPct\nST57mO3zSX5h+FFK0vrY12lYTLK0Gu8Abq2q46vqQ60aSXJxkr1Jvpnk4SS/luTIVu1J0hKj6usu\nSnJvkieTHEyyJ8nzW7Wn0TPJ0mq8BLh7BO38D8AvAVuAVwHnAP9sBO1KEoyur/uvwKur6gXADwNH\nAr86gnY1IiZZGkiSW4CzgA8nWUjyI0lekOTqJH+R5MEkv5Jk2e9UknOTfLX7xfZhIIdqq6o+UlV/\nUlV/VVX7gd8DXt3kwCSpz4j7uoeq6rG+ou8CLx3qAWmsTLI0kKo6G/gT4LKqOq6q/gz4TWDxF9jf\nBd4MvHXpZ5NsAT4F/Aq90amvsbqk6acYza9KSZvcqPu6JK9J8iTwFPAPgQ8M72g0biZZWpMkRwAX\nAe+qqqeq6gHg14E3LVP9fODuqvpkVX2HXifyyIDt/GNgO/BvhxK4JK1C676uqj7bnS48Gfi/gQeG\nGL7GzCRLa7UFOAp4sK/sQeCkZer+LeChxZWqqv71Q0lyIfB/Aa9dMqQuSaPSvK/r6u4H/hC4Zs2R\nauKYZGmtHgO+Q2+C6KIfAvYvU/cAcMriSpL0ry8nyXnAbwP/c1V9ed3RStLaNO3rljgS+NtriFET\nyiRLa1JV3wWuBd6b5PgkLwHeDvzuMtX/E/BjSX6+uxXDLwJ/81D7TnI2vcnu/7CqPj/86CVpMI37\nujcm+aFu+SXAe4Gbh30MGh+TLK3HPwWeBu4HPgt8HLhqaaXuVN/rgSuArwOn0bt0+VD+T3qTTG/s\nru5ZSPKfhxy7JA2qVV/3MuD/TfJ0V+9e4H8dauQaq/ROGUuSJGmYHMmSJElqwCRLkiSpAZMsSZKk\nBkyyJEmSGjDJkiRJauDIcQcAsGXLltq2bdth6zz99NMce+yxowloiIx7tIx7tJbGvXfv3seq6sVj\nDGnirdTfjfu7MM72N2vb427fY1992wP3dVU19tcrX/nKWsmtt966Yp1JZNyjZdyjtTRu4PaagD5l\nkl8r9Xfj/i6Ms/3N2va42/fYV2/Qvs7ThZIkSQ2YZEmSJDVgkiVJktSASZYkSVIDJlmSJEkNTMQt\nHFZj165dA5VJ0rRb2rfZ10nTxZEsSZKkBkyyJEmSGhgoyUryQJIvJ7kjye1d2YlJbkpyX/f+wq48\nST6UZF+SO5O8ouUBSJIkTaLVjGSdVVWnV9X2bv1y4OaqOg24uVsHeC1wWvfaCXxkWMFKkiRNi/Wc\nLrwA2NMt7wEu7Cu/urvz/OeAE5JsXUc7kiRJU2fQqwsL+OMkBfyHqtoNzFTVgW77I8BMt3wS8FDf\nZx/uyg70lZFkJ72RLmZmZpifnz9sAAsLC8zPzzM7O/t921b67Dgtxj1tjHu0jFuSNp5Bk6zXVNX+\nJH8DuCnJV/s3VlV1CdjAukRtN8D27dtrbm7usPXn5+eZm5tb9hLmHTt2rKbpkVqMe9oY92gZtyRt\nPAOdLqyq/d37QeDTwBnAo4unAbv3g131/cApfR8/uSuTJEnaNFZMspIcm+T4xWXgp4G7gOuBi7tq\nFwPXdcvXA2/urjI8E3iy77SiJE20JEck+WKSG7r1U5Pc1l0x/Ykkz+vKj+7W93Xbt40zbkmTZ5DT\nhTPAp5Ms1v94Vf1hki8A1ya5BHgQeENX/0bgfGAf8C3grUOPWpLaeRtwD/D8bv19wPur6pok/x64\nhN5V05cAj1fVS5Nc1NX7Ry0D8w7w0nRZMcmqqvuBly9T/nXgnGXKC7h0KNFJ0gglORn4GeC9wNvT\n+3V5NvC/dFX2ALvoJVkXdMsAnwQ+nCRdHyhJ0/fsQklq6APAO4Dju/UXAU9U1TPd+uLV0tB3JXVV\nPZPkya7+Y0t3upqrqfuv2Fzuaup+La7sHOcVo5u17XG377G3a9skS5KAJD8LHKyqvUnmhrnv1VxN\n3X/F5kqnA1tcWT3OK0Y3a9vjbt9jb9e2SZYk9bwa+Lkk5wPH0JuT9UF6N1Q+shvN6r9aevFK6oeT\nHAm8APj66MOWNKl8QLQkAVX1rqo6uaq2ARcBt1TVG4Fbgdd11ZZeSb14hfXruvrOx5L0PSZZknR4\n76Q3CX4fvTlXV3blVwIv6srfzrPPb5UkwNOFkvR9qmoemO+W76d3A+aldf4SeP1IA5M0VRzJkiRJ\nasAkS5IkqQGTLEmSpAZMsiRJkhowyZIkSWrAJEuSJKkBkyxJkqQGTLIkSZIaMMmSJElqwCRLkiSp\nAZMsSZKkBjbEswt37dp12HVJkqRRcyRLkiSpgYGTrCRHJPlikhu69VOT3JZkX5JPJHleV350t76v\n276tTeiSJEmTazUjWW8D7ulbfx/w/qp6KfA4cElXfgnweFf+/q6eJEnSpjJQkpXkZOBngI926wHO\nBj7ZVdkDXNgtX9Ct020/p6svSZK0aQw68f0DwDuA47v1FwFPVNUz3frDwEnd8knAQwBV9UySJ7v6\nj/XvMMlOYCfAzMwM8/Pzhw1gYWGB+fl5ZmdnVwx2pX2N0mLc08a4R8u4JWnjWTHJSvKzwMGq2ptk\nblgNV9VuYDfA9u3ba27u8Luen59nbm5uoCsHd+zYMYQIh2Mx7mlj3KNl3JK08QwykvVq4OeSnA8c\nAzwf+CBwQpIju9Gsk4H9Xf39wCnAw0mOBF4AfH3okUuSJE2wFedkVdW7qurkqtoGXATcUlVvBG4F\nXtdVuxi4rlu+vlun235LVdVQo5YkSZpw67lP1juBtyfZR2/O1ZVd+ZXAi7rytwOXry9ESZKk6bOq\nO75X1Tww3y3fD5yxTJ2/BF4/hNgkSZKmlnd8lyRJasAkS5IkqQGTLEnqJDkmyeeTfCnJ3Une05X7\nGDFJq2aSJUnP+jZwdlW9HDgdOC/JmfgYMUlrYJIlSZ3qWehWj+pehY8Rk7QGq7q6UJI2uiRHAHuB\nlwK/BXyNET5GrP9RRSs9RqzFI43G+aikzdr2uNv32Nu1bZIlSX2q6rvA6UlOAD4N/OgQ9jnwY8T6\nH1W00mPEWjxCbJyPStqsbY+7fY+9XdueLpSkZVTVE/SebPGTdI8R6zYt9xgxfIyYpKVMsiSpk+TF\n3QgWSX4QOBe4Bx8jJmkNPF0oSc/aCuzp5mX9AHBtVd2Q5CvANUl+Ffgiz32M2O90jxH7Br3nu0oS\nYJIlSd9TVXcCP7FMuY8Rk7Rqni6UJElqwCRLkiSpAZMsSZKkBkyyJEmSGjDJkiRJasAkS5IkqQGT\nLEmSpAZWTLKSHJPk80m+lOTuJO/pyk9NcluSfUk+keR5XfnR3fq+bvu2tocgSZI0eQYZyfo2cHZV\nvRw4HTgvyZnA+4D3V9VLgceBS7r6lwCPd+Xv7+pJkiRtKismWdWz0K0e1b0KOBv4ZFe+B7iwW76g\nW6fbfk6SDC1iSZKkKTDQY3W653jtBV4K/BbwNeCJqnqmq/IwcFK3fBLwEEBVPZPkSeBFwGNL9rkT\n2AkwMzPD/Pz8YWNYWFhgfn6e2dnZFeNdaV+jtBj3tDHu0TJuSdp4Bkqyquq7wOnd0+k/Dfzoehuu\nqt3AboDNVCVNAAAb4UlEQVTt27fX3NzcYevPz88zNzfHrl27Vtz3jh071hve0CzGPW2Me7SMW5I2\nnlVdXVhVTwC3Aj8JnJBkMUk7GdjfLe8HTgHotr8A+PpQopUkSZoSg1xd+OJuBIskPwicC9xDL9l6\nXVftYuC6bvn6bp1u+y1VVcMMWpIkadINcrpwK7Cnm5f1A8C1VXVDkq8A1yT5VeCLwJVd/SuB30my\nD/gGcFGDuCVJkibaiklWVd0J/MQy5fcDZyxT/pfA64cSnSRJ0pTyju+SJEkNmGRJkiQ1YJIlSZLU\ngEmWJElSAwPdjFSSNHmWuznzIDdsljQajmRJkiQ1YJIlSZLUgEmWJElSAyZZkgQkOSXJrUm+kuTu\nJG/ryk9MclOS+7r3F3blSfKhJPuS3JnkFeM9AkmTxiRLknqeAX65ql4GnAlcmuRlwOXAzVV1GnBz\ntw7wWuC07rUT+MjoQ5Y0yUyyJAmoqgNV9afd8lPAPcBJwAXAnq7aHuDCbvkC4Orq+RxwQpKtIw5b\n0gTzFg6StESSbfSe2XobMFNVB7pNjwAz3fJJwEN9H3u4KzvAEkl20hvtYmZmhvn5+UO2vbCw8L3t\ns7Ozq479cPseRH/7o7ZZ2x53+x57u7ZNsiSpT5LjgD8Afqmqvpnke9uqqpLUavdZVbuB3QDbt2+v\nubm5Q9adn59ncfta7nm1Y8eOVX/mUO2P2mZte9zte+zt2vZ0oSR1khxFL8H6var6VFf86OJpwO79\nYFe+Hzil7+Mnd2WSBJhkSRLQu1oQuBK4p6p+o2/T9cDF3fLFwHV95W/urjI8E3iy77SiJHm6UJI6\nrwbeBHw5yR1d2T8HrgCuTXIJ8CDwhm7bjcD5wD7gW8BbRxuupElnkiVJQFV9FsghNp+zTP0CLm0a\nlKSp5ulCSZKkBkyyJEmSGlgxyfJRE5IkSas3yEiWj5qQJElapRWTLB81IUmStHqrurpwmI+aWM1j\nJuDZW98P8piJcT4eYKlxP65grYx7tIxbkjaegZOsYT9qYjWPmYBnb30/yGMm7r333uesr+XRFMMy\n7scVrJVxj5ZxS9LGM9DVhT5qQpIkaXUGubrQR01IkiSt0iCnC33UhCRJ0iqtmGT5qAlJkqTV847v\nkiRJDZhkSZIkNWCSJUmS1IBJliRJUgMmWZIkSQ2YZEmSJDVgkiVJktSASZYkSVIDJlmSJEkNmGRJ\nkiQ1YJIlSZLUgEmWJElSAyZZkiRJDZhkSVInyVVJDia5q6/sxCQ3Jbmve39hV54kH0qyL8mdSV4x\nvsglTSKTLEl61seA85aUXQ7cXFWnATd36wCvBU7rXjuBj4woRklTwiRLkjpV9RngG0uKLwD2dMt7\ngAv7yq+uns8BJyTZOppIJU2DI8cdgCRNuJmqOtAtPwLMdMsnAQ/11Xu4KzvAEkl20hvtYmZmhvn5\n+UM2trCw8L3ts7Ozqw72cPseRH/7o7ZZ2x53+x57u7ZNsiRpQFVVSWoNn9sN7AbYvn17zc3NHbLu\n/Pw8i9t37dq16hjvvffe56yvdh/97Y/aZm173O177O3aXvF0oRNBJW1yjy6eBuzeD3bl+4FT+uqd\n3JVJEjDYnKyP4URQSZvX9cDF3fLFwHV95W/uflyeCTzZd1pRklZOspwIKmmzSPL7wH8DZpM8nOQS\n4Arg3CT3AX+vWwe4Ebgf2Af8NvBPxhCypAm21jlZI50ICs9OThvHRND1GPeEwrUy7tEy7slQVTsO\nsemcZeoWcGnbiCRNs3VPfB/FRFB4dnLaWiaC7thxqH6zvXFPKFwr4x4t45akjWet98lyIqgkSdJh\nrDXJciKoJEnSYax4urCbCDoHbEnyMPBuehM/r+0mhT4IvKGrfiNwPr2JoN8C3togZkmSpIm3YpLl\nRFBJkqTV2xR3fF9usvxaJtBLkiQNygdES5IkNWCSJUmS1IBJliRJUgObYk6WJG1WzkmVxseRLEmS\npAZMsiRJkhowyZIkSWrAJEuSJKkBkyxJkqQGTLIkSZIa8BYOkrTJLL2Fg7d0kNrYtEmWnYwkSWrJ\n04WSJEkNbNqRrKW8K7Kkzaq/r5udnR1fINIG40iWJElSAyZZkiRJDXi68DCcHC9pM3L6hDQczZKs\nJOcBHwSOAD5aVVe0amtU7HgkLbUR+7rl+KNTWr0mSVaSI4DfAs4FHga+kOT6qvpKi/bGyY5H2rw2\nU1+3lD86pZW1Gsk6A9hXVfcDJLkGuADY8B1PKyZz0kSyr1ulw/VdLa9sNCnUOLRKsk4CHupbfxh4\nVaO2JsrSf7Szs7Mr/kMe1j/+QRKxYSRra413khLFSYplkvg/olXbtH3dcobxXVnrPlq1PYz+e5DY\n5ubmVqzTyoEDB54TY6t/85PWv+zates5/49uEUuqavg7TV4HnFdVv9Ctvwl4VVVd1ldnJ7CzW50F\n7l1ht1uAx4YebHvGPVrGPVpL435JVb14XMGM2iB9XVe+mv5u3N+Fcba/Wdsed/se++oN1Ne1Gsna\nD5zSt35yV/Y9VbUb2D3oDpPcXlXbhxPe6Bj3aBn3aE1r3EO0Yl8Hq+vvxv03HWf7m7Xtcbfvsbdr\nu9V9sr4AnJbk1CTPAy4Crm/UliSNi32dpENqMpJVVc8kuQz4I3qXNV9VVXe3aEuSxsW+TtLhNLtP\nVlXdCNw4xF0OfGpxwhj3aBn3aE1r3EOzAfu6cba/Wdsed/seeyNNJr5LkiRtdj67UJIkqYGpSLKS\nnJfk3iT7klw+7ngOJclVSQ4muauv7MQkNyW5r3t/4ThjXE6SU5LcmuQrSe5O8raufKJjT3JMks8n\n+VIX93u68lOT3NZ9Xz7RTUieKEmOSPLFJDd06xMfM0CSB5J8OckdSW7vyib6ezJNxtnXLffftnF7\nY+svD9H2riT7u+O/I8n5jdoea397mPabH/+4++zDtP+xJH/ed+ynD63RqproF73JpF8Dfhh4HvAl\n4GXjjusQsf4U8Argrr6yXwMu75YvB9437jiXiXsr8Ipu+Xjgz4CXTXrsQIDjuuWjgNuAM4FrgYu6\n8n8P/O/jjnWZ2N8OfBy4oVuf+Ji72B4Atiwpm+jvybS8xt3XLffftnF7Y+svD9H2LuCfjeC4x9rf\nHqb95sc/7j77MO1/DHhdizanYSTre4+tqKq/AhYfWzFxquozwDeWFF8A7OmW9wAXjjSoAVTVgar6\n0275KeAeeneynujYq2ehWz2qexVwNvDJrnzi4k5yMvAzwEe79TDhMa9gor8nU2Rq+rphGGd/eYi2\nR2Lc/e1h2m9u3H32YdpvZhqSrOUeWzGSL8SQzFTVgW75EWBmnMGsJMk24CfoZfgTH3t32u0O4CBw\nE72RgCeq6pmuyiR+Xz4AvAP46279RUx+zIsK+OMke9O7izlMwfdkSoy7r1vuv+2ojfu7dFmSO7vT\nic1Pe4+7v13SPozg+MfdZy9tv6oWj/293bG/P8nRw2pvGpKsDaN6Y5QTezlnkuOAPwB+qaq+2b9t\nUmOvqu9W1en07rR9BvCjYw7psJL8LHCwqvaOO5Y1ek1VvQJ4LXBpkp/q3zip3xMN5LD/bUdtDN+l\njwB/GzgdOAD8esvGxt3fLtP+SI5/3H320vaT/Djwri6O/xE4EXjnsNqbhiRroMdWTLBHk2wF6N4P\njjmeZSU5it4/uN+rqk91xVMRO0BVPQHcCvwkcEKSxXvATdr35dXAzyV5gN7poLOBDzLZMX9PVe3v\n3g8Cn6bXSU7N92TCjbWvO8R/21Eb23epqh7t/gf818Bv0/D4x93fLtf+KI+/a2+sfXZf++d1p1Cr\nqr4N/D8M8dinIcma9sdWXA9c3C1fDFw3xliW1c0JuhK4p6p+o2/TRMee5MVJTuiWfxA4l978gluB\n13XVJiruqnpXVZ1cVdvofZdvqao3MsExL0pybJLjF5eBnwbuYsK/J1NkbH3dYf7bjtrYvkuLCU7n\nH9Do+Mfd3x6q/VEc/7j77EO0/9W+5Db05oMN79hbzKYf9gs4n94VEF8D/sW44zlMnL9Pb5j1O/TO\nK19Cb77NzcB9wH8BThx3nMvE/Rp6Q9N3And0r/MnPXbg7wBf7OK+C/iXXfkPA58H9gH/ETh63LEe\nIv45nr26cOJj7mL8Uve6e/Hf4qR/T6bpNa6+7lD/bRu3Obb+8hBt/w7w5a4/uR7Y2qjtsfa3h2m/\n+fGPu88+TPu3dMd+F/C7dFcgDuPlHd8lSZIamIbThZIkSVPHJEuSJKkBkyxJkqQGTLIkSZIaMMmS\nJElqwCRLkiSpAZMsSZKkBkyyJEmSGjDJkiRJasAkS5IkqQGTLEmSpAZMsiRJkhowyZIkSWrAJEuS\nJKkBkywNLMlskjuSPJXkF1eo+5Yknz3M9vkkvzD8KCVJmgxHjjsATZV3ALdW1emjajDJzcDZwFFV\n9cyo2pUkab0cydJqvAS4e1SNJXkjcNSo2pMkaZhMsjSQJLcAZwEfTrKQ5EeSvCDJ1Un+IsmDSX4l\nybLfqSTnJvlqkieTfBjICu29AHg3vdEzSZKmjkmWBlJVZwN/AlxWVcdV1Z8Bvwm8APhh4O8Cbwbe\nuvSzSbYAnwJ+BdgCfA149QpN/hvgI8AjwzoGSZJGySRLa5LkCOAi4F1V9VRVPQD8OvCmZaqfD9xd\nVZ+squ8AH+AwyVOS7fSSsN8ceuCSJI2ISZbWagu9+VIP9pU9CJy0TN2/BTy0uFJV1b/erzvd+O+A\ntznRXZI0zUyytFaPAd+hNxl+0Q8B+5epewA4ZXElSfrXl3g+sB34RJJHgC905Q8n+Z/WG7QkSaNi\nkqU1qarvAtcC701yfJKXAG8HfneZ6v8J+LEkP5/kSOAXgb95iF0/SW/k6/TudX5X/krgtiEegiRJ\nTZlkaT3+KfA0cD/wWeDjwFVLK1XVY8DrgSuArwOnAf91uR1WzyOLL+Avuk2PVtVfDf8QJElqI73p\nMZIkSRomR7IkSZIaMMmSJElqwCRLkiSpAZMsSZKkBo4cdwAAW7ZsqW3bto07DJ5++mmOPfbYcYcB\nTE4skxIHGMuhTFIse/fufayqXjzuOCRpEkxEkrVt2zZuv/32cYfB/Pw8c3Nz4w4DmJxYJiUOMJZD\nmaRYkjy4ci1J2hw8XShJktSASZYkSVIDJlmSJEkNmGRJkiQ1YJIlSZLUwERcXThsu3btOuy6JElS\na45kSZIkNWCSJUmS1IBJliRJUgMmWZIkSQ2YZEmSJDVgkiVJktSASZYkSVIDJlmSJEkNmGRJkiQ1\nYJIlSZLUgEmWJElSAyZZkiRJDZhkSZIkNWCSJUmS1IBJliRJUgMmWZIkSQ2YZEmSJDVgkiVJktSA\nSZYkSVIDR447gFHYtWvXQGWSJEnD4kiWJElSAyZZkiRJDZhkSZIkNWCSJUmS1MCKSVaSq5IcTHJX\nX9muJPuT3NG9zu/b9q4k+5Lcm+TvtwpckiRpkg0ykvUx4Lxlyt9fVad3rxsBkrwMuAj4se4z/y7J\nEcMKVpIkaVqsmGRV1WeAbwy4vwuAa6rq21X158A+4Ix1xCdJkjSVUlUrV0q2ATdU1Y9367uAtwDf\nBG4HfrmqHk/yYeBzVfW7Xb0rgf9cVZ9cZp87gZ0AMzMzr7zmmmuGcDg9Bw4cWLHO1q1bv69sYWGB\n4447bmhxrMekxDIpcYCxHMokxXLWWWftrart445DkibBWm9G+hHgXwPVvf868I9Xs4Oq2g3sBti+\nfXvNzc2tMZTvN8iNRnfs2PF9ZfPz8wwzjvWYlFgmJQ4wlkOZpFgkSc9a09WFVfVoVX23qv4a+G2e\nPSW4Hzilr+rJXZkkSdKmsqYkK0n/ubZ/ACxeeXg9cFGSo5OcCpwGfH59IUqSJE2fFU8XJvl9YA7Y\nkuRh4N3AXJLT6Z0ufAD43wCq6u4k1wJfAZ4BLq2q77YJXZIkaXKtmGRV1fdPXoIrD1P/vcB71xOU\nJEnStPOO75IkSQ2YZEmSJDVgkiVJktSASZYkSVIDJlmSJEkNmGRJkiQ1YJIlSZLUgEmWJElSA2t9\nQPREGeSB0JIkSaPkSJYkSVIDJlmSJEkNmGRJkiQ1YJIlSZLUwIaY+L4WSyfLO3lekiQNkyNZkiRJ\nDZhkSZIkNWCSJUmS1IBJliRJUgMmWZIkSQ2YZEmSJDVgkiVJktTApr1P1lK7du1idnb2OffL8t5Z\nkiRprRzJkiRJasAkS5IkqQGTLEmSpAZMsiRJkhowyZIkSWrAJEuSJKkBkyxJkqQGTLIkSZIaWDHJ\nSnJVkoNJ7uorOzHJTUnu695f2JUnyYeS7EtyZ5JXtAxekiRpUg0ykvUx4LwlZZcDN1fVacDN3TrA\na4HTutdO4CPDCVOSJGm6rJhkVdVngG8sKb4A2NMt7wEu7Cu/uno+B5yQZOuwgpUkSZoWqaqVKyXb\ngBuq6se79Seq6oRuOcDjVXVCkhuAK6rqs922m4F3VtXty+xzJ73RLmZmZl55zTXXrPkgDhw4sObP\n9jv66KP59re//b31rVvHlx8uLCxw3HHHja39SYsDjOVQJimWs846a29VbR93HJI0Cdb9gOiqqiQr\nZ2rf/7ndwG6A7du319zc3JpjGNaDnGdnZ7n33nu/t75jx46h7Hct5ufnWc/fZKPFAcZyKJMUiyTp\nWWu9uvDRxdOA3fvBrnw/cEpfvZO7MkmSpE1lrUnW9cDF3fLFwHV95W/urjI8E3iyqoZzLk+SJGmK\nrHi6MMnvA3PAliQPA+8GrgCuTXIJ8CDwhq76jcD5wD7gW8BbG8QsSZI08VZMsqrqUBOTzlmmbgGX\nrjcoSZKkaecd3yVJkhowyZIkSWrAJEuSJKkBkyxJkqQGTLIkSZIaMMmSJElqwCRLkiSpAZMsSZKk\nBtb9gOiNbOmDp4f1IGpJkrTxOZIlSZLUgEmWJElSAyZZkiRJDZhkSZIkNWCSJUmS1IBJliRJUgMm\nWZIkSQ2YZEmSJDVgkiVJktSASZYkSVIDPlZnnXz0jiRJWo4jWZIkSQ04krUKjlJJkqRBOZIlSZLU\ngEmWJElSAyZZkiRJDUzdnCznRUmSpGkwdUnWtFkuKTRRlCRp4/N0oSRJUgMmWZIkSQ2s63RhkgeA\np4DvAs9U1fYkJwKfALYBDwBvqKrH1xemJEnSdBnGSNZZVXV6VW3v1i8Hbq6q04Cbu3VJkqRNpcXp\nwguAPd3yHuDCBm1IkiRNtFTV2j+c/DnwOFDAf6iq3UmeqKoTuu0BHl9cX/LZncBOgJmZmVdec801\nA7V54MCBNce7kqOPPppvf/vb69rH1q1bn7O+XLxL6yxnYWGB4447bl2xDMOkxAHGciiTFMtZZ521\nt29UW5I2tfXewuE1VbU/yd8Abkry1f6NVVVJls3iqmo3sBtg+/btNTc3N1CDLW9/MDs7y7333ruu\nfezYseM568vFu7TOcubn5xn0b9LSpMQBxnIokxSLJOlZ6zpdWFX7u/eDwKeBM4BHk2wF6N4PrjdI\nSZKkabPmkawkxwI/UFVPdcs/Dfwr4HrgYuCK7v26YQQ6LbzRqCRJgvWdLpwBPt2bdsWRwMer6g+T\nfAG4NsklwIPAG9YfpiRJ0nRZc5JVVfcDL1+m/OvAOesJaqNbOtrl6JckSRuPd3yXJElqwCRLkiSp\ngfXewkFDsNzpQi/JlyRpujmSJUmS1IBJliRJUgMmWZIkSQ2YZEmSJDVgkiVJktSASZYkSVIDJlmS\nJEkNmGRJkiQ1YJIlSZLUgEmWJElSAyZZkiRJDZhkSZIkNWCSJUmS1IBJliRJUgNHjjsADWbXrl2H\nXZckSZPFkSxJkqQGHMmaUsuNZDm6JUnS5DDJmlAHDhwwaZIkaYp5ulCSJKkBR7I2ECfHS5I0ORzJ\nkiRJasCRrE3G0S5JkkbDJEvPsTTpmp2dHU8gkiRNOU8XSpIkNWCSJUmS1ICnCzewQeZbOSdLkqQ2\nTLK0okEmyzuhXpKk52qWZCU5D/ggcATw0aq6olVbGq1hjZCtNjE71CR8EzxJ0iRqkmQlOQL4LeBc\n4GHgC0mur6qvtGhPm0eL5K31fiRJm1OrkawzgH1VdT9AkmuACwCTLH3POJOWVm0PY5RvnA//9sHj\nkjQ8qarh7zR5HXBeVf1Ct/4m4FVVdVlfnZ3Azm51Frh36IGs3hbgsXEH0ZmUWCYlDjCWQ5mkWGar\n6vhxByFJk2BsE9+rajewe1ztLyfJ7VW1fdxxwOTEMilxgLEcyqTFMu4YJGlStLpP1n7glL71k7sy\nSZKkTaFVkvUF4LQkpyZ5HnARcH2jtiRJkiZOk9OFVfVMksuAP6J3C4erquruFm0N2SSdvpyUWCYl\nDjCWQzEWSZpATSa+S5IkbXY+u1CSJKkBkyxJkqQGTLKAJA8k+XKSO0Z9CXqSq5IcTHJXX9mJSW5K\ncl/3/sIxxrIryf7ub3NHkvNHFMspSW5N8pUkdyd5W1c+8r/NYWIZ+d8myTFJPp/kS10s7+nKT01y\nW5J9ST7RXXAyjjg+luTP+/4mp7eMQ5ImmXOy6CVZwPaqGvkNHZP8FLAAXF1VP96V/Rrwjaq6Isnl\nwAur6p1jimUXsFBV/7Z1+0ti2Qpsrao/TXI8sBe4EHgLI/7bHCaWNzDiv02SAMdW1UKSo4DPAm8D\n3g58qqquSfLv4f9v7/5BqgrDOI5/HyQiapBIHIKIWhoiLCgIHCQoaooggiBwTKihtZYgcKxGh4hw\n6A+S/aOpIKGmhsr+gC1BQWE6hFRLUP4a3vfCRdSm97zG/X1APJ575fx48HIez/vce3gtaaRCjiHg\noaTbpY5tZva/8JWsyiQ9Bb4t2H0YGM3bo6QTeq0sVUialvQyb/8ApoCNVKjNMlkap+Rn/nFV/hKw\nD2g1NsXrskwOMzPL3GQlAh5FxIt8u5/aeiVN5+2vQG/NMMDpiHiTlxMbWbpsFxGbgZ3AcyrXZkEW\nqFCbiOiKiElgFngMfADmJP3OT/lMA03gwhySWjUZzjW5HBGrS+cwM1up3GQl/ZJ2AYeAU3nZbEVQ\nWs+teYVgBNgK9AHTwMUmDx4R64Bx4Iyk7+2PNV2bRbJUqY2kP5L6SHdS2ANsa+K4/8oREduBsznP\nbmA9UHyZ28xspXKTBUj6kr/PAndJJ66aZvIcUGseaLZWEEkz+WQ6D1yhwdrkWZ9x4LqkO3l3ldos\nlqVmbfLx54AJYC/QHRGtDxdu9DZWbTkO5qVVSfoFXKP+a8nMrJqOb7IiYm0eZiYi1gIHgHfL/1Zx\nD4DBvD0I3K8VpNXQZEdoqDZ5sPoqMCXpUttDjddmqSw1ahMRPRHRnbfXAPtJM2ITwNH8tOJ1WSLH\n+7YGOEhzYbVfS2Zm1XT8uwsjYgvp6hWk2wzdkDTc4PFvAgPABmAGOA/cA8aATcAn4Jik4gPpS2QZ\nIC2HCfgInGybiSqZpR94BrwF5vPuc6RZqEZrs0yW4zRcm4jYQRps7yL9kzQm6UL+O75FWqJ7BZzI\nV5OazvEE6AECmASG2gbkzcw6Ssc3WWZmZmYldPxyoZmZmVkJbrLMzMzMCnCTZWZmZlaAmywzMzOz\nAtxkmZmZmRXgJsvMzMysADdZZmZmZgX8BRTmQ2DdcZfBAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fae62ded5c0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "hist_data = dict()\n",
    "keys = ['fold 0', 'fold 1', 'fold 2', 'fold 3', 'fold 4']\n",
    "for i, key in enumerate(keys):\n",
    "    hist_data[key] = fold_errors[i]\n",
    "    \n",
    "\n",
    "plt.figure()\n",
    "hist_dataframe = pd.DataFrame(hist_data, columns=keys)\n",
    "_ = hist_dataframe.hist(color='k', alpha=0.5, bins=50, figsize=(10, 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Moving To Sequential Data\n",
    "\n",
    "So we next will try recurrent neural networks. This means munging the data into meaningful sequences of a set length for a discrete stock. Fortunately Pandas, NumPy and Python in general makes this a fairly painless process to get on with.\n",
    "\n",
    "We need some new information now, namely the date and the close, as we will provide the network with a sequence of close values over a given set of days and ask it to predict the next days close. Once we have the information the first step is to sort it by symbol and then date."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>symbol</th>\n",
       "      <th>close</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>251</th>\n",
       "      <td>2010-01-04</td>\n",
       "      <td>A</td>\n",
       "      <td>31.300001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>718</th>\n",
       "      <td>2010-01-05</td>\n",
       "      <td>A</td>\n",
       "      <td>30.960001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1186</th>\n",
       "      <td>2010-01-06</td>\n",
       "      <td>A</td>\n",
       "      <td>30.850001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1654</th>\n",
       "      <td>2010-01-07</td>\n",
       "      <td>A</td>\n",
       "      <td>30.809999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2122</th>\n",
       "      <td>2010-01-08</td>\n",
       "      <td>A</td>\n",
       "      <td>30.800000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            date symbol      close\n",
       "251   2010-01-04      A  31.300001\n",
       "718   2010-01-05      A  30.960001\n",
       "1186  2010-01-06      A  30.850001\n",
       "1654  2010-01-07      A  30.809999\n",
       "2122  2010-01-08      A  30.800000"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sequence_desired_cols = ['date', 'symbol', 'close']\n",
    "sequence_data = dataframe[sequence_desired_cols]\n",
    "sequence_data = sequence_data.sort_values(by=['symbol', 'date'])\n",
    "sequence_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have the sorted information we can loop through it and make sequences of stock close prices. We can make a new supervised learning dataset by having the x data as the sequence of closes and y data as the close value the next day for the model to predict.\n",
    "\n",
    "Annoyingly, because of the irregular sampling of this dataset, and presumably because the stock market closes at the weekend, the maximum sequence we can get a good amount of examples is of length three, with a target prediction (monday -> thursday). Having a llonger sequence would probably help provide more context, but any sequence is better than none!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "sequence_length = 3\n",
    "\n",
    "all_data = deque()\n",
    "\n",
    "previous_date = None\n",
    "previous_class = None\n",
    "reset = False\n",
    "close_deque = deque()\n",
    "\n",
    "for index, row in sequence_data.iterrows():\n",
    "    current_date = parse(row['date'])\n",
    "    current_class = row['symbol']\n",
    "    \n",
    "    first_value = previous_date == None and previous_class == None\n",
    "    if not first_value:\n",
    "        successive_date = current_date == (previous_date + timedelta(days=1))\n",
    "    successive_class = current_class == previous_class\n",
    "    append = first_value or (successive_date and successive_class)\n",
    "\n",
    "    if append:\n",
    "        close_deque.append(row['close'])\n",
    "        \n",
    "        if len(close_deque) > sequence_length:\n",
    "            target_close = close_deque.pop()\n",
    "            all_data.append((list(close_deque), [target_close]))\n",
    "            reset = True\n",
    "            \n",
    "    if not append or reset:\n",
    "        close_deque.clear()\n",
    "        previous_date = None\n",
    "        previous_class = None\n",
    "        reset = False\n",
    "        \n",
    "    previous_date = current_date\n",
    "    previous_class = current_class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we quickly turn the list of tuples of supervised data into two seperate and more managable structures using the zip function. We also reshape y_data so the RNN will happily take it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x shape: (146379, 3, 1) \n",
      "y shape: (146379, 1)\n",
      "sequence difference mean: 1.08033354515 \n",
      "sequence difference max:  401.913324667 \n",
      "sequence difference min:  0.0 \n",
      "sequence difference std:  2.8896750188\n"
     ]
    }
   ],
   "source": [
    "x_data, y_data = list(zip(*all_data))\n",
    "x_data = np.array(x_data).reshape((-1, sequence_length, 1))\n",
    "y_data = np.array(y_data)\n",
    "print(\"x shape:\", x_data.shape, \n",
    "      \"\\ny shape:\", y_data.shape)\n",
    "\n",
    "all_distances = []\n",
    "for i in range(len(x_data)):\n",
    "    all_distances.append(np.abs(np.mean(x_data[i]) - y_data[i][0]))\n",
    "all_distances = np.array(all_distances)\n",
    "\n",
    "print(\"sequence difference mean:\", np.mean(all_distances), \n",
    "      \"\\nsequence difference max: \", np.max(all_distances), \n",
    "      \"\\nsequence difference min: \", np.min(all_distances), \n",
    "      \"\\nsequence difference std: \", np.std(all_distances))\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building A LSTM\n",
    "\n",
    "Now we build a recurrent neural network from the network parameters defined below. It has much of the same functionality as before, except for ofcourse the archetecture begins with a set of recurrent layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Again reset the graph\n",
    "tf.reset_default_graph()\n",
    "\n",
    "# Network parameters\n",
    "rnn_layer_sizes = [64, 32, 16]\n",
    "net_hidden_sizes = [8, 4]\n",
    "amount_epochs = 50\n",
    "learning_rate = 0.001\n",
    "batch_size = 32\n",
    "l2_strength = 0.001\n",
    "non_linearity = tf.nn.relu\n",
    "dropout_amount = 0.4\n",
    "max_grad_norm = 5.0\n",
    "decay = True\n",
    "cell_type = \"rnn\"\n",
    "\n",
    "# The input to the graph - the targets (close) and the inputs\n",
    "# (open). Also a placeholder to pass a variable dropout rate. \n",
    "net_input = tf.placeholder(tf.float32, shape=[None, sequence_length, 1])\n",
    "net_target = tf.placeholder(tf.float32, shape=[None, 1])\n",
    "dropout_prob = tf.placeholder(tf.float32)\n",
    "\n",
    "# Create a stacked GRU with dropout using the rnn_layer_sizes\n",
    "# list to for the lstm memory size.\n",
    "cells = []\n",
    "for num_units in rnn_layer_sizes:\n",
    "    if cell_type == \"rnn\":\n",
    "        cell = tf.contrib.rnn.BasicRNNCell(num_units)\n",
    "    elif cell_type == \"gru\":\n",
    "        cell = tf.contrib.rnn.GRUCell(num_units)\n",
    "    elif cell_type == \"lstm\":\n",
    "        cell = tf.contrib.rnn.LSTMCell(num_units)\n",
    "    cell = tf.contrib.rnn.DropoutWrapper(cell=cell, \n",
    "                                         output_keep_prob=dropout_prob)\n",
    "    cells.append(cell)\n",
    "rnn_cell = tf.contrib.rnn.MultiRNNCell(cells)\n",
    "rnn_out, _ = tf.nn.dynamic_rnn(cell=rnn_cell,\n",
    "                               inputs=net_input,\n",
    "                               sequence_length=None,\n",
    "                               dtype=tf.float32)\n",
    "rnn_output = rnn_out[:, -1, :]\n",
    "                        \n",
    "# L2 regularisation to penalise the weights from growing too\n",
    "# large. Useful to prevent overfitting.\n",
    "regulariser = tf.contrib.layers.l2_regularizer(scale=l2_strength)\n",
    "\n",
    "# Build the network from the list of dimensions. Apply l2 and\n",
    "# dropout regularisation to the layers.\n",
    "net = rnn_output\n",
    "for size in net_hidden_sizes:\n",
    "    net = tf.layers.dense(inputs=net, \n",
    "                          units=size, \n",
    "                          activation=non_linearity, \n",
    "                          kernel_regularizer=regulariser)\n",
    "    net = tf.layers.dropout(inputs=net,\n",
    "                            rate=dropout_prob)\n",
    "    \n",
    "# The models prediction has a linear output. \n",
    "net_output = tf.layers.dense(inputs=net,\n",
    "                             units=1, \n",
    "                             activation=None, \n",
    "                             kernel_regularizer=regulariser)   \n",
    "\n",
    "# The main loss for penalising the network on how well it does.\n",
    "loss = tf.losses.mean_squared_error(labels=net_target, \n",
    "                                    predictions=net_output)\n",
    "\n",
    "# TensorFlows manner of applying l2 to the loss.\n",
    "l2_variables = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)\n",
    "l2_loss = tf.contrib.layers.apply_regularization(regulariser, \n",
    "                                                 l2_variables)\n",
    "cost = loss + l2_loss\n",
    "\n",
    "# Train and initialisation TensorFlow operations to be ran\n",
    "# in the session.\n",
    "trainables = tf.trainable_variables()\n",
    "gradients = tf.gradients(cost, trainables)\n",
    "\n",
    "# Clip the gradients by a pre-defined max norm.\n",
    "clipped_gradients, _ = tf.clip_by_global_norm(gradients, max_grad_norm)\n",
    "\n",
    "# Add the clipped gradients to the optimizer.\n",
    "batch = tf.Variable(0)\n",
    "if decay:\n",
    "    decayed_learning_rate = tf.train.exponential_decay(learning_rate,\n",
    "                                                       batch * batch_size,\n",
    "                                                       117103,\n",
    "                                                       0.999,\n",
    "                                                       staircase=True)\n",
    "else:\n",
    "    decayed_learning_rate = learning_rate\n",
    "optimiser = tf.train.AdamOptimizer(decayed_learning_rate)\n",
    "train_op = optimiser.apply_gradients(zip(gradients, trainables), global_step=batch)\n",
    "\n",
    "init_op = tf.global_variables_initializer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training The LSTM\n",
    "\n",
    "Again, we train the lstm in mostly the same manner as before. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "with tf.Session() as sess:\n",
    "    \n",
    "    amount_folds = 5\n",
    "    k_folds = KFold(n_splits=amount_folds)\n",
    "    fold_errors = []\n",
    "    fold_iteration = 0\n",
    "    \n",
    "    # Cross validate the dataset, using K-Fold.\n",
    "    for train_indices, test_indices in k_folds.split(x_data):\n",
    "\n",
    "        # Each new fold, reinitialise the network.\n",
    "        sess.run(init_op)\n",
    "        \n",
    "        # Training phase.\n",
    "        for epoch in range(amount_epochs):\n",
    "            \n",
    "            # Each new epoch, reshuffle the train set.\n",
    "            random_train_indices = np.random.permutation(train_indices)\n",
    "                \n",
    "            # Loop over the train set and optimise the network.\n",
    "            for begin in range(0, len(train_indices), batch_size):\n",
    "                end = begin + batch_size\n",
    "                batch_x = x_data[random_train_indices][begin:end]\n",
    "                batch_y = y_data[random_train_indices][begin:end]\n",
    "                \n",
    "                sess.run(train_op, feed_dict={\n",
    "                    net_input: batch_x,\n",
    "                    net_target: batch_y,\n",
    "                    dropout_prob: dropout_amount\n",
    "                })\n",
    "                \n",
    "            # Testing phase - record the error over the test set.\n",
    "            epoch_error = []\n",
    "            for begin in range(0, len(test_indices), batch_size):\n",
    "                end = begin + batch_size \n",
    "                batch_x = x_data[test_indices][begin:end]\n",
    "                batch_y = y_data[test_indices][begin:end]\n",
    "\n",
    "                error = sess.run(loss, feed_dict={\n",
    "                    net_input: batch_x,\n",
    "                    net_target: batch_y,\n",
    "                    dropout_prob: 1.0\n",
    "                }) \n",
    "                epoch_error.append(error)\n",
    "        \n",
    "            epoch_error = np.array(epoch_error).reshape((-1))    \n",
    "                \n",
    "            print('\\nepoch:', epoch,\n",
    "                  '\\nmean: ', np.mean(epoch_error), \n",
    "                  '\\nstd:  ', np.std(epoch_error),\n",
    "                  '\\nmax:  ', np.max(epoch_error),\n",
    "                  '\\nmin:  ', np.min(epoch_error))\n",
    " \n",
    "                \n",
    "        # Testing phase - record the error over the test set.\n",
    "        all_error = []\n",
    "        for begin in range(0, len(test_indices), batch_size):\n",
    "            end = begin + batch_size \n",
    "            batch_x = x_data[test_indices][begin:end]\n",
    "            batch_y = y_data[test_indices][begin:end]\n",
    "            \n",
    "            error = sess.run(loss, feed_dict={\n",
    "                net_input: batch_x,\n",
    "                net_target: batch_y,\n",
    "                dropout_prob: 1.0\n",
    "            }) \n",
    "            all_error.append(error)\n",
    "        \n",
    "        all_error = np.array(all_error).reshape((-1))\n",
    "        fold_errors.append(all_error)\n",
    "        \n",
    "        print(\"\\nFold iteration:  \", fold_iteration,\n",
    "              \"\\nError mean:      \", np.mean(all_error),\n",
    "              \"\\nError deviation: \", np.std(all_error),\n",
    "              \"\\n\")\n",
    "        fold_iteration += 1      \n",
    "        \n",
    "    fold_errors = np.array(fold_errors).reshape((amount_folds, -1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusions\n",
    "\n",
    "The results from the RNN were very poor so I didn't bother to include them.\n",
    "\n",
    "It's unclear why the recurrent neural network proved even harder to train, but it could be down to the data and the nature of the sequences; the sequences and the target predictions often had very vast differences which means the prediction problem is very hard. Hopefully this notebook has highlighted a few issue when we try to predict the stock market, and tilts our heads to new ways to tackle similar problems."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

Can i have your Price.csv please?

Please

@Blunderchips and @ricky321u the price.csv is available at Kaggle. Check this link: https://www.kaggle.com/dgawlik/nyse/data