Skip to content

Instantly share code, notes, and snippets.

@qazwsxal

qazwsxal/gym-demo.ipynb

Last active January 28, 2023 13:56
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save qazwsxal/6cc1c5cf16a23ae6ea8d5c369828fa80 to your computer and use it in GitHub Desktop.
Save qazwsxal/6cc1c5cf16a23ae6ea8d5c369828fa80 to your computer and use it in GitHub Desktop.
Download ZIP
OpenAI Gym Lab Demo (Durham Students Click Here!)
Display the source blob
Display the rendered blob
Raw
gym-demo.ipynb
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "OpenAI Gym Demo.ipynb",
"provenance": [],
"collapsed_sections": [],
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/qazwsxal/6cc1c5cf16a23ae6ea8d5c369828fa80/gym-demo.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "QTNU1mwGB1ZD"
},
"source": [
"**Initialise**"
]
},
{
"cell_type": "code",
"metadata": {
"id": "1tfkgPuF3MbI",
"outputId": "c38d0a71-0619-418c-ebaf-197bf56aa336",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 833
}
},
"source": [
"%%bash\n",
"# This Cell is only used to make sure cartpole runs \n",
"\n",
"# install required system dependencies\n",
"apt-get install -y xvfb x11-utils\n",
"\n",
"# install required python dependencies (might need to install additional gym extras depending)\n",
"pip install pyvirtualdisplay PyOpenGL PyOpenGL-accelerate\n"
],
"execution_count": 1,
"outputs": [
{
"output_type": "stream",
"text": [
"Reading package lists...\n",
"Building dependency tree...\n",
"Reading state information...\n",
"The following additional packages will be installed:\n",
" libxxf86dga1\n",
"Suggested packages:\n",
" mesa-utils\n",
"The following NEW packages will be installed:\n",
" libxxf86dga1 x11-utils xvfb\n",
"0 upgraded, 3 newly installed, 0 to remove and 21 not upgraded.\n",
"Need to get 993 kB of archives.\n",
"After this operation, 2,977 kB of additional disk space will be used.\n",
"Get:1 http://archive.ubuntu.com/ubuntu bionic/main amd64 libxxf86dga1 amd64 2:1.1.4-1 [13.7 kB]\n",
"Get:2 http://archive.ubuntu.com/ubuntu bionic/main amd64 x11-utils amd64 7.7+3build1 [196 kB]\n",
"Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 xvfb amd64 2:1.19.6-1ubuntu4.7 [783 kB]\n",
"Fetched 993 kB in 2s (518 kB/s)\n",
"Selecting previously unselected package libxxf86dga1:amd64.\r\n",
"(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 144611 files and directories currently installed.)\r\n",
"Preparing to unpack .../libxxf86dga1_2%3a1.1.4-1_amd64.deb ...\r\n",
"Unpacking libxxf86dga1:amd64 (2:1.1.4-1) ...\r\n",
"Selecting previously unselected package x11-utils.\r\n",
"Preparing to unpack .../x11-utils_7.7+3build1_amd64.deb ...\r\n",
"Unpacking x11-utils (7.7+3build1) ...\r\n",
"Selecting previously unselected package xvfb.\r\n",
"Preparing to unpack .../xvfb_2%3a1.19.6-1ubuntu4.7_amd64.deb ...\r\n",
"Unpacking xvfb (2:1.19.6-1ubuntu4.7) ...\r\n",
"Setting up xvfb (2:1.19.6-1ubuntu4.7) ...\r\n",
"Setting up libxxf86dga1:amd64 (2:1.1.4-1) ...\r\n",
"Setting up x11-utils (7.7+3build1) ...\r\n",
"Processing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\n",
"Processing triggers for libc-bin (2.27-3ubuntu1.2) ...\r\n",
"/sbin/ldconfig.real: /usr/local/lib/python3.6/dist-packages/ideep4py/lib/libmkldnn.so.0 is not a symbolic link\r\n",
"\r\n",
"Collecting pyvirtualdisplay\n",
" Downloading https://files.pythonhosted.org/packages/d0/8a/643043cc70791367bee2d19eb20e00ed1a246ac48e5dbe57bbbcc8be40a9/PyVirtualDisplay-1.3.2-py2.py3-none-any.whl\n",
"Requirement already satisfied: PyOpenGL in /usr/local/lib/python3.6/dist-packages (3.1.5)\n",
"Collecting PyOpenGL-accelerate\n",
" Downloading https://files.pythonhosted.org/packages/a2/3c/f42a62b7784c04b20f8b88d6c8ad04f4f20b0767b721102418aad94d8389/PyOpenGL-accelerate-3.1.5.tar.gz (538kB)\n",
"Collecting EasyProcess\n",
" Downloading https://files.pythonhosted.org/packages/48/3c/75573613641c90c6d094059ac28adb748560d99bd27ee6f80cce398f404e/EasyProcess-0.3-py2.py3-none-any.whl\n",
"Building wheels for collected packages: PyOpenGL-accelerate\n",
" Building wheel for PyOpenGL-accelerate (setup.py): started\n",
" Building wheel for PyOpenGL-accelerate (setup.py): finished with status 'done'\n",
" Created wheel for PyOpenGL-accelerate: filename=PyOpenGL_accelerate-3.1.5-cp36-cp36m-linux_x86_64.whl size=1593646 sha256=4dc2b8873280c38ddd9560bb7a7781c52009ff635150efde0954fb4ebbeeb191\n",
" Stored in directory: /root/.cache/pip/wheels/bd/21/77/99670ceca25fddb3c2b60a7ae44644b8253d1006e8ec417bcc\n",
"Successfully built PyOpenGL-accelerate\n",
"Installing collected packages: EasyProcess, pyvirtualdisplay, PyOpenGL-accelerate\n",
"Successfully installed EasyProcess-0.3 PyOpenGL-accelerate-3.1.5 pyvirtualdisplay-1.3.2\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "uKXoZoJ93Wsd"
},
"source": [
"# This Cell is only used to make sure cartpole runs \n",
"\n",
"import pyvirtualdisplay\n",
"\n",
"\n",
"_display = pyvirtualdisplay.Display(visible=False, # use False with Xvfb\n",
" size=(1400, 900))\n",
"_ = _display.start()"
],
"execution_count": 2,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "_TZefME0MTvA"
},
"source": [
"# this is a Deep Q Learning (DQN) agent including replay memory and a target network \n",
"# you can write a brief 8-10 line abstract detailing your submission and experiments here\n",
"# the code is based on https://github.com/seungeunrho/minimalRL/blob/master/dqn.py, which is released under the MIT licesne\n",
"# make sure you reference any code you have studied as above, with one comment line per reference\n",
"\n",
"# imports\n",
"import gym\n",
"import collections\n",
"import random\n",
"import numpy as np\n",
"import torch\n",
"import torch.nn as nn\n",
"import torch.nn.functional as F\n",
"import torch.optim as optim\n",
"import matplotlib.pyplot as plt\n",
"\n",
"# hyperparameters\n",
"learning_rate = 0.0005\n",
"gamma = 0.98\n",
"buffer_limit = 50000\n",
"batch_size = 32\n",
"video_every = 25\n",
"print_every = 5\n",
"\n",
"class ReplayBuffer():\n",
" def __init__(self):\n",
" self.buffer = collections.deque(maxlen=buffer_limit)\n",
" \n",
" def put(self, transition):\n",
" self.buffer.append(transition)\n",
" \n",
" def sample(self, n):\n",
" mini_batch = random.sample(self.buffer, n)\n",
" s_lst, a_lst, r_lst, s_prime_lst, done_mask_lst = [], [], [], [], []\n",
" \n",
" for transition in mini_batch:\n",
" s, a, r, s_prime, done_mask = transition\n",
" s_lst.append(s)\n",
" a_lst.append([a])\n",
" r_lst.append([r])\n",
" s_prime_lst.append(s_prime)\n",
" done_mask_lst.append([done_mask])\n",
"\n",
" return torch.tensor(s_lst, dtype=torch.float), torch.tensor(a_lst), \\\n",
" torch.tensor(r_lst), torch.tensor(s_prime_lst, dtype=torch.float), \\\n",
" torch.tensor(done_mask_lst)\n",
" \n",
" def size(self):\n",
" return len(self.buffer)\n",
"\n",
"class QNetwork(nn.Module):\n",
" def __init__(self, insize, outsize):\n",
" super(QNetwork, self).__init__()\n",
" self.fc1 = nn.Linear(insize, 256)\n",
" self.fc2 = nn.Linear(256, 84)\n",
" self.fc3 = nn.Linear(84, outsize)\n",
"\n",
" def forward(self, x):\n",
" x = x.view(x.size(0),-1)\n",
" x = F.relu(self.fc1(x))\n",
" x = F.relu(self.fc2(x))\n",
" x = self.fc3(x)\n",
" return x\n",
" \n",
" def sample_action(self, obs, epsilon):\n",
" out = self.forward(obs)\n",
" coin = random.random()\n",
" if coin < epsilon:\n",
" return random.randint(0,1)\n",
" else : \n",
" return out.argmax().item()\n",
" \n",
"def train(q, q_target, memory, optimizer):\n",
" for i in range(10):\n",
" s,a,r,s_prime,done_mask = memory.sample(batch_size)\n",
"\n",
" q_out = q(s)\n",
" q_a = q_out.gather(1,a)\n",
" max_q_prime = q_target(s_prime).max(1)[0].unsqueeze(1)\n",
" target = r + gamma * max_q_prime * done_mask\n",
" loss = F.smooth_l1_loss(q_a, target)\n",
" # Q(s,a) = R(s,a) + γ*Q_targ(s_prime)*done_mask\n",
" optimizer.zero_grad()\n",
" loss.backward()\n",
" optimizer.step()\n"
],
"execution_count": 3,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "4ck-chjFdScJ"
},
"source": [
"**Train**\n",
"\n",
"← You can download the videos from the videos folder in the files on the left"
]
},
{
"cell_type": "code",
"metadata": {
"id": "q6MTebC0u_wI"
},
"source": [
"# setup the environment, and record a video every 50 episodes.\n",
"env = gym.make('CartPole-v0')\n",
"env = gym.wrappers.Monitor(env, \"./video\", video_callable=lambda episode_id: (episode_id%video_every)==0,force=True)"
],
"execution_count": 4,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "SrlpvIidvJxh",
"outputId": "8f97c21b-a432-4f8f-a363-5a113a0034f7",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
}
},
"source": [
"env.reset()"
],
"execution_count": 5,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"array([-0.03431689, 0.04510229, -0.02069988, -0.0475874 ])"
]
},
"metadata": {
"tags": []
},
"execution_count": 5
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "2rdPwFaivOuf",
"outputId": "8659a719-3541-4745-fb1a-7caadf56c11f",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 286
}
},
"source": [
"plt.imshow(env.render(mode='rgb_array'))"
],
"execution_count": 6,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"<matplotlib.image.AxesImage at 0x7fcbb20fbc88>"
]
},
"metadata": {
"tags": []
},
"execution_count": 6
},
{
"output_type": "display_data",
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAATPUlEQVR4nO3df6zddZ3n8eerP2gBHUvlWjpt2TLaDYOzazFXxOgkDMYZZCcLk7gGdheJQ9LZBBNNzO7CbLKjyWJm4o64Zkd2OwHB1RWZUaQhuNpBNhM3EShYEagMVylLuy0tSIGCVnr73j/ut3iAlnvur55+7nk+kpPz/b6/n+857084fXHu537PPakqJEntWDDoBiRJU2NwS1JjDG5JaozBLUmNMbglqTEGtyQ1Zs6CO8kFSR5JMpbkqrl6HkkaNpmL67iTLAT+AfgAsAO4F7i0qh6e9SeTpCEzV++4zwHGqupnVfUr4Gbgojl6LkkaKovm6HFXAU/07O8A3n20waeeemqtXbt2jlqRpPZs376dp556Kkc6NlfBPakkG4ANAKeffjpbtmwZVCuSdNwZHR096rG5WirZCazp2V/d1V5WVRurarSqRkdGRuaoDUmaf+YquO8F1iU5I8kJwCXApjl6LkkaKnOyVFJVB5N8DPgOsBC4oaoemovnkqRhM2dr3FV1B3DHXD2+JA0rPzkpSY0xuCWpMQa3JDXG4JakxhjcktQYg1uSGmNwS1JjDG5JaozBLUmNMbglqTEGtyQ1xuCWpMYY3JLUGINbkhpjcEtSYwxuSWqMwS1JjTG4JakxM/rqsiTbgeeBceBgVY0mWQ58HVgLbAc+XFXPzKxNSdJhs/GO+/eqan1VjXb7VwF3VtU64M5uX5I0S+ZiqeQi4KZu+ybg4jl4DkkaWjMN7gK+m+S+JBu62oqq2tVt7wZWzPA5JEk9ZrTGDbyvqnYmeQuwOclPeg9WVSWpI53YBf0GgNNPP32GbUjS8JjRO+6q2tnd7wFuBc4BnkyyEqC733OUczdW1WhVjY6MjMykDUkaKtMO7iQnJ3nj4W3g94EHgU3A5d2wy4HbZtqkJOnXZrJUsgK4Ncnhx/mfVfW/ktwL3JLkCuBx4MMzb1OSdNi0g7uqfga84wj1p4H3z6QpSdLR+clJSWqMwS1JjTG4JakxBrckNcbglqTGGNyS1BiDW5IaY3BLUmMMbklqjMEtSY0xuCWpMQa3JDXG4JakxhjcktQYg1uSGmNwS1JjDG5JaozBLUmNMbglqTGTBneSG5LsSfJgT215ks1JHu3uT+nqSfKFJGNJHkjyzrlsXpKGUT/vuG8ELnhV7SrgzqpaB9zZ7QN8EFjX3TYA181Om5KkwyYN7qr6e+DnrypfBNzUbd8EXNxT/3JN+AGwLMnK2WpWkjT9Ne4VVbWr294NrOi2VwFP9Izb0dVeI8mGJFuSbNm7d+8025Ck4TPjX05WVQE1jfM2VtVoVY2OjIzMtA1JGhrTDe4nDy+BdPd7uvpOYE3PuNVdTZI0S6Yb3JuAy7vty4Hbeuof6a4uORd4tmdJRZI0CxZNNiDJ14DzgFOT7AD+DPhz4JYkVwCPAx/uht8BXAiMAS8CH52DniVpqE0a3FV16VEOvf8IYwu4cqZNSZKOzk9OSlJjDG5JaozBLUmNMbglqTEGtyQ1xuCWpMYY3JLUGINbkhpjcEtSYwxuSWqMwS1JjTG4JakxBrckNcbglqTGGNyS1BiDW5IaY3BLUmMMbklqzKTBneSGJHuSPNhT+1SSnUm2drcLe45dnWQsySNJ/mCuGpekYdXPO+4bgQuOUL+2qtZ3tzsAkpwFXAK8vTvni0kWzlazkqQ+gruq/h74eZ+PdxFwc1UdqKrHmPi293Nm0J8k6VVmssb9sSQPdEspp3S1VcATPWN2dLXXSLIhyZYkW/bu3TuDNiRpuEw3uK8D3gqsB3YBfznVB6iqjVU1WlWjIyMj02xDkobPtIK7qp6sqvGqOgT8Nb9eDtkJrOkZurqrSZJmybSCO8nKnt0/Ag5fcbIJuCTJkiRnAOuAe2bWoiSp16LJBiT5GnAecGqSHcCfAeclWQ8UsB34E4CqeijJLcDDwEHgyqoan5vWJWk4TRrcVXXpEcrXv874a4BrZtKUJOno/OSkJDXG4JakxhjcktQYg1uSGmNwS1JjJr2qRJrvXtj7OOO/+gVLl53GCScvG3Q70qQMbg2dgwdeZPv/vpEafwmAF/ZsZ/xXL3L67/5rRn77dwfcnTQ5g1tDp8YP8vzObRw6+KtBtyJNi2vcktQYg1uSGmNwS1JjDG5JaozBLUmNMbg1dBaecCJvOv2fvKa+77H7OdRdIigdzwxuDZ0Fixaz9JTffE39xad3UIcODaAjaWoMbklqjMEtSY0xuCWpMZMGd5I1Se5K8nCSh5J8vKsvT7I5yaPd/SldPUm+kGQsyQNJ3jnXk5CkYdLPO+6DwCer6izgXODKJGcBVwF3VtU64M5uH+CDTHy7+zpgA3DdrHctSUNs0uCuql1VdX+3/TywDVgFXATc1A27Cbi4274I+HJN+AGwLMnKWe9ckobUlNa4k6wFzgbuBlZU1a7u0G5gRbe9Cnii57QdXe3Vj7UhyZYkW/bu3TvFtiVpePUd3EneAHwD+ERVPdd7rKoKqKk8cVVtrKrRqhodGRmZyqmSNNT6Cu4ki5kI7a9W1Te78pOHl0C6+z1dfSewpuf01V1NkjQL+rmqJMD1wLaq+lzPoU3A5d325cBtPfWPdFeXnAs827OkIkmaoX6+Aee9wGXAj5Ns7Wp/Cvw5cEuSK4DHgQ93x+4ALgTGgBeBj85qx5I05CYN7qr6PpCjHH7/EcYXcOUM+5IkHYWfnJSkxhjcktQYg1uSGmNwS1JjDG5JaozBLUmNMbglqTEGtyQ1xuCWpMYY3JLUGINbkhpjcEtSYwxuDaWT37KWBYtOeEXt0MEDvLDnsQF1JPXP4NZQesNp61iweOkraodeOsD+3Y8OqCOpfwa3JDXG4JakxhjcktQYg1uSGtPPlwWvSXJXkoeTPJTk4139U0l2Jtna3S7sOefqJGNJHknyB3M5AUkaNv18WfBB4JNVdX+SNwL3JdncHbu2qv5z7+AkZwGXAG8HfhP4uyT/uKrGZ7NxSRpWk77jrqpdVXV/t/08sA1Y9TqnXATcXFUHquoxJr7t/ZzZaFaSNMU17iRrgbOBu7vSx5I8kOSGJKd0tVXAEz2n7eD1g16SNAV9B3eSNwDfAD5RVc8B1wFvBdYDu4C/nMoTJ9mQZEuSLXv37p3KqZI01PoK7iSLmQjtr1bVNwGq6smqGq+qQ8Bf8+vlkJ3Amp7TV3e1V6iqjVU1WlWjIyMjM5mDJA2Vfq4qCXA9sK2qPtdTX9kz7I+AB7vtTcAlSZYkOQNYB9wzey1L0nDr56qS9wKXAT9OsrWr/SlwaZL1QAHbgT8BqKqHktwCPMzEFSlXekWJJM2eSYO7qr4P5AiH7nidc64BrplBX5Kko/CTk5LUGINbkhpjcEtSYwxuSWqMwS1JjTG4JakxBrckNcbglqTGGNyS1BiDW5IaY3BLUmMMbg2lLFjA0mWnvaZ+4Nm9HBp/aQAdSf0zuDWUFixczPK3vfYb9fZt/yHjB14cQEdS//r5s65SM+69914+85nP9DX27DVL+We/8xuvqP3ylwf46B//MS8cODTp+cuXL+eLX/wiS5YsmVav0nQZ3JpXnnzySb71rW/1N/h9Z3LB28/j4KHDwVuMj+/n29/+Nj9/7heTnr5y5UrGx/1T8zr2DG4NrWIBP3nu3fzfX5wJwMK8xJknfnfAXUmTM7g1tP7fL97K9hffTnW/6hmvxTz+4lkcKv9Z6PjmLyc1tMZr4cuhfdjeA2t46dDiAXUk9aefLwtemuSeJD9K8lCST3f1M5LcnWQsydeTnNDVl3T7Y93xtXM7BWl6liz4BQs4+IraqhPHOGHBgQF1JPWnn3fcB4Dzq+odwHrggiTnAn8BXFtVbwOeAa7oxl8BPNPVr+3GScedFUsf57d/425OXriPF57fyTNPP8qC/f8Hv9tax7t+viy4gP3d7uLuVsD5wL/s6jcBnwKuAy7qtgH+FvivSdI9jnTc2Dq2m9z63yjgnm072fX0fkJxyJeqjnN9/RYmyULgPuBtwF8BPwX2VdXhnzN3AKu67VXAEwBVdTDJs8CbgaeO9vi7d+/ms5/97LQmIPXatm1b32O3797H9t37XlGbSmTv37+fz3/+8yxe7Jq4Zt/u3buPeqyv4K6Jnx3XJ1kG3AqcOdOmkmwANgCsWrWKyy67bKYPKbF582a+9KUvHZPnOumkk7j00ks58cQTj8nzabh85StfOeqxKV33VFX7ktwFvAdYlmRR9657NbCzG7YTWAPsSLIIeBPw9BEeayOwEWB0dLROO+21fzdCmqpTTjnlmD3XggULWLFiBSeddNIxe04Nj9f7Sa6fq0pGunfaJDkR+ACwDbgL+FA37HLgtm57U7dPd/x7rm9L0uzp5x33SuCmbp17AXBLVd2e5GHg5iT/CfghcH03/nrgfyQZA34OXDIHfUvS0OrnqpIHgLOPUP8Z8Jo/r1ZVvwT+xax0J0l6DT85KUmNMbglqTH+NR3NKytWrODiiy8+Js+1fPlyFi5ceEyeS+plcGteede73sWtt9466DakOeVSiSQ1xuCWpMYY3JLUGINbkhpjcEtSYwxuSWqMwS1JjTG4JakxBrckNcbglqTGGNyS1BiDW5IaY3BLUmMMbklqTD9fFrw0yT1JfpTkoSSf7uo3Jnksydbutr6rJ8kXkowleSDJO+d6EpI0TPr5e9wHgPOran+SxcD3k3y7O/Zvq+pvXzX+g8C67vZu4LruXpI0CyZ9x10T9ne7i7tbvc4pFwFf7s77AbAsycqZtypJgj7XuJMsTLIV2ANsrqq7u0PXdMsh1yZZ0tVWAU/0nL6jq0mSZkFfwV1V41W1HlgNnJPkd4CrgTOBdwHLgX8/lSdOsiHJliRb9u7dO8W2JWl4TemqkqraB9wFXFBVu7rlkAPAl4BzumE7gTU9p63uaq9+rI1VNVpVoyMjI9PrXpKGUD9XlYwkWdZtnwh8APjJ4XXrJAEuBh7sTtkEfKS7uuRc4Nmq2jUn3UvSEOrnqpKVwE1JFjIR9LdU1e1JvpdkBAiwFfg33fg7gAuBMeBF4KOz37YkDa9Jg7uqHgDOPkL9/KOML+DKmbcmSToSPzkpSY0xuCWpMQa3JDXG4JakxhjcktQYg1uSGmNwS1JjDG5JaozBLUmNMbglqTEGtyQ1xuCWpMYY3JLUGINbkhpjcEtSYwxuSWqMwS1JjTG4JakxBrckNcbglqTGGNyS1BiDW5Iak6oadA8keR54ZNB9zJFTgacG3cQcmK/zgvk7N+fVln9UVSNHOrDoWHdyFI9U1eigm5gLSbbMx7nN13nB/J2b85o/XCqRpMYY3JLUmOMluDcOuoE5NF/nNl/nBfN3bs5rnjgufjkpSerf8fKOW5LUp4EHd5ILkjySZCzJVYPuZ6qS3JBkT5IHe2rLk2xO8mh3f0pXT5IvdHN9IMk7B9f560uyJsldSR5O8lCSj3f1pueWZGmSe5L8qJvXp7v6GUnu7vr/epITuvqSbn+sO752kP1PJsnCJD9Mcnu3P1/mtT3Jj5NsTbKlqzX9WpyJgQZ3koXAXwEfBM4CLk1y1iB7moYbgQteVbsKuLOq1gF3dvswMc913W0DcN0x6nE6DgKfrKqzgHOBK7v/Nq3P7QBwflW9A1gPXJDkXOAvgGur6m3AM8AV3fgrgGe6+rXduOPZx4FtPfvzZV4Av1dV63su/Wv9tTh9VTWwG/Ae4Ds9+1cDVw+yp2nOYy3wYM/+I8DKbnslE9epA/x34NIjjTveb8BtwAfm09yAk4D7gXcz8QGORV395dcl8B3gPd32om5cBt37UeazmokAOx+4Hch8mFfX43bg1FfV5s1rcaq3QS+VrAKe6Nnf0dVat6KqdnXbu4EV3XaT8+1+jD4buJt5MLduOWErsAfYDPwU2FdVB7shvb2/PK/u+LPAm49tx337PPDvgEPd/puZH/MCKOC7Se5LsqGrNf9anK7j5ZOT81ZVVZJmL91J8gbgG8Anquq5JC8fa3VuVTUOrE+yDLgVOHPALc1Ykj8E9lTVfUnOG3Q/c+B9VbUzyVuAzUl+0nuw1dfidA36HfdOYE3P/uqu1ronk6wE6O73dPWm5ptkMROh/dWq+mZXnhdzA6iqfcBdTCwhLEty+I1Mb+8vz6s7/ibg6WPcaj/eC/zzJNuBm5lYLvkvtD8vAKpqZ3e/h4n/2Z7DPHotTtWgg/teYF33m+8TgEuATQPuaTZsAi7vti9nYn34cP0j3W+9zwWe7flR77iSibfW1wPbqupzPYeanluSke6dNklOZGLdfhsTAf6hbtir53V4vh8CvlfdwunxpKqurqrVVbWWiX9H36uqf0Xj8wJIcnKSNx7eBn4feJDGX4szMuhFduBC4B+YWGf8D4PuZxr9fw3YBbzExFraFUysFd4JPAr8HbC8GxsmrqL5KfBjYHTQ/b/OvN7HxLriA8DW7nZh63MD/inww25eDwL/sav/FnAPMAb8DbCkqy/t9se647816Dn0McfzgNvny7y6Ofyouz10OCdafy3O5OYnJyWpMYNeKpEkTZHBLUmNMbglqTEGtyQ1xuCWpMYY3JLUGINbkhpjcEtSY/4/1YKMefUVpcIAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"tags": [],
"needs_background": "light"
}
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "5stHkFq4UztI",
"outputId": "c7a6a422-905b-4510-e2f5-368ee8dc9439",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 1000
}
},
"source": [
"# reproducible environment and action spaces, do not change lines 6-11 here (tools > settings > editor > show line numbers)\n",
"seed = 742\n",
"torch.manual_seed(seed)\n",
"env.seed(seed)\n",
"random.seed(seed)\n",
"np.random.seed(seed)\n",
"env.action_space.seed(seed)\n",
"\n",
"q = QNetwork(np.array(env.observation_space.shape).prod(), env.action_space.n)\n",
"q_target = QNetwork(np.array(env.observation_space.shape).prod(), env.action_space.n)\n",
"q_target.load_state_dict(q.state_dict())\n",
"memory = ReplayBuffer()\n",
"\n",
"score = 0.0\n",
"marking = []\n",
"optimizer = optim.Adam(q.parameters(), lr=learning_rate)\n",
"\n",
"for n_episode in range(1001):\n",
" epsilon = max(0.01, 0.08 - 0.01*(n_episode/200)) # linear annealing from 8% to 1%\n",
" s = env.reset()\n",
" done = False\n",
" score = 0.0\n",
"\n",
" while True:\n",
"\n",
" a = q.sample_action(torch.from_numpy(s).float().unsqueeze(0), epsilon)\n",
" s_prime, r, done, info = env.step(a)\n",
" done_mask = 0.0 if done else 1.0\n",
" memory.put((s,a,r/100.0,s_prime, done_mask))\n",
" s = s_prime\n",
"\n",
" score += r\n",
" if done:\n",
" break\n",
" \n",
" if memory.size()>2000:\n",
" train(q, q_target, memory, optimizer)\n",
"\n",
" # do not change lines 44-48 here, they are for marking the submission log\n",
" marking.append(score)\n",
" if n_episode%100 == 0:\n",
" print(\"marking, episode: {}, score: {:.1f}, mean_score: {:.2f}, std_score: {:.2f}\".format(\n",
" n_episode, score, np.array(marking).mean(), np.array(marking).std()))\n",
" marking = []\n",
"\n",
" # you can change this part, and print any data you like (so long as it doesn't start with \"marking\")\n",
" if n_episode%print_every==0 and n_episode!=0:\n",
" q_target.load_state_dict(q.state_dict())\n",
" print(\"episode: {}, score: {:.1f}, epsilon: {:.2f}\".format(n_episode, score, epsilon))"
],
"execution_count": 7,
"outputs": [
{
"output_type": "stream",
"text": [
"marking, episode: 0, score: 9.0, mean_score: 9.00, std_score: 0.00\n",
"episode: 5, score: 10.0, epsilon: 0.08\n",
"episode: 10, score: 9.0, epsilon: 0.08\n",
"episode: 15, score: 8.0, epsilon: 0.08\n",
"episode: 20, score: 9.0, epsilon: 0.08\n",
"episode: 25, score: 10.0, epsilon: 0.08\n",
"episode: 30, score: 10.0, epsilon: 0.08\n",
"episode: 35, score: 9.0, epsilon: 0.08\n",
"episode: 40, score: 8.0, epsilon: 0.08\n",
"episode: 45, score: 10.0, epsilon: 0.08\n",
"episode: 50, score: 8.0, epsilon: 0.08\n",
"episode: 55, score: 11.0, epsilon: 0.08\n",
"episode: 60, score: 10.0, epsilon: 0.08\n",
"episode: 65, score: 11.0, epsilon: 0.08\n",
"episode: 70, score: 8.0, epsilon: 0.08\n",
"episode: 75, score: 10.0, epsilon: 0.08\n",
"episode: 80, score: 9.0, epsilon: 0.08\n",
"episode: 85, score: 10.0, epsilon: 0.08\n",
"episode: 90, score: 9.0, epsilon: 0.08\n",
"episode: 95, score: 10.0, epsilon: 0.08\n",
"marking, episode: 100, score: 8.0, mean_score: 9.61, std_score: 1.05\n",
"episode: 100, score: 8.0, epsilon: 0.07\n",
"episode: 105, score: 10.0, epsilon: 0.07\n",
"episode: 110, score: 9.0, epsilon: 0.07\n",
"episode: 115, score: 12.0, epsilon: 0.07\n",
"episode: 120, score: 8.0, epsilon: 0.07\n",
"episode: 125, score: 11.0, epsilon: 0.07\n",
"episode: 130, score: 12.0, epsilon: 0.07\n",
"episode: 135, score: 10.0, epsilon: 0.07\n",
"episode: 140, score: 9.0, epsilon: 0.07\n",
"episode: 145, score: 10.0, epsilon: 0.07\n",
"episode: 150, score: 10.0, epsilon: 0.07\n",
"episode: 155, score: 8.0, epsilon: 0.07\n",
"episode: 160, score: 10.0, epsilon: 0.07\n",
"episode: 165, score: 10.0, epsilon: 0.07\n",
"episode: 170, score: 8.0, epsilon: 0.07\n",
"episode: 175, score: 12.0, epsilon: 0.07\n",
"episode: 180, score: 9.0, epsilon: 0.07\n",
"episode: 185, score: 9.0, epsilon: 0.07\n",
"episode: 190, score: 10.0, epsilon: 0.07\n",
"episode: 195, score: 10.0, epsilon: 0.07\n",
"marking, episode: 200, score: 10.0, mean_score: 9.56, std_score: 0.99\n",
"episode: 200, score: 10.0, epsilon: 0.07\n",
"episode: 205, score: 8.0, epsilon: 0.07\n",
"episode: 210, score: 11.0, epsilon: 0.07\n",
"episode: 215, score: 8.0, epsilon: 0.07\n",
"episode: 220, score: 9.0, epsilon: 0.07\n",
"episode: 225, score: 10.0, epsilon: 0.07\n",
"episode: 230, score: 18.0, epsilon: 0.07\n",
"episode: 235, score: 11.0, epsilon: 0.07\n",
"episode: 240, score: 11.0, epsilon: 0.07\n",
"episode: 245, score: 12.0, epsilon: 0.07\n",
"episode: 250, score: 10.0, epsilon: 0.07\n",
"episode: 255, score: 40.0, epsilon: 0.07\n",
"episode: 260, score: 31.0, epsilon: 0.07\n",
"episode: 265, score: 28.0, epsilon: 0.07\n",
"episode: 270, score: 36.0, epsilon: 0.07\n",
"episode: 275, score: 50.0, epsilon: 0.07\n",
"episode: 280, score: 136.0, epsilon: 0.07\n",
"episode: 285, score: 84.0, epsilon: 0.07\n",
"episode: 290, score: 200.0, epsilon: 0.07\n",
"episode: 295, score: 179.0, epsilon: 0.07\n",
"marking, episode: 300, score: 177.0, mean_score: 50.69, std_score: 62.06\n",
"episode: 300, score: 177.0, epsilon: 0.07\n",
"episode: 305, score: 113.0, epsilon: 0.06\n",
"episode: 310, score: 123.0, epsilon: 0.06\n",
"episode: 315, score: 164.0, epsilon: 0.06\n",
"episode: 320, score: 104.0, epsilon: 0.06\n",
"episode: 325, score: 186.0, epsilon: 0.06\n",
"episode: 330, score: 115.0, epsilon: 0.06\n",
"episode: 335, score: 134.0, epsilon: 0.06\n",
"episode: 340, score: 156.0, epsilon: 0.06\n",
"episode: 345, score: 200.0, epsilon: 0.06\n",
"episode: 350, score: 123.0, epsilon: 0.06\n",
"episode: 355, score: 200.0, epsilon: 0.06\n",
"episode: 360, score: 200.0, epsilon: 0.06\n",
"episode: 365, score: 200.0, epsilon: 0.06\n",
"episode: 370, score: 200.0, epsilon: 0.06\n",
"episode: 375, score: 200.0, epsilon: 0.06\n",
"episode: 380, score: 200.0, epsilon: 0.06\n",
"episode: 385, score: 170.0, epsilon: 0.06\n",
"episode: 390, score: 168.0, epsilon: 0.06\n",
"episode: 395, score: 173.0, epsilon: 0.06\n",
"marking, episode: 400, score: 179.0, mean_score: 167.21, std_score: 39.24\n",
"episode: 400, score: 179.0, epsilon: 0.06\n",
"episode: 405, score: 200.0, epsilon: 0.06\n",
"episode: 410, score: 200.0, epsilon: 0.06\n",
"episode: 415, score: 158.0, epsilon: 0.06\n",
"episode: 420, score: 200.0, epsilon: 0.06\n",
"episode: 425, score: 186.0, epsilon: 0.06\n",
"episode: 430, score: 200.0, epsilon: 0.06\n",
"episode: 435, score: 151.0, epsilon: 0.06\n",
"episode: 440, score: 165.0, epsilon: 0.06\n",
"episode: 445, score: 166.0, epsilon: 0.06\n",
"episode: 450, score: 175.0, epsilon: 0.06\n",
"episode: 455, score: 190.0, epsilon: 0.06\n",
"episode: 460, score: 191.0, epsilon: 0.06\n",
"episode: 465, score: 119.0, epsilon: 0.06\n",
"episode: 470, score: 200.0, epsilon: 0.06\n",
"episode: 475, score: 200.0, epsilon: 0.06\n",
"episode: 480, score: 200.0, epsilon: 0.06\n",
"episode: 485, score: 159.0, epsilon: 0.06\n",
"episode: 490, score: 133.0, epsilon: 0.06\n",
"episode: 495, score: 93.0, epsilon: 0.06\n",
"marking, episode: 500, score: 163.0, mean_score: 176.26, std_score: 25.89\n",
"episode: 500, score: 163.0, epsilon: 0.06\n",
"episode: 505, score: 154.0, epsilon: 0.05\n",
"episode: 510, score: 200.0, epsilon: 0.05\n",
"episode: 515, score: 200.0, epsilon: 0.05\n",
"episode: 520, score: 171.0, epsilon: 0.05\n",
"episode: 525, score: 182.0, epsilon: 0.05\n",
"episode: 530, score: 200.0, epsilon: 0.05\n",
"episode: 535, score: 200.0, epsilon: 0.05\n",
"episode: 540, score: 164.0, epsilon: 0.05\n",
"episode: 545, score: 160.0, epsilon: 0.05\n",
"episode: 550, score: 198.0, epsilon: 0.05\n",
"episode: 555, score: 176.0, epsilon: 0.05\n",
"episode: 560, score: 200.0, epsilon: 0.05\n",
"episode: 565, score: 178.0, epsilon: 0.05\n",
"episode: 570, score: 200.0, epsilon: 0.05\n",
"episode: 575, score: 197.0, epsilon: 0.05\n",
"episode: 580, score: 200.0, epsilon: 0.05\n",
"episode: 585, score: 164.0, epsilon: 0.05\n",
"episode: 590, score: 166.0, epsilon: 0.05\n",
"episode: 595, score: 126.0, epsilon: 0.05\n",
"marking, episode: 600, score: 151.0, mean_score: 174.15, std_score: 22.91\n",
"episode: 600, score: 151.0, epsilon: 0.05\n",
"episode: 605, score: 200.0, epsilon: 0.05\n",
"episode: 610, score: 150.0, epsilon: 0.05\n",
"episode: 615, score: 174.0, epsilon: 0.05\n",
"episode: 620, score: 147.0, epsilon: 0.05\n",
"episode: 625, score: 191.0, epsilon: 0.05\n",
"episode: 630, score: 161.0, epsilon: 0.05\n",
"episode: 635, score: 160.0, epsilon: 0.05\n",
"episode: 640, score: 169.0, epsilon: 0.05\n",
"episode: 645, score: 162.0, epsilon: 0.05\n",
"episode: 650, score: 170.0, epsilon: 0.05\n",
"episode: 655, score: 189.0, epsilon: 0.05\n",
"episode: 660, score: 151.0, epsilon: 0.05\n",
"episode: 665, score: 154.0, epsilon: 0.05\n",
"episode: 670, score: 166.0, epsilon: 0.05\n",
"episode: 675, score: 149.0, epsilon: 0.05\n",
"episode: 680, score: 166.0, epsilon: 0.05\n",
"episode: 685, score: 183.0, epsilon: 0.05\n",
"episode: 690, score: 193.0, epsilon: 0.05\n",
"episode: 695, score: 200.0, epsilon: 0.05\n",
"marking, episode: 700, score: 160.0, mean_score: 169.23, std_score: 22.15\n",
"episode: 700, score: 160.0, epsilon: 0.04\n",
"episode: 705, score: 200.0, epsilon: 0.04\n",
"episode: 710, score: 197.0, epsilon: 0.04\n",
"episode: 715, score: 153.0, epsilon: 0.04\n",
"episode: 720, score: 200.0, epsilon: 0.04\n",
"episode: 725, score: 149.0, epsilon: 0.04\n",
"episode: 730, score: 196.0, epsilon: 0.04\n",
"episode: 735, score: 197.0, epsilon: 0.04\n",
"episode: 740, score: 200.0, epsilon: 0.04\n",
"episode: 745, score: 155.0, epsilon: 0.04\n",
"episode: 750, score: 200.0, epsilon: 0.04\n",
"episode: 755, score: 200.0, epsilon: 0.04\n",
"episode: 760, score: 190.0, epsilon: 0.04\n",
"episode: 765, score: 173.0, epsilon: 0.04\n",
"episode: 770, score: 200.0, epsilon: 0.04\n",
"episode: 775, score: 200.0, epsilon: 0.04\n",
"episode: 780, score: 200.0, epsilon: 0.04\n",
"episode: 785, score: 200.0, epsilon: 0.04\n",
"episode: 790, score: 200.0, epsilon: 0.04\n",
"episode: 795, score: 200.0, epsilon: 0.04\n",
"marking, episode: 800, score: 168.0, mean_score: 181.54, std_score: 25.20\n",
"episode: 800, score: 168.0, epsilon: 0.04\n",
"episode: 805, score: 200.0, epsilon: 0.04\n",
"episode: 810, score: 200.0, epsilon: 0.04\n",
"episode: 815, score: 200.0, epsilon: 0.04\n",
"episode: 820, score: 200.0, epsilon: 0.04\n",
"episode: 825, score: 147.0, epsilon: 0.04\n",
"episode: 830, score: 198.0, epsilon: 0.04\n",
"episode: 835, score: 200.0, epsilon: 0.04\n",
"episode: 840, score: 200.0, epsilon: 0.04\n",
"episode: 845, score: 200.0, epsilon: 0.04\n",
"episode: 850, score: 200.0, epsilon: 0.04\n",
"episode: 855, score: 200.0, epsilon: 0.04\n",
"episode: 860, score: 200.0, epsilon: 0.04\n",
"episode: 865, score: 200.0, epsilon: 0.04\n",
"episode: 870, score: 200.0, epsilon: 0.04\n",
"episode: 875, score: 154.0, epsilon: 0.04\n",
"episode: 880, score: 189.0, epsilon: 0.04\n",
"episode: 885, score: 200.0, epsilon: 0.04\n",
"episode: 890, score: 192.0, epsilon: 0.04\n",
"episode: 895, score: 200.0, epsilon: 0.04\n",
"marking, episode: 900, score: 200.0, mean_score: 188.28, std_score: 20.29\n",
"episode: 900, score: 200.0, epsilon: 0.04\n",
"episode: 905, score: 192.0, epsilon: 0.03\n",
"episode: 910, score: 190.0, epsilon: 0.03\n",
"episode: 915, score: 200.0, epsilon: 0.03\n",
"episode: 920, score: 200.0, epsilon: 0.03\n",
"episode: 925, score: 158.0, epsilon: 0.03\n",
"episode: 930, score: 179.0, epsilon: 0.03\n",
"episode: 935, score: 121.0, epsilon: 0.03\n",
"episode: 940, score: 116.0, epsilon: 0.03\n",
"episode: 945, score: 180.0, epsilon: 0.03\n",
"episode: 950, score: 146.0, epsilon: 0.03\n",
"episode: 955, score: 188.0, epsilon: 0.03\n",
"episode: 960, score: 155.0, epsilon: 0.03\n",
"episode: 965, score: 139.0, epsilon: 0.03\n",
"episode: 970, score: 172.0, epsilon: 0.03\n",
"episode: 975, score: 200.0, epsilon: 0.03\n",
"episode: 980, score: 185.0, epsilon: 0.03\n",
"episode: 985, score: 177.0, epsilon: 0.03\n",
"episode: 990, score: 140.0, epsilon: 0.03\n",
"episode: 995, score: 19.0, epsilon: 0.03\n",
"marking, episode: 1000, score: 140.0, mean_score: 175.90, std_score: 28.52\n",
"episode: 1000, score: 140.0, epsilon: 0.03\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "iMGV0rM4To6A"
},
"source": [
"# Custom environment\n",
"# \"Play your cards right\"\n",
"drawn_count = 7\n",
"highest_card = 9\n",
"# Construct our action space\n",
"obv_space = gym.spaces.MultiBinary([drawn_count, highest_card+1])\n",
"act_space = gym.spaces.Discrete(2)\n",
"\n",
"class Brucey(gym.Env):\n",
" reward_range = (0,1)\n",
" action_space = act_space\n",
" observation_space = obv_space\n",
" _max_episode_steps = drawn_count-1\n",
" deck = [x for x in range(1,highest_card+1)]\n",
" cards = \"-123456789\"\n",
" guess='LH'\n",
"\n",
" def __init__(self):\n",
" super().__init__()\n",
" \n",
" def reset(self):\n",
" self.steps=1\n",
" self.hidden_cards = random.sample(self.deck, drawn_count)\n",
" self.guesses=[]\n",
" obs = [0] * drawn_count\n",
" obs[:self.steps] = self.hidden_cards[:self.steps]\n",
" obs = np.array(obs)\n",
" # Fancy onehot encoding\n",
" onehot = np.zeros((obs.size, highest_card+1))\n",
" onehot[np.arange(obs.size),obs] = 1\n",
" return onehot\n",
"\n",
" def step(self, action):\n",
" self.guesses.append(action)\n",
" info = dict()\n",
" cardhigher = self.hidden_cards[self.steps] > self.hidden_cards[self.steps-1] \n",
" self.steps += 1 \n",
" obs = [0] * drawn_count\n",
" obs[:self.steps] = self.hidden_cards[:self.steps]\n",
" obs = np.array(obs)\n",
" # Fancy onehot encoding\n",
" onehot = np.zeros((obs.size, highest_card+1))\n",
" onehot[np.arange(obs.size),obs] = 1\n",
" if cardhigher == action:\n",
" reward = 1\n",
" if self.steps != len(self.hidden_cards):\n",
" done = False\n",
" else:\n",
" done = True\n",
" else:\n",
" reward = 0\n",
" done = True\n",
" \n",
"\n",
" return onehot, reward, done, info\n",
"\n",
" def render(self):\n",
" obs = [0] * drawn_count\n",
" obs[:self.steps] = self.hidden_cards[:self.steps]\n",
" rend = \"{}\\n{}\\n{}\".format(\n",
" ''.join(self.cards[i] for i in obs),\n",
" ''.join(self.guess[i] for i in self.guesses),\n",
" ''.join(self.cards[i] for i in self.hidden_cards),\n",
" )\n",
" print(rend)\n",
" return rend\n",
" \n",
"\n"
],
"execution_count": 14,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "1wG7Ya0zXCFO",
"outputId": "3e8de38b-804a-4f29-f905-f377df01d97e",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 86
}
},
"source": [
"env = Brucey()\n",
"env.reset()\n",
"env.render()\n"
],
"execution_count": 16,
"outputs": [
{
"output_type": "stream",
"text": [
"6------\n",
"\n",
"6327198\n"
],
"name": "stdout"
},
{
"output_type": "execute_result",
"data": {
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "string"
},
"text/plain": [
"'6------\\n\\n6327198'"
]
},
"metadata": {
"tags": []
},
"execution_count": 16
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "knroQcFT0Ldl",
"outputId": "3ff036bf-f621-44d5-8599-297bda1ae0d5",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 136
}
},
"source": [
"env.step(0)"
],
"execution_count": 17,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"(array([[0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],\n",
" [0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],\n",
" [1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
" [1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
" [1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
" [1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
" [1., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]), 1, False, {})"
]
},
"metadata": {
"tags": []
},
"execution_count": 17
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "xD1Hnh-S0PwF",
"outputId": "d7d96d74-83da-4b9d-eab9-2fb17d00310b",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 136
}
},
"source": [
"env.step(1)"
],
"execution_count": 18,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"(array([[0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],\n",
" [0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],\n",
" [0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],\n",
" [1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
" [1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
" [1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
" [1., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]), 0, True, {})"
]
},
"metadata": {
"tags": []
},
"execution_count": 18
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "JSrSc_qa0ReY",
"outputId": "75b685b1-78df-4144-9a5a-88ad2cd8fe23",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 86
}
},
"source": [
"env.render()"
],
"execution_count": 19,
"outputs": [
{
"output_type": "stream",
"text": [
"632----\n",
"LH\n",
"6327198\n"
],
"name": "stdout"
},
{
"output_type": "execute_result",
"data": {
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "string"
},
"text/plain": [
"'632----\\nLH\\n6327198'"
]
},
"metadata": {
"tags": []
},
"execution_count": 19
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "zvjqta75drdA",
"outputId": "57808a42-ea6f-4e8f-a35e-6557cb03ee75",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 1000
}
},
"source": [
"seed = 742\n",
"torch.manual_seed(seed)\n",
"env.seed(seed)\n",
"random.seed(seed)\n",
"np.random.seed(seed)\n",
"env.action_space.seed(seed)\n",
"\n",
"q = QNetwork(drawn_count * (highest_card+1), 2)\n",
"q_target = QNetwork(drawn_count * (highest_card+1), 2)\n",
"q_target.load_state_dict(q.state_dict())\n",
"memory = ReplayBuffer()\n",
"\n",
"score = 0.0\n",
"scores = []\n",
"marking = []\n",
"optimizer = optim.Adam(q.parameters(), lr=learning_rate)\n",
"\n",
"for n_episode in range(10001):\n",
" epsilon = 0.08\n",
" s = env.reset()\n",
" done = False\n",
" score = 0.0\n",
"\n",
" while True:\n",
"\n",
" a = q.sample_action(torch.from_numpy(s).float().unsqueeze(0), epsilon)\n",
" s_prime, r, done, info = env.step(a)\n",
" done_mask = 0.0 if done else 1.0\n",
" memory.put((s,a,r/100.0,s_prime, done_mask))\n",
" s = s_prime\n",
"\n",
" score += r\n",
" if done:\n",
" break\n",
" \n",
" if memory.size()>2000:\n",
" train(q, q_target, memory, optimizer)\n",
" scores.append(score)\n",
" # do not change lines 44-48 here, they are for marking the submission log\n",
" marking.append(score)\n",
" if n_episode%100 == 0:\n",
" print(\"marking, episode: {}, score: {:.1f}, mean_score: {:.2f}, std_score: {:.2f}\".format(\n",
" n_episode, score, np.array(marking).mean(), np.array(marking).std()))\n",
" marking = []\n",
"\n",
" # you can change this part, and print any data you like (so long as it doesn't start with \"marking\")\n",
" if n_episode%print_every==0 and n_episode!=0:\n",
" q_target.load_state_dict(q.state_dict())\n",
" print(\"episode: {}, score: {:.1f}, epsilon: {:.2f}\".format(n_episode, score, epsilon))"
],
"execution_count": 20,
"outputs": [
{
"output_type": "stream",
"text": [
"marking, episode: 0, score: 2.0, mean_score: 2.00, std_score: 0.00\n",
"episode: 5, score: 0.0, epsilon: 0.08\n",
"episode: 10, score: 0.0, epsilon: 0.08\n",
"episode: 15, score: 0.0, epsilon: 0.08\n",
"episode: 20, score: 0.0, epsilon: 0.08\n",
"episode: 25, score: 1.0, epsilon: 0.08\n",
"episode: 30, score: 0.0, epsilon: 0.08\n",
"episode: 35, score: 0.0, epsilon: 0.08\n",
"episode: 40, score: 1.0, epsilon: 0.08\n",
"episode: 45, score: 3.0, epsilon: 0.08\n",
"episode: 50, score: 1.0, epsilon: 0.08\n",
"episode: 55, score: 1.0, epsilon: 0.08\n",
"episode: 60, score: 0.0, epsilon: 0.08\n",
"episode: 65, score: 0.0, epsilon: 0.08\n",
"episode: 70, score: 0.0, epsilon: 0.08\n",
"episode: 75, score: 1.0, epsilon: 0.08\n",
"episode: 80, score: 2.0, epsilon: 0.08\n",
"episode: 85, score: 0.0, epsilon: 0.08\n",
"episode: 90, score: 0.0, epsilon: 0.08\n",
"episode: 95, score: 1.0, epsilon: 0.08\n",
"marking, episode: 100, score: 2.0, mean_score: 0.81, std_score: 1.09\n",
"episode: 100, score: 2.0, epsilon: 0.08\n",
"episode: 105, score: 1.0, epsilon: 0.08\n",
"episode: 110, score: 0.0, epsilon: 0.08\n",
"episode: 115, score: 1.0, epsilon: 0.08\n",
"episode: 120, score: 2.0, epsilon: 0.08\n",
"episode: 125, score: 0.0, epsilon: 0.08\n",
"episode: 130, score: 2.0, epsilon: 0.08\n",
"episode: 135, score: 0.0, epsilon: 0.08\n",
"episode: 140, score: 0.0, epsilon: 0.08\n",
"episode: 145, score: 0.0, epsilon: 0.08\n",
"episode: 150, score: 0.0, epsilon: 0.08\n",
"episode: 155, score: 0.0, epsilon: 0.08\n",
"episode: 160, score: 0.0, epsilon: 0.08\n",
"episode: 165, score: 2.0, epsilon: 0.08\n",
"episode: 170, score: 1.0, epsilon: 0.08\n",
"episode: 175, score: 1.0, epsilon: 0.08\n",
"episode: 180, score: 0.0, epsilon: 0.08\n",
"episode: 185, score: 2.0, epsilon: 0.08\n",
"episode: 190, score: 4.0, epsilon: 0.08\n",
"episode: 195, score: 1.0, epsilon: 0.08\n",
"marking, episode: 200, score: 0.0, mean_score: 0.79, std_score: 1.00\n",
"episode: 200, score: 0.0, epsilon: 0.08\n",
"episode: 205, score: 0.0, epsilon: 0.08\n",
"episode: 210, score: 0.0, epsilon: 0.08\n",
"episode: 215, score: 0.0, epsilon: 0.08\n",
"episode: 220, score: 4.0, epsilon: 0.08\n",
"episode: 225, score: 1.0, epsilon: 0.08\n",
"episode: 230, score: 0.0, epsilon: 0.08\n",
"episode: 235, score: 2.0, epsilon: 0.08\n",
"episode: 240, score: 0.0, epsilon: 0.08\n",
"episode: 245, score: 1.0, epsilon: 0.08\n",
"episode: 250, score: 0.0, epsilon: 0.08\n",
"episode: 255, score: 0.0, epsilon: 0.08\n",
"episode: 260, score: 2.0, epsilon: 0.08\n",
"episode: 265, score: 2.0, epsilon: 0.08\n",
"episode: 270, score: 0.0, epsilon: 0.08\n",
"episode: 275, score: 0.0, epsilon: 0.08\n",
"episode: 280, score: 1.0, epsilon: 0.08\n",
"episode: 285, score: 0.0, epsilon: 0.08\n",
"episode: 290, score: 0.0, epsilon: 0.08\n",
"episode: 295, score: 0.0, epsilon: 0.08\n",
"marking, episode: 300, score: 1.0, mean_score: 0.71, std_score: 1.03\n",
"episode: 300, score: 1.0, epsilon: 0.08\n",
"episode: 305, score: 3.0, epsilon: 0.08\n",
"episode: 310, score: 0.0, epsilon: 0.08\n",
"episode: 315, score: 0.0, epsilon: 0.08\n",
"episode: 320, score: 1.0, epsilon: 0.08\n",
"episode: 325, score: 2.0, epsilon: 0.08\n",
"episode: 330, score: 0.0, epsilon: 0.08\n",
"episode: 335, score: 0.0, epsilon: 0.08\n",
"episode: 340, score: 0.0, epsilon: 0.08\n",
"episode: 345, score: 1.0, epsilon: 0.08\n",
"episode: 350, score: 4.0, epsilon: 0.08\n",
"episode: 355, score: 2.0, epsilon: 0.08\n",
"episode: 360, score: 2.0, epsilon: 0.08\n",
"episode: 365, score: 1.0, epsilon: 0.08\n",
"episode: 370, score: 4.0, epsilon: 0.08\n",
"episode: 375, score: 0.0, epsilon: 0.08\n",
"episode: 380, score: 1.0, epsilon: 0.08\n",
"episode: 385, score: 0.0, epsilon: 0.08\n",
"episode: 390, score: 0.0, epsilon: 0.08\n",
"episode: 395, score: 0.0, epsilon: 0.08\n",
"marking, episode: 400, score: 0.0, mean_score: 0.88, std_score: 1.12\n",
"episode: 400, score: 0.0, epsilon: 0.08\n",
"episode: 405, score: 1.0, epsilon: 0.08\n",
"episode: 410, score: 1.0, epsilon: 0.08\n",
"episode: 415, score: 0.0, epsilon: 0.08\n",
"episode: 420, score: 0.0, epsilon: 0.08\n",
"episode: 425, score: 0.0, epsilon: 0.08\n",
"episode: 430, score: 0.0, epsilon: 0.08\n",
"episode: 435, score: 0.0, epsilon: 0.08\n",
"episode: 440, score: 1.0, epsilon: 0.08\n",
"episode: 445, score: 3.0, epsilon: 0.08\n",
"episode: 450, score: 2.0, epsilon: 0.08\n",
"episode: 455, score: 3.0, epsilon: 0.08\n",
"episode: 460, score: 0.0, epsilon: 0.08\n",
"episode: 465, score: 1.0, epsilon: 0.08\n",
"episode: 470, score: 0.0, epsilon: 0.08\n",
"episode: 475, score: 0.0, epsilon: 0.08\n",
"episode: 480, score: 0.0, epsilon: 0.08\n",
"episode: 485, score: 1.0, epsilon: 0.08\n",
"episode: 490, score: 1.0, epsilon: 0.08\n",
"episode: 495, score: 0.0, epsilon: 0.08\n",
"marking, episode: 500, score: 1.0, mean_score: 0.69, std_score: 0.98\n",
"episode: 500, score: 1.0, epsilon: 0.08\n",
"episode: 505, score: 0.0, epsilon: 0.08\n",
"episode: 510, score: 1.0, epsilon: 0.08\n",
"episode: 515, score: 2.0, epsilon: 0.08\n",
"episode: 520, score: 0.0, epsilon: 0.08\n",
"episode: 525, score: 2.0, epsilon: 0.08\n",
"episode: 530, score: 2.0, epsilon: 0.08\n",
"episode: 535, score: 0.0, epsilon: 0.08\n",
"episode: 540, score: 2.0, epsilon: 0.08\n",
"episode: 545, score: 1.0, epsilon: 0.08\n",
"episode: 550, score: 0.0, epsilon: 0.08\n",
"episode: 555, score: 0.0, epsilon: 0.08\n",
"episode: 560, score: 0.0, epsilon: 0.08\n",
"episode: 565, score: 0.0, epsilon: 0.08\n",
"episode: 570, score: 1.0, epsilon: 0.08\n",
"episode: 575, score: 1.0, epsilon: 0.08\n",
"episode: 580, score: 1.0, epsilon: 0.08\n",
"episode: 585, score: 3.0, epsilon: 0.08\n",
"episode: 590, score: 0.0, epsilon: 0.08\n",
"episode: 595, score: 0.0, epsilon: 0.08\n",
"marking, episode: 600, score: 1.0, mean_score: 0.73, std_score: 0.82\n",
"episode: 600, score: 1.0, epsilon: 0.08\n",
"episode: 605, score: 0.0, epsilon: 0.08\n",
"episode: 610, score: 0.0, epsilon: 0.08\n",
"episode: 615, score: 1.0, epsilon: 0.08\n",
"episode: 620, score: 1.0, epsilon: 0.08\n",
"episode: 625, score: 0.0, epsilon: 0.08\n",
"episode: 630, score: 1.0, epsilon: 0.08\n",
"episode: 635, score: 1.0, epsilon: 0.08\n",
"episode: 640, score: 0.0, epsilon: 0.08\n",
"episode: 645, score: 0.0, epsilon: 0.08\n",
"episode: 650, score: 0.0, epsilon: 0.08\n",
"episode: 655, score: 1.0, epsilon: 0.08\n",
"episode: 660, score: 1.0, epsilon: 0.08\n",
"episode: 665, score: 0.0, epsilon: 0.08\n",
"episode: 670, score: 1.0, epsilon: 0.08\n",
"episode: 675, score: 0.0, epsilon: 0.08\n",
"episode: 680, score: 1.0, epsilon: 0.08\n",
"episode: 685, score: 2.0, epsilon: 0.08\n",
"episode: 690, score: 2.0, epsilon: 0.08\n",
"episode: 695, score: 1.0, epsilon: 0.08\n",
"marking, episode: 700, score: 2.0, mean_score: 0.77, std_score: 0.96\n",
"episode: 700, score: 2.0, epsilon: 0.08\n",
"episode: 705, score: 0.0, epsilon: 0.08\n",
"episode: 710, score: 3.0, epsilon: 0.08\n",
"episode: 715, score: 0.0, epsilon: 0.08\n",
"episode: 720, score: 0.0, epsilon: 0.08\n",
"episode: 725, score: 0.0, epsilon: 0.08\n",
"episode: 730, score: 0.0, epsilon: 0.08\n",
"episode: 735, score: 2.0, epsilon: 0.08\n",
"episode: 740, score: 1.0, epsilon: 0.08\n",
"episode: 745, score: 0.0, epsilon: 0.08\n",
"episode: 750, score: 1.0, epsilon: 0.08\n",
"episode: 755, score: 0.0, epsilon: 0.08\n",
"episode: 760, score: 0.0, epsilon: 0.08\n",
"episode: 765, score: 1.0, epsilon: 0.08\n",
"episode: 770, score: 1.0, epsilon: 0.08\n",
"episode: 775, score: 0.0, epsilon: 0.08\n",
"episode: 780, score: 1.0, epsilon: 0.08\n",
"episode: 785, score: 3.0, epsilon: 0.08\n",
"episode: 790, score: 0.0, epsilon: 0.08\n",
"episode: 795, score: 0.0, epsilon: 0.08\n",
"marking, episode: 800, score: 3.0, mean_score: 0.76, std_score: 0.97\n",
"episode: 800, score: 3.0, epsilon: 0.08\n",
"episode: 805, score: 1.0, epsilon: 0.08\n",
"episode: 810, score: 1.0, epsilon: 0.08\n",
"episode: 815, score: 1.0, epsilon: 0.08\n",
"episode: 820, score: 0.0, epsilon: 0.08\n",
"episode: 825, score: 3.0, epsilon: 0.08\n",
"episode: 830, score: 1.0, epsilon: 0.08\n",
"episode: 835, score: 0.0, epsilon: 0.08\n",
"episode: 840, score: 0.0, epsilon: 0.08\n",
"episode: 845, score: 2.0, epsilon: 0.08\n",
"episode: 850, score: 0.0, epsilon: 0.08\n",
"episode: 855, score: 0.0, epsilon: 0.08\n",
"episode: 860, score: 1.0, epsilon: 0.08\n",
"episode: 865, score: 0.0, epsilon: 0.08\n",
"episode: 870, score: 0.0, epsilon: 0.08\n",
"episode: 875, score: 2.0, epsilon: 0.08\n",
"episode: 880, score: 0.0, epsilon: 0.08\n",
"episode: 885, score: 1.0, epsilon: 0.08\n",
"episode: 890, score: 2.0, epsilon: 0.08\n",
"episode: 895, score: 0.0, epsilon: 0.08\n",
"marking, episode: 900, score: 0.0, mean_score: 0.71, std_score: 0.84\n",
"episode: 900, score: 0.0, epsilon: 0.08\n",
"episode: 905, score: 1.0, epsilon: 0.08\n",
"episode: 910, score: 1.0, epsilon: 0.08\n",
"episode: 915, score: 1.0, epsilon: 0.08\n",
"episode: 920, score: 2.0, epsilon: 0.08\n",
"episode: 925, score: 1.0, epsilon: 0.08\n",
"episode: 930, score: 1.0, epsilon: 0.08\n",
"episode: 935, score: 0.0, epsilon: 0.08\n",
"episode: 940, score: 1.0, epsilon: 0.08\n",
"episode: 945, score: 0.0, epsilon: 0.08\n",
"episode: 950, score: 1.0, epsilon: 0.08\n",
"episode: 955, score: 1.0, epsilon: 0.08\n",
"episode: 960, score: 0.0, epsilon: 0.08\n",
"episode: 965, score: 2.0, epsilon: 0.08\n",
"episode: 970, score: 1.0, epsilon: 0.08\n",
"episode: 975, score: 0.0, epsilon: 0.08\n",
"episode: 980, score: 1.0, epsilon: 0.08\n",
"episode: 985, score: 2.0, epsilon: 0.08\n",
"episode: 990, score: 1.0, epsilon: 0.08\n",
"episode: 995, score: 1.0, epsilon: 0.08\n",
"marking, episode: 1000, score: 0.0, mean_score: 0.90, std_score: 0.90\n",
"episode: 1000, score: 0.0, epsilon: 0.08\n",
"episode: 1005, score: 0.0, epsilon: 0.08\n",
"episode: 1010, score: 2.0, epsilon: 0.08\n",
"episode: 1015, score: 0.0, epsilon: 0.08\n",
"episode: 1020, score: 0.0, epsilon: 0.08\n",
"episode: 1025, score: 2.0, epsilon: 0.08\n",
"episode: 1030, score: 0.0, epsilon: 0.08\n",
"episode: 1035, score: 2.0, epsilon: 0.08\n",
"episode: 1040, score: 1.0, epsilon: 0.08\n",
"episode: 1045, score: 5.0, epsilon: 0.08\n",
"episode: 1050, score: 0.0, epsilon: 0.08\n",
"episode: 1055, score: 2.0, epsilon: 0.08\n",
"episode: 1060, score: 3.0, epsilon: 0.08\n",
"episode: 1065, score: 0.0, epsilon: 0.08\n",
"episode: 1070, score: 0.0, epsilon: 0.08\n",
"episode: 1075, score: 1.0, epsilon: 0.08\n",
"episode: 1080, score: 3.0, epsilon: 0.08\n",
"episode: 1085, score: 0.0, epsilon: 0.08\n",
"episode: 1090, score: 0.0, epsilon: 0.08\n",
"episode: 1095, score: 2.0, epsilon: 0.08\n",
"marking, episode: 1100, score: 0.0, mean_score: 0.93, std_score: 1.07\n",
"episode: 1100, score: 0.0, epsilon: 0.08\n",
"episode: 1105, score: 2.0, epsilon: 0.08\n",
"episode: 1110, score: 5.0, epsilon: 0.08\n",
"episode: 1115, score: 0.0, epsilon: 0.08\n",
"episode: 1120, score: 0.0, epsilon: 0.08\n",
"episode: 1125, score: 0.0, epsilon: 0.08\n",
"episode: 1130, score: 3.0, epsilon: 0.08\n",
"episode: 1135, score: 0.0, epsilon: 0.08\n",
"episode: 1140, score: 0.0, epsilon: 0.08\n",
"episode: 1145, score: 2.0, epsilon: 0.08\n",
"episode: 1150, score: 6.0, epsilon: 0.08\n",
"episode: 1155, score: 2.0, epsilon: 0.08\n",
"episode: 1160, score: 2.0, epsilon: 0.08\n",
"episode: 1165, score: 2.0, epsilon: 0.08\n",
"episode: 1170, score: 1.0, epsilon: 0.08\n",
"episode: 1175, score: 2.0, epsilon: 0.08\n",
"episode: 1180, score: 5.0, epsilon: 0.08\n",
"episode: 1185, score: 1.0, epsilon: 0.08\n",
"episode: 1190, score: 1.0, epsilon: 0.08\n",
"episode: 1195, score: 0.0, epsilon: 0.08\n",
"marking, episode: 1200, score: 3.0, mean_score: 1.44, std_score: 1.68\n",
"episode: 1200, score: 3.0, epsilon: 0.08\n",
"episode: 1205, score: 2.0, epsilon: 0.08\n",
"episode: 1210, score: 0.0, epsilon: 0.08\n",
"episode: 1215, score: 0.0, epsilon: 0.08\n",
"episode: 1220, score: 2.0, epsilon: 0.08\n",
"episode: 1225, score: 1.0, epsilon: 0.08\n",
"episode: 1230, score: 2.0, epsilon: 0.08\n",
"episode: 1235, score: 0.0, epsilon: 0.08\n",
"episode: 1240, score: 1.0, epsilon: 0.08\n",
"episode: 1245, score: 3.0, epsilon: 0.08\n",
"episode: 1250, score: 6.0, epsilon: 0.08\n",
"episode: 1255, score: 2.0, epsilon: 0.08\n",
"episode: 1260, score: 0.0, epsilon: 0.08\n",
"episode: 1265, score: 1.0, epsilon: 0.08\n",
"episode: 1270, score: 4.0, epsilon: 0.08\n",
"episode: 1275, score: 1.0, epsilon: 0.08\n",
"episode: 1280, score: 3.0, epsilon: 0.08\n",
"episode: 1285, score: 1.0, epsilon: 0.08\n",
"episode: 1290, score: 0.0, epsilon: 0.08\n",
"episode: 1295, score: 0.0, epsilon: 0.08\n",
"marking, episode: 1300, score: 6.0, mean_score: 1.66, std_score: 1.77\n",
"episode: 1300, score: 6.0, epsilon: 0.08\n",
"episode: 1305, score: 1.0, epsilon: 0.08\n",
"episode: 1310, score: 3.0, epsilon: 0.08\n",
"episode: 1315, score: 2.0, epsilon: 0.08\n",
"episode: 1320, score: 1.0, epsilon: 0.08\n",
"episode: 1325, score: 0.0, epsilon: 0.08\n",
"episode: 1330, score: 2.0, epsilon: 0.08\n",
"episode: 1335, score: 3.0, epsilon: 0.08\n",
"episode: 1340, score: 1.0, epsilon: 0.08\n",
"episode: 1345, score: 1.0, epsilon: 0.08\n",
"episode: 1350, score: 5.0, epsilon: 0.08\n",
"episode: 1355, score: 0.0, epsilon: 0.08\n",
"episode: 1360, score: 3.0, epsilon: 0.08\n",
"episode: 1365, score: 0.0, epsilon: 0.08\n",
"episode: 1370, score: 1.0, epsilon: 0.08\n",
"episode: 1375, score: 3.0, epsilon: 0.08\n",
"episode: 1380, score: 5.0, epsilon: 0.08\n",
"episode: 1385, score: 0.0, epsilon: 0.08\n",
"episode: 1390, score: 1.0, epsilon: 0.08\n",
"episode: 1395, score: 0.0, epsilon: 0.08\n",
"marking, episode: 1400, score: 1.0, mean_score: 1.87, std_score: 1.81\n",
"episode: 1400, score: 1.0, epsilon: 0.08\n",
"episode: 1405, score: 3.0, epsilon: 0.08\n",
"episode: 1410, score: 3.0, epsilon: 0.08\n",
"episode: 1415, score: 0.0, epsilon: 0.08\n",
"episode: 1420, score: 5.0, epsilon: 0.08\n",
"episode: 1425, score: 0.0, epsilon: 0.08\n",
"episode: 1430, score: 5.0, epsilon: 0.08\n",
"episode: 1435, score: 2.0, epsilon: 0.08\n",
"episode: 1440, score: 0.0, epsilon: 0.08\n",
"episode: 1445, score: 4.0, epsilon: 0.08\n",
"episode: 1450, score: 0.0, epsilon: 0.08\n",
"episode: 1455, score: 5.0, epsilon: 0.08\n",
"episode: 1460, score: 2.0, epsilon: 0.08\n",
"episode: 1465, score: 0.0, epsilon: 0.08\n",
"episode: 1470, score: 4.0, epsilon: 0.08\n",
"episode: 1475, score: 4.0, epsilon: 0.08\n",
"episode: 1480, score: 2.0, epsilon: 0.08\n",
"episode: 1485, score: 2.0, epsilon: 0.08\n",
"episode: 1490, score: 0.0, epsilon: 0.08\n",
"episode: 1495, score: 1.0, epsilon: 0.08\n",
"marking, episode: 1500, score: 1.0, mean_score: 2.37, std_score: 2.00\n",
"episode: 1500, score: 1.0, epsilon: 0.08\n",
"episode: 1505, score: 5.0, epsilon: 0.08\n",
"episode: 1510, score: 0.0, epsilon: 0.08\n",
"episode: 1515, score: 1.0, epsilon: 0.08\n",
"episode: 1520, score: 2.0, epsilon: 0.08\n",
"episode: 1525, score: 3.0, epsilon: 0.08\n",
"episode: 1530, score: 1.0, epsilon: 0.08\n",
"episode: 1535, score: 1.0, epsilon: 0.08\n",
"episode: 1540, score: 1.0, epsilon: 0.08\n",
"episode: 1545, score: 4.0, epsilon: 0.08\n",
"episode: 1550, score: 1.0, epsilon: 0.08\n",
"episode: 1555, score: 1.0, epsilon: 0.08\n",
"episode: 1560, score: 5.0, epsilon: 0.08\n",
"episode: 1565, score: 0.0, epsilon: 0.08\n",
"episode: 1570, score: 3.0, epsilon: 0.08\n",
"episode: 1575, score: 1.0, epsilon: 0.08\n",
"episode: 1580, score: 1.0, epsilon: 0.08\n",
"episode: 1585, score: 2.0, epsilon: 0.08\n",
"episode: 1590, score: 2.0, epsilon: 0.08\n",
"episode: 1595, score: 6.0, epsilon: 0.08\n",
"marking, episode: 1600, score: 1.0, mean_score: 2.13, std_score: 1.86\n",
"episode: 1600, score: 1.0, epsilon: 0.08\n",
"episode: 1605, score: 5.0, epsilon: 0.08\n",
"episode: 1610, score: 6.0, epsilon: 0.08\n",
"episode: 1615, score: 2.0, epsilon: 0.08\n",
"episode: 1620, score: 6.0, epsilon: 0.08\n",
"episode: 1625, score: 1.0, epsilon: 0.08\n",
"episode: 1630, score: 3.0, epsilon: 0.08\n",
"episode: 1635, score: 3.0, epsilon: 0.08\n",
"episode: 1640, score: 0.0, epsilon: 0.08\n",
"episode: 1645, score: 1.0, epsilon: 0.08\n",
"episode: 1650, score: 2.0, epsilon: 0.08\n",
"episode: 1655, score: 1.0, epsilon: 0.08\n",
"episode: 1660, score: 1.0, epsilon: 0.08\n",
"episode: 1665, score: 5.0, epsilon: 0.08\n",
"episode: 1670, score: 0.0, epsilon: 0.08\n",
"episode: 1675, score: 0.0, epsilon: 0.08\n",
"episode: 1680, score: 0.0, epsilon: 0.08\n",
"episode: 1685, score: 2.0, epsilon: 0.08\n",
"episode: 1690, score: 1.0, epsilon: 0.08\n",
"episode: 1695, score: 0.0, epsilon: 0.08\n",
"marking, episode: 1700, score: 4.0, mean_score: 2.11, std_score: 1.97\n",
"episode: 1700, score: 4.0, epsilon: 0.08\n",
"episode: 1705, score: 1.0, epsilon: 0.08\n",
"episode: 1710, score: 5.0, epsilon: 0.08\n",
"episode: 1715, score: 2.0, epsilon: 0.08\n",
"episode: 1720, score: 2.0, epsilon: 0.08\n",
"episode: 1725, score: 2.0, epsilon: 0.08\n",
"episode: 1730, score: 3.0, epsilon: 0.08\n",
"episode: 1735, score: 3.0, epsilon: 0.08\n",
"episode: 1740, score: 1.0, epsilon: 0.08\n",
"episode: 1745, score: 6.0, epsilon: 0.08\n",
"episode: 1750, score: 0.0, epsilon: 0.08\n",
"episode: 1755, score: 0.0, epsilon: 0.08\n",
"episode: 1760, score: 6.0, epsilon: 0.08\n",
"episode: 1765, score: 0.0, epsilon: 0.08\n",
"episode: 1770, score: 1.0, epsilon: 0.08\n",
"episode: 1775, score: 1.0, epsilon: 0.08\n",
"episode: 1780, score: 2.0, epsilon: 0.08\n",
"episode: 1785, score: 5.0, epsilon: 0.08\n",
"episode: 1790, score: 4.0, epsilon: 0.08\n",
"episode: 1795, score: 2.0, epsilon: 0.08\n",
"marking, episode: 1800, score: 2.0, mean_score: 1.88, std_score: 1.73\n",
"episode: 1800, score: 2.0, epsilon: 0.08\n",
"episode: 1805, score: 2.0, epsilon: 0.08\n",
"episode: 1810, score: 5.0, epsilon: 0.08\n",
"episode: 1815, score: 3.0, epsilon: 0.08\n",
"episode: 1820, score: 0.0, epsilon: 0.08\n",
"episode: 1825, score: 2.0, epsilon: 0.08\n",
"episode: 1830, score: 6.0, epsilon: 0.08\n",
"episode: 1835, score: 0.0, epsilon: 0.08\n",
"episode: 1840, score: 0.0, epsilon: 0.08\n",
"episode: 1845, score: 6.0, epsilon: 0.08\n",
"episode: 1850, score: 3.0, epsilon: 0.08\n",
"episode: 1855, score: 0.0, epsilon: 0.08\n",
"episode: 1860, score: 3.0, epsilon: 0.08\n",
"episode: 1865, score: 1.0, epsilon: 0.08\n",
"episode: 1870, score: 2.0, epsilon: 0.08\n",
"episode: 1875, score: 1.0, epsilon: 0.08\n",
"episode: 1880, score: 0.0, epsilon: 0.08\n",
"episode: 1885, score: 3.0, epsilon: 0.08\n",
"episode: 1890, score: 0.0, epsilon: 0.08\n",
"episode: 1895, score: 0.0, epsilon: 0.08\n",
"marking, episode: 1900, score: 3.0, mean_score: 2.26, std_score: 1.98\n",
"episode: 1900, score: 3.0, epsilon: 0.08\n",
"episode: 1905, score: 2.0, epsilon: 0.08\n",
"episode: 1910, score: 1.0, epsilon: 0.08\n",
"episode: 1915, score: 0.0, epsilon: 0.08\n",
"episode: 1920, score: 6.0, epsilon: 0.08\n",
"episode: 1925, score: 3.0, epsilon: 0.08\n",
"episode: 1930, score: 0.0, epsilon: 0.08\n",
"episode: 1935, score: 3.0, epsilon: 0.08\n",
"episode: 1940, score: 1.0, epsilon: 0.08\n",
"episode: 1945, score: 4.0, epsilon: 0.08\n",
"episode: 1950, score: 0.0, epsilon: 0.08\n",
"episode: 1955, score: 1.0, epsilon: 0.08\n",
"episode: 1960, score: 6.0, epsilon: 0.08\n",
"episode: 1965, score: 1.0, epsilon: 0.08\n",
"episode: 1970, score: 1.0, epsilon: 0.08\n",
"episode: 1975, score: 0.0, epsilon: 0.08\n",
"episode: 1980, score: 0.0, epsilon: 0.08\n",
"episode: 1985, score: 4.0, epsilon: 0.08\n",
"episode: 1990, score: 0.0, epsilon: 0.08\n",
"episode: 1995, score: 4.0, epsilon: 0.08\n",
"marking, episode: 2000, score: 6.0, mean_score: 2.03, std_score: 1.79\n",
"episode: 2000, score: 6.0, epsilon: 0.08\n",
"episode: 2005, score: 3.0, epsilon: 0.08\n",
"episode: 2010, score: 3.0, epsilon: 0.08\n",
"episode: 2015, score: 0.0, epsilon: 0.08\n",
"episode: 2020, score: 5.0, epsilon: 0.08\n",
"episode: 2025, score: 6.0, epsilon: 0.08\n",
"episode: 2030, score: 3.0, epsilon: 0.08\n",
"episode: 2035, score: 1.0, epsilon: 0.08\n",
"episode: 2040, score: 1.0, epsilon: 0.08\n",
"episode: 2045, score: 2.0, epsilon: 0.08\n",
"episode: 2050, score: 6.0, epsilon: 0.08\n",
"episode: 2055, score: 1.0, epsilon: 0.08\n",
"episode: 2060, score: 4.0, epsilon: 0.08\n",
"episode: 2065, score: 4.0, epsilon: 0.08\n",
"episode: 2070, score: 0.0, epsilon: 0.08\n",
"episode: 2075, score: 1.0, epsilon: 0.08\n",
"episode: 2080, score: 3.0, epsilon: 0.08\n",
"episode: 2085, score: 2.0, epsilon: 0.08\n",
"episode: 2090, score: 1.0, epsilon: 0.08\n",
"episode: 2095, score: 1.0, epsilon: 0.08\n",
"marking, episode: 2100, score: 4.0, mean_score: 2.23, std_score: 2.02\n",
"episode: 2100, score: 4.0, epsilon: 0.08\n",
"episode: 2105, score: 5.0, epsilon: 0.08\n",
"episode: 2110, score: 1.0, epsilon: 0.08\n",
"episode: 2115, score: 3.0, epsilon: 0.08\n",
"episode: 2120, score: 6.0, epsilon: 0.08\n",
"episode: 2125, score: 1.0, epsilon: 0.08\n",
"episode: 2130, score: 4.0, epsilon: 0.08\n",
"episode: 2135, score: 1.0, epsilon: 0.08\n",
"episode: 2140, score: 1.0, epsilon: 0.08\n",
"episode: 2145, score: 2.0, epsilon: 0.08\n",
"episode: 2150, score: 0.0, epsilon: 0.08\n",
"episode: 2155, score: 0.0, epsilon: 0.08\n",
"episode: 2160, score: 3.0, epsilon: 0.08\n",
"episode: 2165, score: 1.0, epsilon: 0.08\n",
"episode: 2170, score: 2.0, epsilon: 0.08\n",
"episode: 2175, score: 0.0, epsilon: 0.08\n",
"episode: 2180, score: 6.0, epsilon: 0.08\n",
"episode: 2185, score: 1.0, epsilon: 0.08\n",
"episode: 2190, score: 1.0, epsilon: 0.08\n",
"episode: 2195, score: 1.0, epsilon: 0.08\n",
"marking, episode: 2200, score: 2.0, mean_score: 2.03, std_score: 2.03\n",
"episode: 2200, score: 2.0, epsilon: 0.08\n",
"episode: 2205, score: 2.0, epsilon: 0.08\n",
"episode: 2210, score: 3.0, epsilon: 0.08\n",
"episode: 2215, score: 4.0, epsilon: 0.08\n",
"episode: 2220, score: 5.0, epsilon: 0.08\n",
"episode: 2225, score: 2.0, epsilon: 0.08\n",
"episode: 2230, score: 4.0, epsilon: 0.08\n",
"episode: 2235, score: 0.0, epsilon: 0.08\n",
"episode: 2240, score: 0.0, epsilon: 0.08\n",
"episode: 2245, score: 3.0, epsilon: 0.08\n",
"episode: 2250, score: 3.0, epsilon: 0.08\n",
"episode: 2255, score: 2.0, epsilon: 0.08\n",
"episode: 2260, score: 5.0, epsilon: 0.08\n",
"episode: 2265, score: 2.0, epsilon: 0.08\n",
"episode: 2270, score: 4.0, epsilon: 0.08\n",
"episode: 2275, score: 6.0, epsilon: 0.08\n",
"episode: 2280, score: 5.0, epsilon: 0.08\n",
"episode: 2285, score: 1.0, epsilon: 0.08\n",
"episode: 2290, score: 6.0, epsilon: 0.08\n",
"episode: 2295, score: 6.0, epsilon: 0.08\n",
"marking, episode: 2300, score: 6.0, mean_score: 2.27, std_score: 2.03\n",
"episode: 2300, score: 6.0, epsilon: 0.08\n",
"episode: 2305, score: 1.0, epsilon: 0.08\n",
"episode: 2310, score: 6.0, epsilon: 0.08\n",
"episode: 2315, score: 2.0, epsilon: 0.08\n",
"episode: 2320, score: 3.0, epsilon: 0.08\n",
"episode: 2325, score: 2.0, epsilon: 0.08\n",
"episode: 2330, score: 0.0, epsilon: 0.08\n",
"episode: 2335, score: 2.0, epsilon: 0.08\n",
"episode: 2340, score: 1.0, epsilon: 0.08\n",
"episode: 2345, score: 1.0, epsilon: 0.08\n",
"episode: 2350, score: 0.0, epsilon: 0.08\n",
"episode: 2355, score: 1.0, epsilon: 0.08\n",
"episode: 2360, score: 6.0, epsilon: 0.08\n",
"episode: 2365, score: 2.0, epsilon: 0.08\n",
"episode: 2370, score: 1.0, epsilon: 0.08\n",
"episode: 2375, score: 3.0, epsilon: 0.08\n",
"episode: 2380, score: 0.0, epsilon: 0.08\n",
"episode: 2385, score: 4.0, epsilon: 0.08\n",
"episode: 2390, score: 6.0, epsilon: 0.08\n",
"episode: 2395, score: 0.0, epsilon: 0.08\n",
"marking, episode: 2400, score: 6.0, mean_score: 2.21, std_score: 2.04\n",
"episode: 2400, score: 6.0, epsilon: 0.08\n",
"episode: 2405, score: 0.0, epsilon: 0.08\n",
"episode: 2410, score: 0.0, epsilon: 0.08\n",
"episode: 2415, score: 2.0, epsilon: 0.08\n",
"episode: 2420, score: 1.0, epsilon: 0.08\n",
"episode: 2425, score: 3.0, epsilon: 0.08\n",
"episode: 2430, score: 2.0, epsilon: 0.08\n",
"episode: 2435, score: 5.0, epsilon: 0.08\n",
"episode: 2440, score: 0.0, epsilon: 0.08\n",
"episode: 2445, score: 4.0, epsilon: 0.08\n",
"episode: 2450, score: 5.0, epsilon: 0.08\n",
"episode: 2455, score: 0.0, epsilon: 0.08\n",
"episode: 2460, score: 0.0, epsilon: 0.08\n",
"episode: 2465, score: 4.0, epsilon: 0.08\n",
"episode: 2470, score: 1.0, epsilon: 0.08\n",
"episode: 2475, score: 6.0, epsilon: 0.08\n",
"episode: 2480, score: 2.0, epsilon: 0.08\n",
"episode: 2485, score: 5.0, epsilon: 0.08\n",
"episode: 2490, score: 3.0, epsilon: 0.08\n",
"episode: 2495, score: 3.0, epsilon: 0.08\n",
"marking, episode: 2500, score: 3.0, mean_score: 2.40, std_score: 2.11\n",
"episode: 2500, score: 3.0, epsilon: 0.08\n",
"episode: 2505, score: 0.0, epsilon: 0.08\n",
"episode: 2510, score: 0.0, epsilon: 0.08\n",
"episode: 2515, score: 6.0, epsilon: 0.08\n",
"episode: 2520, score: 2.0, epsilon: 0.08\n",
"episode: 2525, score: 0.0, epsilon: 0.08\n",
"episode: 2530, score: 6.0, epsilon: 0.08\n",
"episode: 2535, score: 0.0, epsilon: 0.08\n",
"episode: 2540, score: 0.0, epsilon: 0.08\n",
"episode: 2545, score: 1.0, epsilon: 0.08\n",
"episode: 2550, score: 0.0, epsilon: 0.08\n",
"episode: 2555, score: 0.0, epsilon: 0.08\n",
"episode: 2560, score: 5.0, epsilon: 0.08\n",
"episode: 2565, score: 0.0, epsilon: 0.08\n",
"episode: 2570, score: 6.0, epsilon: 0.08\n",
"episode: 2575, score: 2.0, epsilon: 0.08\n",
"episode: 2580, score: 0.0, epsilon: 0.08\n",
"episode: 2585, score: 6.0, epsilon: 0.08\n",
"episode: 2590, score: 0.0, epsilon: 0.08\n",
"episode: 2595, score: 6.0, epsilon: 0.08\n",
"marking, episode: 2600, score: 0.0, mean_score: 2.31, std_score: 2.16\n",
"episode: 2600, score: 0.0, epsilon: 0.08\n",
"episode: 2605, score: 0.0, epsilon: 0.08\n",
"episode: 2610, score: 0.0, epsilon: 0.08\n",
"episode: 2615, score: 0.0, epsilon: 0.08\n",
"episode: 2620, score: 1.0, epsilon: 0.08\n",
"episode: 2625, score: 0.0, epsilon: 0.08\n",
"episode: 2630, score: 4.0, epsilon: 0.08\n",
"episode: 2635, score: 1.0, epsilon: 0.08\n",
"episode: 2640, score: 1.0, epsilon: 0.08\n",
"episode: 2645, score: 2.0, epsilon: 0.08\n",
"episode: 2650, score: 3.0, epsilon: 0.08\n",
"episode: 2655, score: 0.0, epsilon: 0.08\n",
"episode: 2660, score: 0.0, epsilon: 0.08\n",
"episode: 2665, score: 1.0, epsilon: 0.08\n",
"episode: 2670, score: 5.0, epsilon: 0.08\n",
"episode: 2675, score: 3.0, epsilon: 0.08\n",
"episode: 2680, score: 6.0, epsilon: 0.08\n",
"episode: 2685, score: 3.0, epsilon: 0.08\n",
"episode: 2690, score: 3.0, epsilon: 0.08\n",
"episode: 2695, score: 1.0, epsilon: 0.08\n",
"marking, episode: 2700, score: 4.0, mean_score: 1.81, std_score: 1.72\n",
"episode: 2700, score: 4.0, epsilon: 0.08\n",
"episode: 2705, score: 1.0, epsilon: 0.08\n",
"episode: 2710, score: 0.0, epsilon: 0.08\n",
"episode: 2715, score: 6.0, epsilon: 0.08\n",
"episode: 2720, score: 2.0, epsilon: 0.08\n",
"episode: 2725, score: 0.0, epsilon: 0.08\n",
"episode: 2730, score: 0.0, epsilon: 0.08\n",
"episode: 2735, score: 0.0, epsilon: 0.08\n",
"episode: 2740, score: 1.0, epsilon: 0.08\n",
"episode: 2745, score: 0.0, epsilon: 0.08\n",
"episode: 2750, score: 0.0, epsilon: 0.08\n",
"episode: 2755, score: 4.0, epsilon: 0.08\n",
"episode: 2760, score: 1.0, epsilon: 0.08\n",
"episode: 2765, score: 0.0, epsilon: 0.08\n",
"episode: 2770, score: 3.0, epsilon: 0.08\n",
"episode: 2775, score: 0.0, epsilon: 0.08\n",
"episode: 2780, score: 2.0, epsilon: 0.08\n",
"episode: 2785, score: 0.0, epsilon: 0.08\n",
"episode: 2790, score: 0.0, epsilon: 0.08\n",
"episode: 2795, score: 3.0, epsilon: 0.08\n",
"marking, episode: 2800, score: 0.0, mean_score: 2.11, std_score: 2.05\n",
"episode: 2800, score: 0.0, epsilon: 0.08\n",
"episode: 2805, score: 5.0, epsilon: 0.08\n",
"episode: 2810, score: 3.0, epsilon: 0.08\n",
"episode: 2815, score: 3.0, epsilon: 0.08\n",
"episode: 2820, score: 2.0, epsilon: 0.08\n",
"episode: 2825, score: 1.0, epsilon: 0.08\n",
"episode: 2830, score: 1.0, epsilon: 0.08\n",
"episode: 2835, score: 2.0, epsilon: 0.08\n",
"episode: 2840, score: 2.0, epsilon: 0.08\n",
"episode: 2845, score: 2.0, epsilon: 0.08\n",
"episode: 2850, score: 5.0, epsilon: 0.08\n",
"episode: 2855, score: 2.0, epsilon: 0.08\n",
"episode: 2860, score: 5.0, epsilon: 0.08\n",
"episode: 2865, score: 1.0, epsilon: 0.08\n",
"episode: 2870, score: 1.0, epsilon: 0.08\n",
"episode: 2875, score: 2.0, epsilon: 0.08\n",
"episode: 2880, score: 3.0, epsilon: 0.08\n",
"episode: 2885, score: 3.0, epsilon: 0.08\n",
"episode: 2890, score: 5.0, epsilon: 0.08\n",
"episode: 2895, score: 5.0, epsilon: 0.08\n",
"marking, episode: 2900, score: 1.0, mean_score: 2.44, std_score: 1.86\n",
"episode: 2900, score: 1.0, epsilon: 0.08\n",
"episode: 2905, score: 0.0, epsilon: 0.08\n",
"episode: 2910, score: 1.0, epsilon: 0.08\n",
"episode: 2915, score: 1.0, epsilon: 0.08\n",
"episode: 2920, score: 0.0, epsilon: 0.08\n",
"episode: 2925, score: 1.0, epsilon: 0.08\n",
"episode: 2930, score: 0.0, epsilon: 0.08\n",
"episode: 2935, score: 3.0, epsilon: 0.08\n",
"episode: 2940, score: 2.0, epsilon: 0.08\n",
"episode: 2945, score: 6.0, epsilon: 0.08\n",
"episode: 2950, score: 2.0, epsilon: 0.08\n",
"episode: 2955, score: 3.0, epsilon: 0.08\n",
"episode: 2960, score: 0.0, epsilon: 0.08\n",
"episode: 2965, score: 3.0, epsilon: 0.08\n",
"episode: 2970, score: 3.0, epsilon: 0.08\n",
"episode: 2975, score: 2.0, epsilon: 0.08\n",
"episode: 2980, score: 1.0, epsilon: 0.08\n",
"episode: 2985, score: 4.0, epsilon: 0.08\n",
"episode: 2990, score: 1.0, epsilon: 0.08\n",
"episode: 2995, score: 0.0, epsilon: 0.08\n",
"marking, episode: 3000, score: 2.0, mean_score: 2.40, std_score: 2.05\n",
"episode: 3000, score: 2.0, epsilon: 0.08\n",
"episode: 3005, score: 2.0, epsilon: 0.08\n",
"episode: 3010, score: 0.0, epsilon: 0.08\n",
"episode: 3015, score: 3.0, epsilon: 0.08\n",
"episode: 3020, score: 0.0, epsilon: 0.08\n",
"episode: 3025, score: 3.0, epsilon: 0.08\n",
"episode: 3030, score: 0.0, epsilon: 0.08\n",
"episode: 3035, score: 2.0, epsilon: 0.08\n",
"episode: 3040, score: 0.0, epsilon: 0.08\n",
"episode: 3045, score: 2.0, epsilon: 0.08\n",
"episode: 3050, score: 3.0, epsilon: 0.08\n",
"episode: 3055, score: 5.0, epsilon: 0.08\n",
"episode: 3060, score: 0.0, epsilon: 0.08\n",
"episode: 3065, score: 1.0, epsilon: 0.08\n",
"episode: 3070, score: 6.0, epsilon: 0.08\n",
"episode: 3075, score: 1.0, epsilon: 0.08\n",
"episode: 3080, score: 0.0, epsilon: 0.08\n",
"episode: 3085, score: 6.0, epsilon: 0.08\n",
"episode: 3090, score: 3.0, epsilon: 0.08\n",
"episode: 3095, score: 0.0, epsilon: 0.08\n",
"marking, episode: 3100, score: 1.0, mean_score: 2.22, std_score: 2.04\n",
"episode: 3100, score: 1.0, epsilon: 0.08\n",
"episode: 3105, score: 1.0, epsilon: 0.08\n",
"episode: 3110, score: 2.0, epsilon: 0.08\n",
"episode: 3115, score: 1.0, epsilon: 0.08\n",
"episode: 3120, score: 0.0, epsilon: 0.08\n",
"episode: 3125, score: 1.0, epsilon: 0.08\n",
"episode: 3130, score: 4.0, epsilon: 0.08\n",
"episode: 3135, score: 1.0, epsilon: 0.08\n",
"episode: 3140, score: 0.0, epsilon: 0.08\n",
"episode: 3145, score: 2.0, epsilon: 0.08\n",
"episode: 3150, score: 0.0, epsilon: 0.08\n",
"episode: 3155, score: 1.0, epsilon: 0.08\n",
"episode: 3160, score: 1.0, epsilon: 0.08\n",
"episode: 3165, score: 0.0, epsilon: 0.08\n",
"episode: 3170, score: 0.0, epsilon: 0.08\n",
"episode: 3175, score: 3.0, epsilon: 0.08\n",
"episode: 3180, score: 0.0, epsilon: 0.08\n",
"episode: 3185, score: 5.0, epsilon: 0.08\n",
"episode: 3190, score: 1.0, epsilon: 0.08\n",
"episode: 3195, score: 6.0, epsilon: 0.08\n",
"marking, episode: 3200, score: 5.0, mean_score: 1.90, std_score: 2.07\n",
"episode: 3200, score: 5.0, epsilon: 0.08\n",
"episode: 3205, score: 0.0, epsilon: 0.08\n",
"episode: 3210, score: 2.0, epsilon: 0.08\n",
"episode: 3215, score: 1.0, epsilon: 0.08\n",
"episode: 3220, score: 1.0, epsilon: 0.08\n",
"episode: 3225, score: 6.0, epsilon: 0.08\n",
"episode: 3230, score: 0.0, epsilon: 0.08\n",
"episode: 3235, score: 3.0, epsilon: 0.08\n",
"episode: 3240, score: 6.0, epsilon: 0.08\n",
"episode: 3245, score: 0.0, epsilon: 0.08\n",
"episode: 3250, score: 0.0, epsilon: 0.08\n",
"episode: 3255, score: 1.0, epsilon: 0.08\n",
"episode: 3260, score: 1.0, epsilon: 0.08\n",
"episode: 3265, score: 1.0, epsilon: 0.08\n",
"episode: 3270, score: 0.0, epsilon: 0.08\n",
"episode: 3275, score: 5.0, epsilon: 0.08\n",
"episode: 3280, score: 2.0, epsilon: 0.08\n",
"episode: 3285, score: 4.0, epsilon: 0.08\n",
"episode: 3290, score: 6.0, epsilon: 0.08\n",
"episode: 3295, score: 0.0, epsilon: 0.08\n",
"marking, episode: 3300, score: 6.0, mean_score: 1.91, std_score: 2.04\n",
"episode: 3300, score: 6.0, epsilon: 0.08\n",
"episode: 3305, score: 6.0, epsilon: 0.08\n",
"episode: 3310, score: 6.0, epsilon: 0.08\n",
"episode: 3315, score: 0.0, epsilon: 0.08\n",
"episode: 3320, score: 0.0, epsilon: 0.08\n",
"episode: 3325, score: 1.0, epsilon: 0.08\n",
"episode: 3330, score: 3.0, epsilon: 0.08\n",
"episode: 3335, score: 2.0, epsilon: 0.08\n",
"episode: 3340, score: 2.0, epsilon: 0.08\n",
"episode: 3345, score: 0.0, epsilon: 0.08\n",
"episode: 3350, score: 3.0, epsilon: 0.08\n",
"episode: 3355, score: 2.0, epsilon: 0.08\n",
"episode: 3360, score: 3.0, epsilon: 0.08\n",
"episode: 3365, score: 2.0, epsilon: 0.08\n",
"episode: 3370, score: 3.0, epsilon: 0.08\n",
"episode: 3375, score: 4.0, epsilon: 0.08\n",
"episode: 3380, score: 6.0, epsilon: 0.08\n",
"episode: 3385, score: 1.0, epsilon: 0.08\n",
"episode: 3390, score: 1.0, epsilon: 0.08\n",
"episode: 3395, score: 0.0, epsilon: 0.08\n",
"marking, episode: 3400, score: 6.0, mean_score: 2.17, std_score: 2.13\n",
"episode: 3400, score: 6.0, epsilon: 0.08\n",
"episode: 3405, score: 1.0, epsilon: 0.08\n",
"episode: 3410, score: 6.0, epsilon: 0.08\n",
"episode: 3415, score: 1.0, epsilon: 0.08\n",
"episode: 3420, score: 3.0, epsilon: 0.08\n",
"episode: 3425, score: 3.0, epsilon: 0.08\n",
"episode: 3430, score: 1.0, epsilon: 0.08\n",
"episode: 3435, score: 2.0, epsilon: 0.08\n",
"episode: 3440, score: 0.0, epsilon: 0.08\n",
"episode: 3445, score: 3.0, epsilon: 0.08\n",
"episode: 3450, score: 1.0, epsilon: 0.08\n",
"episode: 3455, score: 5.0, epsilon: 0.08\n",
"episode: 3460, score: 2.0, epsilon: 0.08\n",
"episode: 3465, score: 0.0, epsilon: 0.08\n",
"episode: 3470, score: 1.0, epsilon: 0.08\n",
"episode: 3475, score: 6.0, epsilon: 0.08\n",
"episode: 3480, score: 2.0, epsilon: 0.08\n",
"episode: 3485, score: 1.0, epsilon: 0.08\n",
"episode: 3490, score: 0.0, epsilon: 0.08\n",
"episode: 3495, score: 4.0, epsilon: 0.08\n",
"marking, episode: 3500, score: 0.0, mean_score: 2.18, std_score: 1.97\n",
"episode: 3500, score: 0.0, epsilon: 0.08\n",
"episode: 3505, score: 0.0, epsilon: 0.08\n",
"episode: 3510, score: 3.0, epsilon: 0.08\n",
"episode: 3515, score: 2.0, epsilon: 0.08\n",
"episode: 3520, score: 0.0, epsilon: 0.08\n",
"episode: 3525, score: 1.0, epsilon: 0.08\n",
"episode: 3530, score: 2.0, epsilon: 0.08\n",
"episode: 3535, score: 2.0, epsilon: 0.08\n",
"episode: 3540, score: 6.0, epsilon: 0.08\n",
"episode: 3545, score: 0.0, epsilon: 0.08\n",
"episode: 3550, score: 4.0, epsilon: 0.08\n",
"episode: 3555, score: 1.0, epsilon: 0.08\n",
"episode: 3560, score: 1.0, epsilon: 0.08\n",
"episode: 3565, score: 1.0, epsilon: 0.08\n",
"episode: 3570, score: 0.0, epsilon: 0.08\n",
"episode: 3575, score: 6.0, epsilon: 0.08\n",
"episode: 3580, score: 6.0, epsilon: 0.08\n",
"episode: 3585, score: 2.0, epsilon: 0.08\n",
"episode: 3590, score: 1.0, epsilon: 0.08\n",
"episode: 3595, score: 0.0, epsilon: 0.08\n",
"marking, episode: 3600, score: 1.0, mean_score: 2.49, std_score: 2.05\n",
"episode: 3600, score: 1.0, epsilon: 0.08\n",
"episode: 3605, score: 6.0, epsilon: 0.08\n",
"episode: 3610, score: 4.0, epsilon: 0.08\n",
"episode: 3615, score: 6.0, epsilon: 0.08\n",
"episode: 3620, score: 6.0, epsilon: 0.08\n",
"episode: 3625, score: 0.0, epsilon: 0.08\n",
"episode: 3630, score: 3.0, epsilon: 0.08\n",
"episode: 3635, score: 0.0, epsilon: 0.08\n",
"episode: 3640, score: 0.0, epsilon: 0.08\n",
"episode: 3645, score: 0.0, epsilon: 0.08\n",
"episode: 3650, score: 5.0, epsilon: 0.08\n",
"episode: 3655, score: 0.0, epsilon: 0.08\n",
"episode: 3660, score: 0.0, epsilon: 0.08\n",
"episode: 3665, score: 0.0, epsilon: 0.08\n",
"episode: 3670, score: 0.0, epsilon: 0.08\n",
"episode: 3675, score: 3.0, epsilon: 0.08\n",
"episode: 3680, score: 1.0, epsilon: 0.08\n",
"episode: 3685, score: 0.0, epsilon: 0.08\n",
"episode: 3690, score: 0.0, epsilon: 0.08\n",
"episode: 3695, score: 2.0, epsilon: 0.08\n",
"marking, episode: 3700, score: 2.0, mean_score: 2.33, std_score: 2.06\n",
"episode: 3700, score: 2.0, epsilon: 0.08\n",
"episode: 3705, score: 4.0, epsilon: 0.08\n",
"episode: 3710, score: 2.0, epsilon: 0.08\n",
"episode: 3715, score: 1.0, epsilon: 0.08\n",
"episode: 3720, score: 2.0, epsilon: 0.08\n",
"episode: 3725, score: 2.0, epsilon: 0.08\n",
"episode: 3730, score: 0.0, epsilon: 0.08\n",
"episode: 3735, score: 0.0, epsilon: 0.08\n",
"episode: 3740, score: 0.0, epsilon: 0.08\n",
"episode: 3745, score: 6.0, epsilon: 0.08\n",
"episode: 3750, score: 0.0, epsilon: 0.08\n",
"episode: 3755, score: 1.0, epsilon: 0.08\n",
"episode: 3760, score: 3.0, epsilon: 0.08\n",
"episode: 3765, score: 6.0, epsilon: 0.08\n",
"episode: 3770, score: 2.0, epsilon: 0.08\n",
"episode: 3775, score: 2.0, epsilon: 0.08\n",
"episode: 3780, score: 3.0, epsilon: 0.08\n",
"episode: 3785, score: 1.0, epsilon: 0.08\n",
"episode: 3790, score: 0.0, epsilon: 0.08\n",
"episode: 3795, score: 6.0, epsilon: 0.08\n",
"marking, episode: 3800, score: 2.0, mean_score: 2.37, std_score: 2.11\n",
"episode: 3800, score: 2.0, epsilon: 0.08\n",
"episode: 3805, score: 0.0, epsilon: 0.08\n",
"episode: 3810, score: 0.0, epsilon: 0.08\n",
"episode: 3815, score: 4.0, epsilon: 0.08\n",
"episode: 3820, score: 6.0, epsilon: 0.08\n",
"episode: 3825, score: 3.0, epsilon: 0.08\n",
"episode: 3830, score: 3.0, epsilon: 0.08\n",
"episode: 3835, score: 5.0, epsilon: 0.08\n",
"episode: 3840, score: 0.0, epsilon: 0.08\n",
"episode: 3845, score: 0.0, epsilon: 0.08\n",
"episode: 3850, score: 1.0, epsilon: 0.08\n",
"episode: 3855, score: 6.0, epsilon: 0.08\n",
"episode: 3860, score: 0.0, epsilon: 0.08\n",
"episode: 3865, score: 4.0, epsilon: 0.08\n",
"episode: 3870, score: 1.0, epsilon: 0.08\n",
"episode: 3875, score: 0.0, epsilon: 0.08\n",
"episode: 3880, score: 4.0, epsilon: 0.08\n",
"episode: 3885, score: 1.0, epsilon: 0.08\n",
"episode: 3890, score: 3.0, epsilon: 0.08\n",
"episode: 3895, score: 1.0, epsilon: 0.08\n",
"marking, episode: 3900, score: 3.0, mean_score: 2.16, std_score: 1.94\n",
"episode: 3900, score: 3.0, epsilon: 0.08\n",
"episode: 3905, score: 1.0, epsilon: 0.08\n",
"episode: 3910, score: 1.0, epsilon: 0.08\n",
"episode: 3915, score: 1.0, epsilon: 0.08\n",
"episode: 3920, score: 2.0, epsilon: 0.08\n",
"episode: 3925, score: 1.0, epsilon: 0.08\n",
"episode: 3930, score: 1.0, epsilon: 0.08\n",
"episode: 3935, score: 1.0, epsilon: 0.08\n",
"episode: 3940, score: 6.0, epsilon: 0.08\n",
"episode: 3945, score: 2.0, epsilon: 0.08\n",
"episode: 3950, score: 1.0, epsilon: 0.08\n",
"episode: 3955, score: 2.0, epsilon: 0.08\n",
"episode: 3960, score: 6.0, epsilon: 0.08\n",
"episode: 3965, score: 2.0, epsilon: 0.08\n",
"episode: 3970, score: 1.0, epsilon: 0.08\n",
"episode: 3975, score: 6.0, epsilon: 0.08\n",
"episode: 3980, score: 3.0, epsilon: 0.08\n",
"episode: 3985, score: 0.0, epsilon: 0.08\n",
"episode: 3990, score: 0.0, epsilon: 0.08\n",
"episode: 3995, score: 6.0, epsilon: 0.08\n",
"marking, episode: 4000, score: 4.0, mean_score: 2.64, std_score: 2.19\n",
"episode: 4000, score: 4.0, epsilon: 0.08\n",
"episode: 4005, score: 6.0, epsilon: 0.08\n",
"episode: 4010, score: 1.0, epsilon: 0.08\n",
"episode: 4015, score: 2.0, epsilon: 0.08\n",
"episode: 4020, score: 0.0, epsilon: 0.08\n",
"episode: 4025, score: 1.0, epsilon: 0.08\n",
"episode: 4030, score: 5.0, epsilon: 0.08\n",
"episode: 4035, score: 1.0, epsilon: 0.08\n",
"episode: 4040, score: 2.0, epsilon: 0.08\n",
"episode: 4045, score: 0.0, epsilon: 0.08\n",
"episode: 4050, score: 0.0, epsilon: 0.08\n",
"episode: 4055, score: 6.0, epsilon: 0.08\n",
"episode: 4060, score: 3.0, epsilon: 0.08\n",
"episode: 4065, score: 1.0, epsilon: 0.08\n",
"episode: 4070, score: 0.0, epsilon: 0.08\n",
"episode: 4075, score: 0.0, epsilon: 0.08\n",
"episode: 4080, score: 2.0, epsilon: 0.08\n",
"episode: 4085, score: 1.0, epsilon: 0.08\n",
"episode: 4090, score: 2.0, epsilon: 0.08\n",
"episode: 4095, score: 1.0, epsilon: 0.08\n",
"marking, episode: 4100, score: 6.0, mean_score: 2.20, std_score: 1.89\n",
"episode: 4100, score: 6.0, epsilon: 0.08\n",
"episode: 4105, score: 4.0, epsilon: 0.08\n",
"episode: 4110, score: 6.0, epsilon: 0.08\n",
"episode: 4115, score: 2.0, epsilon: 0.08\n",
"episode: 4120, score: 3.0, epsilon: 0.08\n",
"episode: 4125, score: 2.0, epsilon: 0.08\n",
"episode: 4130, score: 0.0, epsilon: 0.08\n",
"episode: 4135, score: 2.0, epsilon: 0.08\n",
"episode: 4140, score: 0.0, epsilon: 0.08\n",
"episode: 4145, score: 0.0, epsilon: 0.08\n",
"episode: 4150, score: 1.0, epsilon: 0.08\n",
"episode: 4155, score: 4.0, epsilon: 0.08\n",
"episode: 4160, score: 3.0, epsilon: 0.08\n",
"episode: 4165, score: 6.0, epsilon: 0.08\n",
"episode: 4170, score: 0.0, epsilon: 0.08\n",
"episode: 4175, score: 2.0, epsilon: 0.08\n",
"episode: 4180, score: 4.0, epsilon: 0.08\n",
"episode: 4185, score: 3.0, epsilon: 0.08\n",
"episode: 4190, score: 6.0, epsilon: 0.08\n",
"episode: 4195, score: 4.0, epsilon: 0.08\n",
"marking, episode: 4200, score: 1.0, mean_score: 2.55, std_score: 2.17\n",
"episode: 4200, score: 1.0, epsilon: 0.08\n",
"episode: 4205, score: 1.0, epsilon: 0.08\n",
"episode: 4210, score: 0.0, epsilon: 0.08\n",
"episode: 4215, score: 0.0, epsilon: 0.08\n",
"episode: 4220, score: 2.0, epsilon: 0.08\n",
"episode: 4225, score: 6.0, epsilon: 0.08\n",
"episode: 4230, score: 1.0, epsilon: 0.08\n",
"episode: 4235, score: 4.0, epsilon: 0.08\n",
"episode: 4240, score: 4.0, epsilon: 0.08\n",
"episode: 4245, score: 1.0, epsilon: 0.08\n",
"episode: 4250, score: 1.0, epsilon: 0.08\n",
"episode: 4255, score: 0.0, epsilon: 0.08\n",
"episode: 4260, score: 0.0, epsilon: 0.08\n",
"episode: 4265, score: 2.0, epsilon: 0.08\n",
"episode: 4270, score: 6.0, epsilon: 0.08\n",
"episode: 4275, score: 2.0, epsilon: 0.08\n",
"episode: 4280, score: 1.0, epsilon: 0.08\n",
"episode: 4285, score: 0.0, epsilon: 0.08\n",
"episode: 4290, score: 4.0, epsilon: 0.08\n",
"episode: 4295, score: 3.0, epsilon: 0.08\n",
"marking, episode: 4300, score: 5.0, mean_score: 2.40, std_score: 2.05\n",
"episode: 4300, score: 5.0, epsilon: 0.08\n",
"episode: 4305, score: 3.0, epsilon: 0.08\n",
"episode: 4310, score: 0.0, epsilon: 0.08\n",
"episode: 4315, score: 0.0, epsilon: 0.08\n",
"episode: 4320, score: 6.0, epsilon: 0.08\n",
"episode: 4325, score: 2.0, epsilon: 0.08\n",
"episode: 4330, score: 0.0, epsilon: 0.08\n",
"episode: 4335, score: 2.0, epsilon: 0.08\n",
"episode: 4340, score: 0.0, epsilon: 0.08\n",
"episode: 4345, score: 1.0, epsilon: 0.08\n",
"episode: 4350, score: 0.0, epsilon: 0.08\n",
"episode: 4355, score: 3.0, epsilon: 0.08\n",
"episode: 4360, score: 6.0, epsilon: 0.08\n",
"episode: 4365, score: 6.0, epsilon: 0.08\n",
"episode: 4370, score: 0.0, epsilon: 0.08\n",
"episode: 4375, score: 6.0, epsilon: 0.08\n",
"episode: 4380, score: 2.0, epsilon: 0.08\n",
"episode: 4385, score: 6.0, epsilon: 0.08\n",
"episode: 4390, score: 3.0, epsilon: 0.08\n",
"episode: 4395, score: 1.0, epsilon: 0.08\n",
"marking, episode: 4400, score: 1.0, mean_score: 2.31, std_score: 1.97\n",
"episode: 4400, score: 1.0, epsilon: 0.08\n",
"episode: 4405, score: 1.0, epsilon: 0.08\n",
"episode: 4410, score: 1.0, epsilon: 0.08\n",
"episode: 4415, score: 2.0, epsilon: 0.08\n",
"episode: 4420, score: 0.0, epsilon: 0.08\n",
"episode: 4425, score: 1.0, epsilon: 0.08\n",
"episode: 4430, score: 6.0, epsilon: 0.08\n",
"episode: 4435, score: 0.0, epsilon: 0.08\n",
"episode: 4440, score: 2.0, epsilon: 0.08\n",
"episode: 4445, score: 2.0, epsilon: 0.08\n",
"episode: 4450, score: 0.0, epsilon: 0.08\n",
"episode: 4455, score: 2.0, epsilon: 0.08\n",
"episode: 4460, score: 6.0, epsilon: 0.08\n",
"episode: 4465, score: 6.0, epsilon: 0.08\n",
"episode: 4470, score: 4.0, epsilon: 0.08\n",
"episode: 4475, score: 1.0, epsilon: 0.08\n",
"episode: 4480, score: 1.0, epsilon: 0.08\n",
"episode: 4485, score: 0.0, epsilon: 0.08\n",
"episode: 4490, score: 5.0, epsilon: 0.08\n",
"episode: 4495, score: 3.0, epsilon: 0.08\n",
"marking, episode: 4500, score: 1.0, mean_score: 2.48, std_score: 2.06\n",
"episode: 4500, score: 1.0, epsilon: 0.08\n",
"episode: 4505, score: 2.0, epsilon: 0.08\n",
"episode: 4510, score: 6.0, epsilon: 0.08\n",
"episode: 4515, score: 3.0, epsilon: 0.08\n",
"episode: 4520, score: 1.0, epsilon: 0.08\n",
"episode: 4525, score: 3.0, epsilon: 0.08\n",
"episode: 4530, score: 5.0, epsilon: 0.08\n",
"episode: 4535, score: 3.0, epsilon: 0.08\n",
"episode: 4540, score: 4.0, epsilon: 0.08\n",
"episode: 4545, score: 0.0, epsilon: 0.08\n",
"episode: 4550, score: 1.0, epsilon: 0.08\n",
"episode: 4555, score: 6.0, epsilon: 0.08\n",
"episode: 4560, score: 2.0, epsilon: 0.08\n",
"episode: 4565, score: 6.0, epsilon: 0.08\n",
"episode: 4570, score: 1.0, epsilon: 0.08\n",
"episode: 4575, score: 5.0, epsilon: 0.08\n",
"episode: 4580, score: 6.0, epsilon: 0.08\n",
"episode: 4585, score: 6.0, epsilon: 0.08\n",
"episode: 4590, score: 6.0, epsilon: 0.08\n",
"episode: 4595, score: 1.0, epsilon: 0.08\n",
"marking, episode: 4600, score: 6.0, mean_score: 2.18, std_score: 2.05\n",
"episode: 4600, score: 6.0, epsilon: 0.08\n",
"episode: 4605, score: 1.0, epsilon: 0.08\n",
"episode: 4610, score: 2.0, epsilon: 0.08\n",
"episode: 4615, score: 1.0, epsilon: 0.08\n",
"episode: 4620, score: 0.0, epsilon: 0.08\n",
"episode: 4625, score: 2.0, epsilon: 0.08\n",
"episode: 4630, score: 0.0, epsilon: 0.08\n",
"episode: 4635, score: 0.0, epsilon: 0.08\n",
"episode: 4640, score: 6.0, epsilon: 0.08\n",
"episode: 4645, score: 5.0, epsilon: 0.08\n",
"episode: 4650, score: 5.0, epsilon: 0.08\n",
"episode: 4655, score: 6.0, epsilon: 0.08\n",
"episode: 4660, score: 6.0, epsilon: 0.08\n",
"episode: 4665, score: 2.0, epsilon: 0.08\n",
"episode: 4670, score: 4.0, epsilon: 0.08\n",
"episode: 4675, score: 5.0, epsilon: 0.08\n",
"episode: 4680, score: 0.0, epsilon: 0.08\n",
"episode: 4685, score: 4.0, epsilon: 0.08\n",
"episode: 4690, score: 0.0, epsilon: 0.08\n",
"episode: 4695, score: 4.0, epsilon: 0.08\n",
"marking, episode: 4700, score: 2.0, mean_score: 2.02, std_score: 2.02\n",
"episode: 4700, score: 2.0, epsilon: 0.08\n",
"episode: 4705, score: 1.0, epsilon: 0.08\n",
"episode: 4710, score: 5.0, epsilon: 0.08\n",
"episode: 4715, score: 0.0, epsilon: 0.08\n",
"episode: 4720, score: 4.0, epsilon: 0.08\n",
"episode: 4725, score: 6.0, epsilon: 0.08\n",
"episode: 4730, score: 1.0, epsilon: 0.08\n",
"episode: 4735, score: 1.0, epsilon: 0.08\n",
"episode: 4740, score: 3.0, epsilon: 0.08\n",
"episode: 4745, score: 0.0, epsilon: 0.08\n",
"episode: 4750, score: 3.0, epsilon: 0.08\n",
"episode: 4755, score: 1.0, epsilon: 0.08\n",
"episode: 4760, score: 3.0, epsilon: 0.08\n",
"episode: 4765, score: 1.0, epsilon: 0.08\n",
"episode: 4770, score: 1.0, epsilon: 0.08\n",
"episode: 4775, score: 1.0, epsilon: 0.08\n",
"episode: 4780, score: 1.0, epsilon: 0.08\n",
"episode: 4785, score: 3.0, epsilon: 0.08\n",
"episode: 4790, score: 4.0, epsilon: 0.08\n",
"episode: 4795, score: 4.0, epsilon: 0.08\n",
"marking, episode: 4800, score: 0.0, mean_score: 1.92, std_score: 1.82\n",
"episode: 4800, score: 0.0, epsilon: 0.08\n",
"episode: 4805, score: 6.0, epsilon: 0.08\n",
"episode: 4810, score: 5.0, epsilon: 0.08\n",
"episode: 4815, score: 0.0, epsilon: 0.08\n",
"episode: 4820, score: 6.0, epsilon: 0.08\n",
"episode: 4825, score: 2.0, epsilon: 0.08\n",
"episode: 4830, score: 6.0, epsilon: 0.08\n",
"episode: 4835, score: 0.0, epsilon: 0.08\n",
"episode: 4840, score: 6.0, epsilon: 0.08\n",
"episode: 4845, score: 0.0, epsilon: 0.08\n",
"episode: 4850, score: 5.0, epsilon: 0.08\n",
"episode: 4855, score: 3.0, epsilon: 0.08\n",
"episode: 4860, score: 0.0, epsilon: 0.08\n",
"episode: 4865, score: 0.0, epsilon: 0.08\n",
"episode: 4870, score: 6.0, epsilon: 0.08\n",
"episode: 4875, score: 1.0, epsilon: 0.08\n",
"episode: 4880, score: 3.0, epsilon: 0.08\n",
"episode: 4885, score: 6.0, epsilon: 0.08\n",
"episode: 4890, score: 0.0, epsilon: 0.08\n",
"episode: 4895, score: 4.0, epsilon: 0.08\n",
"marking, episode: 4900, score: 2.0, mean_score: 2.49, std_score: 2.07\n",
"episode: 4900, score: 2.0, epsilon: 0.08\n",
"episode: 4905, score: 0.0, epsilon: 0.08\n",
"episode: 4910, score: 5.0, epsilon: 0.08\n",
"episode: 4915, score: 6.0, epsilon: 0.08\n",
"episode: 4920, score: 0.0, epsilon: 0.08\n",
"episode: 4925, score: 0.0, epsilon: 0.08\n",
"episode: 4930, score: 3.0, epsilon: 0.08\n",
"episode: 4935, score: 0.0, epsilon: 0.08\n",
"episode: 4940, score: 0.0, epsilon: 0.08\n",
"episode: 4945, score: 3.0, epsilon: 0.08\n",
"episode: 4950, score: 1.0, epsilon: 0.08\n",
"episode: 4955, score: 0.0, epsilon: 0.08\n",
"episode: 4960, score: 2.0, epsilon: 0.08\n",
"episode: 4965, score: 0.0, epsilon: 0.08\n",
"episode: 4970, score: 6.0, epsilon: 0.08\n",
"episode: 4975, score: 1.0, epsilon: 0.08\n",
"episode: 4980, score: 1.0, epsilon: 0.08\n",
"episode: 4985, score: 4.0, epsilon: 0.08\n",
"episode: 4990, score: 3.0, epsilon: 0.08\n",
"episode: 4995, score: 1.0, epsilon: 0.08\n",
"marking, episode: 5000, score: 5.0, mean_score: 1.51, std_score: 1.74\n",
"episode: 5000, score: 5.0, epsilon: 0.08\n",
"episode: 5005, score: 1.0, epsilon: 0.08\n",
"episode: 5010, score: 0.0, epsilon: 0.08\n",
"episode: 5015, score: 2.0, epsilon: 0.08\n",
"episode: 5020, score: 2.0, epsilon: 0.08\n",
"episode: 5025, score: 5.0, epsilon: 0.08\n",
"episode: 5030, score: 3.0, epsilon: 0.08\n",
"episode: 5035, score: 1.0, epsilon: 0.08\n",
"episode: 5040, score: 0.0, epsilon: 0.08\n",
"episode: 5045, score: 3.0, epsilon: 0.08\n",
"episode: 5050, score: 1.0, epsilon: 0.08\n",
"episode: 5055, score: 6.0, epsilon: 0.08\n",
"episode: 5060, score: 0.0, epsilon: 0.08\n",
"episode: 5065, score: 5.0, epsilon: 0.08\n",
"episode: 5070, score: 6.0, epsilon: 0.08\n",
"episode: 5075, score: 1.0, epsilon: 0.08\n",
"episode: 5080, score: 0.0, epsilon: 0.08\n",
"episode: 5085, score: 5.0, epsilon: 0.08\n",
"episode: 5090, score: 2.0, epsilon: 0.08\n",
"episode: 5095, score: 6.0, epsilon: 0.08\n",
"marking, episode: 5100, score: 1.0, mean_score: 2.85, std_score: 2.23\n",
"episode: 5100, score: 1.0, epsilon: 0.08\n",
"episode: 5105, score: 0.0, epsilon: 0.08\n",
"episode: 5110, score: 3.0, epsilon: 0.08\n",
"episode: 5115, score: 1.0, epsilon: 0.08\n",
"episode: 5120, score: 1.0, epsilon: 0.08\n",
"episode: 5125, score: 1.0, epsilon: 0.08\n",
"episode: 5130, score: 0.0, epsilon: 0.08\n",
"episode: 5135, score: 4.0, epsilon: 0.08\n",
"episode: 5140, score: 3.0, epsilon: 0.08\n",
"episode: 5145, score: 4.0, epsilon: 0.08\n",
"episode: 5150, score: 0.0, epsilon: 0.08\n",
"episode: 5155, score: 3.0, epsilon: 0.08\n",
"episode: 5160, score: 0.0, epsilon: 0.08\n",
"episode: 5165, score: 0.0, epsilon: 0.08\n",
"episode: 5170, score: 3.0, epsilon: 0.08\n",
"episode: 5175, score: 0.0, epsilon: 0.08\n",
"episode: 5180, score: 2.0, epsilon: 0.08\n",
"episode: 5185, score: 0.0, epsilon: 0.08\n",
"episode: 5190, score: 0.0, epsilon: 0.08\n",
"episode: 5195, score: 1.0, epsilon: 0.08\n",
"marking, episode: 5200, score: 3.0, mean_score: 2.12, std_score: 2.01\n",
"episode: 5200, score: 3.0, epsilon: 0.08\n",
"episode: 5205, score: 3.0, epsilon: 0.08\n",
"episode: 5210, score: 3.0, epsilon: 0.08\n",
"episode: 5215, score: 1.0, epsilon: 0.08\n",
"episode: 5220, score: 3.0, epsilon: 0.08\n",
"episode: 5225, score: 1.0, epsilon: 0.08\n",
"episode: 5230, score: 4.0, epsilon: 0.08\n",
"episode: 5235, score: 0.0, epsilon: 0.08\n",
"episode: 5240, score: 2.0, epsilon: 0.08\n",
"episode: 5245, score: 0.0, epsilon: 0.08\n",
"episode: 5250, score: 0.0, epsilon: 0.08\n",
"episode: 5255, score: 3.0, epsilon: 0.08\n",
"episode: 5260, score: 3.0, epsilon: 0.08\n",
"episode: 5265, score: 6.0, epsilon: 0.08\n",
"episode: 5270, score: 0.0, epsilon: 0.08\n",
"episode: 5275, score: 4.0, epsilon: 0.08\n",
"episode: 5280, score: 3.0, epsilon: 0.08\n",
"episode: 5285, score: 2.0, epsilon: 0.08\n",
"episode: 5290, score: 0.0, epsilon: 0.08\n",
"episode: 5295, score: 3.0, epsilon: 0.08\n",
"marking, episode: 5300, score: 6.0, mean_score: 2.60, std_score: 2.03\n",
"episode: 5300, score: 6.0, epsilon: 0.08\n",
"episode: 5305, score: 2.0, epsilon: 0.08\n",
"episode: 5310, score: 2.0, epsilon: 0.08\n",
"episode: 5315, score: 6.0, epsilon: 0.08\n",
"episode: 5320, score: 6.0, epsilon: 0.08\n",
"episode: 5325, score: 0.0, epsilon: 0.08\n",
"episode: 5330, score: 1.0, epsilon: 0.08\n",
"episode: 5335, score: 2.0, epsilon: 0.08\n",
"episode: 5340, score: 2.0, epsilon: 0.08\n",
"episode: 5345, score: 6.0, epsilon: 0.08\n",
"episode: 5350, score: 1.0, epsilon: 0.08\n",
"episode: 5355, score: 0.0, epsilon: 0.08\n",
"episode: 5360, score: 5.0, epsilon: 0.08\n",
"episode: 5365, score: 2.0, epsilon: 0.08\n",
"episode: 5370, score: 2.0, epsilon: 0.08\n",
"episode: 5375, score: 3.0, epsilon: 0.08\n",
"episode: 5380, score: 2.0, epsilon: 0.08\n",
"episode: 5385, score: 2.0, epsilon: 0.08\n",
"episode: 5390, score: 3.0, epsilon: 0.08\n",
"episode: 5395, score: 5.0, epsilon: 0.08\n",
"marking, episode: 5400, score: 1.0, mean_score: 2.39, std_score: 1.95\n",
"episode: 5400, score: 1.0, epsilon: 0.08\n",
"episode: 5405, score: 2.0, epsilon: 0.08\n",
"episode: 5410, score: 2.0, epsilon: 0.08\n",
"episode: 5415, score: 5.0, epsilon: 0.08\n",
"episode: 5420, score: 5.0, epsilon: 0.08\n",
"episode: 5425, score: 0.0, epsilon: 0.08\n",
"episode: 5430, score: 3.0, epsilon: 0.08\n",
"episode: 5435, score: 1.0, epsilon: 0.08\n",
"episode: 5440, score: 3.0, epsilon: 0.08\n",
"episode: 5445, score: 0.0, epsilon: 0.08\n",
"episode: 5450, score: 3.0, epsilon: 0.08\n",
"episode: 5455, score: 1.0, epsilon: 0.08\n",
"episode: 5460, score: 1.0, epsilon: 0.08\n",
"episode: 5465, score: 2.0, epsilon: 0.08\n",
"episode: 5470, score: 2.0, epsilon: 0.08\n",
"episode: 5475, score: 1.0, epsilon: 0.08\n",
"episode: 5480, score: 3.0, epsilon: 0.08\n",
"episode: 5485, score: 1.0, epsilon: 0.08\n",
"episode: 5490, score: 1.0, epsilon: 0.08\n",
"episode: 5495, score: 2.0, epsilon: 0.08\n",
"marking, episode: 5500, score: 0.0, mean_score: 2.23, std_score: 1.83\n",
"episode: 5500, score: 0.0, epsilon: 0.08\n",
"episode: 5505, score: 3.0, epsilon: 0.08\n",
"episode: 5510, score: 5.0, epsilon: 0.08\n",
"episode: 5515, score: 0.0, epsilon: 0.08\n",
"episode: 5520, score: 0.0, epsilon: 0.08\n",
"episode: 5525, score: 2.0, epsilon: 0.08\n",
"episode: 5530, score: 0.0, epsilon: 0.08\n",
"episode: 5535, score: 6.0, epsilon: 0.08\n",
"episode: 5540, score: 0.0, epsilon: 0.08\n",
"episode: 5545, score: 2.0, epsilon: 0.08\n",
"episode: 5550, score: 0.0, epsilon: 0.08\n",
"episode: 5555, score: 1.0, epsilon: 0.08\n",
"episode: 5560, score: 0.0, epsilon: 0.08\n",
"episode: 5565, score: 3.0, epsilon: 0.08\n",
"episode: 5570, score: 6.0, epsilon: 0.08\n",
"episode: 5575, score: 1.0, epsilon: 0.08\n",
"episode: 5580, score: 2.0, epsilon: 0.08\n",
"episode: 5585, score: 6.0, epsilon: 0.08\n",
"episode: 5590, score: 1.0, epsilon: 0.08\n",
"episode: 5595, score: 5.0, epsilon: 0.08\n",
"marking, episode: 5600, score: 0.0, mean_score: 1.88, std_score: 1.94\n",
"episode: 5600, score: 0.0, epsilon: 0.08\n",
"episode: 5605, score: 0.0, epsilon: 0.08\n",
"episode: 5610, score: 4.0, epsilon: 0.08\n",
"episode: 5615, score: 6.0, epsilon: 0.08\n",
"episode: 5620, score: 4.0, epsilon: 0.08\n",
"episode: 5625, score: 0.0, epsilon: 0.08\n",
"episode: 5630, score: 3.0, epsilon: 0.08\n",
"episode: 5635, score: 6.0, epsilon: 0.08\n",
"episode: 5640, score: 1.0, epsilon: 0.08\n",
"episode: 5645, score: 2.0, epsilon: 0.08\n",
"episode: 5650, score: 5.0, epsilon: 0.08\n",
"episode: 5655, score: 1.0, epsilon: 0.08\n",
"episode: 5660, score: 2.0, epsilon: 0.08\n",
"episode: 5665, score: 0.0, epsilon: 0.08\n",
"episode: 5670, score: 1.0, epsilon: 0.08\n",
"episode: 5675, score: 2.0, epsilon: 0.08\n",
"episode: 5680, score: 0.0, epsilon: 0.08\n",
"episode: 5685, score: 3.0, epsilon: 0.08\n",
"episode: 5690, score: 2.0, epsilon: 0.08\n",
"episode: 5695, score: 3.0, epsilon: 0.08\n",
"marking, episode: 5700, score: 4.0, mean_score: 1.90, std_score: 1.82\n",
"episode: 5700, score: 4.0, epsilon: 0.08\n",
"episode: 5705, score: 1.0, epsilon: 0.08\n",
"episode: 5710, score: 1.0, epsilon: 0.08\n",
"episode: 5715, score: 2.0, epsilon: 0.08\n",
"episode: 5720, score: 2.0, epsilon: 0.08\n",
"episode: 5725, score: 2.0, epsilon: 0.08\n",
"episode: 5730, score: 4.0, epsilon: 0.08\n",
"episode: 5735, score: 0.0, epsilon: 0.08\n",
"episode: 5740, score: 0.0, epsilon: 0.08\n",
"episode: 5745, score: 3.0, epsilon: 0.08\n",
"episode: 5750, score: 0.0, epsilon: 0.08\n",
"episode: 5755, score: 0.0, epsilon: 0.08\n",
"episode: 5760, score: 2.0, epsilon: 0.08\n",
"episode: 5765, score: 4.0, epsilon: 0.08\n",
"episode: 5770, score: 3.0, epsilon: 0.08\n",
"episode: 5775, score: 3.0, epsilon: 0.08\n",
"episode: 5780, score: 0.0, epsilon: 0.08\n",
"episode: 5785, score: 6.0, epsilon: 0.08\n",
"episode: 5790, score: 2.0, epsilon: 0.08\n",
"episode: 5795, score: 1.0, epsilon: 0.08\n",
"marking, episode: 5800, score: 1.0, mean_score: 2.20, std_score: 1.98\n",
"episode: 5800, score: 1.0, epsilon: 0.08\n",
"episode: 5805, score: 1.0, epsilon: 0.08\n",
"episode: 5810, score: 6.0, epsilon: 0.08\n",
"episode: 5815, score: 0.0, epsilon: 0.08\n",
"episode: 5820, score: 6.0, epsilon: 0.08\n",
"episode: 5825, score: 0.0, epsilon: 0.08\n",
"episode: 5830, score: 6.0, epsilon: 0.08\n",
"episode: 5835, score: 1.0, epsilon: 0.08\n",
"episode: 5840, score: 6.0, epsilon: 0.08\n",
"episode: 5845, score: 0.0, epsilon: 0.08\n",
"episode: 5850, score: 4.0, epsilon: 0.08\n",
"episode: 5855, score: 6.0, epsilon: 0.08\n",
"episode: 5860, score: 1.0, epsilon: 0.08\n",
"episode: 5865, score: 2.0, epsilon: 0.08\n",
"episode: 5870, score: 1.0, epsilon: 0.08\n",
"episode: 5875, score: 5.0, epsilon: 0.08\n",
"episode: 5880, score: 2.0, epsilon: 0.08\n",
"episode: 5885, score: 6.0, epsilon: 0.08\n",
"episode: 5890, score: 2.0, epsilon: 0.08\n",
"episode: 5895, score: 0.0, epsilon: 0.08\n",
"marking, episode: 5900, score: 1.0, mean_score: 2.44, std_score: 2.16\n",
"episode: 5900, score: 1.0, epsilon: 0.08\n",
"episode: 5905, score: 0.0, epsilon: 0.08\n",
"episode: 5910, score: 1.0, epsilon: 0.08\n",
"episode: 5915, score: 3.0, epsilon: 0.08\n",
"episode: 5920, score: 6.0, epsilon: 0.08\n",
"episode: 5925, score: 0.0, epsilon: 0.08\n",
"episode: 5930, score: 1.0, epsilon: 0.08\n",
"episode: 5935, score: 6.0, epsilon: 0.08\n",
"episode: 5940, score: 1.0, epsilon: 0.08\n",
"episode: 5945, score: 0.0, epsilon: 0.08\n",
"episode: 5950, score: 1.0, epsilon: 0.08\n",
"episode: 5955, score: 0.0, epsilon: 0.08\n",
"episode: 5960, score: 0.0, epsilon: 0.08\n",
"episode: 5965, score: 2.0, epsilon: 0.08\n",
"episode: 5970, score: 0.0, epsilon: 0.08\n",
"episode: 5975, score: 1.0, epsilon: 0.08\n",
"episode: 5980, score: 2.0, epsilon: 0.08\n",
"episode: 5985, score: 4.0, epsilon: 0.08\n",
"episode: 5990, score: 1.0, epsilon: 0.08\n",
"episode: 5995, score: 6.0, epsilon: 0.08\n",
"marking, episode: 6000, score: 1.0, mean_score: 1.94, std_score: 1.85\n",
"episode: 6000, score: 1.0, epsilon: 0.08\n",
"episode: 6005, score: 1.0, epsilon: 0.08\n",
"episode: 6010, score: 3.0, epsilon: 0.08\n",
"episode: 6015, score: 0.0, epsilon: 0.08\n",
"episode: 6020, score: 1.0, epsilon: 0.08\n",
"episode: 6025, score: 1.0, epsilon: 0.08\n",
"episode: 6030, score: 5.0, epsilon: 0.08\n",
"episode: 6035, score: 4.0, epsilon: 0.08\n",
"episode: 6040, score: 1.0, epsilon: 0.08\n",
"episode: 6045, score: 2.0, epsilon: 0.08\n",
"episode: 6050, score: 3.0, epsilon: 0.08\n",
"episode: 6055, score: 0.0, epsilon: 0.08\n",
"episode: 6060, score: 6.0, epsilon: 0.08\n",
"episode: 6065, score: 1.0, epsilon: 0.08\n",
"episode: 6070, score: 5.0, epsilon: 0.08\n",
"episode: 6075, score: 4.0, epsilon: 0.08\n",
"episode: 6080, score: 0.0, epsilon: 0.08\n",
"episode: 6085, score: 0.0, epsilon: 0.08\n",
"episode: 6090, score: 2.0, epsilon: 0.08\n",
"episode: 6095, score: 6.0, epsilon: 0.08\n",
"marking, episode: 6100, score: 0.0, mean_score: 2.40, std_score: 2.12\n",
"episode: 6100, score: 0.0, epsilon: 0.08\n",
"episode: 6105, score: 1.0, epsilon: 0.08\n",
"episode: 6110, score: 6.0, epsilon: 0.08\n",
"episode: 6115, score: 4.0, epsilon: 0.08\n",
"episode: 6120, score: 1.0, epsilon: 0.08\n",
"episode: 6125, score: 2.0, epsilon: 0.08\n",
"episode: 6130, score: 2.0, epsilon: 0.08\n",
"episode: 6135, score: 0.0, epsilon: 0.08\n",
"episode: 6140, score: 1.0, epsilon: 0.08\n",
"episode: 6145, score: 2.0, epsilon: 0.08\n",
"episode: 6150, score: 2.0, epsilon: 0.08\n",
"episode: 6155, score: 2.0, epsilon: 0.08\n",
"episode: 6160, score: 4.0, epsilon: 0.08\n",
"episode: 6165, score: 0.0, epsilon: 0.08\n",
"episode: 6170, score: 0.0, epsilon: 0.08\n",
"episode: 6175, score: 6.0, epsilon: 0.08\n",
"episode: 6180, score: 1.0, epsilon: 0.08\n",
"episode: 6185, score: 1.0, epsilon: 0.08\n",
"episode: 6190, score: 6.0, epsilon: 0.08\n",
"episode: 6195, score: 2.0, epsilon: 0.08\n",
"marking, episode: 6200, score: 1.0, mean_score: 2.15, std_score: 2.01\n",
"episode: 6200, score: 1.0, epsilon: 0.08\n",
"episode: 6205, score: 0.0, epsilon: 0.08\n",
"episode: 6210, score: 0.0, epsilon: 0.08\n",
"episode: 6215, score: 0.0, epsilon: 0.08\n",
"episode: 6220, score: 6.0, epsilon: 0.08\n",
"episode: 6225, score: 6.0, epsilon: 0.08\n",
"episode: 6230, score: 0.0, epsilon: 0.08\n",
"episode: 6235, score: 4.0, epsilon: 0.08\n",
"episode: 6240, score: 0.0, epsilon: 0.08\n",
"episode: 6245, score: 6.0, epsilon: 0.08\n",
"episode: 6250, score: 3.0, epsilon: 0.08\n",
"episode: 6255, score: 6.0, epsilon: 0.08\n",
"episode: 6260, score: 4.0, epsilon: 0.08\n",
"episode: 6265, score: 1.0, epsilon: 0.08\n",
"episode: 6270, score: 1.0, epsilon: 0.08\n",
"episode: 6275, score: 6.0, epsilon: 0.08\n",
"episode: 6280, score: 1.0, epsilon: 0.08\n",
"episode: 6285, score: 2.0, epsilon: 0.08\n",
"episode: 6290, score: 3.0, epsilon: 0.08\n",
"episode: 6295, score: 2.0, epsilon: 0.08\n",
"marking, episode: 6300, score: 2.0, mean_score: 2.37, std_score: 2.12\n",
"episode: 6300, score: 2.0, epsilon: 0.08\n",
"episode: 6305, score: 1.0, epsilon: 0.08\n",
"episode: 6310, score: 0.0, epsilon: 0.08\n",
"episode: 6315, score: 3.0, epsilon: 0.08\n",
"episode: 6320, score: 3.0, epsilon: 0.08\n",
"episode: 6325, score: 0.0, epsilon: 0.08\n",
"episode: 6330, score: 6.0, epsilon: 0.08\n",
"episode: 6335, score: 4.0, epsilon: 0.08\n",
"episode: 6340, score: 0.0, epsilon: 0.08\n",
"episode: 6345, score: 6.0, epsilon: 0.08\n",
"episode: 6350, score: 1.0, epsilon: 0.08\n",
"episode: 6355, score: 1.0, epsilon: 0.08\n",
"episode: 6360, score: 6.0, epsilon: 0.08\n",
"episode: 6365, score: 0.0, epsilon: 0.08\n",
"episode: 6370, score: 4.0, epsilon: 0.08\n",
"episode: 6375, score: 0.0, epsilon: 0.08\n",
"episode: 6380, score: 0.0, epsilon: 0.08\n",
"episode: 6385, score: 1.0, epsilon: 0.08\n",
"episode: 6390, score: 0.0, epsilon: 0.08\n",
"episode: 6395, score: 3.0, epsilon: 0.08\n",
"marking, episode: 6400, score: 6.0, mean_score: 2.23, std_score: 2.06\n",
"episode: 6400, score: 6.0, epsilon: 0.08\n",
"episode: 6405, score: 1.0, epsilon: 0.08\n",
"episode: 6410, score: 1.0, epsilon: 0.08\n",
"episode: 6415, score: 0.0, epsilon: 0.08\n",
"episode: 6420, score: 6.0, epsilon: 0.08\n",
"episode: 6425, score: 0.0, epsilon: 0.08\n",
"episode: 6430, score: 3.0, epsilon: 0.08\n",
"episode: 6435, score: 4.0, epsilon: 0.08\n",
"episode: 6440, score: 5.0, epsilon: 0.08\n",
"episode: 6445, score: 6.0, epsilon: 0.08\n",
"episode: 6450, score: 6.0, epsilon: 0.08\n",
"episode: 6455, score: 2.0, epsilon: 0.08\n",
"episode: 6460, score: 0.0, epsilon: 0.08\n",
"episode: 6465, score: 0.0, epsilon: 0.08\n",
"episode: 6470, score: 2.0, epsilon: 0.08\n",
"episode: 6475, score: 4.0, epsilon: 0.08\n",
"episode: 6480, score: 3.0, epsilon: 0.08\n",
"episode: 6485, score: 3.0, epsilon: 0.08\n",
"episode: 6490, score: 4.0, epsilon: 0.08\n",
"episode: 6495, score: 1.0, epsilon: 0.08\n",
"marking, episode: 6500, score: 0.0, mean_score: 2.43, std_score: 2.08\n",
"episode: 6500, score: 0.0, epsilon: 0.08\n",
"episode: 6505, score: 6.0, epsilon: 0.08\n",
"episode: 6510, score: 0.0, epsilon: 0.08\n",
"episode: 6515, score: 1.0, epsilon: 0.08\n",
"episode: 6520, score: 1.0, epsilon: 0.08\n",
"episode: 6525, score: 2.0, epsilon: 0.08\n",
"episode: 6530, score: 1.0, epsilon: 0.08\n",
"episode: 6535, score: 4.0, epsilon: 0.08\n",
"episode: 6540, score: 0.0, epsilon: 0.08\n",
"episode: 6545, score: 2.0, epsilon: 0.08\n",
"episode: 6550, score: 4.0, epsilon: 0.08\n",
"episode: 6555, score: 2.0, epsilon: 0.08\n",
"episode: 6560, score: 5.0, epsilon: 0.08\n",
"episode: 6565, score: 5.0, epsilon: 0.08\n",
"episode: 6570, score: 1.0, epsilon: 0.08\n",
"episode: 6575, score: 0.0, epsilon: 0.08\n",
"episode: 6580, score: 0.0, epsilon: 0.08\n",
"episode: 6585, score: 1.0, epsilon: 0.08\n",
"episode: 6590, score: 2.0, epsilon: 0.08\n",
"episode: 6595, score: 2.0, epsilon: 0.08\n",
"marking, episode: 6600, score: 4.0, mean_score: 2.34, std_score: 2.13\n",
"episode: 6600, score: 4.0, epsilon: 0.08\n",
"episode: 6605, score: 1.0, epsilon: 0.08\n",
"episode: 6610, score: 2.0, epsilon: 0.08\n",
"episode: 6615, score: 0.0, epsilon: 0.08\n",
"episode: 6620, score: 3.0, epsilon: 0.08\n",
"episode: 6625, score: 0.0, epsilon: 0.08\n",
"episode: 6630, score: 2.0, epsilon: 0.08\n",
"episode: 6635, score: 0.0, epsilon: 0.08\n",
"episode: 6640, score: 1.0, epsilon: 0.08\n",
"episode: 6645, score: 6.0, epsilon: 0.08\n",
"episode: 6650, score: 2.0, epsilon: 0.08\n",
"episode: 6655, score: 2.0, epsilon: 0.08\n",
"episode: 6660, score: 0.0, epsilon: 0.08\n",
"episode: 6665, score: 6.0, epsilon: 0.08\n",
"episode: 6670, score: 4.0, epsilon: 0.08\n",
"episode: 6675, score: 0.0, epsilon: 0.08\n",
"episode: 6680, score: 0.0, epsilon: 0.08\n",
"episode: 6685, score: 3.0, epsilon: 0.08\n",
"episode: 6690, score: 0.0, epsilon: 0.08\n",
"episode: 6695, score: 0.0, epsilon: 0.08\n",
"marking, episode: 6700, score: 6.0, mean_score: 2.18, std_score: 2.03\n",
"episode: 6700, score: 6.0, epsilon: 0.08\n",
"episode: 6705, score: 0.0, epsilon: 0.08\n",
"episode: 6710, score: 3.0, epsilon: 0.08\n",
"episode: 6715, score: 3.0, epsilon: 0.08\n",
"episode: 6720, score: 6.0, epsilon: 0.08\n",
"episode: 6725, score: 3.0, epsilon: 0.08\n",
"episode: 6730, score: 0.0, epsilon: 0.08\n",
"episode: 6735, score: 0.0, epsilon: 0.08\n",
"episode: 6740, score: 4.0, epsilon: 0.08\n",
"episode: 6745, score: 0.0, epsilon: 0.08\n",
"episode: 6750, score: 5.0, epsilon: 0.08\n",
"episode: 6755, score: 2.0, epsilon: 0.08\n",
"episode: 6760, score: 1.0, epsilon: 0.08\n",
"episode: 6765, score: 0.0, epsilon: 0.08\n",
"episode: 6770, score: 6.0, epsilon: 0.08\n",
"episode: 6775, score: 1.0, epsilon: 0.08\n",
"episode: 6780, score: 3.0, epsilon: 0.08\n",
"episode: 6785, score: 0.0, epsilon: 0.08\n",
"episode: 6790, score: 6.0, epsilon: 0.08\n",
"episode: 6795, score: 0.0, epsilon: 0.08\n",
"marking, episode: 6800, score: 0.0, mean_score: 2.70, std_score: 2.16\n",
"episode: 6800, score: 0.0, epsilon: 0.08\n",
"episode: 6805, score: 1.0, epsilon: 0.08\n",
"episode: 6810, score: 2.0, epsilon: 0.08\n",
"episode: 6815, score: 0.0, epsilon: 0.08\n",
"episode: 6820, score: 0.0, epsilon: 0.08\n",
"episode: 6825, score: 0.0, epsilon: 0.08\n",
"episode: 6830, score: 0.0, epsilon: 0.08\n",
"episode: 6835, score: 0.0, epsilon: 0.08\n",
"episode: 6840, score: 3.0, epsilon: 0.08\n",
"episode: 6845, score: 4.0, epsilon: 0.08\n",
"episode: 6850, score: 2.0, epsilon: 0.08\n",
"episode: 6855, score: 1.0, epsilon: 0.08\n",
"episode: 6860, score: 0.0, epsilon: 0.08\n",
"episode: 6865, score: 1.0, epsilon: 0.08\n",
"episode: 6870, score: 1.0, epsilon: 0.08\n",
"episode: 6875, score: 2.0, epsilon: 0.08\n",
"episode: 6880, score: 6.0, epsilon: 0.08\n",
"episode: 6885, score: 6.0, epsilon: 0.08\n",
"episode: 6890, score: 6.0, epsilon: 0.08\n",
"episode: 6895, score: 1.0, epsilon: 0.08\n",
"marking, episode: 6900, score: 0.0, mean_score: 2.40, std_score: 2.19\n",
"episode: 6900, score: 0.0, epsilon: 0.08\n",
"episode: 6905, score: 1.0, epsilon: 0.08\n",
"episode: 6910, score: 1.0, epsilon: 0.08\n",
"episode: 6915, score: 1.0, epsilon: 0.08\n",
"episode: 6920, score: 4.0, epsilon: 0.08\n",
"episode: 6925, score: 3.0, epsilon: 0.08\n",
"episode: 6930, score: 0.0, epsilon: 0.08\n",
"episode: 6935, score: 3.0, epsilon: 0.08\n",
"episode: 6940, score: 6.0, epsilon: 0.08\n",
"episode: 6945, score: 6.0, epsilon: 0.08\n",
"episode: 6950, score: 6.0, epsilon: 0.08\n",
"episode: 6955, score: 0.0, epsilon: 0.08\n",
"episode: 6960, score: 2.0, epsilon: 0.08\n",
"episode: 6965, score: 0.0, epsilon: 0.08\n",
"episode: 6970, score: 6.0, epsilon: 0.08\n",
"episode: 6975, score: 4.0, epsilon: 0.08\n",
"episode: 6980, score: 3.0, epsilon: 0.08\n",
"episode: 6985, score: 0.0, epsilon: 0.08\n",
"episode: 6990, score: 1.0, epsilon: 0.08\n",
"episode: 6995, score: 0.0, epsilon: 0.08\n",
"marking, episode: 7000, score: 6.0, mean_score: 2.42, std_score: 2.09\n",
"episode: 7000, score: 6.0, epsilon: 0.08\n",
"episode: 7005, score: 1.0, epsilon: 0.08\n",
"episode: 7010, score: 3.0, epsilon: 0.08\n",
"episode: 7015, score: 4.0, epsilon: 0.08\n",
"episode: 7020, score: 0.0, epsilon: 0.08\n",
"episode: 7025, score: 2.0, epsilon: 0.08\n",
"episode: 7030, score: 0.0, epsilon: 0.08\n",
"episode: 7035, score: 3.0, epsilon: 0.08\n",
"episode: 7040, score: 2.0, epsilon: 0.08\n",
"episode: 7045, score: 2.0, epsilon: 0.08\n",
"episode: 7050, score: 3.0, epsilon: 0.08\n",
"episode: 7055, score: 5.0, epsilon: 0.08\n",
"episode: 7060, score: 1.0, epsilon: 0.08\n",
"episode: 7065, score: 0.0, epsilon: 0.08\n",
"episode: 7070, score: 1.0, epsilon: 0.08\n",
"episode: 7075, score: 3.0, epsilon: 0.08\n",
"episode: 7080, score: 0.0, epsilon: 0.08\n",
"episode: 7085, score: 6.0, epsilon: 0.08\n",
"episode: 7090, score: 0.0, epsilon: 0.08\n",
"episode: 7095, score: 1.0, epsilon: 0.08\n",
"marking, episode: 7100, score: 6.0, mean_score: 2.52, std_score: 2.03\n",
"episode: 7100, score: 6.0, epsilon: 0.08\n",
"episode: 7105, score: 0.0, epsilon: 0.08\n",
"episode: 7110, score: 6.0, epsilon: 0.08\n",
"episode: 7115, score: 1.0, epsilon: 0.08\n",
"episode: 7120, score: 0.0, epsilon: 0.08\n",
"episode: 7125, score: 2.0, epsilon: 0.08\n",
"episode: 7130, score: 0.0, epsilon: 0.08\n",
"episode: 7135, score: 1.0, epsilon: 0.08\n",
"episode: 7140, score: 0.0, epsilon: 0.08\n",
"episode: 7145, score: 5.0, epsilon: 0.08\n",
"episode: 7150, score: 6.0, epsilon: 0.08\n",
"episode: 7155, score: 6.0, epsilon: 0.08\n",
"episode: 7160, score: 0.0, epsilon: 0.08\n",
"episode: 7165, score: 6.0, epsilon: 0.08\n",
"episode: 7170, score: 6.0, epsilon: 0.08\n",
"episode: 7175, score: 0.0, epsilon: 0.08\n",
"episode: 7180, score: 5.0, epsilon: 0.08\n",
"episode: 7185, score: 6.0, epsilon: 0.08\n",
"episode: 7190, score: 1.0, epsilon: 0.08\n",
"episode: 7195, score: 1.0, epsilon: 0.08\n",
"marking, episode: 7200, score: 0.0, mean_score: 2.45, std_score: 2.21\n",
"episode: 7200, score: 0.0, epsilon: 0.08\n",
"episode: 7205, score: 6.0, epsilon: 0.08\n",
"episode: 7210, score: 1.0, epsilon: 0.08\n",
"episode: 7215, score: 1.0, epsilon: 0.08\n",
"episode: 7220, score: 1.0, epsilon: 0.08\n",
"episode: 7225, score: 1.0, epsilon: 0.08\n",
"episode: 7230, score: 1.0, epsilon: 0.08\n",
"episode: 7235, score: 6.0, epsilon: 0.08\n",
"episode: 7240, score: 0.0, epsilon: 0.08\n",
"episode: 7245, score: 0.0, epsilon: 0.08\n",
"episode: 7250, score: 3.0, epsilon: 0.08\n",
"episode: 7255, score: 1.0, epsilon: 0.08\n",
"episode: 7260, score: 1.0, epsilon: 0.08\n",
"episode: 7265, score: 0.0, epsilon: 0.08\n",
"episode: 7270, score: 3.0, epsilon: 0.08\n",
"episode: 7275, score: 0.0, epsilon: 0.08\n",
"episode: 7280, score: 1.0, epsilon: 0.08\n",
"episode: 7285, score: 6.0, epsilon: 0.08\n",
"episode: 7290, score: 1.0, epsilon: 0.08\n",
"episode: 7295, score: 6.0, epsilon: 0.08\n",
"marking, episode: 7300, score: 1.0, mean_score: 2.20, std_score: 2.11\n",
"episode: 7300, score: 1.0, epsilon: 0.08\n",
"episode: 7305, score: 2.0, epsilon: 0.08\n",
"episode: 7310, score: 1.0, epsilon: 0.08\n",
"episode: 7315, score: 2.0, epsilon: 0.08\n",
"episode: 7320, score: 0.0, epsilon: 0.08\n",
"episode: 7325, score: 5.0, epsilon: 0.08\n",
"episode: 7330, score: 6.0, epsilon: 0.08\n",
"episode: 7335, score: 2.0, epsilon: 0.08\n",
"episode: 7340, score: 0.0, epsilon: 0.08\n",
"episode: 7345, score: 0.0, epsilon: 0.08\n",
"episode: 7350, score: 1.0, epsilon: 0.08\n",
"episode: 7355, score: 0.0, epsilon: 0.08\n",
"episode: 7360, score: 0.0, epsilon: 0.08\n",
"episode: 7365, score: 1.0, epsilon: 0.08\n",
"episode: 7370, score: 6.0, epsilon: 0.08\n",
"episode: 7375, score: 0.0, epsilon: 0.08\n",
"episode: 7380, score: 3.0, epsilon: 0.08\n",
"episode: 7385, score: 0.0, epsilon: 0.08\n",
"episode: 7390, score: 1.0, epsilon: 0.08\n",
"episode: 7395, score: 3.0, epsilon: 0.08\n",
"marking, episode: 7400, score: 6.0, mean_score: 1.91, std_score: 1.93\n",
"episode: 7400, score: 6.0, epsilon: 0.08\n",
"episode: 7405, score: 0.0, epsilon: 0.08\n",
"episode: 7410, score: 4.0, epsilon: 0.08\n",
"episode: 7415, score: 2.0, epsilon: 0.08\n",
"episode: 7420, score: 0.0, epsilon: 0.08\n",
"episode: 7425, score: 4.0, epsilon: 0.08\n",
"episode: 7430, score: 0.0, epsilon: 0.08\n",
"episode: 7435, score: 0.0, epsilon: 0.08\n",
"episode: 7440, score: 6.0, epsilon: 0.08\n",
"episode: 7445, score: 2.0, epsilon: 0.08\n",
"episode: 7450, score: 5.0, epsilon: 0.08\n",
"episode: 7455, score: 2.0, epsilon: 0.08\n",
"episode: 7460, score: 6.0, epsilon: 0.08\n",
"episode: 7465, score: 6.0, epsilon: 0.08\n",
"episode: 7470, score: 5.0, epsilon: 0.08\n",
"episode: 7475, score: 6.0, epsilon: 0.08\n",
"episode: 7480, score: 6.0, epsilon: 0.08\n",
"episode: 7485, score: 0.0, epsilon: 0.08\n",
"episode: 7490, score: 0.0, epsilon: 0.08\n",
"episode: 7495, score: 1.0, epsilon: 0.08\n",
"marking, episode: 7500, score: 2.0, mean_score: 2.49, std_score: 2.21\n",
"episode: 7500, score: 2.0, epsilon: 0.08\n",
"episode: 7505, score: 6.0, epsilon: 0.08\n",
"episode: 7510, score: 3.0, epsilon: 0.08\n",
"episode: 7515, score: 5.0, epsilon: 0.08\n",
"episode: 7520, score: 6.0, epsilon: 0.08\n",
"episode: 7525, score: 0.0, epsilon: 0.08\n",
"episode: 7530, score: 2.0, epsilon: 0.08\n",
"episode: 7535, score: 2.0, epsilon: 0.08\n",
"episode: 7540, score: 2.0, epsilon: 0.08\n",
"episode: 7545, score: 0.0, epsilon: 0.08\n",
"episode: 7550, score: 5.0, epsilon: 0.08\n",
"episode: 7555, score: 3.0, epsilon: 0.08\n",
"episode: 7560, score: 6.0, epsilon: 0.08\n",
"episode: 7565, score: 0.0, epsilon: 0.08\n",
"episode: 7570, score: 1.0, epsilon: 0.08\n",
"episode: 7575, score: 4.0, epsilon: 0.08\n",
"episode: 7580, score: 1.0, epsilon: 0.08\n",
"episode: 7585, score: 0.0, epsilon: 0.08\n",
"episode: 7590, score: 2.0, epsilon: 0.08\n",
"episode: 7595, score: 3.0, epsilon: 0.08\n",
"marking, episode: 7600, score: 2.0, mean_score: 2.29, std_score: 1.98\n",
"episode: 7600, score: 2.0, epsilon: 0.08\n",
"episode: 7605, score: 1.0, epsilon: 0.08\n",
"episode: 7610, score: 1.0, epsilon: 0.08\n",
"episode: 7615, score: 1.0, epsilon: 0.08\n",
"episode: 7620, score: 0.0, epsilon: 0.08\n",
"episode: 7625, score: 1.0, epsilon: 0.08\n",
"episode: 7630, score: 0.0, epsilon: 0.08\n",
"episode: 7635, score: 0.0, epsilon: 0.08\n",
"episode: 7640, score: 4.0, epsilon: 0.08\n",
"episode: 7645, score: 1.0, epsilon: 0.08\n",
"episode: 7650, score: 2.0, epsilon: 0.08\n",
"episode: 7655, score: 2.0, epsilon: 0.08\n",
"episode: 7660, score: 3.0, epsilon: 0.08\n",
"episode: 7665, score: 1.0, epsilon: 0.08\n",
"episode: 7670, score: 6.0, epsilon: 0.08\n",
"episode: 7675, score: 6.0, epsilon: 0.08\n",
"episode: 7680, score: 0.0, epsilon: 0.08\n",
"episode: 7685, score: 1.0, epsilon: 0.08\n",
"episode: 7690, score: 0.0, epsilon: 0.08\n",
"episode: 7695, score: 0.0, epsilon: 0.08\n",
"marking, episode: 7700, score: 1.0, mean_score: 2.47, std_score: 2.01\n",
"episode: 7700, score: 1.0, epsilon: 0.08\n",
"episode: 7705, score: 4.0, epsilon: 0.08\n",
"episode: 7710, score: 3.0, epsilon: 0.08\n",
"episode: 7715, score: 3.0, epsilon: 0.08\n",
"episode: 7720, score: 1.0, epsilon: 0.08\n",
"episode: 7725, score: 0.0, epsilon: 0.08\n",
"episode: 7730, score: 3.0, epsilon: 0.08\n",
"episode: 7735, score: 0.0, epsilon: 0.08\n",
"episode: 7740, score: 0.0, epsilon: 0.08\n",
"episode: 7745, score: 2.0, epsilon: 0.08\n",
"episode: 7750, score: 1.0, epsilon: 0.08\n",
"episode: 7755, score: 0.0, epsilon: 0.08\n",
"episode: 7760, score: 3.0, epsilon: 0.08\n",
"episode: 7765, score: 6.0, epsilon: 0.08\n",
"episode: 7770, score: 1.0, epsilon: 0.08\n",
"episode: 7775, score: 2.0, epsilon: 0.08\n",
"episode: 7780, score: 2.0, epsilon: 0.08\n",
"episode: 7785, score: 3.0, epsilon: 0.08\n",
"episode: 7790, score: 4.0, epsilon: 0.08\n",
"episode: 7795, score: 0.0, epsilon: 0.08\n",
"marking, episode: 7800, score: 3.0, mean_score: 2.25, std_score: 1.88\n",
"episode: 7800, score: 3.0, epsilon: 0.08\n",
"episode: 7805, score: 1.0, epsilon: 0.08\n",
"episode: 7810, score: 1.0, epsilon: 0.08\n",
"episode: 7815, score: 2.0, epsilon: 0.08\n",
"episode: 7820, score: 3.0, epsilon: 0.08\n",
"episode: 7825, score: 2.0, epsilon: 0.08\n",
"episode: 7830, score: 1.0, epsilon: 0.08\n",
"episode: 7835, score: 1.0, epsilon: 0.08\n",
"episode: 7840, score: 1.0, epsilon: 0.08\n",
"episode: 7845, score: 0.0, epsilon: 0.08\n",
"episode: 7850, score: 0.0, epsilon: 0.08\n",
"episode: 7855, score: 2.0, epsilon: 0.08\n",
"episode: 7860, score: 0.0, epsilon: 0.08\n",
"episode: 7865, score: 2.0, epsilon: 0.08\n",
"episode: 7870, score: 2.0, epsilon: 0.08\n",
"episode: 7875, score: 6.0, epsilon: 0.08\n",
"episode: 7880, score: 3.0, epsilon: 0.08\n",
"episode: 7885, score: 6.0, epsilon: 0.08\n",
"episode: 7890, score: 4.0, epsilon: 0.08\n",
"episode: 7895, score: 0.0, epsilon: 0.08\n",
"marking, episode: 7900, score: 0.0, mean_score: 2.13, std_score: 1.96\n",
"episode: 7900, score: 0.0, epsilon: 0.08\n",
"episode: 7905, score: 1.0, epsilon: 0.08\n",
"episode: 7910, score: 5.0, epsilon: 0.08\n",
"episode: 7915, score: 1.0, epsilon: 0.08\n",
"episode: 7920, score: 0.0, epsilon: 0.08\n",
"episode: 7925, score: 0.0, epsilon: 0.08\n",
"episode: 7930, score: 4.0, epsilon: 0.08\n",
"episode: 7935, score: 1.0, epsilon: 0.08\n",
"episode: 7940, score: 3.0, epsilon: 0.08\n",
"episode: 7945, score: 3.0, epsilon: 0.08\n",
"episode: 7950, score: 6.0, epsilon: 0.08\n",
"episode: 7955, score: 3.0, epsilon: 0.08\n",
"episode: 7960, score: 0.0, epsilon: 0.08\n",
"episode: 7965, score: 2.0, epsilon: 0.08\n",
"episode: 7970, score: 2.0, epsilon: 0.08\n",
"episode: 7975, score: 6.0, epsilon: 0.08\n",
"episode: 7980, score: 6.0, epsilon: 0.08\n",
"episode: 7985, score: 6.0, epsilon: 0.08\n",
"episode: 7990, score: 3.0, epsilon: 0.08\n",
"episode: 7995, score: 2.0, epsilon: 0.08\n",
"marking, episode: 8000, score: 0.0, mean_score: 2.37, std_score: 2.02\n",
"episode: 8000, score: 0.0, epsilon: 0.08\n",
"episode: 8005, score: 3.0, epsilon: 0.08\n",
"episode: 8010, score: 0.0, epsilon: 0.08\n",
"episode: 8015, score: 0.0, epsilon: 0.08\n",
"episode: 8020, score: 3.0, epsilon: 0.08\n",
"episode: 8025, score: 5.0, epsilon: 0.08\n",
"episode: 8030, score: 6.0, epsilon: 0.08\n",
"episode: 8035, score: 1.0, epsilon: 0.08\n",
"episode: 8040, score: 5.0, epsilon: 0.08\n",
"episode: 8045, score: 5.0, epsilon: 0.08\n",
"episode: 8050, score: 0.0, epsilon: 0.08\n",
"episode: 8055, score: 4.0, epsilon: 0.08\n",
"episode: 8060, score: 3.0, epsilon: 0.08\n",
"episode: 8065, score: 5.0, epsilon: 0.08\n",
"episode: 8070, score: 0.0, epsilon: 0.08\n",
"episode: 8075, score: 0.0, epsilon: 0.08\n",
"episode: 8080, score: 3.0, epsilon: 0.08\n",
"episode: 8085, score: 6.0, epsilon: 0.08\n",
"episode: 8090, score: 1.0, epsilon: 0.08\n",
"episode: 8095, score: 1.0, epsilon: 0.08\n",
"marking, episode: 8100, score: 1.0, mean_score: 2.16, std_score: 2.00\n",
"episode: 8100, score: 1.0, epsilon: 0.08\n",
"episode: 8105, score: 0.0, epsilon: 0.08\n",
"episode: 8110, score: 0.0, epsilon: 0.08\n",
"episode: 8115, score: 2.0, epsilon: 0.08\n",
"episode: 8120, score: 1.0, epsilon: 0.08\n",
"episode: 8125, score: 1.0, epsilon: 0.08\n",
"episode: 8130, score: 1.0, epsilon: 0.08\n",
"episode: 8135, score: 0.0, epsilon: 0.08\n",
"episode: 8140, score: 3.0, epsilon: 0.08\n",
"episode: 8145, score: 1.0, epsilon: 0.08\n",
"episode: 8150, score: 3.0, epsilon: 0.08\n",
"episode: 8155, score: 0.0, epsilon: 0.08\n",
"episode: 8160, score: 0.0, epsilon: 0.08\n",
"episode: 8165, score: 2.0, epsilon: 0.08\n",
"episode: 8170, score: 1.0, epsilon: 0.08\n",
"episode: 8175, score: 2.0, epsilon: 0.08\n",
"episode: 8180, score: 2.0, epsilon: 0.08\n",
"episode: 8185, score: 0.0, epsilon: 0.08\n",
"episode: 8190, score: 1.0, epsilon: 0.08\n",
"episode: 8195, score: 0.0, epsilon: 0.08\n",
"marking, episode: 8200, score: 0.0, mean_score: 2.14, std_score: 1.90\n",
"episode: 8200, score: 0.0, epsilon: 0.08\n",
"episode: 8205, score: 3.0, epsilon: 0.08\n",
"episode: 8210, score: 1.0, epsilon: 0.08\n",
"episode: 8215, score: 0.0, epsilon: 0.08\n",
"episode: 8220, score: 6.0, epsilon: 0.08\n",
"episode: 8225, score: 6.0, epsilon: 0.08\n",
"episode: 8230, score: 2.0, epsilon: 0.08\n",
"episode: 8235, score: 6.0, epsilon: 0.08\n",
"episode: 8240, score: 3.0, epsilon: 0.08\n",
"episode: 8245, score: 1.0, epsilon: 0.08\n",
"episode: 8250, score: 0.0, epsilon: 0.08\n",
"episode: 8255, score: 0.0, epsilon: 0.08\n",
"episode: 8260, score: 5.0, epsilon: 0.08\n",
"episode: 8265, score: 0.0, epsilon: 0.08\n",
"episode: 8270, score: 2.0, epsilon: 0.08\n",
"episode: 8275, score: 4.0, epsilon: 0.08\n",
"episode: 8280, score: 0.0, epsilon: 0.08\n",
"episode: 8285, score: 0.0, epsilon: 0.08\n",
"episode: 8290, score: 2.0, epsilon: 0.08\n",
"episode: 8295, score: 5.0, epsilon: 0.08\n",
"marking, episode: 8300, score: 0.0, mean_score: 2.32, std_score: 2.12\n",
"episode: 8300, score: 0.0, epsilon: 0.08\n",
"episode: 8305, score: 6.0, epsilon: 0.08\n",
"episode: 8310, score: 1.0, epsilon: 0.08\n",
"episode: 8315, score: 3.0, epsilon: 0.08\n",
"episode: 8320, score: 0.0, epsilon: 0.08\n",
"episode: 8325, score: 0.0, epsilon: 0.08\n",
"episode: 8330, score: 2.0, epsilon: 0.08\n",
"episode: 8335, score: 6.0, epsilon: 0.08\n",
"episode: 8340, score: 0.0, epsilon: 0.08\n",
"episode: 8345, score: 0.0, epsilon: 0.08\n",
"episode: 8350, score: 2.0, epsilon: 0.08\n",
"episode: 8355, score: 0.0, epsilon: 0.08\n",
"episode: 8360, score: 1.0, epsilon: 0.08\n",
"episode: 8365, score: 6.0, epsilon: 0.08\n",
"episode: 8370, score: 2.0, epsilon: 0.08\n",
"episode: 8375, score: 3.0, epsilon: 0.08\n",
"episode: 8380, score: 5.0, epsilon: 0.08\n",
"episode: 8385, score: 3.0, epsilon: 0.08\n",
"episode: 8390, score: 0.0, epsilon: 0.08\n",
"episode: 8395, score: 1.0, epsilon: 0.08\n",
"marking, episode: 8400, score: 1.0, mean_score: 1.97, std_score: 1.92\n",
"episode: 8400, score: 1.0, epsilon: 0.08\n",
"episode: 8405, score: 0.0, epsilon: 0.08\n",
"episode: 8410, score: 5.0, epsilon: 0.08\n",
"episode: 8415, score: 5.0, epsilon: 0.08\n",
"episode: 8420, score: 0.0, epsilon: 0.08\n",
"episode: 8425, score: 0.0, epsilon: 0.08\n",
"episode: 8430, score: 2.0, epsilon: 0.08\n",
"episode: 8435, score: 1.0, epsilon: 0.08\n",
"episode: 8440, score: 0.0, epsilon: 0.08\n",
"episode: 8445, score: 2.0, epsilon: 0.08\n",
"episode: 8450, score: 4.0, epsilon: 0.08\n",
"episode: 8455, score: 3.0, epsilon: 0.08\n",
"episode: 8460, score: 0.0, epsilon: 0.08\n",
"episode: 8465, score: 3.0, epsilon: 0.08\n",
"episode: 8470, score: 2.0, epsilon: 0.08\n",
"episode: 8475, score: 1.0, epsilon: 0.08\n",
"episode: 8480, score: 0.0, epsilon: 0.08\n",
"episode: 8485, score: 1.0, epsilon: 0.08\n",
"episode: 8490, score: 1.0, epsilon: 0.08\n",
"episode: 8495, score: 3.0, epsilon: 0.08\n",
"marking, episode: 8500, score: 0.0, mean_score: 2.28, std_score: 2.08\n",
"episode: 8500, score: 0.0, epsilon: 0.08\n",
"episode: 8505, score: 2.0, epsilon: 0.08\n",
"episode: 8510, score: 1.0, epsilon: 0.08\n",
"episode: 8515, score: 0.0, epsilon: 0.08\n",
"episode: 8520, score: 6.0, epsilon: 0.08\n",
"episode: 8525, score: 2.0, epsilon: 0.08\n",
"episode: 8530, score: 2.0, epsilon: 0.08\n",
"episode: 8535, score: 2.0, epsilon: 0.08\n",
"episode: 8540, score: 0.0, epsilon: 0.08\n",
"episode: 8545, score: 0.0, epsilon: 0.08\n",
"episode: 8550, score: 1.0, epsilon: 0.08\n",
"episode: 8555, score: 3.0, epsilon: 0.08\n",
"episode: 8560, score: 0.0, epsilon: 0.08\n",
"episode: 8565, score: 1.0, epsilon: 0.08\n",
"episode: 8570, score: 0.0, epsilon: 0.08\n",
"episode: 8575, score: 3.0, epsilon: 0.08\n",
"episode: 8580, score: 6.0, epsilon: 0.08\n",
"episode: 8585, score: 3.0, epsilon: 0.08\n",
"episode: 8590, score: 1.0, epsilon: 0.08\n",
"episode: 8595, score: 6.0, epsilon: 0.08\n",
"marking, episode: 8600, score: 1.0, mean_score: 2.30, std_score: 2.17\n",
"episode: 8600, score: 1.0, epsilon: 0.08\n",
"episode: 8605, score: 2.0, epsilon: 0.08\n",
"episode: 8610, score: 1.0, epsilon: 0.08\n",
"episode: 8615, score: 1.0, epsilon: 0.08\n",
"episode: 8620, score: 0.0, epsilon: 0.08\n",
"episode: 8625, score: 0.0, epsilon: 0.08\n",
"episode: 8630, score: 1.0, epsilon: 0.08\n",
"episode: 8635, score: 2.0, epsilon: 0.08\n",
"episode: 8640, score: 1.0, epsilon: 0.08\n",
"episode: 8645, score: 6.0, epsilon: 0.08\n",
"episode: 8650, score: 6.0, epsilon: 0.08\n",
"episode: 8655, score: 4.0, epsilon: 0.08\n",
"episode: 8660, score: 3.0, epsilon: 0.08\n",
"episode: 8665, score: 1.0, epsilon: 0.08\n",
"episode: 8670, score: 0.0, epsilon: 0.08\n",
"episode: 8675, score: 0.0, epsilon: 0.08\n",
"episode: 8680, score: 1.0, epsilon: 0.08\n",
"episode: 8685, score: 2.0, epsilon: 0.08\n",
"episode: 8690, score: 0.0, epsilon: 0.08\n",
"episode: 8695, score: 0.0, epsilon: 0.08\n",
"marking, episode: 8700, score: 1.0, mean_score: 2.33, std_score: 2.14\n",
"episode: 8700, score: 1.0, epsilon: 0.08\n",
"episode: 8705, score: 4.0, epsilon: 0.08\n",
"episode: 8710, score: 4.0, epsilon: 0.08\n",
"episode: 8715, score: 4.0, epsilon: 0.08\n",
"episode: 8720, score: 2.0, epsilon: 0.08\n",
"episode: 8725, score: 6.0, epsilon: 0.08\n",
"episode: 8730, score: 6.0, epsilon: 0.08\n",
"episode: 8735, score: 0.0, epsilon: 0.08\n",
"episode: 8740, score: 4.0, epsilon: 0.08\n",
"episode: 8745, score: 6.0, epsilon: 0.08\n",
"episode: 8750, score: 2.0, epsilon: 0.08\n",
"episode: 8755, score: 4.0, epsilon: 0.08\n",
"episode: 8760, score: 1.0, epsilon: 0.08\n",
"episode: 8765, score: 6.0, epsilon: 0.08\n",
"episode: 8770, score: 6.0, epsilon: 0.08\n",
"episode: 8775, score: 1.0, epsilon: 0.08\n",
"episode: 8780, score: 3.0, epsilon: 0.08\n",
"episode: 8785, score: 3.0, epsilon: 0.08\n",
"episode: 8790, score: 3.0, epsilon: 0.08\n",
"episode: 8795, score: 0.0, epsilon: 0.08\n",
"marking, episode: 8800, score: 1.0, mean_score: 2.42, std_score: 2.07\n",
"episode: 8800, score: 1.0, epsilon: 0.08\n",
"episode: 8805, score: 2.0, epsilon: 0.08\n",
"episode: 8810, score: 3.0, epsilon: 0.08\n",
"episode: 8815, score: 2.0, epsilon: 0.08\n",
"episode: 8820, score: 0.0, epsilon: 0.08\n",
"episode: 8825, score: 1.0, epsilon: 0.08\n",
"episode: 8830, score: 3.0, epsilon: 0.08\n",
"episode: 8835, score: 2.0, epsilon: 0.08\n",
"episode: 8840, score: 5.0, epsilon: 0.08\n",
"episode: 8845, score: 1.0, epsilon: 0.08\n",
"episode: 8850, score: 1.0, epsilon: 0.08\n",
"episode: 8855, score: 5.0, epsilon: 0.08\n",
"episode: 8860, score: 6.0, epsilon: 0.08\n",
"episode: 8865, score: 0.0, epsilon: 0.08\n",
"episode: 8870, score: 5.0, epsilon: 0.08\n",
"episode: 8875, score: 0.0, epsilon: 0.08\n",
"episode: 8880, score: 2.0, epsilon: 0.08\n",
"episode: 8885, score: 6.0, epsilon: 0.08\n",
"episode: 8890, score: 2.0, epsilon: 0.08\n",
"episode: 8895, score: 0.0, epsilon: 0.08\n",
"marking, episode: 8900, score: 0.0, mean_score: 2.23, std_score: 1.82\n",
"episode: 8900, score: 0.0, epsilon: 0.08\n",
"episode: 8905, score: 0.0, epsilon: 0.08\n",
"episode: 8910, score: 3.0, epsilon: 0.08\n",
"episode: 8915, score: 6.0, epsilon: 0.08\n",
"episode: 8920, score: 0.0, epsilon: 0.08\n",
"episode: 8925, score: 1.0, epsilon: 0.08\n",
"episode: 8930, score: 6.0, epsilon: 0.08\n",
"episode: 8935, score: 1.0, epsilon: 0.08\n",
"episode: 8940, score: 4.0, epsilon: 0.08\n",
"episode: 8945, score: 1.0, epsilon: 0.08\n",
"episode: 8950, score: 2.0, epsilon: 0.08\n",
"episode: 8955, score: 2.0, epsilon: 0.08\n",
"episode: 8960, score: 4.0, epsilon: 0.08\n",
"episode: 8965, score: 3.0, epsilon: 0.08\n",
"episode: 8970, score: 2.0, epsilon: 0.08\n",
"episode: 8975, score: 3.0, epsilon: 0.08\n",
"episode: 8980, score: 0.0, epsilon: 0.08\n",
"episode: 8985, score: 1.0, epsilon: 0.08\n",
"episode: 8990, score: 0.0, epsilon: 0.08\n",
"episode: 8995, score: 3.0, epsilon: 0.08\n",
"marking, episode: 9000, score: 2.0, mean_score: 2.30, std_score: 2.07\n",
"episode: 9000, score: 2.0, epsilon: 0.08\n",
"episode: 9005, score: 1.0, epsilon: 0.08\n",
"episode: 9010, score: 5.0, epsilon: 0.08\n",
"episode: 9015, score: 1.0, epsilon: 0.08\n",
"episode: 9020, score: 1.0, epsilon: 0.08\n",
"episode: 9025, score: 1.0, epsilon: 0.08\n",
"episode: 9030, score: 2.0, epsilon: 0.08\n",
"episode: 9035, score: 1.0, epsilon: 0.08\n",
"episode: 9040, score: 6.0, epsilon: 0.08\n",
"episode: 9045, score: 2.0, epsilon: 0.08\n",
"episode: 9050, score: 1.0, epsilon: 0.08\n",
"episode: 9055, score: 0.0, epsilon: 0.08\n",
"episode: 9060, score: 2.0, epsilon: 0.08\n",
"episode: 9065, score: 1.0, epsilon: 0.08\n",
"episode: 9070, score: 6.0, epsilon: 0.08\n",
"episode: 9075, score: 1.0, epsilon: 0.08\n",
"episode: 9080, score: 3.0, epsilon: 0.08\n",
"episode: 9085, score: 0.0, epsilon: 0.08\n",
"episode: 9090, score: 3.0, epsilon: 0.08\n",
"episode: 9095, score: 0.0, epsilon: 0.08\n",
"marking, episode: 9100, score: 1.0, mean_score: 2.40, std_score: 2.01\n",
"episode: 9100, score: 1.0, epsilon: 0.08\n",
"episode: 9105, score: 3.0, epsilon: 0.08\n",
"episode: 9110, score: 5.0, epsilon: 0.08\n",
"episode: 9115, score: 2.0, epsilon: 0.08\n",
"episode: 9120, score: 6.0, epsilon: 0.08\n",
"episode: 9125, score: 3.0, epsilon: 0.08\n",
"episode: 9130, score: 2.0, epsilon: 0.08\n",
"episode: 9135, score: 4.0, epsilon: 0.08\n",
"episode: 9140, score: 0.0, epsilon: 0.08\n",
"episode: 9145, score: 6.0, epsilon: 0.08\n",
"episode: 9150, score: 1.0, epsilon: 0.08\n",
"episode: 9155, score: 6.0, epsilon: 0.08\n",
"episode: 9160, score: 0.0, epsilon: 0.08\n",
"episode: 9165, score: 1.0, epsilon: 0.08\n",
"episode: 9170, score: 0.0, epsilon: 0.08\n",
"episode: 9175, score: 5.0, epsilon: 0.08\n",
"episode: 9180, score: 2.0, epsilon: 0.08\n",
"episode: 9185, score: 0.0, epsilon: 0.08\n",
"episode: 9190, score: 2.0, epsilon: 0.08\n",
"episode: 9195, score: 0.0, epsilon: 0.08\n",
"marking, episode: 9200, score: 1.0, mean_score: 2.70, std_score: 2.26\n",
"episode: 9200, score: 1.0, epsilon: 0.08\n",
"episode: 9205, score: 1.0, epsilon: 0.08\n",
"episode: 9210, score: 0.0, epsilon: 0.08\n",
"episode: 9215, score: 2.0, epsilon: 0.08\n",
"episode: 9220, score: 6.0, epsilon: 0.08\n",
"episode: 9225, score: 6.0, epsilon: 0.08\n",
"episode: 9230, score: 0.0, epsilon: 0.08\n",
"episode: 9235, score: 4.0, epsilon: 0.08\n",
"episode: 9240, score: 2.0, epsilon: 0.08\n",
"episode: 9245, score: 0.0, epsilon: 0.08\n",
"episode: 9250, score: 0.0, epsilon: 0.08\n",
"episode: 9255, score: 1.0, epsilon: 0.08\n",
"episode: 9260, score: 0.0, epsilon: 0.08\n",
"episode: 9265, score: 2.0, epsilon: 0.08\n",
"episode: 9270, score: 3.0, epsilon: 0.08\n",
"episode: 9275, score: 0.0, epsilon: 0.08\n",
"episode: 9280, score: 0.0, epsilon: 0.08\n",
"episode: 9285, score: 0.0, epsilon: 0.08\n",
"episode: 9290, score: 2.0, epsilon: 0.08\n",
"episode: 9295, score: 0.0, epsilon: 0.08\n",
"marking, episode: 9300, score: 0.0, mean_score: 2.12, std_score: 2.09\n",
"episode: 9300, score: 0.0, epsilon: 0.08\n",
"episode: 9305, score: 5.0, epsilon: 0.08\n",
"episode: 9310, score: 2.0, epsilon: 0.08\n",
"episode: 9315, score: 2.0, epsilon: 0.08\n",
"episode: 9320, score: 6.0, epsilon: 0.08\n",
"episode: 9325, score: 6.0, epsilon: 0.08\n",
"episode: 9330, score: 2.0, epsilon: 0.08\n",
"episode: 9335, score: 6.0, epsilon: 0.08\n",
"episode: 9340, score: 1.0, epsilon: 0.08\n",
"episode: 9345, score: 4.0, epsilon: 0.08\n",
"episode: 9350, score: 3.0, epsilon: 0.08\n",
"episode: 9355, score: 0.0, epsilon: 0.08\n",
"episode: 9360, score: 6.0, epsilon: 0.08\n",
"episode: 9365, score: 6.0, epsilon: 0.08\n",
"episode: 9370, score: 1.0, epsilon: 0.08\n",
"episode: 9375, score: 1.0, epsilon: 0.08\n",
"episode: 9380, score: 6.0, epsilon: 0.08\n",
"episode: 9385, score: 5.0, epsilon: 0.08\n",
"episode: 9390, score: 2.0, epsilon: 0.08\n",
"episode: 9395, score: 5.0, epsilon: 0.08\n",
"marking, episode: 9400, score: 4.0, mean_score: 2.68, std_score: 2.31\n",
"episode: 9400, score: 4.0, epsilon: 0.08\n",
"episode: 9405, score: 0.0, epsilon: 0.08\n",
"episode: 9410, score: 2.0, epsilon: 0.08\n",
"episode: 9415, score: 2.0, epsilon: 0.08\n",
"episode: 9420, score: 4.0, epsilon: 0.08\n",
"episode: 9425, score: 0.0, epsilon: 0.08\n",
"episode: 9430, score: 0.0, epsilon: 0.08\n",
"episode: 9435, score: 1.0, epsilon: 0.08\n",
"episode: 9440, score: 0.0, epsilon: 0.08\n",
"episode: 9445, score: 1.0, epsilon: 0.08\n",
"episode: 9450, score: 2.0, epsilon: 0.08\n",
"episode: 9455, score: 4.0, epsilon: 0.08\n",
"episode: 9460, score: 0.0, epsilon: 0.08\n",
"episode: 9465, score: 2.0, epsilon: 0.08\n",
"episode: 9470, score: 0.0, epsilon: 0.08\n",
"episode: 9475, score: 0.0, epsilon: 0.08\n",
"episode: 9480, score: 4.0, epsilon: 0.08\n",
"episode: 9485, score: 0.0, epsilon: 0.08\n",
"episode: 9490, score: 0.0, epsilon: 0.08\n",
"episode: 9495, score: 6.0, epsilon: 0.08\n",
"marking, episode: 9500, score: 3.0, mean_score: 2.20, std_score: 2.02\n",
"episode: 9500, score: 3.0, epsilon: 0.08\n",
"episode: 9505, score: 6.0, epsilon: 0.08\n",
"episode: 9510, score: 2.0, epsilon: 0.08\n",
"episode: 9515, score: 0.0, epsilon: 0.08\n",
"episode: 9520, score: 0.0, epsilon: 0.08\n",
"episode: 9525, score: 4.0, epsilon: 0.08\n",
"episode: 9530, score: 0.0, epsilon: 0.08\n",
"episode: 9535, score: 2.0, epsilon: 0.08\n",
"episode: 9540, score: 6.0, epsilon: 0.08\n",
"episode: 9545, score: 0.0, epsilon: 0.08\n",
"episode: 9550, score: 2.0, epsilon: 0.08\n",
"episode: 9555, score: 1.0, epsilon: 0.08\n",
"episode: 9560, score: 0.0, epsilon: 0.08\n",
"episode: 9565, score: 0.0, epsilon: 0.08\n",
"episode: 9570, score: 6.0, epsilon: 0.08\n",
"episode: 9575, score: 5.0, epsilon: 0.08\n",
"episode: 9580, score: 6.0, epsilon: 0.08\n",
"episode: 9585, score: 3.0, epsilon: 0.08\n",
"episode: 9590, score: 6.0, epsilon: 0.08\n",
"episode: 9595, score: 1.0, epsilon: 0.08\n",
"marking, episode: 9600, score: 6.0, mean_score: 2.19, std_score: 2.18\n",
"episode: 9600, score: 6.0, epsilon: 0.08\n",
"episode: 9605, score: 4.0, epsilon: 0.08\n",
"episode: 9610, score: 0.0, epsilon: 0.08\n",
"episode: 9615, score: 0.0, epsilon: 0.08\n",
"episode: 9620, score: 0.0, epsilon: 0.08\n",
"episode: 9625, score: 1.0, epsilon: 0.08\n",
"episode: 9630, score: 2.0, epsilon: 0.08\n",
"episode: 9635, score: 2.0, epsilon: 0.08\n",
"episode: 9640, score: 6.0, epsilon: 0.08\n",
"episode: 9645, score: 4.0, epsilon: 0.08\n",
"episode: 9650, score: 6.0, epsilon: 0.08\n",
"episode: 9655, score: 6.0, epsilon: 0.08\n",
"episode: 9660, score: 3.0, epsilon: 0.08\n",
"episode: 9665, score: 0.0, epsilon: 0.08\n",
"episode: 9670, score: 0.0, epsilon: 0.08\n",
"episode: 9675, score: 0.0, epsilon: 0.08\n",
"episode: 9680, score: 0.0, epsilon: 0.08\n",
"episode: 9685, score: 3.0, epsilon: 0.08\n",
"episode: 9690, score: 3.0, epsilon: 0.08\n",
"episode: 9695, score: 1.0, epsilon: 0.08\n",
"marking, episode: 9700, score: 0.0, mean_score: 2.32, std_score: 2.16\n",
"episode: 9700, score: 0.0, epsilon: 0.08\n",
"episode: 9705, score: 2.0, epsilon: 0.08\n",
"episode: 9710, score: 6.0, epsilon: 0.08\n",
"episode: 9715, score: 4.0, epsilon: 0.08\n",
"episode: 9720, score: 3.0, epsilon: 0.08\n",
"episode: 9725, score: 0.0, epsilon: 0.08\n",
"episode: 9730, score: 1.0, epsilon: 0.08\n",
"episode: 9735, score: 1.0, epsilon: 0.08\n",
"episode: 9740, score: 3.0, epsilon: 0.08\n",
"episode: 9745, score: 1.0, epsilon: 0.08\n",
"episode: 9750, score: 5.0, epsilon: 0.08\n",
"episode: 9755, score: 6.0, epsilon: 0.08\n",
"episode: 9760, score: 1.0, epsilon: 0.08\n",
"episode: 9765, score: 2.0, epsilon: 0.08\n",
"episode: 9770, score: 3.0, epsilon: 0.08\n",
"episode: 9775, score: 3.0, epsilon: 0.08\n",
"episode: 9780, score: 0.0, epsilon: 0.08\n",
"episode: 9785, score: 0.0, epsilon: 0.08\n",
"episode: 9790, score: 0.0, epsilon: 0.08\n",
"episode: 9795, score: 3.0, epsilon: 0.08\n",
"marking, episode: 9800, score: 6.0, mean_score: 2.34, std_score: 2.05\n",
"episode: 9800, score: 6.0, epsilon: 0.08\n",
"episode: 9805, score: 4.0, epsilon: 0.08\n",
"episode: 9810, score: 0.0, epsilon: 0.08\n",
"episode: 9815, score: 5.0, epsilon: 0.08\n",
"episode: 9820, score: 0.0, epsilon: 0.08\n",
"episode: 9825, score: 3.0, epsilon: 0.08\n",
"episode: 9830, score: 0.0, epsilon: 0.08\n",
"episode: 9835, score: 2.0, epsilon: 0.08\n",
"episode: 9840, score: 1.0, epsilon: 0.08\n",
"episode: 9845, score: 0.0, epsilon: 0.08\n",
"episode: 9850, score: 5.0, epsilon: 0.08\n",
"episode: 9855, score: 2.0, epsilon: 0.08\n",
"episode: 9860, score: 2.0, epsilon: 0.08\n",
"episode: 9865, score: 3.0, epsilon: 0.08\n",
"episode: 9870, score: 0.0, epsilon: 0.08\n",
"episode: 9875, score: 2.0, epsilon: 0.08\n",
"episode: 9880, score: 3.0, epsilon: 0.08\n",
"episode: 9885, score: 1.0, epsilon: 0.08\n",
"episode: 9890, score: 4.0, epsilon: 0.08\n",
"episode: 9895, score: 0.0, epsilon: 0.08\n",
"marking, episode: 9900, score: 6.0, mean_score: 2.53, std_score: 2.19\n",
"episode: 9900, score: 6.0, epsilon: 0.08\n",
"episode: 9905, score: 0.0, epsilon: 0.08\n",
"episode: 9910, score: 0.0, epsilon: 0.08\n",
"episode: 9915, score: 5.0, epsilon: 0.08\n",
"episode: 9920, score: 5.0, epsilon: 0.08\n",
"episode: 9925, score: 1.0, epsilon: 0.08\n",
"episode: 9930, score: 1.0, epsilon: 0.08\n",
"episode: 9935, score: 2.0, epsilon: 0.08\n",
"episode: 9940, score: 1.0, epsilon: 0.08\n",
"episode: 9945, score: 6.0, epsilon: 0.08\n",
"episode: 9950, score: 1.0, epsilon: 0.08\n",
"episode: 9955, score: 2.0, epsilon: 0.08\n",
"episode: 9960, score: 0.0, epsilon: 0.08\n",
"episode: 9965, score: 0.0, epsilon: 0.08\n",
"episode: 9970, score: 0.0, epsilon: 0.08\n",
"episode: 9975, score: 2.0, epsilon: 0.08\n",
"episode: 9980, score: 2.0, epsilon: 0.08\n",
"episode: 9985, score: 2.0, epsilon: 0.08\n",
"episode: 9990, score: 2.0, epsilon: 0.08\n",
"episode: 9995, score: 0.0, epsilon: 0.08\n",
"marking, episode: 10000, score: 5.0, mean_score: 2.35, std_score: 2.07\n",
"episode: 10000, score: 5.0, epsilon: 0.08\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "LPgQkpSWrb7C"
},
"source": [
"class OptimalPolicy(object):\n",
" def __init__(self):\n",
" super().__init__()\n",
"\n",
" def calc(self, x):\n",
" cardsleft = torch.ones(highest_card+1)\n",
" for card in x:\n",
" newval = card.argmax().item()\n",
" cardsleft[newval] = 0\n",
" if newval == 0:\n",
" break\n",
" else:\n",
" val = newval\n",
" lower_sum = cardsleft[1:val].sum()\n",
" higher_sum = cardsleft[val:].sum()\n",
" action = int(lower_sum < higher_sum)\n",
" return action\n",
" \n",
" def sample_action(self, obs, epsilon):\n",
" out = self.calc(obs)\n",
" coin = random.random()\n",
" if coin < epsilon:\n",
" return random.randint(0,1)\n",
" else : \n",
" return out\n",
"\n",
"class RandomPolicy(object):\n",
" def __init__(self):\n",
" super().__init__()\n",
"\n",
" def sample_action(self, obs, epsilon):\n",
" return random.randint(0,1)\n"
],
"execution_count": 21,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "Us3xPaDIreGe"
},
"source": [
"good_pol = OptimalPolicy()\n",
"bad_pol = RandomPolicy()"
],
"execution_count": 22,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "f8bNTJxIsJs_",
"outputId": "e35da819-01c5-480f-ebba-a2c8ec45e793",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 1000
}
},
"source": [
"marking = []\n",
"opt_scores = []\n",
"for n_episode in range(10001):\n",
" epsilon = 0.08\n",
" s = env.reset()\n",
" done = False\n",
" score = 0.0\n",
"\n",
" while True:\n",
" a = good_pol.sample_action(torch.from_numpy(s).float(), epsilon)\n",
" s_prime, r, done, info = env.step(a)\n",
" s = s_prime\n",
" score += r\n",
" if done:\n",
" break\n",
" opt_scores.append(score)\n",
"\n",
" # do not change lines 44-48 here, they are for marking the submission log\n",
" marking.append(score)\n",
" if n_episode%100 == 0:\n",
" print(\"marking, episode: {}, score: {:.1f}, mean_score: {:.2f}, std_score: {:.2f}\".format(\n",
" n_episode, score, np.array(marking).mean(), np.array(marking).std()))\n",
" marking = []\n",
"\n",
" # you can change this part, and print any data you like (so long as it doesn't start with \"marking\")\n",
" if n_episode%print_every==0 and n_episode!=0:\n",
" print(\"episode: {}, score: {:.1f}, epsilon: {:.2f}\".format(n_episode, score, epsilon))"
],
"execution_count": 23,
"outputs": [
{
"output_type": "stream",
"text": [
"marking, episode: 0, score: 1.0, mean_score: 1.00, std_score: 0.00\n",
"episode: 5, score: 0.0, epsilon: 0.08\n",
"episode: 10, score: 6.0, epsilon: 0.08\n",
"episode: 15, score: 2.0, epsilon: 0.08\n",
"episode: 20, score: 0.0, epsilon: 0.08\n",
"episode: 25, score: 4.0, epsilon: 0.08\n",
"episode: 30, score: 4.0, epsilon: 0.08\n",
"episode: 35, score: 6.0, epsilon: 0.08\n",
"episode: 40, score: 1.0, epsilon: 0.08\n",
"episode: 45, score: 6.0, epsilon: 0.08\n",
"episode: 50, score: 0.0, epsilon: 0.08\n",
"episode: 55, score: 0.0, epsilon: 0.08\n",
"episode: 60, score: 2.0, epsilon: 0.08\n",
"episode: 65, score: 3.0, epsilon: 0.08\n",
"episode: 70, score: 4.0, epsilon: 0.08\n",
"episode: 75, score: 4.0, epsilon: 0.08\n",
"episode: 80, score: 4.0, epsilon: 0.08\n",
"episode: 85, score: 4.0, epsilon: 0.08\n",
"episode: 90, score: 2.0, epsilon: 0.08\n",
"episode: 95, score: 3.0, epsilon: 0.08\n",
"marking, episode: 100, score: 2.0, mean_score: 2.18, std_score: 1.97\n",
"episode: 100, score: 2.0, epsilon: 0.08\n",
"episode: 105, score: 5.0, epsilon: 0.08\n",
"episode: 110, score: 3.0, epsilon: 0.08\n",
"episode: 115, score: 0.0, epsilon: 0.08\n",
"episode: 120, score: 2.0, epsilon: 0.08\n",
"episode: 125, score: 0.0, epsilon: 0.08\n",
"episode: 130, score: 0.0, epsilon: 0.08\n",
"episode: 135, score: 0.0, epsilon: 0.08\n",
"episode: 140, score: 1.0, epsilon: 0.08\n",
"episode: 145, score: 6.0, epsilon: 0.08\n",
"episode: 150, score: 0.0, epsilon: 0.08\n",
"episode: 155, score: 2.0, epsilon: 0.08\n",
"episode: 160, score: 0.0, epsilon: 0.08\n",
"episode: 165, score: 6.0, epsilon: 0.08\n",
"episode: 170, score: 4.0, epsilon: 0.08\n",
"episode: 175, score: 6.0, epsilon: 0.08\n",
"episode: 180, score: 5.0, epsilon: 0.08\n",
"episode: 185, score: 1.0, epsilon: 0.08\n",
"episode: 190, score: 1.0, epsilon: 0.08\n",
"episode: 195, score: 1.0, epsilon: 0.08\n",
"marking, episode: 200, score: 3.0, mean_score: 2.13, std_score: 2.06\n",
"episode: 200, score: 3.0, epsilon: 0.08\n",
"episode: 205, score: 0.0, epsilon: 0.08\n",
"episode: 210, score: 0.0, epsilon: 0.08\n",
"episode: 215, score: 6.0, epsilon: 0.08\n",
"episode: 220, score: 0.0, epsilon: 0.08\n",
"episode: 225, score: 0.0, epsilon: 0.08\n",
"episode: 230, score: 5.0, epsilon: 0.08\n",
"episode: 235, score: 1.0, epsilon: 0.08\n",
"episode: 240, score: 2.0, epsilon: 0.08\n",
"episode: 245, score: 1.0, epsilon: 0.08\n",
"episode: 250, score: 6.0, epsilon: 0.08\n",
"episode: 255, score: 6.0, epsilon: 0.08\n",
"episode: 260, score: 0.0, epsilon: 0.08\n",
"episode: 265, score: 0.0, epsilon: 0.08\n",
"episode: 270, score: 0.0, epsilon: 0.08\n",
"episode: 275, score: 0.0, epsilon: 0.08\n",
"episode: 280, score: 1.0, epsilon: 0.08\n",
"episode: 285, score: 1.0, epsilon: 0.08\n",
"episode: 290, score: 1.0, epsilon: 0.08\n",
"episode: 295, score: 6.0, epsilon: 0.08\n",
"marking, episode: 300, score: 3.0, mean_score: 2.42, std_score: 2.23\n",
"episode: 300, score: 3.0, epsilon: 0.08\n",
"episode: 305, score: 0.0, epsilon: 0.08\n",
"episode: 310, score: 0.0, epsilon: 0.08\n",
"episode: 315, score: 2.0, epsilon: 0.08\n",
"episode: 320, score: 5.0, epsilon: 0.08\n",
"episode: 325, score: 6.0, epsilon: 0.08\n",
"episode: 330, score: 4.0, epsilon: 0.08\n",
"episode: 335, score: 0.0, epsilon: 0.08\n",
"episode: 340, score: 1.0, epsilon: 0.08\n",
"episode: 345, score: 4.0, epsilon: 0.08\n",
"episode: 350, score: 1.0, epsilon: 0.08\n",
"episode: 355, score: 0.0, epsilon: 0.08\n",
"episode: 360, score: 6.0, epsilon: 0.08\n",
"episode: 365, score: 3.0, epsilon: 0.08\n",
"episode: 370, score: 2.0, epsilon: 0.08\n",
"episode: 375, score: 1.0, epsilon: 0.08\n",
"episode: 380, score: 6.0, epsilon: 0.08\n",
"episode: 385, score: 1.0, epsilon: 0.08\n",
"episode: 390, score: 4.0, epsilon: 0.08\n",
"episode: 395, score: 1.0, epsilon: 0.08\n",
"marking, episode: 400, score: 1.0, mean_score: 2.51, std_score: 2.14\n",
"episode: 400, score: 1.0, epsilon: 0.08\n",
"episode: 405, score: 0.0, epsilon: 0.08\n",
"episode: 410, score: 1.0, epsilon: 0.08\n",
"episode: 415, score: 0.0, epsilon: 0.08\n",
"episode: 420, score: 6.0, epsilon: 0.08\n",
"episode: 425, score: 3.0, epsilon: 0.08\n",
"episode: 430, score: 3.0, epsilon: 0.08\n",
"episode: 435, score: 0.0, epsilon: 0.08\n",
"episode: 440, score: 6.0, epsilon: 0.08\n",
"episode: 445, score: 1.0, epsilon: 0.08\n",
"episode: 450, score: 6.0, epsilon: 0.08\n",
"episode: 455, score: 6.0, epsilon: 0.08\n",
"episode: 460, score: 0.0, epsilon: 0.08\n",
"episode: 465, score: 3.0, epsilon: 0.08\n",
"episode: 470, score: 0.0, epsilon: 0.08\n",
"episode: 475, score: 6.0, epsilon: 0.08\n",
"episode: 480, score: 4.0, epsilon: 0.08\n",
"episode: 485, score: 3.0, epsilon: 0.08\n",
"episode: 490, score: 2.0, epsilon: 0.08\n",
"episode: 495, score: 2.0, epsilon: 0.08\n",
"marking, episode: 500, score: 3.0, mean_score: 2.69, std_score: 2.24\n",
"episode: 500, score: 3.0, epsilon: 0.08\n",
"episode: 505, score: 0.0, epsilon: 0.08\n",
"episode: 510, score: 6.0, epsilon: 0.08\n",
"episode: 515, score: 6.0, epsilon: 0.08\n",
"episode: 520, score: 2.0, epsilon: 0.08\n",
"episode: 525, score: 3.0, epsilon: 0.08\n",
"episode: 530, score: 0.0, epsilon: 0.08\n",
"episode: 535, score: 6.0, epsilon: 0.08\n",
"episode: 540, score: 6.0, epsilon: 0.08\n",
"episode: 545, score: 2.0, epsilon: 0.08\n",
"episode: 550, score: 6.0, epsilon: 0.08\n",
"episode: 555, score: 6.0, epsilon: 0.08\n",
"episode: 560, score: 0.0, epsilon: 0.08\n",
"episode: 565, score: 6.0, epsilon: 0.08\n",
"episode: 570, score: 4.0, epsilon: 0.08\n",
"episode: 575, score: 0.0, epsilon: 0.08\n",
"episode: 580, score: 6.0, epsilon: 0.08\n",
"episode: 585, score: 0.0, epsilon: 0.08\n",
"episode: 590, score: 3.0, epsilon: 0.08\n",
"episode: 595, score: 6.0, epsilon: 0.08\n",
"marking, episode: 600, score: 0.0, mean_score: 2.45, std_score: 2.34\n",
"episode: 600, score: 0.0, epsilon: 0.08\n",
"episode: 605, score: 6.0, epsilon: 0.08\n",
"episode: 610, score: 0.0, epsilon: 0.08\n",
"episode: 615, score: 4.0, epsilon: 0.08\n",
"episode: 620, score: 2.0, epsilon: 0.08\n",
"episode: 625, score: 0.0, epsilon: 0.08\n",
"episode: 630, score: 2.0, epsilon: 0.08\n",
"episode: 635, score: 6.0, epsilon: 0.08\n",
"episode: 640, score: 0.0, epsilon: 0.08\n",
"episode: 645, score: 4.0, epsilon: 0.08\n",
"episode: 650, score: 1.0, epsilon: 0.08\n",
"episode: 655, score: 6.0, epsilon: 0.08\n",
"episode: 660, score: 2.0, epsilon: 0.08\n",
"episode: 665, score: 0.0, epsilon: 0.08\n",
"episode: 670, score: 3.0, epsilon: 0.08\n",
"episode: 675, score: 6.0, epsilon: 0.08\n",
"episode: 680, score: 6.0, epsilon: 0.08\n",
"episode: 685, score: 0.0, epsilon: 0.08\n",
"episode: 690, score: 2.0, epsilon: 0.08\n",
"episode: 695, score: 0.0, epsilon: 0.08\n",
"marking, episode: 700, score: 0.0, mean_score: 2.16, std_score: 2.25\n",
"episode: 700, score: 0.0, epsilon: 0.08\n",
"episode: 705, score: 0.0, epsilon: 0.08\n",
"episode: 710, score: 6.0, epsilon: 0.08\n",
"episode: 715, score: 6.0, epsilon: 0.08\n",
"episode: 720, score: 5.0, epsilon: 0.08\n",
"episode: 725, score: 6.0, epsilon: 0.08\n",
"episode: 730, score: 0.0, epsilon: 0.08\n",
"episode: 735, score: 3.0, epsilon: 0.08\n",
"episode: 740, score: 3.0, epsilon: 0.08\n",
"episode: 745, score: 2.0, epsilon: 0.08\n",
"episode: 750, score: 1.0, epsilon: 0.08\n",
"episode: 755, score: 1.0, epsilon: 0.08\n",
"episode: 760, score: 1.0, epsilon: 0.08\n",
"episode: 765, score: 0.0, epsilon: 0.08\n",
"episode: 770, score: 2.0, epsilon: 0.08\n",
"episode: 775, score: 1.0, epsilon: 0.08\n",
"episode: 780, score: 1.0, epsilon: 0.08\n",
"episode: 785, score: 2.0, epsilon: 0.08\n",
"episode: 790, score: 2.0, epsilon: 0.08\n",
"episode: 795, score: 5.0, epsilon: 0.08\n",
"marking, episode: 800, score: 0.0, mean_score: 2.55, std_score: 2.29\n",
"episode: 800, score: 0.0, epsilon: 0.08\n",
"episode: 805, score: 2.0, epsilon: 0.08\n",
"episode: 810, score: 1.0, epsilon: 0.08\n",
"episode: 815, score: 2.0, epsilon: 0.08\n",
"episode: 820, score: 4.0, epsilon: 0.08\n",
"episode: 825, score: 0.0, epsilon: 0.08\n",
"episode: 830, score: 6.0, epsilon: 0.08\n",
"episode: 835, score: 0.0, epsilon: 0.08\n",
"episode: 840, score: 2.0, epsilon: 0.08\n",
"episode: 845, score: 0.0, epsilon: 0.08\n",
"episode: 850, score: 1.0, epsilon: 0.08\n",
"episode: 855, score: 1.0, epsilon: 0.08\n",
"episode: 860, score: 3.0, epsilon: 0.08\n",
"episode: 865, score: 0.0, epsilon: 0.08\n",
"episode: 870, score: 0.0, epsilon: 0.08\n",
"episode: 875, score: 6.0, epsilon: 0.08\n",
"episode: 880, score: 1.0, epsilon: 0.08\n",
"episode: 885, score: 2.0, epsilon: 0.08\n",
"episode: 890, score: 6.0, epsilon: 0.08\n",
"episode: 895, score: 1.0, epsilon: 0.08\n",
"marking, episode: 900, score: 6.0, mean_score: 2.09, std_score: 1.92\n",
"episode: 900, score: 6.0, epsilon: 0.08\n",
"episode: 905, score: 6.0, epsilon: 0.08\n",
"episode: 910, score: 5.0, epsilon: 0.08\n",
"episode: 915, score: 3.0, epsilon: 0.08\n",
"episode: 920, score: 2.0, epsilon: 0.08\n",
"episode: 925, score: 4.0, epsilon: 0.08\n",
"episode: 930, score: 2.0, epsilon: 0.08\n",
"episode: 935, score: 0.0, epsilon: 0.08\n",
"episode: 940, score: 2.0, epsilon: 0.08\n",
"episode: 945, score: 3.0, epsilon: 0.08\n",
"episode: 950, score: 6.0, epsilon: 0.08\n",
"episode: 955, score: 1.0, epsilon: 0.08\n",
"episode: 960, score: 6.0, epsilon: 0.08\n",
"episode: 965, score: 1.0, epsilon: 0.08\n",
"episode: 970, score: 0.0, epsilon: 0.08\n",
"episode: 975, score: 2.0, epsilon: 0.08\n",
"episode: 980, score: 0.0, epsilon: 0.08\n",
"episode: 985, score: 6.0, epsilon: 0.08\n",
"episode: 990, score: 0.0, epsilon: 0.08\n",
"episode: 995, score: 0.0, epsilon: 0.08\n",
"marking, episode: 1000, score: 1.0, mean_score: 2.31, std_score: 2.17\n",
"episode: 1000, score: 1.0, epsilon: 0.08\n",
"episode: 1005, score: 6.0, epsilon: 0.08\n",
"episode: 1010, score: 1.0, epsilon: 0.08\n",
"episode: 1015, score: 0.0, epsilon: 0.08\n",
"episode: 1020, score: 3.0, epsilon: 0.08\n",
"episode: 1025, score: 1.0, epsilon: 0.08\n",
"episode: 1030, score: 0.0, epsilon: 0.08\n",
"episode: 1035, score: 2.0, epsilon: 0.08\n",
"episode: 1040, score: 2.0, epsilon: 0.08\n",
"episode: 1045, score: 1.0, epsilon: 0.08\n",
"episode: 1050, score: 0.0, epsilon: 0.08\n",
"episode: 1055, score: 0.0, epsilon: 0.08\n",
"episode: 1060, score: 0.0, epsilon: 0.08\n",
"episode: 1065, score: 4.0, epsilon: 0.08\n",
"episode: 1070, score: 6.0, epsilon: 0.08\n",
"episode: 1075, score: 2.0, epsilon: 0.08\n",
"episode: 1080, score: 6.0, epsilon: 0.08\n",
"episode: 1085, score: 3.0, epsilon: 0.08\n",
"episode: 1090, score: 6.0, epsilon: 0.08\n",
"episode: 1095, score: 3.0, epsilon: 0.08\n",
"marking, episode: 1100, score: 3.0, mean_score: 2.58, std_score: 2.14\n",
"episode: 1100, score: 3.0, epsilon: 0.08\n",
"episode: 1105, score: 1.0, epsilon: 0.08\n",
"episode: 1110, score: 3.0, epsilon: 0.08\n",
"episode: 1115, score: 2.0, epsilon: 0.08\n",
"episode: 1120, score: 6.0, epsilon: 0.08\n",
"episode: 1125, score: 6.0, epsilon: 0.08\n",
"episode: 1130, score: 4.0, epsilon: 0.08\n",
"episode: 1135, score: 3.0, epsilon: 0.08\n",
"episode: 1140, score: 5.0, epsilon: 0.08\n",
"episode: 1145, score: 0.0, epsilon: 0.08\n",
"episode: 1150, score: 4.0, epsilon: 0.08\n",
"episode: 1155, score: 0.0, epsilon: 0.08\n",
"episode: 1160, score: 0.0, epsilon: 0.08\n",
"episode: 1165, score: 1.0, epsilon: 0.08\n",
"episode: 1170, score: 5.0, epsilon: 0.08\n",
"episode: 1175, score: 2.0, epsilon: 0.08\n",
"episode: 1180, score: 2.0, epsilon: 0.08\n",
"episode: 1185, score: 6.0, epsilon: 0.08\n",
"episode: 1190, score: 5.0, epsilon: 0.08\n",
"episode: 1195, score: 3.0, epsilon: 0.08\n",
"marking, episode: 1200, score: 6.0, mean_score: 2.55, std_score: 2.15\n",
"episode: 1200, score: 6.0, epsilon: 0.08\n",
"episode: 1205, score: 2.0, epsilon: 0.08\n",
"episode: 1210, score: 3.0, epsilon: 0.08\n",
"episode: 1215, score: 6.0, epsilon: 0.08\n",
"episode: 1220, score: 0.0, epsilon: 0.08\n",
"episode: 1225, score: 0.0, epsilon: 0.08\n",
"episode: 1230, score: 1.0, epsilon: 0.08\n",
"episode: 1235, score: 2.0, epsilon: 0.08\n",
"episode: 1240, score: 1.0, epsilon: 0.08\n",
"episode: 1245, score: 5.0, epsilon: 0.08\n",
"episode: 1250, score: 0.0, epsilon: 0.08\n",
"episode: 1255, score: 4.0, epsilon: 0.08\n",
"episode: 1260, score: 6.0, epsilon: 0.08\n",
"episode: 1265, score: 6.0, epsilon: 0.08\n",
"episode: 1270, score: 1.0, epsilon: 0.08\n",
"episode: 1275, score: 3.0, epsilon: 0.08\n",
"episode: 1280, score: 1.0, epsilon: 0.08\n",
"episode: 1285, score: 2.0, epsilon: 0.08\n",
"episode: 1290, score: 2.0, epsilon: 0.08\n",
"episode: 1295, score: 6.0, epsilon: 0.08\n",
"marking, episode: 1300, score: 2.0, mean_score: 2.46, std_score: 2.12\n",
"episode: 1300, score: 2.0, epsilon: 0.08\n",
"episode: 1305, score: 0.0, epsilon: 0.08\n",
"episode: 1310, score: 0.0, epsilon: 0.08\n",
"episode: 1315, score: 0.0, epsilon: 0.08\n",
"episode: 1320, score: 0.0, epsilon: 0.08\n",
"episode: 1325, score: 0.0, epsilon: 0.08\n",
"episode: 1330, score: 1.0, epsilon: 0.08\n",
"episode: 1335, score: 6.0, epsilon: 0.08\n",
"episode: 1340, score: 6.0, epsilon: 0.08\n",
"episode: 1345, score: 0.0, epsilon: 0.08\n",
"episode: 1350, score: 1.0, epsilon: 0.08\n",
"episode: 1355, score: 1.0, epsilon: 0.08\n",
"episode: 1360, score: 1.0, epsilon: 0.08\n",
"episode: 1365, score: 2.0, epsilon: 0.08\n",
"episode: 1370, score: 2.0, epsilon: 0.08\n",
"episode: 1375, score: 5.0, epsilon: 0.08\n",
"episode: 1380, score: 0.0, epsilon: 0.08\n",
"episode: 1385, score: 2.0, epsilon: 0.08\n",
"episode: 1390, score: 0.0, epsilon: 0.08\n",
"episode: 1395, score: 1.0, epsilon: 0.08\n",
"marking, episode: 1400, score: 6.0, mean_score: 2.10, std_score: 2.10\n",
"episode: 1400, score: 6.0, epsilon: 0.08\n",
"episode: 1405, score: 3.0, epsilon: 0.08\n",
"episode: 1410, score: 2.0, epsilon: 0.08\n",
"episode: 1415, score: 6.0, epsilon: 0.08\n",
"episode: 1420, score: 0.0, epsilon: 0.08\n",
"episode: 1425, score: 0.0, epsilon: 0.08\n",
"episode: 1430, score: 1.0, epsilon: 0.08\n",
"episode: 1435, score: 0.0, epsilon: 0.08\n",
"episode: 1440, score: 0.0, epsilon: 0.08\n",
"episode: 1445, score: 0.0, epsilon: 0.08\n",
"episode: 1450, score: 1.0, epsilon: 0.08\n",
"episode: 1455, score: 4.0, epsilon: 0.08\n",
"episode: 1460, score: 2.0, epsilon: 0.08\n",
"episode: 1465, score: 3.0, epsilon: 0.08\n",
"episode: 1470, score: 2.0, epsilon: 0.08\n",
"episode: 1475, score: 6.0, epsilon: 0.08\n",
"episode: 1480, score: 3.0, epsilon: 0.08\n",
"episode: 1485, score: 4.0, epsilon: 0.08\n",
"episode: 1490, score: 3.0, epsilon: 0.08\n",
"episode: 1495, score: 1.0, epsilon: 0.08\n",
"marking, episode: 1500, score: 1.0, mean_score: 2.47, std_score: 2.24\n",
"episode: 1500, score: 1.0, epsilon: 0.08\n",
"episode: 1505, score: 2.0, epsilon: 0.08\n",
"episode: 1510, score: 6.0, epsilon: 0.08\n",
"episode: 1515, score: 4.0, epsilon: 0.08\n",
"episode: 1520, score: 6.0, epsilon: 0.08\n",
"episode: 1525, score: 2.0, epsilon: 0.08\n",
"episode: 1530, score: 6.0, epsilon: 0.08\n",
"episode: 1535, score: 1.0, epsilon: 0.08\n",
"episode: 1540, score: 4.0, epsilon: 0.08\n",
"episode: 1545, score: 2.0, epsilon: 0.08\n",
"episode: 1550, score: 0.0, epsilon: 0.08\n",
"episode: 1555, score: 0.0, epsilon: 0.08\n",
"episode: 1560, score: 2.0, epsilon: 0.08\n",
"episode: 1565, score: 6.0, epsilon: 0.08\n",
"episode: 1570, score: 0.0, epsilon: 0.08\n",
"episode: 1575, score: 2.0, epsilon: 0.08\n",
"episode: 1580, score: 0.0, epsilon: 0.08\n",
"episode: 1585, score: 6.0, epsilon: 0.08\n",
"episode: 1590, score: 2.0, epsilon: 0.08\n",
"episode: 1595, score: 4.0, epsilon: 0.08\n",
"marking, episode: 1600, score: 3.0, mean_score: 2.78, std_score: 2.13\n",
"episode: 1600, score: 3.0, epsilon: 0.08\n",
"episode: 1605, score: 6.0, epsilon: 0.08\n",
"episode: 1610, score: 6.0, epsilon: 0.08\n",
"episode: 1615, score: 6.0, epsilon: 0.08\n",
"episode: 1620, score: 0.0, epsilon: 0.08\n",
"episode: 1625, score: 1.0, epsilon: 0.08\n",
"episode: 1630, score: 0.0, epsilon: 0.08\n",
"episode: 1635, score: 1.0, epsilon: 0.08\n",
"episode: 1640, score: 2.0, epsilon: 0.08\n",
"episode: 1645, score: 2.0, epsilon: 0.08\n",
"episode: 1650, score: 0.0, epsilon: 0.08\n",
"episode: 1655, score: 6.0, epsilon: 0.08\n",
"episode: 1660, score: 3.0, epsilon: 0.08\n",
"episode: 1665, score: 1.0, epsilon: 0.08\n",
"episode: 1670, score: 0.0, epsilon: 0.08\n",
"episode: 1675, score: 1.0, epsilon: 0.08\n",
"episode: 1680, score: 2.0, epsilon: 0.08\n",
"episode: 1685, score: 2.0, epsilon: 0.08\n",
"episode: 1690, score: 6.0, epsilon: 0.08\n",
"episode: 1695, score: 6.0, epsilon: 0.08\n",
"marking, episode: 1700, score: 0.0, mean_score: 2.37, std_score: 2.32\n",
"episode: 1700, score: 0.0, epsilon: 0.08\n",
"episode: 1705, score: 1.0, epsilon: 0.08\n",
"episode: 1710, score: 0.0, epsilon: 0.08\n",
"episode: 1715, score: 4.0, epsilon: 0.08\n",
"episode: 1720, score: 6.0, epsilon: 0.08\n",
"episode: 1725, score: 0.0, epsilon: 0.08\n",
"episode: 1730, score: 1.0, epsilon: 0.08\n",
"episode: 1735, score: 2.0, epsilon: 0.08\n",
"episode: 1740, score: 1.0, epsilon: 0.08\n",
"episode: 1745, score: 6.0, epsilon: 0.08\n",
"episode: 1750, score: 6.0, epsilon: 0.08\n",
"episode: 1755, score: 1.0, epsilon: 0.08\n",
"episode: 1760, score: 5.0, epsilon: 0.08\n",
"episode: 1765, score: 2.0, epsilon: 0.08\n",
"episode: 1770, score: 4.0, epsilon: 0.08\n",
"episode: 1775, score: 0.0, epsilon: 0.08\n",
"episode: 1780, score: 5.0, epsilon: 0.08\n",
"episode: 1785, score: 6.0, epsilon: 0.08\n",
"episode: 1790, score: 4.0, epsilon: 0.08\n",
"episode: 1795, score: 0.0, epsilon: 0.08\n",
"marking, episode: 1800, score: 6.0, mean_score: 2.87, std_score: 2.28\n",
"episode: 1800, score: 6.0, epsilon: 0.08\n",
"episode: 1805, score: 6.0, epsilon: 0.08\n",
"episode: 1810, score: 0.0, epsilon: 0.08\n",
"episode: 1815, score: 0.0, epsilon: 0.08\n",
"episode: 1820, score: 3.0, epsilon: 0.08\n",
"episode: 1825, score: 6.0, epsilon: 0.08\n",
"episode: 1830, score: 2.0, epsilon: 0.08\n",
"episode: 1835, score: 6.0, epsilon: 0.08\n",
"episode: 1840, score: 6.0, epsilon: 0.08\n",
"episode: 1845, score: 0.0, epsilon: 0.08\n",
"episode: 1850, score: 6.0, epsilon: 0.08\n",
"episode: 1855, score: 2.0, epsilon: 0.08\n",
"episode: 1860, score: 1.0, epsilon: 0.08\n",
"episode: 1865, score: 6.0, epsilon: 0.08\n",
"episode: 1870, score: 6.0, epsilon: 0.08\n",
"episode: 1875, score: 1.0, epsilon: 0.08\n",
"episode: 1880, score: 6.0, epsilon: 0.08\n",
"episode: 1885, score: 0.0, epsilon: 0.08\n",
"episode: 1890, score: 6.0, epsilon: 0.08\n",
"episode: 1895, score: 0.0, epsilon: 0.08\n",
"marking, episode: 1900, score: 0.0, mean_score: 2.81, std_score: 2.25\n",
"episode: 1900, score: 0.0, epsilon: 0.08\n",
"episode: 1905, score: 1.0, epsilon: 0.08\n",
"episode: 1910, score: 3.0, epsilon: 0.08\n",
"episode: 1915, score: 6.0, epsilon: 0.08\n",
"episode: 1920, score: 6.0, epsilon: 0.08\n",
"episode: 1925, score: 0.0, epsilon: 0.08\n",
"episode: 1930, score: 1.0, epsilon: 0.08\n",
"episode: 1935, score: 0.0, epsilon: 0.08\n",
"episode: 1940, score: 0.0, epsilon: 0.08\n",
"episode: 1945, score: 1.0, epsilon: 0.08\n",
"episode: 1950, score: 1.0, epsilon: 0.08\n",
"episode: 1955, score: 1.0, epsilon: 0.08\n",
"episode: 1960, score: 0.0, epsilon: 0.08\n",
"episode: 1965, score: 0.0, epsilon: 0.08\n",
"episode: 1970, score: 6.0, epsilon: 0.08\n",
"episode: 1975, score: 0.0, epsilon: 0.08\n",
"episode: 1980, score: 2.0, epsilon: 0.08\n",
"episode: 1985, score: 2.0, epsilon: 0.08\n",
"episode: 1990, score: 2.0, epsilon: 0.08\n",
"episode: 1995, score: 4.0, epsilon: 0.08\n",
"marking, episode: 2000, score: 3.0, mean_score: 2.35, std_score: 2.22\n",
"episode: 2000, score: 3.0, epsilon: 0.08\n",
"episode: 2005, score: 1.0, epsilon: 0.08\n",
"episode: 2010, score: 2.0, epsilon: 0.08\n",
"episode: 2015, score: 0.0, epsilon: 0.08\n",
"episode: 2020, score: 0.0, epsilon: 0.08\n",
"episode: 2025, score: 0.0, epsilon: 0.08\n",
"episode: 2030, score: 1.0, epsilon: 0.08\n",
"episode: 2035, score: 2.0, epsilon: 0.08\n",
"episode: 2040, score: 3.0, epsilon: 0.08\n",
"episode: 2045, score: 1.0, epsilon: 0.08\n",
"episode: 2050, score: 0.0, epsilon: 0.08\n",
"episode: 2055, score: 1.0, epsilon: 0.08\n",
"episode: 2060, score: 6.0, epsilon: 0.08\n",
"episode: 2065, score: 6.0, epsilon: 0.08\n",
"episode: 2070, score: 3.0, epsilon: 0.08\n",
"episode: 2075, score: 4.0, epsilon: 0.08\n",
"episode: 2080, score: 5.0, epsilon: 0.08\n",
"episode: 2085, score: 2.0, epsilon: 0.08\n",
"episode: 2090, score: 4.0, epsilon: 0.08\n",
"episode: 2095, score: 1.0, epsilon: 0.08\n",
"marking, episode: 2100, score: 3.0, mean_score: 2.21, std_score: 2.16\n",
"episode: 2100, score: 3.0, epsilon: 0.08\n",
"episode: 2105, score: 4.0, epsilon: 0.08\n",
"episode: 2110, score: 1.0, epsilon: 0.08\n",
"episode: 2115, score: 6.0, epsilon: 0.08\n",
"episode: 2120, score: 6.0, epsilon: 0.08\n",
"episode: 2125, score: 1.0, epsilon: 0.08\n",
"episode: 2130, score: 4.0, epsilon: 0.08\n",
"episode: 2135, score: 0.0, epsilon: 0.08\n",
"episode: 2140, score: 6.0, epsilon: 0.08\n",
"episode: 2145, score: 6.0, epsilon: 0.08\n",
"episode: 2150, score: 1.0, epsilon: 0.08\n",
"episode: 2155, score: 2.0, epsilon: 0.08\n",
"episode: 2160, score: 0.0, epsilon: 0.08\n",
"episode: 2165, score: 5.0, epsilon: 0.08\n",
"episode: 2170, score: 4.0, epsilon: 0.08\n",
"episode: 2175, score: 2.0, epsilon: 0.08\n",
"episode: 2180, score: 2.0, epsilon: 0.08\n",
"episode: 2185, score: 6.0, epsilon: 0.08\n",
"episode: 2190, score: 3.0, epsilon: 0.08\n",
"episode: 2195, score: 2.0, epsilon: 0.08\n",
"marking, episode: 2200, score: 1.0, mean_score: 2.48, std_score: 2.28\n",
"episode: 2200, score: 1.0, epsilon: 0.08\n",
"episode: 2205, score: 3.0, epsilon: 0.08\n",
"episode: 2210, score: 6.0, epsilon: 0.08\n",
"episode: 2215, score: 4.0, epsilon: 0.08\n",
"episode: 2220, score: 3.0, epsilon: 0.08\n",
"episode: 2225, score: 4.0, epsilon: 0.08\n",
"episode: 2230, score: 0.0, epsilon: 0.08\n",
"episode: 2235, score: 1.0, epsilon: 0.08\n",
"episode: 2240, score: 1.0, epsilon: 0.08\n",
"episode: 2245, score: 1.0, epsilon: 0.08\n",
"episode: 2250, score: 5.0, epsilon: 0.08\n",
"episode: 2255, score: 3.0, epsilon: 0.08\n",
"episode: 2260, score: 0.0, epsilon: 0.08\n",
"episode: 2265, score: 1.0, epsilon: 0.08\n",
"episode: 2270, score: 1.0, epsilon: 0.08\n",
"episode: 2275, score: 0.0, epsilon: 0.08\n",
"episode: 2280, score: 4.0, epsilon: 0.08\n",
"episode: 2285, score: 3.0, epsilon: 0.08\n",
"episode: 2290, score: 1.0, epsilon: 0.08\n",
"episode: 2295, score: 0.0, epsilon: 0.08\n",
"marking, episode: 2300, score: 0.0, mean_score: 2.41, std_score: 2.01\n",
"episode: 2300, score: 0.0, epsilon: 0.08\n",
"episode: 2305, score: 6.0, epsilon: 0.08\n",
"episode: 2310, score: 6.0, epsilon: 0.08\n",
"episode: 2315, score: 3.0, epsilon: 0.08\n",
"episode: 2320, score: 6.0, epsilon: 0.08\n",
"episode: 2325, score: 1.0, epsilon: 0.08\n",
"episode: 2330, score: 5.0, epsilon: 0.08\n",
"episode: 2335, score: 0.0, epsilon: 0.08\n",
"episode: 2340, score: 0.0, epsilon: 0.08\n",
"episode: 2345, score: 6.0, epsilon: 0.08\n",
"episode: 2350, score: 3.0, epsilon: 0.08\n",
"episode: 2355, score: 4.0, epsilon: 0.08\n",
"episode: 2360, score: 6.0, epsilon: 0.08\n",
"episode: 2365, score: 0.0, epsilon: 0.08\n",
"episode: 2370, score: 1.0, epsilon: 0.08\n",
"episode: 2375, score: 6.0, epsilon: 0.08\n",
"episode: 2380, score: 1.0, epsilon: 0.08\n",
"episode: 2385, score: 0.0, epsilon: 0.08\n",
"episode: 2390, score: 1.0, epsilon: 0.08\n",
"episode: 2395, score: 5.0, epsilon: 0.08\n",
"marking, episode: 2400, score: 6.0, mean_score: 2.65, std_score: 2.19\n",
"episode: 2400, score: 6.0, epsilon: 0.08\n",
"episode: 2405, score: 1.0, epsilon: 0.08\n",
"episode: 2410, score: 0.0, epsilon: 0.08\n",
"episode: 2415, score: 6.0, epsilon: 0.08\n",
"episode: 2420, score: 0.0, epsilon: 0.08\n",
"episode: 2425, score: 0.0, epsilon: 0.08\n",
"episode: 2430, score: 2.0, epsilon: 0.08\n",
"episode: 2435, score: 1.0, epsilon: 0.08\n",
"episode: 2440, score: 2.0, epsilon: 0.08\n",
"episode: 2445, score: 0.0, epsilon: 0.08\n",
"episode: 2450, score: 6.0, epsilon: 0.08\n",
"episode: 2455, score: 0.0, epsilon: 0.08\n",
"episode: 2460, score: 1.0, epsilon: 0.08\n",
"episode: 2465, score: 0.0, epsilon: 0.08\n",
"episode: 2470, score: 3.0, epsilon: 0.08\n",
"episode: 2475, score: 2.0, epsilon: 0.08\n",
"episode: 2480, score: 3.0, epsilon: 0.08\n",
"episode: 2485, score: 6.0, epsilon: 0.08\n",
"episode: 2490, score: 6.0, epsilon: 0.08\n",
"episode: 2495, score: 2.0, epsilon: 0.08\n",
"marking, episode: 2500, score: 1.0, mean_score: 2.32, std_score: 2.03\n",
"episode: 2500, score: 1.0, epsilon: 0.08\n",
"episode: 2505, score: 6.0, epsilon: 0.08\n",
"episode: 2510, score: 0.0, epsilon: 0.08\n",
"episode: 2515, score: 0.0, epsilon: 0.08\n",
"episode: 2520, score: 4.0, epsilon: 0.08\n",
"episode: 2525, score: 6.0, epsilon: 0.08\n",
"episode: 2530, score: 1.0, epsilon: 0.08\n",
"episode: 2535, score: 0.0, epsilon: 0.08\n",
"episode: 2540, score: 0.0, epsilon: 0.08\n",
"episode: 2545, score: 2.0, epsilon: 0.08\n",
"episode: 2550, score: 0.0, epsilon: 0.08\n",
"episode: 2555, score: 6.0, epsilon: 0.08\n",
"episode: 2560, score: 0.0, epsilon: 0.08\n",
"episode: 2565, score: 2.0, epsilon: 0.08\n",
"episode: 2570, score: 0.0, epsilon: 0.08\n",
"episode: 2575, score: 0.0, epsilon: 0.08\n",
"episode: 2580, score: 1.0, epsilon: 0.08\n",
"episode: 2585, score: 6.0, epsilon: 0.08\n",
"episode: 2590, score: 0.0, epsilon: 0.08\n",
"episode: 2595, score: 0.0, epsilon: 0.08\n",
"marking, episode: 2600, score: 3.0, mean_score: 2.23, std_score: 2.06\n",
"episode: 2600, score: 3.0, epsilon: 0.08\n",
"episode: 2605, score: 1.0, epsilon: 0.08\n",
"episode: 2610, score: 1.0, epsilon: 0.08\n",
"episode: 2615, score: 6.0, epsilon: 0.08\n",
"episode: 2620, score: 1.0, epsilon: 0.08\n",
"episode: 2625, score: 1.0, epsilon: 0.08\n",
"episode: 2630, score: 3.0, epsilon: 0.08\n",
"episode: 2635, score: 2.0, epsilon: 0.08\n",
"episode: 2640, score: 3.0, epsilon: 0.08\n",
"episode: 2645, score: 0.0, epsilon: 0.08\n",
"episode: 2650, score: 0.0, epsilon: 0.08\n",
"episode: 2655, score: 1.0, epsilon: 0.08\n",
"episode: 2660, score: 0.0, epsilon: 0.08\n",
"episode: 2665, score: 6.0, epsilon: 0.08\n",
"episode: 2670, score: 6.0, epsilon: 0.08\n",
"episode: 2675, score: 1.0, epsilon: 0.08\n",
"episode: 2680, score: 0.0, epsilon: 0.08\n",
"episode: 2685, score: 2.0, epsilon: 0.08\n",
"episode: 2690, score: 3.0, epsilon: 0.08\n",
"episode: 2695, score: 1.0, epsilon: 0.08\n",
"marking, episode: 2700, score: 1.0, mean_score: 2.09, std_score: 2.05\n",
"episode: 2700, score: 1.0, epsilon: 0.08\n",
"episode: 2705, score: 2.0, epsilon: 0.08\n",
"episode: 2710, score: 0.0, epsilon: 0.08\n",
"episode: 2715, score: 3.0, epsilon: 0.08\n",
"episode: 2720, score: 0.0, epsilon: 0.08\n",
"episode: 2725, score: 1.0, epsilon: 0.08\n",
"episode: 2730, score: 1.0, epsilon: 0.08\n",
"episode: 2735, score: 1.0, epsilon: 0.08\n",
"episode: 2740, score: 0.0, epsilon: 0.08\n",
"episode: 2745, score: 4.0, epsilon: 0.08\n",
"episode: 2750, score: 4.0, epsilon: 0.08\n",
"episode: 2755, score: 6.0, epsilon: 0.08\n",
"episode: 2760, score: 1.0, epsilon: 0.08\n",
"episode: 2765, score: 4.0, epsilon: 0.08\n",
"episode: 2770, score: 0.0, epsilon: 0.08\n",
"episode: 2775, score: 1.0, epsilon: 0.08\n",
"episode: 2780, score: 3.0, epsilon: 0.08\n",
"episode: 2785, score: 2.0, epsilon: 0.08\n",
"episode: 2790, score: 0.0, epsilon: 0.08\n",
"episode: 2795, score: 6.0, epsilon: 0.08\n",
"marking, episode: 2800, score: 3.0, mean_score: 2.32, std_score: 2.09\n",
"episode: 2800, score: 3.0, epsilon: 0.08\n",
"episode: 2805, score: 3.0, epsilon: 0.08\n",
"episode: 2810, score: 4.0, epsilon: 0.08\n",
"episode: 2815, score: 2.0, epsilon: 0.08\n",
"episode: 2820, score: 0.0, epsilon: 0.08\n",
"episode: 2825, score: 5.0, epsilon: 0.08\n",
"episode: 2830, score: 2.0, epsilon: 0.08\n",
"episode: 2835, score: 6.0, epsilon: 0.08\n",
"episode: 2840, score: 3.0, epsilon: 0.08\n",
"episode: 2845, score: 2.0, epsilon: 0.08\n",
"episode: 2850, score: 1.0, epsilon: 0.08\n",
"episode: 2855, score: 2.0, epsilon: 0.08\n",
"episode: 2860, score: 1.0, epsilon: 0.08\n",
"episode: 2865, score: 6.0, epsilon: 0.08\n",
"episode: 2870, score: 3.0, epsilon: 0.08\n",
"episode: 2875, score: 2.0, epsilon: 0.08\n",
"episode: 2880, score: 0.0, epsilon: 0.08\n",
"episode: 2885, score: 6.0, epsilon: 0.08\n",
"episode: 2890, score: 3.0, epsilon: 0.08\n",
"episode: 2895, score: 2.0, epsilon: 0.08\n",
"marking, episode: 2900, score: 1.0, mean_score: 2.46, std_score: 2.21\n",
"episode: 2900, score: 1.0, epsilon: 0.08\n",
"episode: 2905, score: 6.0, epsilon: 0.08\n",
"episode: 2910, score: 6.0, epsilon: 0.08\n",
"episode: 2915, score: 3.0, epsilon: 0.08\n",
"episode: 2920, score: 0.0, epsilon: 0.08\n",
"episode: 2925, score: 2.0, epsilon: 0.08\n",
"episode: 2930, score: 0.0, epsilon: 0.08\n",
"episode: 2935, score: 0.0, epsilon: 0.08\n",
"episode: 2940, score: 6.0, epsilon: 0.08\n",
"episode: 2945, score: 6.0, epsilon: 0.08\n",
"episode: 2950, score: 5.0, epsilon: 0.08\n",
"episode: 2955, score: 2.0, epsilon: 0.08\n",
"episode: 2960, score: 3.0, epsilon: 0.08\n",
"episode: 2965, score: 2.0, epsilon: 0.08\n",
"episode: 2970, score: 0.0, epsilon: 0.08\n",
"episode: 2975, score: 5.0, epsilon: 0.08\n",
"episode: 2980, score: 1.0, epsilon: 0.08\n",
"episode: 2985, score: 0.0, epsilon: 0.08\n",
"episode: 2990, score: 6.0, epsilon: 0.08\n",
"episode: 2995, score: 6.0, epsilon: 0.08\n",
"marking, episode: 3000, score: 6.0, mean_score: 2.69, std_score: 2.34\n",
"episode: 3000, score: 6.0, epsilon: 0.08\n",
"episode: 3005, score: 2.0, epsilon: 0.08\n",
"episode: 3010, score: 0.0, epsilon: 0.08\n",
"episode: 3015, score: 0.0, epsilon: 0.08\n",
"episode: 3020, score: 3.0, epsilon: 0.08\n",
"episode: 3025, score: 0.0, epsilon: 0.08\n",
"episode: 3030, score: 2.0, epsilon: 0.08\n",
"episode: 3035, score: 1.0, epsilon: 0.08\n",
"episode: 3040, score: 0.0, epsilon: 0.08\n",
"episode: 3045, score: 4.0, epsilon: 0.08\n",
"episode: 3050, score: 1.0, epsilon: 0.08\n",
"episode: 3055, score: 1.0, epsilon: 0.08\n",
"episode: 3060, score: 1.0, epsilon: 0.08\n",
"episode: 3065, score: 0.0, epsilon: 0.08\n",
"episode: 3070, score: 0.0, epsilon: 0.08\n",
"episode: 3075, score: 1.0, epsilon: 0.08\n",
"episode: 3080, score: 1.0, epsilon: 0.08\n",
"episode: 3085, score: 3.0, epsilon: 0.08\n",
"episode: 3090, score: 5.0, epsilon: 0.08\n",
"episode: 3095, score: 6.0, epsilon: 0.08\n",
"marking, episode: 3100, score: 1.0, mean_score: 2.13, std_score: 2.02\n",
"episode: 3100, score: 1.0, epsilon: 0.08\n",
"episode: 3105, score: 6.0, epsilon: 0.08\n",
"episode: 3110, score: 3.0, epsilon: 0.08\n",
"episode: 3115, score: 0.0, epsilon: 0.08\n",
"episode: 3120, score: 0.0, epsilon: 0.08\n",
"episode: 3125, score: 4.0, epsilon: 0.08\n",
"episode: 3130, score: 1.0, epsilon: 0.08\n",
"episode: 3135, score: 6.0, epsilon: 0.08\n",
"episode: 3140, score: 0.0, epsilon: 0.08\n",
"episode: 3145, score: 0.0, epsilon: 0.08\n",
"episode: 3150, score: 0.0, epsilon: 0.08\n",
"episode: 3155, score: 4.0, epsilon: 0.08\n",
"episode: 3160, score: 1.0, epsilon: 0.08\n",
"episode: 3165, score: 6.0, epsilon: 0.08\n",
"episode: 3170, score: 2.0, epsilon: 0.08\n",
"episode: 3175, score: 0.0, epsilon: 0.08\n",
"episode: 3180, score: 0.0, epsilon: 0.08\n",
"episode: 3185, score: 0.0, epsilon: 0.08\n",
"episode: 3190, score: 6.0, epsilon: 0.08\n",
"episode: 3195, score: 1.0, epsilon: 0.08\n",
"marking, episode: 3200, score: 0.0, mean_score: 2.90, std_score: 2.41\n",
"episode: 3200, score: 0.0, epsilon: 0.08\n",
"episode: 3205, score: 1.0, epsilon: 0.08\n",
"episode: 3210, score: 3.0, epsilon: 0.08\n",
"episode: 3215, score: 3.0, epsilon: 0.08\n",
"episode: 3220, score: 6.0, epsilon: 0.08\n",
"episode: 3225, score: 1.0, epsilon: 0.08\n",
"episode: 3230, score: 2.0, epsilon: 0.08\n",
"episode: 3235, score: 0.0, epsilon: 0.08\n",
"episode: 3240, score: 6.0, epsilon: 0.08\n",
"episode: 3245, score: 2.0, epsilon: 0.08\n",
"episode: 3250, score: 0.0, epsilon: 0.08\n",
"episode: 3255, score: 0.0, epsilon: 0.08\n",
"episode: 3260, score: 6.0, epsilon: 0.08\n",
"episode: 3265, score: 3.0, epsilon: 0.08\n",
"episode: 3270, score: 4.0, epsilon: 0.08\n",
"episode: 3275, score: 3.0, epsilon: 0.08\n",
"episode: 3280, score: 6.0, epsilon: 0.08\n",
"episode: 3285, score: 2.0, epsilon: 0.08\n",
"episode: 3290, score: 0.0, epsilon: 0.08\n",
"episode: 3295, score: 4.0, epsilon: 0.08\n",
"marking, episode: 3300, score: 4.0, mean_score: 2.48, std_score: 2.22\n",
"episode: 3300, score: 4.0, epsilon: 0.08\n",
"episode: 3305, score: 1.0, epsilon: 0.08\n",
"episode: 3310, score: 6.0, epsilon: 0.08\n",
"episode: 3315, score: 2.0, epsilon: 0.08\n",
"episode: 3320, score: 0.0, epsilon: 0.08\n",
"episode: 3325, score: 6.0, epsilon: 0.08\n",
"episode: 3330, score: 1.0, epsilon: 0.08\n",
"episode: 3335, score: 1.0, epsilon: 0.08\n",
"episode: 3340, score: 1.0, epsilon: 0.08\n",
"episode: 3345, score: 1.0, epsilon: 0.08\n",
"episode: 3350, score: 1.0, epsilon: 0.08\n",
"episode: 3355, score: 4.0, epsilon: 0.08\n",
"episode: 3360, score: 1.0, epsilon: 0.08\n",
"episode: 3365, score: 0.0, epsilon: 0.08\n",
"episode: 3370, score: 5.0, epsilon: 0.08\n",
"episode: 3375, score: 3.0, epsilon: 0.08\n",
"episode: 3380, score: 0.0, epsilon: 0.08\n",
"episode: 3385, score: 5.0, epsilon: 0.08\n",
"episode: 3390, score: 0.0, epsilon: 0.08\n",
"episode: 3395, score: 0.0, epsilon: 0.08\n",
"marking, episode: 3400, score: 1.0, mean_score: 2.03, std_score: 2.08\n",
"episode: 3400, score: 1.0, epsilon: 0.08\n",
"episode: 3405, score: 1.0, epsilon: 0.08\n",
"episode: 3410, score: 0.0, epsilon: 0.08\n",
"episode: 3415, score: 2.0, epsilon: 0.08\n",
"episode: 3420, score: 0.0, epsilon: 0.08\n",
"episode: 3425, score: 1.0, epsilon: 0.08\n",
"episode: 3430, score: 1.0, epsilon: 0.08\n",
"episode: 3435, score: 1.0, epsilon: 0.08\n",
"episode: 3440, score: 3.0, epsilon: 0.08\n",
"episode: 3445, score: 0.0, epsilon: 0.08\n",
"episode: 3450, score: 1.0, epsilon: 0.08\n",
"episode: 3455, score: 3.0, epsilon: 0.08\n",
"episode: 3460, score: 1.0, epsilon: 0.08\n",
"episode: 3465, score: 0.0, epsilon: 0.08\n",
"episode: 3470, score: 0.0, epsilon: 0.08\n",
"episode: 3475, score: 1.0, epsilon: 0.08\n",
"episode: 3480, score: 3.0, epsilon: 0.08\n",
"episode: 3485, score: 0.0, epsilon: 0.08\n",
"episode: 3490, score: 3.0, epsilon: 0.08\n",
"episode: 3495, score: 2.0, epsilon: 0.08\n",
"marking, episode: 3500, score: 6.0, mean_score: 2.01, std_score: 2.11\n",
"episode: 3500, score: 6.0, epsilon: 0.08\n",
"episode: 3505, score: 4.0, epsilon: 0.08\n",
"episode: 3510, score: 6.0, epsilon: 0.08\n",
"episode: 3515, score: 6.0, epsilon: 0.08\n",
"episode: 3520, score: 1.0, epsilon: 0.08\n",
"episode: 3525, score: 6.0, epsilon: 0.08\n",
"episode: 3530, score: 2.0, epsilon: 0.08\n",
"episode: 3535, score: 0.0, epsilon: 0.08\n",
"episode: 3540, score: 2.0, epsilon: 0.08\n",
"episode: 3545, score: 6.0, epsilon: 0.08\n",
"episode: 3550, score: 4.0, epsilon: 0.08\n",
"episode: 3555, score: 2.0, epsilon: 0.08\n",
"episode: 3560, score: 5.0, epsilon: 0.08\n",
"episode: 3565, score: 6.0, epsilon: 0.08\n",
"episode: 3570, score: 0.0, epsilon: 0.08\n",
"episode: 3575, score: 0.0, epsilon: 0.08\n",
"episode: 3580, score: 4.0, epsilon: 0.08\n",
"episode: 3585, score: 0.0, epsilon: 0.08\n",
"episode: 3590, score: 4.0, epsilon: 0.08\n",
"episode: 3595, score: 3.0, epsilon: 0.08\n",
"marking, episode: 3600, score: 0.0, mean_score: 2.90, std_score: 2.26\n",
"episode: 3600, score: 0.0, epsilon: 0.08\n",
"episode: 3605, score: 3.0, epsilon: 0.08\n",
"episode: 3610, score: 4.0, epsilon: 0.08\n",
"episode: 3615, score: 2.0, epsilon: 0.08\n",
"episode: 3620, score: 4.0, epsilon: 0.08\n",
"episode: 3625, score: 3.0, epsilon: 0.08\n",
"episode: 3630, score: 0.0, epsilon: 0.08\n",
"episode: 3635, score: 0.0, epsilon: 0.08\n",
"episode: 3640, score: 0.0, epsilon: 0.08\n",
"episode: 3645, score: 6.0, epsilon: 0.08\n",
"episode: 3650, score: 0.0, epsilon: 0.08\n",
"episode: 3655, score: 1.0, epsilon: 0.08\n",
"episode: 3660, score: 2.0, epsilon: 0.08\n",
"episode: 3665, score: 6.0, epsilon: 0.08\n",
"episode: 3670, score: 6.0, epsilon: 0.08\n",
"episode: 3675, score: 0.0, epsilon: 0.08\n",
"episode: 3680, score: 1.0, epsilon: 0.08\n",
"episode: 3685, score: 2.0, epsilon: 0.08\n",
"episode: 3690, score: 0.0, epsilon: 0.08\n",
"episode: 3695, score: 1.0, epsilon: 0.08\n",
"marking, episode: 3700, score: 0.0, mean_score: 2.46, std_score: 2.12\n",
"episode: 3700, score: 0.0, epsilon: 0.08\n",
"episode: 3705, score: 0.0, epsilon: 0.08\n",
"episode: 3710, score: 1.0, epsilon: 0.08\n",
"episode: 3715, score: 6.0, epsilon: 0.08\n",
"episode: 3720, score: 0.0, epsilon: 0.08\n",
"episode: 3725, score: 0.0, epsilon: 0.08\n",
"episode: 3730, score: 0.0, epsilon: 0.08\n",
"episode: 3735, score: 1.0, epsilon: 0.08\n",
"episode: 3740, score: 1.0, epsilon: 0.08\n",
"episode: 3745, score: 2.0, epsilon: 0.08\n",
"episode: 3750, score: 1.0, epsilon: 0.08\n",
"episode: 3755, score: 6.0, epsilon: 0.08\n",
"episode: 3760, score: 2.0, epsilon: 0.08\n",
"episode: 3765, score: 1.0, epsilon: 0.08\n",
"episode: 3770, score: 2.0, epsilon: 0.08\n",
"episode: 3775, score: 4.0, epsilon: 0.08\n",
"episode: 3780, score: 1.0, epsilon: 0.08\n",
"episode: 3785, score: 2.0, epsilon: 0.08\n",
"episode: 3790, score: 2.0, epsilon: 0.08\n",
"episode: 3795, score: 1.0, epsilon: 0.08\n",
"marking, episode: 3800, score: 3.0, mean_score: 1.90, std_score: 1.89\n",
"episode: 3800, score: 3.0, epsilon: 0.08\n",
"episode: 3805, score: 1.0, epsilon: 0.08\n",
"episode: 3810, score: 1.0, epsilon: 0.08\n",
"episode: 3815, score: 2.0, epsilon: 0.08\n",
"episode: 3820, score: 0.0, epsilon: 0.08\n",
"episode: 3825, score: 6.0, epsilon: 0.08\n",
"episode: 3830, score: 0.0, epsilon: 0.08\n",
"episode: 3835, score: 0.0, epsilon: 0.08\n",
"episode: 3840, score: 0.0, epsilon: 0.08\n",
"episode: 3845, score: 1.0, epsilon: 0.08\n",
"episode: 3850, score: 0.0, epsilon: 0.08\n",
"episode: 3855, score: 6.0, epsilon: 0.08\n",
"episode: 3860, score: 6.0, epsilon: 0.08\n",
"episode: 3865, score: 4.0, epsilon: 0.08\n",
"episode: 3870, score: 0.0, epsilon: 0.08\n",
"episode: 3875, score: 1.0, epsilon: 0.08\n",
"episode: 3880, score: 2.0, epsilon: 0.08\n",
"episode: 3885, score: 1.0, epsilon: 0.08\n",
"episode: 3890, score: 1.0, epsilon: 0.08\n",
"episode: 3895, score: 6.0, epsilon: 0.08\n",
"marking, episode: 3900, score: 1.0, mean_score: 2.58, std_score: 2.23\n",
"episode: 3900, score: 1.0, epsilon: 0.08\n",
"episode: 3905, score: 2.0, epsilon: 0.08\n",
"episode: 3910, score: 0.0, epsilon: 0.08\n",
"episode: 3915, score: 2.0, epsilon: 0.08\n",
"episode: 3920, score: 2.0, epsilon: 0.08\n",
"episode: 3925, score: 1.0, epsilon: 0.08\n",
"episode: 3930, score: 0.0, epsilon: 0.08\n",
"episode: 3935, score: 1.0, epsilon: 0.08\n",
"episode: 3940, score: 1.0, epsilon: 0.08\n",
"episode: 3945, score: 2.0, epsilon: 0.08\n",
"episode: 3950, score: 0.0, epsilon: 0.08\n",
"episode: 3955, score: 2.0, epsilon: 0.08\n",
"episode: 3960, score: 0.0, epsilon: 0.08\n",
"episode: 3965, score: 6.0, epsilon: 0.08\n",
"episode: 3970, score: 0.0, epsilon: 0.08\n",
"episode: 3975, score: 0.0, epsilon: 0.08\n",
"episode: 3980, score: 4.0, epsilon: 0.08\n",
"episode: 3985, score: 5.0, epsilon: 0.08\n",
"episode: 3990, score: 0.0, epsilon: 0.08\n",
"episode: 3995, score: 1.0, epsilon: 0.08\n",
"marking, episode: 4000, score: 0.0, mean_score: 2.15, std_score: 2.11\n",
"episode: 4000, score: 0.0, epsilon: 0.08\n",
"episode: 4005, score: 6.0, epsilon: 0.08\n",
"episode: 4010, score: 0.0, epsilon: 0.08\n",
"episode: 4015, score: 1.0, epsilon: 0.08\n",
"episode: 4020, score: 5.0, epsilon: 0.08\n",
"episode: 4025, score: 1.0, epsilon: 0.08\n",
"episode: 4030, score: 4.0, epsilon: 0.08\n",
"episode: 4035, score: 1.0, epsilon: 0.08\n",
"episode: 4040, score: 3.0, epsilon: 0.08\n",
"episode: 4045, score: 1.0, epsilon: 0.08\n",
"episode: 4050, score: 1.0, epsilon: 0.08\n",
"episode: 4055, score: 2.0, epsilon: 0.08\n",
"episode: 4060, score: 0.0, epsilon: 0.08\n",
"episode: 4065, score: 1.0, epsilon: 0.08\n",
"episode: 4070, score: 2.0, epsilon: 0.08\n",
"episode: 4075, score: 6.0, epsilon: 0.08\n",
"episode: 4080, score: 0.0, epsilon: 0.08\n",
"episode: 4085, score: 5.0, epsilon: 0.08\n",
"episode: 4090, score: 0.0, epsilon: 0.08\n",
"episode: 4095, score: 4.0, epsilon: 0.08\n",
"marking, episode: 4100, score: 0.0, mean_score: 2.14, std_score: 1.96\n",
"episode: 4100, score: 0.0, epsilon: 0.08\n",
"episode: 4105, score: 0.0, epsilon: 0.08\n",
"episode: 4110, score: 2.0, epsilon: 0.08\n",
"episode: 4115, score: 2.0, epsilon: 0.08\n",
"episode: 4120, score: 6.0, epsilon: 0.08\n",
"episode: 4125, score: 0.0, epsilon: 0.08\n",
"episode: 4130, score: 6.0, epsilon: 0.08\n",
"episode: 4135, score: 1.0, epsilon: 0.08\n",
"episode: 4140, score: 6.0, epsilon: 0.08\n",
"episode: 4145, score: 0.0, epsilon: 0.08\n",
"episode: 4150, score: 1.0, epsilon: 0.08\n",
"episode: 4155, score: 0.0, epsilon: 0.08\n",
"episode: 4160, score: 2.0, epsilon: 0.08\n",
"episode: 4165, score: 0.0, epsilon: 0.08\n",
"episode: 4170, score: 3.0, epsilon: 0.08\n",
"episode: 4175, score: 2.0, epsilon: 0.08\n",
"episode: 4180, score: 1.0, epsilon: 0.08\n",
"episode: 4185, score: 1.0, epsilon: 0.08\n",
"episode: 4190, score: 0.0, epsilon: 0.08\n",
"episode: 4195, score: 2.0, epsilon: 0.08\n",
"marking, episode: 4200, score: 4.0, mean_score: 2.48, std_score: 2.30\n",
"episode: 4200, score: 4.0, epsilon: 0.08\n",
"episode: 4205, score: 4.0, epsilon: 0.08\n",
"episode: 4210, score: 1.0, epsilon: 0.08\n",
"episode: 4215, score: 0.0, epsilon: 0.08\n",
"episode: 4220, score: 6.0, epsilon: 0.08\n",
"episode: 4225, score: 2.0, epsilon: 0.08\n",
"episode: 4230, score: 3.0, epsilon: 0.08\n",
"episode: 4235, score: 6.0, epsilon: 0.08\n",
"episode: 4240, score: 0.0, epsilon: 0.08\n",
"episode: 4245, score: 0.0, epsilon: 0.08\n",
"episode: 4250, score: 6.0, epsilon: 0.08\n",
"episode: 4255, score: 6.0, epsilon: 0.08\n",
"episode: 4260, score: 0.0, epsilon: 0.08\n",
"episode: 4265, score: 0.0, epsilon: 0.08\n",
"episode: 4270, score: 0.0, epsilon: 0.08\n",
"episode: 4275, score: 6.0, epsilon: 0.08\n",
"episode: 4280, score: 1.0, epsilon: 0.08\n",
"episode: 4285, score: 3.0, epsilon: 0.08\n",
"episode: 4290, score: 2.0, epsilon: 0.08\n",
"episode: 4295, score: 6.0, epsilon: 0.08\n",
"marking, episode: 4300, score: 3.0, mean_score: 2.65, std_score: 2.16\n",
"episode: 4300, score: 3.0, epsilon: 0.08\n",
"episode: 4305, score: 1.0, epsilon: 0.08\n",
"episode: 4310, score: 0.0, epsilon: 0.08\n",
"episode: 4315, score: 1.0, epsilon: 0.08\n",
"episode: 4320, score: 1.0, epsilon: 0.08\n",
"episode: 4325, score: 0.0, epsilon: 0.08\n",
"episode: 4330, score: 0.0, epsilon: 0.08\n",
"episode: 4335, score: 5.0, epsilon: 0.08\n",
"episode: 4340, score: 1.0, epsilon: 0.08\n",
"episode: 4345, score: 3.0, epsilon: 0.08\n",
"episode: 4350, score: 0.0, epsilon: 0.08\n",
"episode: 4355, score: 6.0, epsilon: 0.08\n",
"episode: 4360, score: 3.0, epsilon: 0.08\n",
"episode: 4365, score: 0.0, epsilon: 0.08\n",
"episode: 4370, score: 6.0, epsilon: 0.08\n",
"episode: 4375, score: 6.0, epsilon: 0.08\n",
"episode: 4380, score: 0.0, epsilon: 0.08\n",
"episode: 4385, score: 0.0, epsilon: 0.08\n",
"episode: 4390, score: 2.0, epsilon: 0.08\n",
"episode: 4395, score: 6.0, epsilon: 0.08\n",
"marking, episode: 4400, score: 1.0, mean_score: 2.62, std_score: 2.28\n",
"episode: 4400, score: 1.0, epsilon: 0.08\n",
"episode: 4405, score: 0.0, epsilon: 0.08\n",
"episode: 4410, score: 0.0, epsilon: 0.08\n",
"episode: 4415, score: 1.0, epsilon: 0.08\n",
"episode: 4420, score: 0.0, epsilon: 0.08\n",
"episode: 4425, score: 4.0, epsilon: 0.08\n",
"episode: 4430, score: 5.0, epsilon: 0.08\n",
"episode: 4435, score: 2.0, epsilon: 0.08\n",
"episode: 4440, score: 1.0, epsilon: 0.08\n",
"episode: 4445, score: 0.0, epsilon: 0.08\n",
"episode: 4450, score: 2.0, epsilon: 0.08\n",
"episode: 4455, score: 2.0, epsilon: 0.08\n",
"episode: 4460, score: 5.0, epsilon: 0.08\n",
"episode: 4465, score: 1.0, epsilon: 0.08\n",
"episode: 4470, score: 4.0, epsilon: 0.08\n",
"episode: 4475, score: 3.0, epsilon: 0.08\n",
"episode: 4480, score: 2.0, epsilon: 0.08\n",
"episode: 4485, score: 6.0, epsilon: 0.08\n",
"episode: 4490, score: 0.0, epsilon: 0.08\n",
"episode: 4495, score: 2.0, epsilon: 0.08\n",
"marking, episode: 4500, score: 4.0, mean_score: 2.28, std_score: 2.11\n",
"episode: 4500, score: 4.0, epsilon: 0.08\n",
"episode: 4505, score: 3.0, epsilon: 0.08\n",
"episode: 4510, score: 5.0, epsilon: 0.08\n",
"episode: 4515, score: 4.0, epsilon: 0.08\n",
"episode: 4520, score: 0.0, epsilon: 0.08\n",
"episode: 4525, score: 2.0, epsilon: 0.08\n",
"episode: 4530, score: 5.0, epsilon: 0.08\n",
"episode: 4535, score: 2.0, epsilon: 0.08\n",
"episode: 4540, score: 1.0, epsilon: 0.08\n",
"episode: 4545, score: 0.0, epsilon: 0.08\n",
"episode: 4550, score: 1.0, epsilon: 0.08\n",
"episode: 4555, score: 1.0, epsilon: 0.08\n",
"episode: 4560, score: 0.0, epsilon: 0.08\n",
"episode: 4565, score: 2.0, epsilon: 0.08\n",
"episode: 4570, score: 4.0, epsilon: 0.08\n",
"episode: 4575, score: 0.0, epsilon: 0.08\n",
"episode: 4580, score: 2.0, epsilon: 0.08\n",
"episode: 4585, score: 1.0, epsilon: 0.08\n",
"episode: 4590, score: 6.0, epsilon: 0.08\n",
"episode: 4595, score: 2.0, epsilon: 0.08\n",
"marking, episode: 4600, score: 1.0, mean_score: 2.10, std_score: 2.04\n",
"episode: 4600, score: 1.0, epsilon: 0.08\n",
"episode: 4605, score: 0.0, epsilon: 0.08\n",
"episode: 4610, score: 0.0, epsilon: 0.08\n",
"episode: 4615, score: 0.0, epsilon: 0.08\n",
"episode: 4620, score: 3.0, epsilon: 0.08\n",
"episode: 4625, score: 0.0, epsilon: 0.08\n",
"episode: 4630, score: 0.0, epsilon: 0.08\n",
"episode: 4635, score: 5.0, epsilon: 0.08\n",
"episode: 4640, score: 1.0, epsilon: 0.08\n",
"episode: 4645, score: 6.0, epsilon: 0.08\n",
"episode: 4650, score: 1.0, epsilon: 0.08\n",
"episode: 4655, score: 2.0, epsilon: 0.08\n",
"episode: 4660, score: 0.0, epsilon: 0.08\n",
"episode: 4665, score: 1.0, epsilon: 0.08\n",
"episode: 4670, score: 6.0, epsilon: 0.08\n",
"episode: 4675, score: 3.0, epsilon: 0.08\n",
"episode: 4680, score: 6.0, epsilon: 0.08\n",
"episode: 4685, score: 0.0, epsilon: 0.08\n",
"episode: 4690, score: 0.0, epsilon: 0.08\n",
"episode: 4695, score: 1.0, epsilon: 0.08\n",
"marking, episode: 4700, score: 6.0, mean_score: 2.35, std_score: 2.20\n",
"episode: 4700, score: 6.0, epsilon: 0.08\n",
"episode: 4705, score: 2.0, epsilon: 0.08\n",
"episode: 4710, score: 5.0, epsilon: 0.08\n",
"episode: 4715, score: 6.0, epsilon: 0.08\n",
"episode: 4720, score: 1.0, epsilon: 0.08\n",
"episode: 4725, score: 2.0, epsilon: 0.08\n",
"episode: 4730, score: 2.0, epsilon: 0.08\n",
"episode: 4735, score: 2.0, epsilon: 0.08\n",
"episode: 4740, score: 0.0, epsilon: 0.08\n",
"episode: 4745, score: 1.0, epsilon: 0.08\n",
"episode: 4750, score: 3.0, epsilon: 0.08\n",
"episode: 4755, score: 1.0, epsilon: 0.08\n",
"episode: 4760, score: 4.0, epsilon: 0.08\n",
"episode: 4765, score: 0.0, epsilon: 0.08\n",
"episode: 4770, score: 4.0, epsilon: 0.08\n",
"episode: 4775, score: 0.0, epsilon: 0.08\n",
"episode: 4780, score: 0.0, epsilon: 0.08\n",
"episode: 4785, score: 3.0, epsilon: 0.08\n",
"episode: 4790, score: 3.0, epsilon: 0.08\n",
"episode: 4795, score: 1.0, epsilon: 0.08\n",
"marking, episode: 4800, score: 0.0, mean_score: 2.37, std_score: 2.11\n",
"episode: 4800, score: 0.0, epsilon: 0.08\n",
"episode: 4805, score: 0.0, epsilon: 0.08\n",
"episode: 4810, score: 1.0, epsilon: 0.08\n",
"episode: 4815, score: 4.0, epsilon: 0.08\n",
"episode: 4820, score: 1.0, epsilon: 0.08\n",
"episode: 4825, score: 4.0, epsilon: 0.08\n",
"episode: 4830, score: 6.0, epsilon: 0.08\n",
"episode: 4835, score: 4.0, epsilon: 0.08\n",
"episode: 4840, score: 1.0, epsilon: 0.08\n",
"episode: 4845, score: 3.0, epsilon: 0.08\n",
"episode: 4850, score: 0.0, epsilon: 0.08\n",
"episode: 4855, score: 0.0, epsilon: 0.08\n",
"episode: 4860, score: 1.0, epsilon: 0.08\n",
"episode: 4865, score: 1.0, epsilon: 0.08\n",
"episode: 4870, score: 3.0, epsilon: 0.08\n",
"episode: 4875, score: 2.0, epsilon: 0.08\n",
"episode: 4880, score: 2.0, epsilon: 0.08\n",
"episode: 4885, score: 6.0, epsilon: 0.08\n",
"episode: 4890, score: 4.0, epsilon: 0.08\n",
"episode: 4895, score: 2.0, epsilon: 0.08\n",
"marking, episode: 4900, score: 2.0, mean_score: 2.31, std_score: 2.12\n",
"episode: 4900, score: 2.0, epsilon: 0.08\n",
"episode: 4905, score: 2.0, epsilon: 0.08\n",
"episode: 4910, score: 0.0, epsilon: 0.08\n",
"episode: 4915, score: 6.0, epsilon: 0.08\n",
"episode: 4920, score: 2.0, epsilon: 0.08\n",
"episode: 4925, score: 6.0, epsilon: 0.08\n",
"episode: 4930, score: 2.0, epsilon: 0.08\n",
"episode: 4935, score: 0.0, epsilon: 0.08\n",
"episode: 4940, score: 1.0, epsilon: 0.08\n",
"episode: 4945, score: 0.0, epsilon: 0.08\n",
"episode: 4950, score: 0.0, epsilon: 0.08\n",
"episode: 4955, score: 2.0, epsilon: 0.08\n",
"episode: 4960, score: 1.0, epsilon: 0.08\n",
"episode: 4965, score: 6.0, epsilon: 0.08\n",
"episode: 4970, score: 0.0, epsilon: 0.08\n",
"episode: 4975, score: 4.0, epsilon: 0.08\n",
"episode: 4980, score: 0.0, epsilon: 0.08\n",
"episode: 4985, score: 1.0, epsilon: 0.08\n",
"episode: 4990, score: 6.0, epsilon: 0.08\n",
"episode: 4995, score: 1.0, epsilon: 0.08\n",
"marking, episode: 5000, score: 1.0, mean_score: 2.56, std_score: 2.21\n",
"episode: 5000, score: 1.0, epsilon: 0.08\n",
"episode: 5005, score: 0.0, epsilon: 0.08\n",
"episode: 5010, score: 1.0, epsilon: 0.08\n",
"episode: 5015, score: 6.0, epsilon: 0.08\n",
"episode: 5020, score: 2.0, epsilon: 0.08\n",
"episode: 5025, score: 3.0, epsilon: 0.08\n",
"episode: 5030, score: 1.0, epsilon: 0.08\n",
"episode: 5035, score: 1.0, epsilon: 0.08\n",
"episode: 5040, score: 4.0, epsilon: 0.08\n",
"episode: 5045, score: 2.0, epsilon: 0.08\n",
"episode: 5050, score: 5.0, epsilon: 0.08\n",
"episode: 5055, score: 6.0, epsilon: 0.08\n",
"episode: 5060, score: 6.0, epsilon: 0.08\n",
"episode: 5065, score: 0.0, epsilon: 0.08\n",
"episode: 5070, score: 4.0, epsilon: 0.08\n",
"episode: 5075, score: 2.0, epsilon: 0.08\n",
"episode: 5080, score: 2.0, epsilon: 0.08\n",
"episode: 5085, score: 0.0, epsilon: 0.08\n",
"episode: 5090, score: 0.0, epsilon: 0.08\n",
"episode: 5095, score: 4.0, epsilon: 0.08\n",
"marking, episode: 5100, score: 3.0, mean_score: 2.38, std_score: 2.04\n",
"episode: 5100, score: 3.0, epsilon: 0.08\n",
"episode: 5105, score: 3.0, epsilon: 0.08\n",
"episode: 5110, score: 3.0, epsilon: 0.08\n",
"episode: 5115, score: 1.0, epsilon: 0.08\n",
"episode: 5120, score: 0.0, epsilon: 0.08\n",
"episode: 5125, score: 1.0, epsilon: 0.08\n",
"episode: 5130, score: 2.0, epsilon: 0.08\n",
"episode: 5135, score: 4.0, epsilon: 0.08\n",
"episode: 5140, score: 1.0, epsilon: 0.08\n",
"episode: 5145, score: 5.0, epsilon: 0.08\n",
"episode: 5150, score: 6.0, epsilon: 0.08\n",
"episode: 5155, score: 0.0, epsilon: 0.08\n",
"episode: 5160, score: 1.0, epsilon: 0.08\n",
"episode: 5165, score: 3.0, epsilon: 0.08\n",
"episode: 5170, score: 1.0, epsilon: 0.08\n",
"episode: 5175, score: 0.0, epsilon: 0.08\n",
"episode: 5180, score: 4.0, epsilon: 0.08\n",
"episode: 5185, score: 0.0, epsilon: 0.08\n",
"episode: 5190, score: 5.0, epsilon: 0.08\n",
"episode: 5195, score: 1.0, epsilon: 0.08\n",
"marking, episode: 5200, score: 0.0, mean_score: 2.20, std_score: 2.09\n",
"episode: 5200, score: 0.0, epsilon: 0.08\n",
"episode: 5205, score: 3.0, epsilon: 0.08\n",
"episode: 5210, score: 2.0, epsilon: 0.08\n",
"episode: 5215, score: 3.0, epsilon: 0.08\n",
"episode: 5220, score: 6.0, epsilon: 0.08\n",
"episode: 5225, score: 3.0, epsilon: 0.08\n",
"episode: 5230, score: 2.0, epsilon: 0.08\n",
"episode: 5235, score: 0.0, epsilon: 0.08\n",
"episode: 5240, score: 1.0, epsilon: 0.08\n",
"episode: 5245, score: 0.0, epsilon: 0.08\n",
"episode: 5250, score: 1.0, epsilon: 0.08\n",
"episode: 5255, score: 0.0, epsilon: 0.08\n",
"episode: 5260, score: 6.0, epsilon: 0.08\n",
"episode: 5265, score: 3.0, epsilon: 0.08\n",
"episode: 5270, score: 0.0, epsilon: 0.08\n",
"episode: 5275, score: 5.0, epsilon: 0.08\n",
"episode: 5280, score: 0.0, epsilon: 0.08\n",
"episode: 5285, score: 5.0, epsilon: 0.08\n",
"episode: 5290, score: 3.0, epsilon: 0.08\n",
"episode: 5295, score: 0.0, epsilon: 0.08\n",
"marking, episode: 5300, score: 2.0, mean_score: 2.29, std_score: 1.97\n",
"episode: 5300, score: 2.0, epsilon: 0.08\n",
"episode: 5305, score: 3.0, epsilon: 0.08\n",
"episode: 5310, score: 0.0, epsilon: 0.08\n",
"episode: 5315, score: 0.0, epsilon: 0.08\n",
"episode: 5320, score: 6.0, epsilon: 0.08\n",
"episode: 5325, score: 0.0, epsilon: 0.08\n",
"episode: 5330, score: 6.0, epsilon: 0.08\n",
"episode: 5335, score: 1.0, epsilon: 0.08\n",
"episode: 5340, score: 2.0, epsilon: 0.08\n",
"episode: 5345, score: 6.0, epsilon: 0.08\n",
"episode: 5350, score: 2.0, epsilon: 0.08\n",
"episode: 5355, score: 2.0, epsilon: 0.08\n",
"episode: 5360, score: 2.0, epsilon: 0.08\n",
"episode: 5365, score: 1.0, epsilon: 0.08\n",
"episode: 5370, score: 0.0, epsilon: 0.08\n",
"episode: 5375, score: 1.0, epsilon: 0.08\n",
"episode: 5380, score: 1.0, epsilon: 0.08\n",
"episode: 5385, score: 1.0, epsilon: 0.08\n",
"episode: 5390, score: 2.0, epsilon: 0.08\n",
"episode: 5395, score: 2.0, epsilon: 0.08\n",
"marking, episode: 5400, score: 6.0, mean_score: 2.36, std_score: 2.17\n",
"episode: 5400, score: 6.0, epsilon: 0.08\n",
"episode: 5405, score: 1.0, epsilon: 0.08\n",
"episode: 5410, score: 2.0, epsilon: 0.08\n",
"episode: 5415, score: 4.0, epsilon: 0.08\n",
"episode: 5420, score: 3.0, epsilon: 0.08\n",
"episode: 5425, score: 0.0, epsilon: 0.08\n",
"episode: 5430, score: 2.0, epsilon: 0.08\n",
"episode: 5435, score: 1.0, epsilon: 0.08\n",
"episode: 5440, score: 3.0, epsilon: 0.08\n",
"episode: 5445, score: 2.0, epsilon: 0.08\n",
"episode: 5450, score: 0.0, epsilon: 0.08\n",
"episode: 5455, score: 4.0, epsilon: 0.08\n",
"episode: 5460, score: 0.0, epsilon: 0.08\n",
"episode: 5465, score: 1.0, epsilon: 0.08\n",
"episode: 5470, score: 4.0, epsilon: 0.08\n",
"episode: 5475, score: 4.0, epsilon: 0.08\n",
"episode: 5480, score: 3.0, epsilon: 0.08\n",
"episode: 5485, score: 2.0, epsilon: 0.08\n",
"episode: 5490, score: 3.0, epsilon: 0.08\n",
"episode: 5495, score: 3.0, epsilon: 0.08\n",
"marking, episode: 5500, score: 1.0, mean_score: 2.64, std_score: 2.11\n",
"episode: 5500, score: 1.0, epsilon: 0.08\n",
"episode: 5505, score: 6.0, epsilon: 0.08\n",
"episode: 5510, score: 1.0, epsilon: 0.08\n",
"episode: 5515, score: 6.0, epsilon: 0.08\n",
"episode: 5520, score: 1.0, epsilon: 0.08\n",
"episode: 5525, score: 6.0, epsilon: 0.08\n",
"episode: 5530, score: 1.0, epsilon: 0.08\n",
"episode: 5535, score: 0.0, epsilon: 0.08\n",
"episode: 5540, score: 0.0, epsilon: 0.08\n",
"episode: 5545, score: 1.0, epsilon: 0.08\n",
"episode: 5550, score: 6.0, epsilon: 0.08\n",
"episode: 5555, score: 1.0, epsilon: 0.08\n",
"episode: 5560, score: 6.0, epsilon: 0.08\n",
"episode: 5565, score: 2.0, epsilon: 0.08\n",
"episode: 5570, score: 2.0, epsilon: 0.08\n",
"episode: 5575, score: 0.0, epsilon: 0.08\n",
"episode: 5580, score: 4.0, epsilon: 0.08\n",
"episode: 5585, score: 2.0, epsilon: 0.08\n",
"episode: 5590, score: 3.0, epsilon: 0.08\n",
"episode: 5595, score: 4.0, epsilon: 0.08\n",
"marking, episode: 5600, score: 2.0, mean_score: 2.44, std_score: 2.29\n",
"episode: 5600, score: 2.0, epsilon: 0.08\n",
"episode: 5605, score: 1.0, epsilon: 0.08\n",
"episode: 5610, score: 2.0, epsilon: 0.08\n",
"episode: 5615, score: 5.0, epsilon: 0.08\n",
"episode: 5620, score: 3.0, epsilon: 0.08\n",
"episode: 5625, score: 6.0, epsilon: 0.08\n",
"episode: 5630, score: 1.0, epsilon: 0.08\n",
"episode: 5635, score: 6.0, epsilon: 0.08\n",
"episode: 5640, score: 3.0, epsilon: 0.08\n",
"episode: 5645, score: 0.0, epsilon: 0.08\n",
"episode: 5650, score: 2.0, epsilon: 0.08\n",
"episode: 5655, score: 3.0, epsilon: 0.08\n",
"episode: 5660, score: 0.0, epsilon: 0.08\n",
"episode: 5665, score: 2.0, epsilon: 0.08\n",
"episode: 5670, score: 2.0, epsilon: 0.08\n",
"episode: 5675, score: 2.0, epsilon: 0.08\n",
"episode: 5680, score: 5.0, epsilon: 0.08\n",
"episode: 5685, score: 4.0, epsilon: 0.08\n",
"episode: 5690, score: 6.0, epsilon: 0.08\n",
"episode: 5695, score: 0.0, epsilon: 0.08\n",
"marking, episode: 5700, score: 2.0, mean_score: 2.38, std_score: 2.12\n",
"episode: 5700, score: 2.0, epsilon: 0.08\n",
"episode: 5705, score: 0.0, epsilon: 0.08\n",
"episode: 5710, score: 6.0, epsilon: 0.08\n",
"episode: 5715, score: 1.0, epsilon: 0.08\n",
"episode: 5720, score: 0.0, epsilon: 0.08\n",
"episode: 5725, score: 6.0, epsilon: 0.08\n",
"episode: 5730, score: 1.0, epsilon: 0.08\n",
"episode: 5735, score: 6.0, epsilon: 0.08\n",
"episode: 5740, score: 3.0, epsilon: 0.08\n",
"episode: 5745, score: 6.0, epsilon: 0.08\n",
"episode: 5750, score: 0.0, epsilon: 0.08\n",
"episode: 5755, score: 6.0, epsilon: 0.08\n",
"episode: 5760, score: 2.0, epsilon: 0.08\n",
"episode: 5765, score: 6.0, epsilon: 0.08\n",
"episode: 5770, score: 4.0, epsilon: 0.08\n",
"episode: 5775, score: 6.0, epsilon: 0.08\n",
"episode: 5780, score: 1.0, epsilon: 0.08\n",
"episode: 5785, score: 6.0, epsilon: 0.08\n",
"episode: 5790, score: 5.0, epsilon: 0.08\n",
"episode: 5795, score: 4.0, epsilon: 0.08\n",
"marking, episode: 5800, score: 6.0, mean_score: 2.76, std_score: 2.25\n",
"episode: 5800, score: 6.0, epsilon: 0.08\n",
"episode: 5805, score: 6.0, epsilon: 0.08\n",
"episode: 5810, score: 1.0, epsilon: 0.08\n",
"episode: 5815, score: 1.0, epsilon: 0.08\n",
"episode: 5820, score: 0.0, epsilon: 0.08\n",
"episode: 5825, score: 5.0, epsilon: 0.08\n",
"episode: 5830, score: 6.0, epsilon: 0.08\n",
"episode: 5835, score: 1.0, epsilon: 0.08\n",
"episode: 5840, score: 1.0, epsilon: 0.08\n",
"episode: 5845, score: 2.0, epsilon: 0.08\n",
"episode: 5850, score: 1.0, epsilon: 0.08\n",
"episode: 5855, score: 6.0, epsilon: 0.08\n",
"episode: 5860, score: 6.0, epsilon: 0.08\n",
"episode: 5865, score: 0.0, epsilon: 0.08\n",
"episode: 5870, score: 1.0, epsilon: 0.08\n",
"episode: 5875, score: 4.0, epsilon: 0.08\n",
"episode: 5880, score: 4.0, epsilon: 0.08\n",
"episode: 5885, score: 1.0, epsilon: 0.08\n",
"episode: 5890, score: 1.0, epsilon: 0.08\n",
"episode: 5895, score: 3.0, epsilon: 0.08\n",
"marking, episode: 5900, score: 6.0, mean_score: 2.56, std_score: 2.27\n",
"episode: 5900, score: 6.0, epsilon: 0.08\n",
"episode: 5905, score: 6.0, epsilon: 0.08\n",
"episode: 5910, score: 2.0, epsilon: 0.08\n",
"episode: 5915, score: 4.0, epsilon: 0.08\n",
"episode: 5920, score: 0.0, epsilon: 0.08\n",
"episode: 5925, score: 1.0, epsilon: 0.08\n",
"episode: 5930, score: 0.0, epsilon: 0.08\n",
"episode: 5935, score: 3.0, epsilon: 0.08\n",
"episode: 5940, score: 0.0, epsilon: 0.08\n",
"episode: 5945, score: 0.0, epsilon: 0.08\n",
"episode: 5950, score: 1.0, epsilon: 0.08\n",
"episode: 5955, score: 6.0, epsilon: 0.08\n",
"episode: 5960, score: 0.0, epsilon: 0.08\n",
"episode: 5965, score: 3.0, epsilon: 0.08\n",
"episode: 5970, score: 2.0, epsilon: 0.08\n",
"episode: 5975, score: 2.0, epsilon: 0.08\n",
"episode: 5980, score: 6.0, epsilon: 0.08\n",
"episode: 5985, score: 0.0, epsilon: 0.08\n",
"episode: 5990, score: 4.0, epsilon: 0.08\n",
"episode: 5995, score: 2.0, epsilon: 0.08\n",
"marking, episode: 6000, score: 3.0, mean_score: 2.12, std_score: 1.98\n",
"episode: 6000, score: 3.0, epsilon: 0.08\n",
"episode: 6005, score: 1.0, epsilon: 0.08\n",
"episode: 6010, score: 2.0, epsilon: 0.08\n",
"episode: 6015, score: 1.0, epsilon: 0.08\n",
"episode: 6020, score: 2.0, epsilon: 0.08\n",
"episode: 6025, score: 3.0, epsilon: 0.08\n",
"episode: 6030, score: 0.0, epsilon: 0.08\n",
"episode: 6035, score: 6.0, epsilon: 0.08\n",
"episode: 6040, score: 3.0, epsilon: 0.08\n",
"episode: 6045, score: 3.0, epsilon: 0.08\n",
"episode: 6050, score: 0.0, epsilon: 0.08\n",
"episode: 6055, score: 0.0, epsilon: 0.08\n",
"episode: 6060, score: 2.0, epsilon: 0.08\n",
"episode: 6065, score: 6.0, epsilon: 0.08\n",
"episode: 6070, score: 0.0, epsilon: 0.08\n",
"episode: 6075, score: 1.0, epsilon: 0.08\n",
"episode: 6080, score: 6.0, epsilon: 0.08\n",
"episode: 6085, score: 4.0, epsilon: 0.08\n",
"episode: 6090, score: 1.0, epsilon: 0.08\n",
"episode: 6095, score: 2.0, epsilon: 0.08\n",
"marking, episode: 6100, score: 2.0, mean_score: 2.20, std_score: 2.07\n",
"episode: 6100, score: 2.0, epsilon: 0.08\n",
"episode: 6105, score: 1.0, epsilon: 0.08\n",
"episode: 6110, score: 2.0, epsilon: 0.08\n",
"episode: 6115, score: 1.0, epsilon: 0.08\n",
"episode: 6120, score: 6.0, epsilon: 0.08\n",
"episode: 6125, score: 5.0, epsilon: 0.08\n",
"episode: 6130, score: 0.0, epsilon: 0.08\n",
"episode: 6135, score: 6.0, epsilon: 0.08\n",
"episode: 6140, score: 6.0, epsilon: 0.08\n",
"episode: 6145, score: 1.0, epsilon: 0.08\n",
"episode: 6150, score: 1.0, epsilon: 0.08\n",
"episode: 6155, score: 0.0, epsilon: 0.08\n",
"episode: 6160, score: 0.0, epsilon: 0.08\n",
"episode: 6165, score: 1.0, epsilon: 0.08\n",
"episode: 6170, score: 0.0, epsilon: 0.08\n",
"episode: 6175, score: 0.0, epsilon: 0.08\n",
"episode: 6180, score: 2.0, epsilon: 0.08\n",
"episode: 6185, score: 0.0, epsilon: 0.08\n",
"episode: 6190, score: 3.0, epsilon: 0.08\n",
"episode: 6195, score: 6.0, epsilon: 0.08\n",
"marking, episode: 6200, score: 6.0, mean_score: 2.07, std_score: 2.06\n",
"episode: 6200, score: 6.0, epsilon: 0.08\n",
"episode: 6205, score: 0.0, epsilon: 0.08\n",
"episode: 6210, score: 0.0, epsilon: 0.08\n",
"episode: 6215, score: 3.0, epsilon: 0.08\n",
"episode: 6220, score: 0.0, epsilon: 0.08\n",
"episode: 6225, score: 0.0, epsilon: 0.08\n",
"episode: 6230, score: 0.0, epsilon: 0.08\n",
"episode: 6235, score: 2.0, epsilon: 0.08\n",
"episode: 6240, score: 3.0, epsilon: 0.08\n",
"episode: 6245, score: 1.0, epsilon: 0.08\n",
"episode: 6250, score: 1.0, epsilon: 0.08\n",
"episode: 6255, score: 0.0, epsilon: 0.08\n",
"episode: 6260, score: 4.0, epsilon: 0.08\n",
"episode: 6265, score: 2.0, epsilon: 0.08\n",
"episode: 6270, score: 1.0, epsilon: 0.08\n",
"episode: 6275, score: 1.0, epsilon: 0.08\n",
"episode: 6280, score: 4.0, epsilon: 0.08\n",
"episode: 6285, score: 0.0, epsilon: 0.08\n",
"episode: 6290, score: 1.0, epsilon: 0.08\n",
"episode: 6295, score: 0.0, epsilon: 0.08\n",
"marking, episode: 6300, score: 5.0, mean_score: 2.34, std_score: 2.10\n",
"episode: 6300, score: 5.0, epsilon: 0.08\n",
"episode: 6305, score: 2.0, epsilon: 0.08\n",
"episode: 6310, score: 2.0, epsilon: 0.08\n",
"episode: 6315, score: 3.0, epsilon: 0.08\n",
"episode: 6320, score: 2.0, epsilon: 0.08\n",
"episode: 6325, score: 6.0, epsilon: 0.08\n",
"episode: 6330, score: 6.0, epsilon: 0.08\n",
"episode: 6335, score: 0.0, epsilon: 0.08\n",
"episode: 6340, score: 1.0, epsilon: 0.08\n",
"episode: 6345, score: 5.0, epsilon: 0.08\n",
"episode: 6350, score: 4.0, epsilon: 0.08\n",
"episode: 6355, score: 4.0, epsilon: 0.08\n",
"episode: 6360, score: 1.0, epsilon: 0.08\n",
"episode: 6365, score: 1.0, epsilon: 0.08\n",
"episode: 6370, score: 6.0, epsilon: 0.08\n",
"episode: 6375, score: 1.0, epsilon: 0.08\n",
"episode: 6380, score: 5.0, epsilon: 0.08\n",
"episode: 6385, score: 0.0, epsilon: 0.08\n",
"episode: 6390, score: 6.0, epsilon: 0.08\n",
"episode: 6395, score: 1.0, epsilon: 0.08\n",
"marking, episode: 6400, score: 2.0, mean_score: 3.10, std_score: 2.16\n",
"episode: 6400, score: 2.0, epsilon: 0.08\n",
"episode: 6405, score: 0.0, epsilon: 0.08\n",
"episode: 6410, score: 0.0, epsilon: 0.08\n",
"episode: 6415, score: 0.0, epsilon: 0.08\n",
"episode: 6420, score: 5.0, epsilon: 0.08\n",
"episode: 6425, score: 2.0, epsilon: 0.08\n",
"episode: 6430, score: 6.0, epsilon: 0.08\n",
"episode: 6435, score: 6.0, epsilon: 0.08\n",
"episode: 6440, score: 6.0, epsilon: 0.08\n",
"episode: 6445, score: 1.0, epsilon: 0.08\n",
"episode: 6450, score: 6.0, epsilon: 0.08\n",
"episode: 6455, score: 6.0, epsilon: 0.08\n",
"episode: 6460, score: 6.0, epsilon: 0.08\n",
"episode: 6465, score: 3.0, epsilon: 0.08\n",
"episode: 6470, score: 2.0, epsilon: 0.08\n",
"episode: 6475, score: 4.0, epsilon: 0.08\n",
"episode: 6480, score: 0.0, epsilon: 0.08\n",
"episode: 6485, score: 5.0, epsilon: 0.08\n",
"episode: 6490, score: 3.0, epsilon: 0.08\n",
"episode: 6495, score: 5.0, epsilon: 0.08\n",
"marking, episode: 6500, score: 1.0, mean_score: 2.57, std_score: 2.20\n",
"episode: 6500, score: 1.0, epsilon: 0.08\n",
"episode: 6505, score: 0.0, epsilon: 0.08\n",
"episode: 6510, score: 2.0, epsilon: 0.08\n",
"episode: 6515, score: 1.0, epsilon: 0.08\n",
"episode: 6520, score: 5.0, epsilon: 0.08\n",
"episode: 6525, score: 6.0, epsilon: 0.08\n",
"episode: 6530, score: 3.0, epsilon: 0.08\n",
"episode: 6535, score: 0.0, epsilon: 0.08\n",
"episode: 6540, score: 2.0, epsilon: 0.08\n",
"episode: 6545, score: 1.0, epsilon: 0.08\n",
"episode: 6550, score: 5.0, epsilon: 0.08\n",
"episode: 6555, score: 2.0, epsilon: 0.08\n",
"episode: 6560, score: 0.0, epsilon: 0.08\n",
"episode: 6565, score: 2.0, epsilon: 0.08\n",
"episode: 6570, score: 6.0, epsilon: 0.08\n",
"episode: 6575, score: 3.0, epsilon: 0.08\n",
"episode: 6580, score: 1.0, epsilon: 0.08\n",
"episode: 6585, score: 0.0, epsilon: 0.08\n",
"episode: 6590, score: 5.0, epsilon: 0.08\n",
"episode: 6595, score: 0.0, epsilon: 0.08\n",
"marking, episode: 6600, score: 0.0, mean_score: 2.46, std_score: 2.07\n",
"episode: 6600, score: 0.0, epsilon: 0.08\n",
"episode: 6605, score: 5.0, epsilon: 0.08\n",
"episode: 6610, score: 2.0, epsilon: 0.08\n",
"episode: 6615, score: 6.0, epsilon: 0.08\n",
"episode: 6620, score: 0.0, epsilon: 0.08\n",
"episode: 6625, score: 1.0, epsilon: 0.08\n",
"episode: 6630, score: 2.0, epsilon: 0.08\n",
"episode: 6635, score: 0.0, epsilon: 0.08\n",
"episode: 6640, score: 0.0, epsilon: 0.08\n",
"episode: 6645, score: 0.0, epsilon: 0.08\n",
"episode: 6650, score: 0.0, epsilon: 0.08\n",
"episode: 6655, score: 5.0, epsilon: 0.08\n",
"episode: 6660, score: 0.0, epsilon: 0.08\n",
"episode: 6665, score: 4.0, epsilon: 0.08\n",
"episode: 6670, score: 6.0, epsilon: 0.08\n",
"episode: 6675, score: 6.0, epsilon: 0.08\n",
"episode: 6680, score: 6.0, epsilon: 0.08\n",
"episode: 6685, score: 4.0, epsilon: 0.08\n",
"episode: 6690, score: 3.0, epsilon: 0.08\n",
"episode: 6695, score: 1.0, epsilon: 0.08\n",
"marking, episode: 6700, score: 4.0, mean_score: 2.53, std_score: 2.13\n",
"episode: 6700, score: 4.0, epsilon: 0.08\n",
"episode: 6705, score: 4.0, epsilon: 0.08\n",
"episode: 6710, score: 1.0, epsilon: 0.08\n",
"episode: 6715, score: 1.0, epsilon: 0.08\n",
"episode: 6720, score: 3.0, epsilon: 0.08\n",
"episode: 6725, score: 2.0, epsilon: 0.08\n",
"episode: 6730, score: 1.0, epsilon: 0.08\n",
"episode: 6735, score: 6.0, epsilon: 0.08\n",
"episode: 6740, score: 6.0, epsilon: 0.08\n",
"episode: 6745, score: 1.0, epsilon: 0.08\n",
"episode: 6750, score: 0.0, epsilon: 0.08\n",
"episode: 6755, score: 0.0, epsilon: 0.08\n",
"episode: 6760, score: 0.0, epsilon: 0.08\n",
"episode: 6765, score: 2.0, epsilon: 0.08\n",
"episode: 6770, score: 4.0, epsilon: 0.08\n",
"episode: 6775, score: 0.0, epsilon: 0.08\n",
"episode: 6780, score: 6.0, epsilon: 0.08\n",
"episode: 6785, score: 0.0, epsilon: 0.08\n",
"episode: 6790, score: 4.0, epsilon: 0.08\n",
"episode: 6795, score: 4.0, epsilon: 0.08\n",
"marking, episode: 6800, score: 1.0, mean_score: 2.40, std_score: 2.10\n",
"episode: 6800, score: 1.0, epsilon: 0.08\n",
"episode: 6805, score: 1.0, epsilon: 0.08\n",
"episode: 6810, score: 6.0, epsilon: 0.08\n",
"episode: 6815, score: 1.0, epsilon: 0.08\n",
"episode: 6820, score: 6.0, epsilon: 0.08\n",
"episode: 6825, score: 0.0, epsilon: 0.08\n",
"episode: 6830, score: 0.0, epsilon: 0.08\n",
"episode: 6835, score: 0.0, epsilon: 0.08\n",
"episode: 6840, score: 1.0, epsilon: 0.08\n",
"episode: 6845, score: 6.0, epsilon: 0.08\n",
"episode: 6850, score: 1.0, epsilon: 0.08\n",
"episode: 6855, score: 1.0, epsilon: 0.08\n",
"episode: 6860, score: 2.0, epsilon: 0.08\n",
"episode: 6865, score: 2.0, epsilon: 0.08\n",
"episode: 6870, score: 2.0, epsilon: 0.08\n",
"episode: 6875, score: 1.0, epsilon: 0.08\n",
"episode: 6880, score: 4.0, epsilon: 0.08\n",
"episode: 6885, score: 2.0, epsilon: 0.08\n",
"episode: 6890, score: 1.0, epsilon: 0.08\n",
"episode: 6895, score: 2.0, epsilon: 0.08\n",
"marking, episode: 6900, score: 1.0, mean_score: 2.34, std_score: 2.17\n",
"episode: 6900, score: 1.0, epsilon: 0.08\n",
"episode: 6905, score: 1.0, epsilon: 0.08\n",
"episode: 6910, score: 3.0, epsilon: 0.08\n",
"episode: 6915, score: 0.0, epsilon: 0.08\n",
"episode: 6920, score: 1.0, epsilon: 0.08\n",
"episode: 6925, score: 0.0, epsilon: 0.08\n",
"episode: 6930, score: 1.0, epsilon: 0.08\n",
"episode: 6935, score: 3.0, epsilon: 0.08\n",
"episode: 6940, score: 1.0, epsilon: 0.08\n",
"episode: 6945, score: 0.0, epsilon: 0.08\n",
"episode: 6950, score: 3.0, epsilon: 0.08\n",
"episode: 6955, score: 6.0, epsilon: 0.08\n",
"episode: 6960, score: 1.0, epsilon: 0.08\n",
"episode: 6965, score: 6.0, epsilon: 0.08\n",
"episode: 6970, score: 2.0, epsilon: 0.08\n",
"episode: 6975, score: 2.0, epsilon: 0.08\n",
"episode: 6980, score: 2.0, epsilon: 0.08\n",
"episode: 6985, score: 3.0, epsilon: 0.08\n",
"episode: 6990, score: 2.0, epsilon: 0.08\n",
"episode: 6995, score: 6.0, epsilon: 0.08\n",
"marking, episode: 7000, score: 4.0, mean_score: 2.37, std_score: 2.24\n",
"episode: 7000, score: 4.0, epsilon: 0.08\n",
"episode: 7005, score: 6.0, epsilon: 0.08\n",
"episode: 7010, score: 0.0, epsilon: 0.08\n",
"episode: 7015, score: 6.0, epsilon: 0.08\n",
"episode: 7020, score: 6.0, epsilon: 0.08\n",
"episode: 7025, score: 1.0, epsilon: 0.08\n",
"episode: 7030, score: 6.0, epsilon: 0.08\n",
"episode: 7035, score: 1.0, epsilon: 0.08\n",
"episode: 7040, score: 0.0, epsilon: 0.08\n",
"episode: 7045, score: 0.0, epsilon: 0.08\n",
"episode: 7050, score: 0.0, epsilon: 0.08\n",
"episode: 7055, score: 5.0, epsilon: 0.08\n",
"episode: 7060, score: 0.0, epsilon: 0.08\n",
"episode: 7065, score: 1.0, epsilon: 0.08\n",
"episode: 7070, score: 4.0, epsilon: 0.08\n",
"episode: 7075, score: 0.0, epsilon: 0.08\n",
"episode: 7080, score: 0.0, epsilon: 0.08\n",
"episode: 7085, score: 0.0, epsilon: 0.08\n",
"episode: 7090, score: 2.0, epsilon: 0.08\n",
"episode: 7095, score: 0.0, epsilon: 0.08\n",
"marking, episode: 7100, score: 1.0, mean_score: 2.57, std_score: 2.40\n",
"episode: 7100, score: 1.0, epsilon: 0.08\n",
"episode: 7105, score: 0.0, epsilon: 0.08\n",
"episode: 7110, score: 0.0, epsilon: 0.08\n",
"episode: 7115, score: 1.0, epsilon: 0.08\n",
"episode: 7120, score: 2.0, epsilon: 0.08\n",
"episode: 7125, score: 1.0, epsilon: 0.08\n",
"episode: 7130, score: 0.0, epsilon: 0.08\n",
"episode: 7135, score: 2.0, epsilon: 0.08\n",
"episode: 7140, score: 5.0, epsilon: 0.08\n",
"episode: 7145, score: 0.0, epsilon: 0.08\n",
"episode: 7150, score: 0.0, epsilon: 0.08\n",
"episode: 7155, score: 3.0, epsilon: 0.08\n",
"episode: 7160, score: 1.0, epsilon: 0.08\n",
"episode: 7165, score: 1.0, epsilon: 0.08\n",
"episode: 7170, score: 0.0, epsilon: 0.08\n",
"episode: 7175, score: 3.0, epsilon: 0.08\n",
"episode: 7180, score: 0.0, epsilon: 0.08\n",
"episode: 7185, score: 0.0, epsilon: 0.08\n",
"episode: 7190, score: 2.0, epsilon: 0.08\n",
"episode: 7195, score: 1.0, epsilon: 0.08\n",
"marking, episode: 7200, score: 3.0, mean_score: 2.40, std_score: 2.05\n",
"episode: 7200, score: 3.0, epsilon: 0.08\n",
"episode: 7205, score: 0.0, epsilon: 0.08\n",
"episode: 7210, score: 6.0, epsilon: 0.08\n",
"episode: 7215, score: 6.0, epsilon: 0.08\n",
"episode: 7220, score: 4.0, epsilon: 0.08\n",
"episode: 7225, score: 0.0, epsilon: 0.08\n",
"episode: 7230, score: 6.0, epsilon: 0.08\n",
"episode: 7235, score: 2.0, epsilon: 0.08\n",
"episode: 7240, score: 0.0, epsilon: 0.08\n",
"episode: 7245, score: 6.0, epsilon: 0.08\n",
"episode: 7250, score: 4.0, epsilon: 0.08\n",
"episode: 7255, score: 1.0, epsilon: 0.08\n",
"episode: 7260, score: 2.0, epsilon: 0.08\n",
"episode: 7265, score: 2.0, epsilon: 0.08\n",
"episode: 7270, score: 5.0, epsilon: 0.08\n",
"episode: 7275, score: 4.0, epsilon: 0.08\n",
"episode: 7280, score: 3.0, epsilon: 0.08\n",
"episode: 7285, score: 6.0, epsilon: 0.08\n",
"episode: 7290, score: 3.0, epsilon: 0.08\n",
"episode: 7295, score: 3.0, epsilon: 0.08\n",
"marking, episode: 7300, score: 4.0, mean_score: 2.52, std_score: 2.15\n",
"episode: 7300, score: 4.0, epsilon: 0.08\n",
"episode: 7305, score: 0.0, epsilon: 0.08\n",
"episode: 7310, score: 0.0, epsilon: 0.08\n",
"episode: 7315, score: 1.0, epsilon: 0.08\n",
"episode: 7320, score: 1.0, epsilon: 0.08\n",
"episode: 7325, score: 5.0, epsilon: 0.08\n",
"episode: 7330, score: 5.0, epsilon: 0.08\n",
"episode: 7335, score: 4.0, epsilon: 0.08\n",
"episode: 7340, score: 2.0, epsilon: 0.08\n",
"episode: 7345, score: 6.0, epsilon: 0.08\n",
"episode: 7350, score: 3.0, epsilon: 0.08\n",
"episode: 7355, score: 6.0, epsilon: 0.08\n",
"episode: 7360, score: 6.0, epsilon: 0.08\n",
"episode: 7365, score: 5.0, epsilon: 0.08\n",
"episode: 7370, score: 6.0, epsilon: 0.08\n",
"episode: 7375, score: 2.0, epsilon: 0.08\n",
"episode: 7380, score: 0.0, epsilon: 0.08\n",
"episode: 7385, score: 0.0, epsilon: 0.08\n",
"episode: 7390, score: 2.0, epsilon: 0.08\n",
"episode: 7395, score: 1.0, epsilon: 0.08\n",
"marking, episode: 7400, score: 4.0, mean_score: 2.17, std_score: 2.04\n",
"episode: 7400, score: 4.0, epsilon: 0.08\n",
"episode: 7405, score: 2.0, epsilon: 0.08\n",
"episode: 7410, score: 2.0, epsilon: 0.08\n",
"episode: 7415, score: 4.0, epsilon: 0.08\n",
"episode: 7420, score: 6.0, epsilon: 0.08\n",
"episode: 7425, score: 0.0, epsilon: 0.08\n",
"episode: 7430, score: 6.0, epsilon: 0.08\n",
"episode: 7435, score: 1.0, epsilon: 0.08\n",
"episode: 7440, score: 6.0, epsilon: 0.08\n",
"episode: 7445, score: 1.0, epsilon: 0.08\n",
"episode: 7450, score: 6.0, epsilon: 0.08\n",
"episode: 7455, score: 1.0, epsilon: 0.08\n",
"episode: 7460, score: 5.0, epsilon: 0.08\n",
"episode: 7465, score: 0.0, epsilon: 0.08\n",
"episode: 7470, score: 0.0, epsilon: 0.08\n",
"episode: 7475, score: 6.0, epsilon: 0.08\n",
"episode: 7480, score: 0.0, epsilon: 0.08\n",
"episode: 7485, score: 0.0, epsilon: 0.08\n",
"episode: 7490, score: 0.0, epsilon: 0.08\n",
"episode: 7495, score: 6.0, epsilon: 0.08\n",
"marking, episode: 7500, score: 1.0, mean_score: 2.30, std_score: 2.27\n",
"episode: 7500, score: 1.0, epsilon: 0.08\n",
"episode: 7505, score: 1.0, epsilon: 0.08\n",
"episode: 7510, score: 2.0, epsilon: 0.08\n",
"episode: 7515, score: 0.0, epsilon: 0.08\n",
"episode: 7520, score: 2.0, epsilon: 0.08\n",
"episode: 7525, score: 1.0, epsilon: 0.08\n",
"episode: 7530, score: 2.0, epsilon: 0.08\n",
"episode: 7535, score: 0.0, epsilon: 0.08\n",
"episode: 7540, score: 6.0, epsilon: 0.08\n",
"episode: 7545, score: 0.0, epsilon: 0.08\n",
"episode: 7550, score: 6.0, epsilon: 0.08\n",
"episode: 7555, score: 6.0, epsilon: 0.08\n",
"episode: 7560, score: 0.0, epsilon: 0.08\n",
"episode: 7565, score: 6.0, epsilon: 0.08\n",
"episode: 7570, score: 4.0, epsilon: 0.08\n",
"episode: 7575, score: 4.0, epsilon: 0.08\n",
"episode: 7580, score: 2.0, epsilon: 0.08\n",
"episode: 7585, score: 6.0, epsilon: 0.08\n",
"episode: 7590, score: 2.0, epsilon: 0.08\n",
"episode: 7595, score: 3.0, epsilon: 0.08\n",
"marking, episode: 7600, score: 1.0, mean_score: 2.57, std_score: 2.19\n",
"episode: 7600, score: 1.0, epsilon: 0.08\n",
"episode: 7605, score: 1.0, epsilon: 0.08\n",
"episode: 7610, score: 0.0, epsilon: 0.08\n",
"episode: 7615, score: 0.0, epsilon: 0.08\n",
"episode: 7620, score: 0.0, epsilon: 0.08\n",
"episode: 7625, score: 5.0, epsilon: 0.08\n",
"episode: 7630, score: 0.0, epsilon: 0.08\n",
"episode: 7635, score: 6.0, epsilon: 0.08\n",
"episode: 7640, score: 0.0, epsilon: 0.08\n",
"episode: 7645, score: 6.0, epsilon: 0.08\n",
"episode: 7650, score: 0.0, epsilon: 0.08\n",
"episode: 7655, score: 5.0, epsilon: 0.08\n",
"episode: 7660, score: 4.0, epsilon: 0.08\n",
"episode: 7665, score: 0.0, epsilon: 0.08\n",
"episode: 7670, score: 4.0, epsilon: 0.08\n",
"episode: 7675, score: 1.0, epsilon: 0.08\n",
"episode: 7680, score: 1.0, epsilon: 0.08\n",
"episode: 7685, score: 3.0, epsilon: 0.08\n",
"episode: 7690, score: 6.0, epsilon: 0.08\n",
"episode: 7695, score: 2.0, epsilon: 0.08\n",
"marking, episode: 7700, score: 3.0, mean_score: 2.55, std_score: 2.16\n",
"episode: 7700, score: 3.0, epsilon: 0.08\n",
"episode: 7705, score: 1.0, epsilon: 0.08\n",
"episode: 7710, score: 0.0, epsilon: 0.08\n",
"episode: 7715, score: 2.0, epsilon: 0.08\n",
"episode: 7720, score: 1.0, epsilon: 0.08\n",
"episode: 7725, score: 4.0, epsilon: 0.08\n",
"episode: 7730, score: 1.0, epsilon: 0.08\n",
"episode: 7735, score: 5.0, epsilon: 0.08\n",
"episode: 7740, score: 1.0, epsilon: 0.08\n",
"episode: 7745, score: 3.0, epsilon: 0.08\n",
"episode: 7750, score: 0.0, epsilon: 0.08\n",
"episode: 7755, score: 1.0, epsilon: 0.08\n",
"episode: 7760, score: 0.0, epsilon: 0.08\n",
"episode: 7765, score: 1.0, epsilon: 0.08\n",
"episode: 7770, score: 3.0, epsilon: 0.08\n",
"episode: 7775, score: 4.0, epsilon: 0.08\n",
"episode: 7780, score: 1.0, epsilon: 0.08\n",
"episode: 7785, score: 2.0, epsilon: 0.08\n",
"episode: 7790, score: 0.0, epsilon: 0.08\n",
"episode: 7795, score: 4.0, epsilon: 0.08\n",
"marking, episode: 7800, score: 0.0, mean_score: 2.09, std_score: 2.15\n",
"episode: 7800, score: 0.0, epsilon: 0.08\n",
"episode: 7805, score: 0.0, epsilon: 0.08\n",
"episode: 7810, score: 6.0, epsilon: 0.08\n",
"episode: 7815, score: 0.0, epsilon: 0.08\n",
"episode: 7820, score: 4.0, epsilon: 0.08\n",
"episode: 7825, score: 6.0, epsilon: 0.08\n",
"episode: 7830, score: 0.0, epsilon: 0.08\n",
"episode: 7835, score: 1.0, epsilon: 0.08\n",
"episode: 7840, score: 1.0, epsilon: 0.08\n",
"episode: 7845, score: 0.0, epsilon: 0.08\n",
"episode: 7850, score: 6.0, epsilon: 0.08\n",
"episode: 7855, score: 1.0, epsilon: 0.08\n",
"episode: 7860, score: 0.0, epsilon: 0.08\n",
"episode: 7865, score: 4.0, epsilon: 0.08\n",
"episode: 7870, score: 1.0, epsilon: 0.08\n",
"episode: 7875, score: 0.0, epsilon: 0.08\n",
"episode: 7880, score: 6.0, epsilon: 0.08\n",
"episode: 7885, score: 0.0, epsilon: 0.08\n",
"episode: 7890, score: 2.0, epsilon: 0.08\n",
"episode: 7895, score: 6.0, epsilon: 0.08\n",
"marking, episode: 7900, score: 1.0, mean_score: 2.29, std_score: 2.17\n",
"episode: 7900, score: 1.0, epsilon: 0.08\n",
"episode: 7905, score: 0.0, epsilon: 0.08\n",
"episode: 7910, score: 3.0, epsilon: 0.08\n",
"episode: 7915, score: 4.0, epsilon: 0.08\n",
"episode: 7920, score: 2.0, epsilon: 0.08\n",
"episode: 7925, score: 0.0, epsilon: 0.08\n",
"episode: 7930, score: 1.0, epsilon: 0.08\n",
"episode: 7935, score: 6.0, epsilon: 0.08\n",
"episode: 7940, score: 1.0, epsilon: 0.08\n",
"episode: 7945, score: 5.0, epsilon: 0.08\n",
"episode: 7950, score: 4.0, epsilon: 0.08\n",
"episode: 7955, score: 0.0, epsilon: 0.08\n",
"episode: 7960, score: 0.0, epsilon: 0.08\n",
"episode: 7965, score: 2.0, epsilon: 0.08\n",
"episode: 7970, score: 1.0, epsilon: 0.08\n",
"episode: 7975, score: 1.0, epsilon: 0.08\n",
"episode: 7980, score: 2.0, epsilon: 0.08\n",
"episode: 7985, score: 1.0, epsilon: 0.08\n",
"episode: 7990, score: 1.0, epsilon: 0.08\n",
"episode: 7995, score: 1.0, epsilon: 0.08\n",
"marking, episode: 8000, score: 1.0, mean_score: 2.31, std_score: 2.12\n",
"episode: 8000, score: 1.0, epsilon: 0.08\n",
"episode: 8005, score: 0.0, epsilon: 0.08\n",
"episode: 8010, score: 6.0, epsilon: 0.08\n",
"episode: 8015, score: 4.0, epsilon: 0.08\n",
"episode: 8020, score: 1.0, epsilon: 0.08\n",
"episode: 8025, score: 6.0, epsilon: 0.08\n",
"episode: 8030, score: 1.0, epsilon: 0.08\n",
"episode: 8035, score: 6.0, epsilon: 0.08\n",
"episode: 8040, score: 2.0, epsilon: 0.08\n",
"episode: 8045, score: 1.0, epsilon: 0.08\n",
"episode: 8050, score: 1.0, epsilon: 0.08\n",
"episode: 8055, score: 4.0, epsilon: 0.08\n",
"episode: 8060, score: 4.0, epsilon: 0.08\n",
"episode: 8065, score: 2.0, epsilon: 0.08\n",
"episode: 8070, score: 6.0, epsilon: 0.08\n",
"episode: 8075, score: 6.0, epsilon: 0.08\n",
"episode: 8080, score: 0.0, epsilon: 0.08\n",
"episode: 8085, score: 1.0, epsilon: 0.08\n",
"episode: 8090, score: 1.0, epsilon: 0.08\n",
"episode: 8095, score: 1.0, epsilon: 0.08\n",
"marking, episode: 8100, score: 0.0, mean_score: 2.57, std_score: 2.23\n",
"episode: 8100, score: 0.0, epsilon: 0.08\n",
"episode: 8105, score: 3.0, epsilon: 0.08\n",
"episode: 8110, score: 0.0, epsilon: 0.08\n",
"episode: 8115, score: 1.0, epsilon: 0.08\n",
"episode: 8120, score: 0.0, epsilon: 0.08\n",
"episode: 8125, score: 1.0, epsilon: 0.08\n",
"episode: 8130, score: 5.0, epsilon: 0.08\n",
"episode: 8135, score: 0.0, epsilon: 0.08\n",
"episode: 8140, score: 5.0, epsilon: 0.08\n",
"episode: 8145, score: 0.0, epsilon: 0.08\n",
"episode: 8150, score: 1.0, epsilon: 0.08\n",
"episode: 8155, score: 1.0, epsilon: 0.08\n",
"episode: 8160, score: 3.0, epsilon: 0.08\n",
"episode: 8165, score: 0.0, epsilon: 0.08\n",
"episode: 8170, score: 5.0, epsilon: 0.08\n",
"episode: 8175, score: 0.0, epsilon: 0.08\n",
"episode: 8180, score: 1.0, epsilon: 0.08\n",
"episode: 8185, score: 0.0, epsilon: 0.08\n",
"episode: 8190, score: 4.0, epsilon: 0.08\n",
"episode: 8195, score: 2.0, epsilon: 0.08\n",
"marking, episode: 8200, score: 6.0, mean_score: 2.74, std_score: 2.21\n",
"episode: 8200, score: 6.0, epsilon: 0.08\n",
"episode: 8205, score: 6.0, epsilon: 0.08\n",
"episode: 8210, score: 1.0, epsilon: 0.08\n",
"episode: 8215, score: 0.0, epsilon: 0.08\n",
"episode: 8220, score: 6.0, epsilon: 0.08\n",
"episode: 8225, score: 0.0, epsilon: 0.08\n",
"episode: 8230, score: 6.0, epsilon: 0.08\n",
"episode: 8235, score: 0.0, epsilon: 0.08\n",
"episode: 8240, score: 4.0, epsilon: 0.08\n",
"episode: 8245, score: 6.0, epsilon: 0.08\n",
"episode: 8250, score: 0.0, epsilon: 0.08\n",
"episode: 8255, score: 1.0, epsilon: 0.08\n",
"episode: 8260, score: 6.0, epsilon: 0.08\n",
"episode: 8265, score: 0.0, epsilon: 0.08\n",
"episode: 8270, score: 4.0, epsilon: 0.08\n",
"episode: 8275, score: 3.0, epsilon: 0.08\n",
"episode: 8280, score: 0.0, epsilon: 0.08\n",
"episode: 8285, score: 0.0, epsilon: 0.08\n",
"episode: 8290, score: 0.0, epsilon: 0.08\n",
"episode: 8295, score: 2.0, epsilon: 0.08\n",
"marking, episode: 8300, score: 1.0, mean_score: 2.59, std_score: 2.25\n",
"episode: 8300, score: 1.0, epsilon: 0.08\n",
"episode: 8305, score: 5.0, epsilon: 0.08\n",
"episode: 8310, score: 6.0, epsilon: 0.08\n",
"episode: 8315, score: 0.0, epsilon: 0.08\n",
"episode: 8320, score: 2.0, epsilon: 0.08\n",
"episode: 8325, score: 6.0, epsilon: 0.08\n",
"episode: 8330, score: 2.0, epsilon: 0.08\n",
"episode: 8335, score: 2.0, epsilon: 0.08\n",
"episode: 8340, score: 1.0, epsilon: 0.08\n",
"episode: 8345, score: 2.0, epsilon: 0.08\n",
"episode: 8350, score: 4.0, epsilon: 0.08\n",
"episode: 8355, score: 1.0, epsilon: 0.08\n",
"episode: 8360, score: 4.0, epsilon: 0.08\n",
"episode: 8365, score: 6.0, epsilon: 0.08\n",
"episode: 8370, score: 0.0, epsilon: 0.08\n",
"episode: 8375, score: 0.0, epsilon: 0.08\n",
"episode: 8380, score: 4.0, epsilon: 0.08\n",
"episode: 8385, score: 3.0, epsilon: 0.08\n",
"episode: 8390, score: 2.0, epsilon: 0.08\n",
"episode: 8395, score: 2.0, epsilon: 0.08\n",
"marking, episode: 8400, score: 3.0, mean_score: 2.47, std_score: 2.03\n",
"episode: 8400, score: 3.0, epsilon: 0.08\n",
"episode: 8405, score: 2.0, epsilon: 0.08\n",
"episode: 8410, score: 2.0, epsilon: 0.08\n",
"episode: 8415, score: 1.0, epsilon: 0.08\n",
"episode: 8420, score: 0.0, epsilon: 0.08\n",
"episode: 8425, score: 1.0, epsilon: 0.08\n",
"episode: 8430, score: 3.0, epsilon: 0.08\n",
"episode: 8435, score: 0.0, epsilon: 0.08\n",
"episode: 8440, score: 2.0, epsilon: 0.08\n",
"episode: 8445, score: 3.0, epsilon: 0.08\n",
"episode: 8450, score: 5.0, epsilon: 0.08\n",
"episode: 8455, score: 0.0, epsilon: 0.08\n",
"episode: 8460, score: 5.0, epsilon: 0.08\n",
"episode: 8465, score: 1.0, epsilon: 0.08\n",
"episode: 8470, score: 2.0, epsilon: 0.08\n",
"episode: 8475, score: 0.0, epsilon: 0.08\n",
"episode: 8480, score: 4.0, epsilon: 0.08\n",
"episode: 8485, score: 0.0, epsilon: 0.08\n",
"episode: 8490, score: 1.0, epsilon: 0.08\n",
"episode: 8495, score: 0.0, epsilon: 0.08\n",
"marking, episode: 8500, score: 2.0, mean_score: 2.61, std_score: 2.15\n",
"episode: 8500, score: 2.0, epsilon: 0.08\n",
"episode: 8505, score: 4.0, epsilon: 0.08\n",
"episode: 8510, score: 3.0, epsilon: 0.08\n",
"episode: 8515, score: 1.0, epsilon: 0.08\n",
"episode: 8520, score: 4.0, epsilon: 0.08\n",
"episode: 8525, score: 1.0, epsilon: 0.08\n",
"episode: 8530, score: 6.0, epsilon: 0.08\n",
"episode: 8535, score: 6.0, epsilon: 0.08\n",
"episode: 8540, score: 4.0, epsilon: 0.08\n",
"episode: 8545, score: 6.0, epsilon: 0.08\n",
"episode: 8550, score: 6.0, epsilon: 0.08\n",
"episode: 8555, score: 4.0, epsilon: 0.08\n",
"episode: 8560, score: 4.0, epsilon: 0.08\n",
"episode: 8565, score: 0.0, epsilon: 0.08\n",
"episode: 8570, score: 0.0, epsilon: 0.08\n",
"episode: 8575, score: 6.0, epsilon: 0.08\n",
"episode: 8580, score: 3.0, epsilon: 0.08\n",
"episode: 8585, score: 6.0, epsilon: 0.08\n",
"episode: 8590, score: 6.0, epsilon: 0.08\n",
"episode: 8595, score: 0.0, epsilon: 0.08\n",
"marking, episode: 8600, score: 3.0, mean_score: 3.30, std_score: 2.37\n",
"episode: 8600, score: 3.0, epsilon: 0.08\n",
"episode: 8605, score: 2.0, epsilon: 0.08\n",
"episode: 8610, score: 2.0, epsilon: 0.08\n",
"episode: 8615, score: 2.0, epsilon: 0.08\n",
"episode: 8620, score: 0.0, epsilon: 0.08\n",
"episode: 8625, score: 3.0, epsilon: 0.08\n",
"episode: 8630, score: 0.0, epsilon: 0.08\n",
"episode: 8635, score: 0.0, epsilon: 0.08\n",
"episode: 8640, score: 2.0, epsilon: 0.08\n",
"episode: 8645, score: 2.0, epsilon: 0.08\n",
"episode: 8650, score: 2.0, epsilon: 0.08\n",
"episode: 8655, score: 1.0, epsilon: 0.08\n",
"episode: 8660, score: 3.0, epsilon: 0.08\n",
"episode: 8665, score: 1.0, epsilon: 0.08\n",
"episode: 8670, score: 1.0, epsilon: 0.08\n",
"episode: 8675, score: 1.0, epsilon: 0.08\n",
"episode: 8680, score: 1.0, epsilon: 0.08\n",
"episode: 8685, score: 1.0, epsilon: 0.08\n",
"episode: 8690, score: 0.0, epsilon: 0.08\n",
"episode: 8695, score: 0.0, epsilon: 0.08\n",
"marking, episode: 8700, score: 1.0, mean_score: 2.32, std_score: 2.18\n",
"episode: 8700, score: 1.0, epsilon: 0.08\n",
"episode: 8705, score: 4.0, epsilon: 0.08\n",
"episode: 8710, score: 6.0, epsilon: 0.08\n",
"episode: 8715, score: 0.0, epsilon: 0.08\n",
"episode: 8720, score: 2.0, epsilon: 0.08\n",
"episode: 8725, score: 2.0, epsilon: 0.08\n",
"episode: 8730, score: 1.0, epsilon: 0.08\n",
"episode: 8735, score: 0.0, epsilon: 0.08\n",
"episode: 8740, score: 0.0, epsilon: 0.08\n",
"episode: 8745, score: 0.0, epsilon: 0.08\n",
"episode: 8750, score: 3.0, epsilon: 0.08\n",
"episode: 8755, score: 1.0, epsilon: 0.08\n",
"episode: 8760, score: 1.0, epsilon: 0.08\n",
"episode: 8765, score: 3.0, epsilon: 0.08\n",
"episode: 8770, score: 0.0, epsilon: 0.08\n",
"episode: 8775, score: 0.0, epsilon: 0.08\n",
"episode: 8780, score: 0.0, epsilon: 0.08\n",
"episode: 8785, score: 0.0, epsilon: 0.08\n",
"episode: 8790, score: 0.0, epsilon: 0.08\n",
"episode: 8795, score: 6.0, epsilon: 0.08\n",
"marking, episode: 8800, score: 1.0, mean_score: 2.40, std_score: 2.30\n",
"episode: 8800, score: 1.0, epsilon: 0.08\n",
"episode: 8805, score: 0.0, epsilon: 0.08\n",
"episode: 8810, score: 2.0, epsilon: 0.08\n",
"episode: 8815, score: 6.0, epsilon: 0.08\n",
"episode: 8820, score: 0.0, epsilon: 0.08\n",
"episode: 8825, score: 6.0, epsilon: 0.08\n",
"episode: 8830, score: 6.0, epsilon: 0.08\n",
"episode: 8835, score: 1.0, epsilon: 0.08\n",
"episode: 8840, score: 1.0, epsilon: 0.08\n",
"episode: 8845, score: 3.0, epsilon: 0.08\n",
"episode: 8850, score: 6.0, epsilon: 0.08\n",
"episode: 8855, score: 3.0, epsilon: 0.08\n",
"episode: 8860, score: 3.0, epsilon: 0.08\n",
"episode: 8865, score: 0.0, epsilon: 0.08\n",
"episode: 8870, score: 0.0, epsilon: 0.08\n",
"episode: 8875, score: 1.0, epsilon: 0.08\n",
"episode: 8880, score: 0.0, epsilon: 0.08\n",
"episode: 8885, score: 6.0, epsilon: 0.08\n",
"episode: 8890, score: 2.0, epsilon: 0.08\n",
"episode: 8895, score: 4.0, epsilon: 0.08\n",
"marking, episode: 8900, score: 0.0, mean_score: 2.76, std_score: 2.32\n",
"episode: 8900, score: 0.0, epsilon: 0.08\n",
"episode: 8905, score: 0.0, epsilon: 0.08\n",
"episode: 8910, score: 0.0, epsilon: 0.08\n",
"episode: 8915, score: 6.0, epsilon: 0.08\n",
"episode: 8920, score: 1.0, epsilon: 0.08\n",
"episode: 8925, score: 3.0, epsilon: 0.08\n",
"episode: 8930, score: 2.0, epsilon: 0.08\n",
"episode: 8935, score: 0.0, epsilon: 0.08\n",
"episode: 8940, score: 1.0, epsilon: 0.08\n",
"episode: 8945, score: 3.0, epsilon: 0.08\n",
"episode: 8950, score: 6.0, epsilon: 0.08\n",
"episode: 8955, score: 2.0, epsilon: 0.08\n",
"episode: 8960, score: 1.0, epsilon: 0.08\n",
"episode: 8965, score: 6.0, epsilon: 0.08\n",
"episode: 8970, score: 6.0, epsilon: 0.08\n",
"episode: 8975, score: 0.0, epsilon: 0.08\n",
"episode: 8980, score: 1.0, epsilon: 0.08\n",
"episode: 8985, score: 1.0, epsilon: 0.08\n",
"episode: 8990, score: 1.0, epsilon: 0.08\n",
"episode: 8995, score: 0.0, epsilon: 0.08\n",
"marking, episode: 9000, score: 1.0, mean_score: 2.40, std_score: 2.14\n",
"episode: 9000, score: 1.0, epsilon: 0.08\n",
"episode: 9005, score: 0.0, epsilon: 0.08\n",
"episode: 9010, score: 0.0, epsilon: 0.08\n",
"episode: 9015, score: 0.0, epsilon: 0.08\n",
"episode: 9020, score: 6.0, epsilon: 0.08\n",
"episode: 9025, score: 4.0, epsilon: 0.08\n",
"episode: 9030, score: 0.0, epsilon: 0.08\n",
"episode: 9035, score: 0.0, epsilon: 0.08\n",
"episode: 9040, score: 1.0, epsilon: 0.08\n",
"episode: 9045, score: 0.0, epsilon: 0.08\n",
"episode: 9050, score: 6.0, epsilon: 0.08\n",
"episode: 9055, score: 2.0, epsilon: 0.08\n",
"episode: 9060, score: 1.0, epsilon: 0.08\n",
"episode: 9065, score: 6.0, epsilon: 0.08\n",
"episode: 9070, score: 4.0, epsilon: 0.08\n",
"episode: 9075, score: 3.0, epsilon: 0.08\n",
"episode: 9080, score: 0.0, epsilon: 0.08\n",
"episode: 9085, score: 2.0, epsilon: 0.08\n",
"episode: 9090, score: 1.0, epsilon: 0.08\n",
"episode: 9095, score: 6.0, epsilon: 0.08\n",
"marking, episode: 9100, score: 6.0, mean_score: 2.70, std_score: 2.41\n",
"episode: 9100, score: 6.0, epsilon: 0.08\n",
"episode: 9105, score: 6.0, epsilon: 0.08\n",
"episode: 9110, score: 3.0, epsilon: 0.08\n",
"episode: 9115, score: 2.0, epsilon: 0.08\n",
"episode: 9120, score: 2.0, epsilon: 0.08\n",
"episode: 9125, score: 3.0, epsilon: 0.08\n",
"episode: 9130, score: 6.0, epsilon: 0.08\n",
"episode: 9135, score: 1.0, epsilon: 0.08\n",
"episode: 9140, score: 0.0, epsilon: 0.08\n",
"episode: 9145, score: 2.0, epsilon: 0.08\n",
"episode: 9150, score: 6.0, epsilon: 0.08\n",
"episode: 9155, score: 1.0, epsilon: 0.08\n",
"episode: 9160, score: 0.0, epsilon: 0.08\n",
"episode: 9165, score: 1.0, epsilon: 0.08\n",
"episode: 9170, score: 6.0, epsilon: 0.08\n",
"episode: 9175, score: 0.0, epsilon: 0.08\n",
"episode: 9180, score: 0.0, epsilon: 0.08\n",
"episode: 9185, score: 0.0, epsilon: 0.08\n",
"episode: 9190, score: 3.0, epsilon: 0.08\n",
"episode: 9195, score: 0.0, epsilon: 0.08\n",
"marking, episode: 9200, score: 2.0, mean_score: 2.42, std_score: 2.22\n",
"episode: 9200, score: 2.0, epsilon: 0.08\n",
"episode: 9205, score: 1.0, epsilon: 0.08\n",
"episode: 9210, score: 4.0, epsilon: 0.08\n",
"episode: 9215, score: 1.0, epsilon: 0.08\n",
"episode: 9220, score: 3.0, epsilon: 0.08\n",
"episode: 9225, score: 1.0, epsilon: 0.08\n",
"episode: 9230, score: 1.0, epsilon: 0.08\n",
"episode: 9235, score: 6.0, epsilon: 0.08\n",
"episode: 9240, score: 1.0, epsilon: 0.08\n",
"episode: 9245, score: 1.0, epsilon: 0.08\n",
"episode: 9250, score: 1.0, epsilon: 0.08\n",
"episode: 9255, score: 4.0, epsilon: 0.08\n",
"episode: 9260, score: 2.0, epsilon: 0.08\n",
"episode: 9265, score: 2.0, epsilon: 0.08\n",
"episode: 9270, score: 0.0, epsilon: 0.08\n",
"episode: 9275, score: 4.0, epsilon: 0.08\n",
"episode: 9280, score: 4.0, epsilon: 0.08\n",
"episode: 9285, score: 1.0, epsilon: 0.08\n",
"episode: 9290, score: 4.0, epsilon: 0.08\n",
"episode: 9295, score: 1.0, epsilon: 0.08\n",
"marking, episode: 9300, score: 4.0, mean_score: 2.28, std_score: 2.06\n",
"episode: 9300, score: 4.0, epsilon: 0.08\n",
"episode: 9305, score: 6.0, epsilon: 0.08\n",
"episode: 9310, score: 3.0, epsilon: 0.08\n",
"episode: 9315, score: 2.0, epsilon: 0.08\n",
"episode: 9320, score: 0.0, epsilon: 0.08\n",
"episode: 9325, score: 0.0, epsilon: 0.08\n",
"episode: 9330, score: 6.0, epsilon: 0.08\n",
"episode: 9335, score: 0.0, epsilon: 0.08\n",
"episode: 9340, score: 4.0, epsilon: 0.08\n",
"episode: 9345, score: 1.0, epsilon: 0.08\n",
"episode: 9350, score: 1.0, epsilon: 0.08\n",
"episode: 9355, score: 1.0, epsilon: 0.08\n",
"episode: 9360, score: 1.0, epsilon: 0.08\n",
"episode: 9365, score: 4.0, epsilon: 0.08\n",
"episode: 9370, score: 0.0, epsilon: 0.08\n",
"episode: 9375, score: 3.0, epsilon: 0.08\n",
"episode: 9380, score: 0.0, epsilon: 0.08\n",
"episode: 9385, score: 6.0, epsilon: 0.08\n",
"episode: 9390, score: 1.0, epsilon: 0.08\n",
"episode: 9395, score: 2.0, epsilon: 0.08\n",
"marking, episode: 9400, score: 0.0, mean_score: 2.25, std_score: 1.94\n",
"episode: 9400, score: 0.0, epsilon: 0.08\n",
"episode: 9405, score: 4.0, epsilon: 0.08\n",
"episode: 9410, score: 6.0, epsilon: 0.08\n",
"episode: 9415, score: 3.0, epsilon: 0.08\n",
"episode: 9420, score: 4.0, epsilon: 0.08\n",
"episode: 9425, score: 0.0, epsilon: 0.08\n",
"episode: 9430, score: 6.0, epsilon: 0.08\n",
"episode: 9435, score: 0.0, epsilon: 0.08\n",
"episode: 9440, score: 4.0, epsilon: 0.08\n",
"episode: 9445, score: 1.0, epsilon: 0.08\n",
"episode: 9450, score: 5.0, epsilon: 0.08\n",
"episode: 9455, score: 6.0, epsilon: 0.08\n",
"episode: 9460, score: 6.0, epsilon: 0.08\n",
"episode: 9465, score: 6.0, epsilon: 0.08\n",
"episode: 9470, score: 4.0, epsilon: 0.08\n",
"episode: 9475, score: 0.0, epsilon: 0.08\n",
"episode: 9480, score: 0.0, epsilon: 0.08\n",
"episode: 9485, score: 4.0, epsilon: 0.08\n",
"episode: 9490, score: 6.0, epsilon: 0.08\n",
"episode: 9495, score: 6.0, epsilon: 0.08\n",
"marking, episode: 9500, score: 1.0, mean_score: 2.91, std_score: 2.27\n",
"episode: 9500, score: 1.0, epsilon: 0.08\n",
"episode: 9505, score: 4.0, epsilon: 0.08\n",
"episode: 9510, score: 0.0, epsilon: 0.08\n",
"episode: 9515, score: 1.0, epsilon: 0.08\n",
"episode: 9520, score: 1.0, epsilon: 0.08\n",
"episode: 9525, score: 1.0, epsilon: 0.08\n",
"episode: 9530, score: 5.0, epsilon: 0.08\n",
"episode: 9535, score: 3.0, epsilon: 0.08\n",
"episode: 9540, score: 6.0, epsilon: 0.08\n",
"episode: 9545, score: 1.0, epsilon: 0.08\n",
"episode: 9550, score: 0.0, epsilon: 0.08\n",
"episode: 9555, score: 0.0, epsilon: 0.08\n",
"episode: 9560, score: 1.0, epsilon: 0.08\n",
"episode: 9565, score: 0.0, epsilon: 0.08\n",
"episode: 9570, score: 1.0, epsilon: 0.08\n",
"episode: 9575, score: 3.0, epsilon: 0.08\n",
"episode: 9580, score: 2.0, epsilon: 0.08\n",
"episode: 9585, score: 6.0, epsilon: 0.08\n",
"episode: 9590, score: 1.0, epsilon: 0.08\n",
"episode: 9595, score: 3.0, epsilon: 0.08\n",
"marking, episode: 9600, score: 1.0, mean_score: 2.25, std_score: 2.16\n",
"episode: 9600, score: 1.0, epsilon: 0.08\n",
"episode: 9605, score: 1.0, epsilon: 0.08\n",
"episode: 9610, score: 4.0, epsilon: 0.08\n",
"episode: 9615, score: 0.0, epsilon: 0.08\n",
"episode: 9620, score: 6.0, epsilon: 0.08\n",
"episode: 9625, score: 1.0, epsilon: 0.08\n",
"episode: 9630, score: 0.0, epsilon: 0.08\n",
"episode: 9635, score: 0.0, epsilon: 0.08\n",
"episode: 9640, score: 1.0, epsilon: 0.08\n",
"episode: 9645, score: 0.0, epsilon: 0.08\n",
"episode: 9650, score: 0.0, epsilon: 0.08\n",
"episode: 9655, score: 1.0, epsilon: 0.08\n",
"episode: 9660, score: 0.0, epsilon: 0.08\n",
"episode: 9665, score: 2.0, epsilon: 0.08\n",
"episode: 9670, score: 6.0, epsilon: 0.08\n",
"episode: 9675, score: 0.0, epsilon: 0.08\n",
"episode: 9680, score: 6.0, epsilon: 0.08\n",
"episode: 9685, score: 1.0, epsilon: 0.08\n",
"episode: 9690, score: 4.0, epsilon: 0.08\n",
"episode: 9695, score: 4.0, epsilon: 0.08\n",
"marking, episode: 9700, score: 1.0, mean_score: 2.32, std_score: 2.14\n",
"episode: 9700, score: 1.0, epsilon: 0.08\n",
"episode: 9705, score: 0.0, epsilon: 0.08\n",
"episode: 9710, score: 6.0, epsilon: 0.08\n",
"episode: 9715, score: 1.0, epsilon: 0.08\n",
"episode: 9720, score: 3.0, epsilon: 0.08\n",
"episode: 9725, score: 3.0, epsilon: 0.08\n",
"episode: 9730, score: 6.0, epsilon: 0.08\n",
"episode: 9735, score: 6.0, epsilon: 0.08\n",
"episode: 9740, score: 0.0, epsilon: 0.08\n",
"episode: 9745, score: 1.0, epsilon: 0.08\n",
"episode: 9750, score: 0.0, epsilon: 0.08\n",
"episode: 9755, score: 0.0, epsilon: 0.08\n",
"episode: 9760, score: 6.0, epsilon: 0.08\n",
"episode: 9765, score: 6.0, epsilon: 0.08\n",
"episode: 9770, score: 0.0, epsilon: 0.08\n",
"episode: 9775, score: 2.0, epsilon: 0.08\n",
"episode: 9780, score: 6.0, epsilon: 0.08\n",
"episode: 9785, score: 0.0, epsilon: 0.08\n",
"episode: 9790, score: 3.0, epsilon: 0.08\n",
"episode: 9795, score: 6.0, epsilon: 0.08\n",
"marking, episode: 9800, score: 3.0, mean_score: 2.66, std_score: 2.30\n",
"episode: 9800, score: 3.0, epsilon: 0.08\n",
"episode: 9805, score: 0.0, epsilon: 0.08\n",
"episode: 9810, score: 0.0, epsilon: 0.08\n",
"episode: 9815, score: 3.0, epsilon: 0.08\n",
"episode: 9820, score: 6.0, epsilon: 0.08\n",
"episode: 9825, score: 6.0, epsilon: 0.08\n",
"episode: 9830, score: 0.0, epsilon: 0.08\n",
"episode: 9835, score: 2.0, epsilon: 0.08\n",
"episode: 9840, score: 0.0, epsilon: 0.08\n",
"episode: 9845, score: 0.0, epsilon: 0.08\n",
"episode: 9850, score: 5.0, epsilon: 0.08\n",
"episode: 9855, score: 2.0, epsilon: 0.08\n",
"episode: 9860, score: 1.0, epsilon: 0.08\n",
"episode: 9865, score: 0.0, epsilon: 0.08\n",
"episode: 9870, score: 2.0, epsilon: 0.08\n",
"episode: 9875, score: 0.0, epsilon: 0.08\n",
"episode: 9880, score: 0.0, epsilon: 0.08\n",
"episode: 9885, score: 6.0, epsilon: 0.08\n",
"episode: 9890, score: 2.0, epsilon: 0.08\n",
"episode: 9895, score: 0.0, epsilon: 0.08\n",
"marking, episode: 9900, score: 1.0, mean_score: 2.01, std_score: 2.09\n",
"episode: 9900, score: 1.0, epsilon: 0.08\n",
"episode: 9905, score: 2.0, epsilon: 0.08\n",
"episode: 9910, score: 0.0, epsilon: 0.08\n",
"episode: 9915, score: 0.0, epsilon: 0.08\n",
"episode: 9920, score: 3.0, epsilon: 0.08\n",
"episode: 9925, score: 2.0, epsilon: 0.08\n",
"episode: 9930, score: 3.0, epsilon: 0.08\n",
"episode: 9935, score: 1.0, epsilon: 0.08\n",
"episode: 9940, score: 2.0, epsilon: 0.08\n",
"episode: 9945, score: 1.0, epsilon: 0.08\n",
"episode: 9950, score: 3.0, epsilon: 0.08\n",
"episode: 9955, score: 1.0, epsilon: 0.08\n",
"episode: 9960, score: 0.0, epsilon: 0.08\n",
"episode: 9965, score: 0.0, epsilon: 0.08\n",
"episode: 9970, score: 2.0, epsilon: 0.08\n",
"episode: 9975, score: 0.0, epsilon: 0.08\n",
"episode: 9980, score: 6.0, epsilon: 0.08\n",
"episode: 9985, score: 2.0, epsilon: 0.08\n",
"episode: 9990, score: 1.0, epsilon: 0.08\n",
"episode: 9995, score: 1.0, epsilon: 0.08\n",
"marking, episode: 10000, score: 0.0, mean_score: 2.00, std_score: 2.15\n",
"episode: 10000, score: 0.0, epsilon: 0.08\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "So-4CSO4d1Ji",
"outputId": "b4dda750-418c-4ac7-c205-0a9033c1ac0f",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 1000
}
},
"source": [
"rand_scores = []\n",
"marking = []\n",
"for n_episode in range(10001):\n",
" epsilon = 0.08\n",
" s = env.reset()\n",
" done = False\n",
" score = 0.0\n",
"\n",
" while True:\n",
" a = bad_pol.sample_action(torch.from_numpy(s).float(), epsilon)\n",
" s_prime, r, done, info = env.step(a)\n",
" s = s_prime\n",
" score += r\n",
" if done:\n",
" break\n",
" rand_scores.append(score)\n",
"\n",
" # do not change lines 44-48 here, they are for marking the submission log\n",
" marking.append(score)\n",
" if n_episode%100 == 0:\n",
" print(\"marking, episode: {}, score: {:.1f}, mean_score: {:.2f}, std_score: {:.2f}\".format(\n",
" n_episode, score, np.array(marking).mean(), np.array(marking).std()))\n",
" marking = []\n",
"\n",
" # you can change this part, and print any data you like (so long as it doesn't start with \"marking\")\n",
" if n_episode%print_every==0 and n_episode!=0:\n",
" print(\"episode: {}, score: {:.1f}, epsilon: {:.2f}\".format(n_episode, score, epsilon))"
],
"execution_count": 24,
"outputs": [
{
"output_type": "stream",
"text": [
"marking, episode: 0, score: 4.0, mean_score: 4.00, std_score: 0.00\n",
"episode: 5, score: 0.0, epsilon: 0.08\n",
"episode: 10, score: 1.0, epsilon: 0.08\n",
"episode: 15, score: 2.0, epsilon: 0.08\n",
"episode: 20, score: 3.0, epsilon: 0.08\n",
"episode: 25, score: 1.0, epsilon: 0.08\n",
"episode: 30, score: 0.0, epsilon: 0.08\n",
"episode: 35, score: 0.0, epsilon: 0.08\n",
"episode: 40, score: 1.0, epsilon: 0.08\n",
"episode: 45, score: 0.0, epsilon: 0.08\n",
"episode: 50, score: 0.0, epsilon: 0.08\n",
"episode: 55, score: 0.0, epsilon: 0.08\n",
"episode: 60, score: 4.0, epsilon: 0.08\n",
"episode: 65, score: 1.0, epsilon: 0.08\n",
"episode: 70, score: 0.0, epsilon: 0.08\n",
"episode: 75, score: 0.0, epsilon: 0.08\n",
"episode: 80, score: 0.0, epsilon: 0.08\n",
"episode: 85, score: 1.0, epsilon: 0.08\n",
"episode: 90, score: 0.0, epsilon: 0.08\n",
"episode: 95, score: 2.0, epsilon: 0.08\n",
"marking, episode: 100, score: 2.0, mean_score: 1.13, std_score: 1.50\n",
"episode: 100, score: 2.0, epsilon: 0.08\n",
"episode: 105, score: 0.0, epsilon: 0.08\n",
"episode: 110, score: 0.0, epsilon: 0.08\n",
"episode: 115, score: 2.0, epsilon: 0.08\n",
"episode: 120, score: 0.0, epsilon: 0.08\n",
"episode: 125, score: 1.0, epsilon: 0.08\n",
"episode: 130, score: 0.0, epsilon: 0.08\n",
"episode: 135, score: 1.0, epsilon: 0.08\n",
"episode: 140, score: 0.0, epsilon: 0.08\n",
"episode: 145, score: 0.0, epsilon: 0.08\n",
"episode: 150, score: 2.0, epsilon: 0.08\n",
"episode: 155, score: 0.0, epsilon: 0.08\n",
"episode: 160, score: 0.0, epsilon: 0.08\n",
"episode: 165, score: 1.0, epsilon: 0.08\n",
"episode: 170, score: 1.0, epsilon: 0.08\n",
"episode: 175, score: 0.0, epsilon: 0.08\n",
"episode: 180, score: 4.0, epsilon: 0.08\n",
"episode: 185, score: 0.0, epsilon: 0.08\n",
"episode: 190, score: 2.0, epsilon: 0.08\n",
"episode: 195, score: 0.0, epsilon: 0.08\n",
"marking, episode: 200, score: 0.0, mean_score: 0.94, std_score: 1.28\n",
"episode: 200, score: 0.0, epsilon: 0.08\n",
"episode: 205, score: 1.0, epsilon: 0.08\n",
"episode: 210, score: 3.0, epsilon: 0.08\n",
"episode: 215, score: 0.0, epsilon: 0.08\n",
"episode: 220, score: 3.0, epsilon: 0.08\n",
"episode: 225, score: 1.0, epsilon: 0.08\n",
"episode: 230, score: 3.0, epsilon: 0.08\n",
"episode: 235, score: 0.0, epsilon: 0.08\n",
"episode: 240, score: 0.0, epsilon: 0.08\n",
"episode: 245, score: 0.0, epsilon: 0.08\n",
"episode: 250, score: 0.0, epsilon: 0.08\n",
"episode: 255, score: 0.0, epsilon: 0.08\n",
"episode: 260, score: 0.0, epsilon: 0.08\n",
"episode: 265, score: 6.0, epsilon: 0.08\n",
"episode: 270, score: 5.0, epsilon: 0.08\n",
"episode: 275, score: 0.0, epsilon: 0.08\n",
"episode: 280, score: 0.0, epsilon: 0.08\n",
"episode: 285, score: 3.0, epsilon: 0.08\n",
"episode: 290, score: 1.0, epsilon: 0.08\n",
"episode: 295, score: 5.0, epsilon: 0.08\n",
"marking, episode: 300, score: 0.0, mean_score: 1.00, std_score: 1.53\n",
"episode: 300, score: 0.0, epsilon: 0.08\n",
"episode: 305, score: 0.0, epsilon: 0.08\n",
"episode: 310, score: 1.0, epsilon: 0.08\n",
"episode: 315, score: 6.0, epsilon: 0.08\n",
"episode: 320, score: 1.0, epsilon: 0.08\n",
"episode: 325, score: 1.0, epsilon: 0.08\n",
"episode: 330, score: 1.0, epsilon: 0.08\n",
"episode: 335, score: 1.0, epsilon: 0.08\n",
"episode: 340, score: 2.0, epsilon: 0.08\n",
"episode: 345, score: 1.0, epsilon: 0.08\n",
"episode: 350, score: 0.0, epsilon: 0.08\n",
"episode: 355, score: 0.0, epsilon: 0.08\n",
"episode: 360, score: 0.0, epsilon: 0.08\n",
"episode: 365, score: 0.0, epsilon: 0.08\n",
"episode: 370, score: 0.0, epsilon: 0.08\n",
"episode: 375, score: 0.0, epsilon: 0.08\n",
"episode: 380, score: 0.0, epsilon: 0.08\n",
"episode: 385, score: 1.0, epsilon: 0.08\n",
"episode: 390, score: 0.0, epsilon: 0.08\n",
"episode: 395, score: 0.0, epsilon: 0.08\n",
"marking, episode: 400, score: 0.0, mean_score: 0.99, std_score: 1.45\n",
"episode: 400, score: 0.0, epsilon: 0.08\n",
"episode: 405, score: 0.0, epsilon: 0.08\n",
"episode: 410, score: 0.0, epsilon: 0.08\n",
"episode: 415, score: 0.0, epsilon: 0.08\n",
"episode: 420, score: 0.0, epsilon: 0.08\n",
"episode: 425, score: 0.0, epsilon: 0.08\n",
"episode: 430, score: 0.0, epsilon: 0.08\n",
"episode: 435, score: 0.0, epsilon: 0.08\n",
"episode: 440, score: 1.0, epsilon: 0.08\n",
"episode: 445, score: 0.0, epsilon: 0.08\n",
"episode: 450, score: 2.0, epsilon: 0.08\n",
"episode: 455, score: 0.0, epsilon: 0.08\n",
"episode: 460, score: 0.0, epsilon: 0.08\n",
"episode: 465, score: 0.0, epsilon: 0.08\n",
"episode: 470, score: 1.0, epsilon: 0.08\n",
"episode: 475, score: 0.0, epsilon: 0.08\n",
"episode: 480, score: 1.0, epsilon: 0.08\n",
"episode: 485, score: 4.0, epsilon: 0.08\n",
"episode: 490, score: 0.0, epsilon: 0.08\n",
"episode: 495, score: 0.0, epsilon: 0.08\n",
"marking, episode: 500, score: 0.0, mean_score: 0.76, std_score: 1.18\n",
"episode: 500, score: 0.0, epsilon: 0.08\n",
"episode: 505, score: 0.0, epsilon: 0.08\n",
"episode: 510, score: 4.0, epsilon: 0.08\n",
"episode: 515, score: 1.0, epsilon: 0.08\n",
"episode: 520, score: 0.0, epsilon: 0.08\n",
"episode: 525, score: 2.0, epsilon: 0.08\n",
"episode: 530, score: 0.0, epsilon: 0.08\n",
"episode: 535, score: 4.0, epsilon: 0.08\n",
"episode: 540, score: 0.0, epsilon: 0.08\n",
"episode: 545, score: 1.0, epsilon: 0.08\n",
"episode: 550, score: 0.0, epsilon: 0.08\n",
"episode: 555, score: 0.0, epsilon: 0.08\n",
"episode: 560, score: 1.0, epsilon: 0.08\n",
"episode: 565, score: 6.0, epsilon: 0.08\n",
"episode: 570, score: 1.0, epsilon: 0.08\n",
"episode: 575, score: 0.0, epsilon: 0.08\n",
"episode: 580, score: 4.0, epsilon: 0.08\n",
"episode: 585, score: 0.0, epsilon: 0.08\n",
"episode: 590, score: 0.0, epsilon: 0.08\n",
"episode: 595, score: 0.0, epsilon: 0.08\n",
"marking, episode: 600, score: 1.0, mean_score: 0.84, std_score: 1.29\n",
"episode: 600, score: 1.0, epsilon: 0.08\n",
"episode: 605, score: 2.0, epsilon: 0.08\n",
"episode: 610, score: 3.0, epsilon: 0.08\n",
"episode: 615, score: 1.0, epsilon: 0.08\n",
"episode: 620, score: 0.0, epsilon: 0.08\n",
"episode: 625, score: 3.0, epsilon: 0.08\n",
"episode: 630, score: 5.0, epsilon: 0.08\n",
"episode: 635, score: 2.0, epsilon: 0.08\n",
"episode: 640, score: 0.0, epsilon: 0.08\n",
"episode: 645, score: 1.0, epsilon: 0.08\n",
"episode: 650, score: 1.0, epsilon: 0.08\n",
"episode: 655, score: 0.0, epsilon: 0.08\n",
"episode: 660, score: 0.0, epsilon: 0.08\n",
"episode: 665, score: 1.0, epsilon: 0.08\n",
"episode: 670, score: 0.0, epsilon: 0.08\n",
"episode: 675, score: 0.0, epsilon: 0.08\n",
"episode: 680, score: 0.0, epsilon: 0.08\n",
"episode: 685, score: 0.0, epsilon: 0.08\n",
"episode: 690, score: 0.0, epsilon: 0.08\n",
"episode: 695, score: 0.0, epsilon: 0.08\n",
"marking, episode: 700, score: 0.0, mean_score: 1.14, std_score: 1.48\n",
"episode: 700, score: 0.0, epsilon: 0.08\n",
"episode: 705, score: 0.0, epsilon: 0.08\n",
"episode: 710, score: 1.0, epsilon: 0.08\n",
"episode: 715, score: 1.0, epsilon: 0.08\n",
"episode: 720, score: 0.0, epsilon: 0.08\n",
"episode: 725, score: 1.0, epsilon: 0.08\n",
"episode: 730, score: 3.0, epsilon: 0.08\n",
"episode: 735, score: 6.0, epsilon: 0.08\n",
"episode: 740, score: 4.0, epsilon: 0.08\n",
"episode: 745, score: 1.0, epsilon: 0.08\n",
"episode: 750, score: 2.0, epsilon: 0.08\n",
"episode: 755, score: 2.0, epsilon: 0.08\n",
"episode: 760, score: 0.0, epsilon: 0.08\n",
"episode: 765, score: 1.0, epsilon: 0.08\n",
"episode: 770, score: 0.0, epsilon: 0.08\n",
"episode: 775, score: 0.0, epsilon: 0.08\n",
"episode: 780, score: 2.0, epsilon: 0.08\n",
"episode: 785, score: 0.0, epsilon: 0.08\n",
"episode: 790, score: 1.0, epsilon: 0.08\n",
"episode: 795, score: 0.0, epsilon: 0.08\n",
"marking, episode: 800, score: 1.0, mean_score: 1.02, std_score: 1.39\n",
"episode: 800, score: 1.0, epsilon: 0.08\n",
"episode: 805, score: 1.0, epsilon: 0.08\n",
"episode: 810, score: 0.0, epsilon: 0.08\n",
"episode: 815, score: 0.0, epsilon: 0.08\n",
"episode: 820, score: 1.0, epsilon: 0.08\n",
"episode: 825, score: 0.0, epsilon: 0.08\n",
"episode: 830, score: 1.0, epsilon: 0.08\n",
"episode: 835, score: 0.0, epsilon: 0.08\n",
"episode: 840, score: 0.0, epsilon: 0.08\n",
"episode: 845, score: 1.0, epsilon: 0.08\n",
"episode: 850, score: 0.0, epsilon: 0.08\n",
"episode: 855, score: 0.0, epsilon: 0.08\n",
"episode: 860, score: 1.0, epsilon: 0.08\n",
"episode: 865, score: 1.0, epsilon: 0.08\n",
"episode: 870, score: 0.0, epsilon: 0.08\n",
"episode: 875, score: 1.0, epsilon: 0.08\n",
"episode: 880, score: 0.0, epsilon: 0.08\n",
"episode: 885, score: 1.0, epsilon: 0.08\n",
"episode: 890, score: 0.0, epsilon: 0.08\n",
"episode: 895, score: 2.0, epsilon: 0.08\n",
"marking, episode: 900, score: 0.0, mean_score: 0.84, std_score: 1.32\n",
"episode: 900, score: 0.0, epsilon: 0.08\n",
"episode: 905, score: 3.0, epsilon: 0.08\n",
"episode: 910, score: 1.0, epsilon: 0.08\n",
"episode: 915, score: 0.0, epsilon: 0.08\n",
"episode: 920, score: 0.0, epsilon: 0.08\n",
"episode: 925, score: 0.0, epsilon: 0.08\n",
"episode: 930, score: 0.0, epsilon: 0.08\n",
"episode: 935, score: 0.0, epsilon: 0.08\n",
"episode: 940, score: 0.0, epsilon: 0.08\n",
"episode: 945, score: 1.0, epsilon: 0.08\n",
"episode: 950, score: 1.0, epsilon: 0.08\n",
"episode: 955, score: 0.0, epsilon: 0.08\n",
"episode: 960, score: 0.0, epsilon: 0.08\n",
"episode: 965, score: 0.0, epsilon: 0.08\n",
"episode: 970, score: 1.0, epsilon: 0.08\n",
"episode: 975, score: 3.0, epsilon: 0.08\n",
"episode: 980, score: 1.0, epsilon: 0.08\n",
"episode: 985, score: 2.0, epsilon: 0.08\n",
"episode: 990, score: 0.0, epsilon: 0.08\n",
"episode: 995, score: 3.0, epsilon: 0.08\n",
"marking, episode: 1000, score: 1.0, mean_score: 0.91, std_score: 1.32\n",
"episode: 1000, score: 1.0, epsilon: 0.08\n",
"episode: 1005, score: 2.0, epsilon: 0.08\n",
"episode: 1010, score: 3.0, epsilon: 0.08\n",
"episode: 1015, score: 3.0, epsilon: 0.08\n",
"episode: 1020, score: 3.0, epsilon: 0.08\n",
"episode: 1025, score: 1.0, epsilon: 0.08\n",
"episode: 1030, score: 2.0, epsilon: 0.08\n",
"episode: 1035, score: 0.0, epsilon: 0.08\n",
"episode: 1040, score: 1.0, epsilon: 0.08\n",
"episode: 1045, score: 0.0, epsilon: 0.08\n",
"episode: 1050, score: 0.0, epsilon: 0.08\n",
"episode: 1055, score: 2.0, epsilon: 0.08\n",
"episode: 1060, score: 0.0, epsilon: 0.08\n",
"episode: 1065, score: 2.0, epsilon: 0.08\n",
"episode: 1070, score: 1.0, epsilon: 0.08\n",
"episode: 1075, score: 2.0, epsilon: 0.08\n",
"episode: 1080, score: 0.0, epsilon: 0.08\n",
"episode: 1085, score: 0.0, epsilon: 0.08\n",
"episode: 1090, score: 6.0, epsilon: 0.08\n",
"episode: 1095, score: 2.0, epsilon: 0.08\n",
"marking, episode: 1100, score: 0.0, mean_score: 1.15, std_score: 1.32\n",
"episode: 1100, score: 0.0, epsilon: 0.08\n",
"episode: 1105, score: 0.0, epsilon: 0.08\n",
"episode: 1110, score: 0.0, epsilon: 0.08\n",
"episode: 1115, score: 5.0, epsilon: 0.08\n",
"episode: 1120, score: 3.0, epsilon: 0.08\n",
"episode: 1125, score: 0.0, epsilon: 0.08\n",
"episode: 1130, score: 0.0, epsilon: 0.08\n",
"episode: 1135, score: 0.0, epsilon: 0.08\n",
"episode: 1140, score: 3.0, epsilon: 0.08\n",
"episode: 1145, score: 0.0, epsilon: 0.08\n",
"episode: 1150, score: 2.0, epsilon: 0.08\n",
"episode: 1155, score: 1.0, epsilon: 0.08\n",
"episode: 1160, score: 2.0, epsilon: 0.08\n",
"episode: 1165, score: 4.0, epsilon: 0.08\n",
"episode: 1170, score: 0.0, epsilon: 0.08\n",
"episode: 1175, score: 1.0, epsilon: 0.08\n",
"episode: 1180, score: 1.0, epsilon: 0.08\n",
"episode: 1185, score: 0.0, epsilon: 0.08\n",
"episode: 1190, score: 0.0, epsilon: 0.08\n",
"episode: 1195, score: 3.0, epsilon: 0.08\n",
"marking, episode: 1200, score: 1.0, mean_score: 0.86, std_score: 1.10\n",
"episode: 1200, score: 1.0, epsilon: 0.08\n",
"episode: 1205, score: 0.0, epsilon: 0.08\n",
"episode: 1210, score: 0.0, epsilon: 0.08\n",
"episode: 1215, score: 2.0, epsilon: 0.08\n",
"episode: 1220, score: 0.0, epsilon: 0.08\n",
"episode: 1225, score: 0.0, epsilon: 0.08\n",
"episode: 1230, score: 0.0, epsilon: 0.08\n",
"episode: 1235, score: 0.0, epsilon: 0.08\n",
"episode: 1240, score: 0.0, epsilon: 0.08\n",
"episode: 1245, score: 0.0, epsilon: 0.08\n",
"episode: 1250, score: 2.0, epsilon: 0.08\n",
"episode: 1255, score: 1.0, epsilon: 0.08\n",
"episode: 1260, score: 0.0, epsilon: 0.08\n",
"episode: 1265, score: 2.0, epsilon: 0.08\n",
"episode: 1270, score: 2.0, epsilon: 0.08\n",
"episode: 1275, score: 2.0, epsilon: 0.08\n",
"episode: 1280, score: 1.0, epsilon: 0.08\n",
"episode: 1285, score: 1.0, epsilon: 0.08\n",
"episode: 1290, score: 0.0, epsilon: 0.08\n",
"episode: 1295, score: 1.0, epsilon: 0.08\n",
"marking, episode: 1300, score: 2.0, mean_score: 1.01, std_score: 1.37\n",
"episode: 1300, score: 2.0, epsilon: 0.08\n",
"episode: 1305, score: 0.0, epsilon: 0.08\n",
"episode: 1310, score: 1.0, epsilon: 0.08\n",
"episode: 1315, score: 0.0, epsilon: 0.08\n",
"episode: 1320, score: 1.0, epsilon: 0.08\n",
"episode: 1325, score: 3.0, epsilon: 0.08\n",
"episode: 1330, score: 0.0, epsilon: 0.08\n",
"episode: 1335, score: 2.0, epsilon: 0.08\n",
"episode: 1340, score: 0.0, epsilon: 0.08\n",
"episode: 1345, score: 0.0, epsilon: 0.08\n",
"episode: 1350, score: 0.0, epsilon: 0.08\n",
"episode: 1355, score: 1.0, epsilon: 0.08\n",
"episode: 1360, score: 1.0, epsilon: 0.08\n",
"episode: 1365, score: 1.0, epsilon: 0.08\n",
"episode: 1370, score: 0.0, epsilon: 0.08\n",
"episode: 1375, score: 2.0, epsilon: 0.08\n",
"episode: 1380, score: 0.0, epsilon: 0.08\n",
"episode: 1385, score: 0.0, epsilon: 0.08\n",
"episode: 1390, score: 1.0, epsilon: 0.08\n",
"episode: 1395, score: 0.0, epsilon: 0.08\n",
"marking, episode: 1400, score: 1.0, mean_score: 0.98, std_score: 1.37\n",
"episode: 1400, score: 1.0, epsilon: 0.08\n",
"episode: 1405, score: 2.0, epsilon: 0.08\n",
"episode: 1410, score: 1.0, epsilon: 0.08\n",
"episode: 1415, score: 0.0, epsilon: 0.08\n",
"episode: 1420, score: 2.0, epsilon: 0.08\n",
"episode: 1425, score: 0.0, epsilon: 0.08\n",
"episode: 1430, score: 0.0, epsilon: 0.08\n",
"episode: 1435, score: 0.0, epsilon: 0.08\n",
"episode: 1440, score: 0.0, epsilon: 0.08\n",
"episode: 1445, score: 1.0, epsilon: 0.08\n",
"episode: 1450, score: 2.0, epsilon: 0.08\n",
"episode: 1455, score: 1.0, epsilon: 0.08\n",
"episode: 1460, score: 1.0, epsilon: 0.08\n",
"episode: 1465, score: 1.0, epsilon: 0.08\n",
"episode: 1470, score: 1.0, epsilon: 0.08\n",
"episode: 1475, score: 0.0, epsilon: 0.08\n",
"episode: 1480, score: 0.0, epsilon: 0.08\n",
"episode: 1485, score: 1.0, epsilon: 0.08\n",
"episode: 1490, score: 0.0, epsilon: 0.08\n",
"episode: 1495, score: 0.0, epsilon: 0.08\n",
"marking, episode: 1500, score: 3.0, mean_score: 0.91, std_score: 1.07\n",
"episode: 1500, score: 3.0, epsilon: 0.08\n",
"episode: 1505, score: 0.0, epsilon: 0.08\n",
"episode: 1510, score: 0.0, epsilon: 0.08\n",
"episode: 1515, score: 0.0, epsilon: 0.08\n",
"episode: 1520, score: 0.0, epsilon: 0.08\n",
"episode: 1525, score: 1.0, epsilon: 0.08\n",
"episode: 1530, score: 0.0, epsilon: 0.08\n",
"episode: 1535, score: 2.0, epsilon: 0.08\n",
"episode: 1540, score: 1.0, epsilon: 0.08\n",
"episode: 1545, score: 5.0, epsilon: 0.08\n",
"episode: 1550, score: 2.0, epsilon: 0.08\n",
"episode: 1555, score: 1.0, epsilon: 0.08\n",
"episode: 1560, score: 0.0, epsilon: 0.08\n",
"episode: 1565, score: 0.0, epsilon: 0.08\n",
"episode: 1570, score: 1.0, epsilon: 0.08\n",
"episode: 1575, score: 5.0, epsilon: 0.08\n",
"episode: 1580, score: 0.0, epsilon: 0.08\n",
"episode: 1585, score: 0.0, epsilon: 0.08\n",
"episode: 1590, score: 0.0, epsilon: 0.08\n",
"episode: 1595, score: 0.0, epsilon: 0.08\n",
"marking, episode: 1600, score: 0.0, mean_score: 0.95, std_score: 1.40\n",
"episode: 1600, score: 0.0, epsilon: 0.08\n",
"episode: 1605, score: 0.0, epsilon: 0.08\n",
"episode: 1610, score: 0.0, epsilon: 0.08\n",
"episode: 1615, score: 0.0, epsilon: 0.08\n",
"episode: 1620, score: 5.0, epsilon: 0.08\n",
"episode: 1625, score: 0.0, epsilon: 0.08\n",
"episode: 1630, score: 0.0, epsilon: 0.08\n",
"episode: 1635, score: 0.0, epsilon: 0.08\n",
"episode: 1640, score: 0.0, epsilon: 0.08\n",
"episode: 1645, score: 1.0, epsilon: 0.08\n",
"episode: 1650, score: 0.0, epsilon: 0.08\n",
"episode: 1655, score: 0.0, epsilon: 0.08\n",
"episode: 1660, score: 3.0, epsilon: 0.08\n",
"episode: 1665, score: 0.0, epsilon: 0.08\n",
"episode: 1670, score: 0.0, epsilon: 0.08\n",
"episode: 1675, score: 1.0, epsilon: 0.08\n",
"episode: 1680, score: 1.0, epsilon: 0.08\n",
"episode: 1685, score: 2.0, epsilon: 0.08\n",
"episode: 1690, score: 1.0, epsilon: 0.08\n",
"episode: 1695, score: 1.0, epsilon: 0.08\n",
"marking, episode: 1700, score: 3.0, mean_score: 1.04, std_score: 1.33\n",
"episode: 1700, score: 3.0, epsilon: 0.08\n",
"episode: 1705, score: 0.0, epsilon: 0.08\n",
"episode: 1710, score: 0.0, epsilon: 0.08\n",
"episode: 1715, score: 0.0, epsilon: 0.08\n",
"episode: 1720, score: 1.0, epsilon: 0.08\n",
"episode: 1725, score: 0.0, epsilon: 0.08\n",
"episode: 1730, score: 3.0, epsilon: 0.08\n",
"episode: 1735, score: 3.0, epsilon: 0.08\n",
"episode: 1740, score: 0.0, epsilon: 0.08\n",
"episode: 1745, score: 0.0, epsilon: 0.08\n",
"episode: 1750, score: 0.0, epsilon: 0.08\n",
"episode: 1755, score: 3.0, epsilon: 0.08\n",
"episode: 1760, score: 5.0, epsilon: 0.08\n",
"episode: 1765, score: 2.0, epsilon: 0.08\n",
"episode: 1770, score: 0.0, epsilon: 0.08\n",
"episode: 1775, score: 5.0, epsilon: 0.08\n",
"episode: 1780, score: 0.0, epsilon: 0.08\n",
"episode: 1785, score: 1.0, epsilon: 0.08\n",
"episode: 1790, score: 3.0, epsilon: 0.08\n",
"episode: 1795, score: 0.0, epsilon: 0.08\n",
"marking, episode: 1800, score: 0.0, mean_score: 0.93, std_score: 1.32\n",
"episode: 1800, score: 0.0, epsilon: 0.08\n",
"episode: 1805, score: 0.0, epsilon: 0.08\n",
"episode: 1810, score: 0.0, epsilon: 0.08\n",
"episode: 1815, score: 3.0, epsilon: 0.08\n",
"episode: 1820, score: 2.0, epsilon: 0.08\n",
"episode: 1825, score: 0.0, epsilon: 0.08\n",
"episode: 1830, score: 0.0, epsilon: 0.08\n",
"episode: 1835, score: 4.0, epsilon: 0.08\n",
"episode: 1840, score: 1.0, epsilon: 0.08\n",
"episode: 1845, score: 0.0, epsilon: 0.08\n",
"episode: 1850, score: 0.0, epsilon: 0.08\n",
"episode: 1855, score: 1.0, epsilon: 0.08\n",
"episode: 1860, score: 1.0, epsilon: 0.08\n",
"episode: 1865, score: 3.0, epsilon: 0.08\n",
"episode: 1870, score: 0.0, epsilon: 0.08\n",
"episode: 1875, score: 0.0, epsilon: 0.08\n",
"episode: 1880, score: 0.0, epsilon: 0.08\n",
"episode: 1885, score: 0.0, epsilon: 0.08\n",
"episode: 1890, score: 1.0, epsilon: 0.08\n",
"episode: 1895, score: 0.0, epsilon: 0.08\n",
"marking, episode: 1900, score: 1.0, mean_score: 0.91, std_score: 1.19\n",
"episode: 1900, score: 1.0, epsilon: 0.08\n",
"episode: 1905, score: 1.0, epsilon: 0.08\n",
"episode: 1910, score: 0.0, epsilon: 0.08\n",
"episode: 1915, score: 0.0, epsilon: 0.08\n",
"episode: 1920, score: 1.0, epsilon: 0.08\n",
"episode: 1925, score: 0.0, epsilon: 0.08\n",
"episode: 1930, score: 1.0, epsilon: 0.08\n",
"episode: 1935, score: 1.0, epsilon: 0.08\n",
"episode: 1940, score: 0.0, epsilon: 0.08\n",
"episode: 1945, score: 0.0, epsilon: 0.08\n",
"episode: 1950, score: 0.0, epsilon: 0.08\n",
"episode: 1955, score: 2.0, epsilon: 0.08\n",
"episode: 1960, score: 0.0, epsilon: 0.08\n",
"episode: 1965, score: 1.0, epsilon: 0.08\n",
"episode: 1970, score: 3.0, epsilon: 0.08\n",
"episode: 1975, score: 1.0, epsilon: 0.08\n",
"episode: 1980, score: 0.0, epsilon: 0.08\n",
"episode: 1985, score: 1.0, epsilon: 0.08\n",
"episode: 1990, score: 3.0, epsilon: 0.08\n",
"episode: 1995, score: 0.0, epsilon: 0.08\n",
"marking, episode: 2000, score: 6.0, mean_score: 1.12, std_score: 1.52\n",
"episode: 2000, score: 6.0, epsilon: 0.08\n",
"episode: 2005, score: 0.0, epsilon: 0.08\n",
"episode: 2010, score: 0.0, epsilon: 0.08\n",
"episode: 2015, score: 0.0, epsilon: 0.08\n",
"episode: 2020, score: 0.0, epsilon: 0.08\n",
"episode: 2025, score: 0.0, epsilon: 0.08\n",
"episode: 2030, score: 0.0, epsilon: 0.08\n",
"episode: 2035, score: 0.0, epsilon: 0.08\n",
"episode: 2040, score: 0.0, epsilon: 0.08\n",
"episode: 2045, score: 1.0, epsilon: 0.08\n",
"episode: 2050, score: 0.0, epsilon: 0.08\n",
"episode: 2055, score: 0.0, epsilon: 0.08\n",
"episode: 2060, score: 1.0, epsilon: 0.08\n",
"episode: 2065, score: 5.0, epsilon: 0.08\n",
"episode: 2070, score: 3.0, epsilon: 0.08\n",
"episode: 2075, score: 2.0, epsilon: 0.08\n",
"episode: 2080, score: 1.0, epsilon: 0.08\n",
"episode: 2085, score: 0.0, epsilon: 0.08\n",
"episode: 2090, score: 1.0, epsilon: 0.08\n",
"episode: 2095, score: 1.0, epsilon: 0.08\n",
"marking, episode: 2100, score: 1.0, mean_score: 0.80, std_score: 1.17\n",
"episode: 2100, score: 1.0, epsilon: 0.08\n",
"episode: 2105, score: 4.0, epsilon: 0.08\n",
"episode: 2110, score: 1.0, epsilon: 0.08\n",
"episode: 2115, score: 0.0, epsilon: 0.08\n",
"episode: 2120, score: 2.0, epsilon: 0.08\n",
"episode: 2125, score: 0.0, epsilon: 0.08\n",
"episode: 2130, score: 0.0, epsilon: 0.08\n",
"episode: 2135, score: 0.0, epsilon: 0.08\n",
"episode: 2140, score: 0.0, epsilon: 0.08\n",
"episode: 2145, score: 1.0, epsilon: 0.08\n",
"episode: 2150, score: 3.0, epsilon: 0.08\n",
"episode: 2155, score: 1.0, epsilon: 0.08\n",
"episode: 2160, score: 1.0, epsilon: 0.08\n",
"episode: 2165, score: 0.0, epsilon: 0.08\n",
"episode: 2170, score: 0.0, epsilon: 0.08\n",
"episode: 2175, score: 0.0, epsilon: 0.08\n",
"episode: 2180, score: 0.0, epsilon: 0.08\n",
"episode: 2185, score: 3.0, epsilon: 0.08\n",
"episode: 2190, score: 0.0, epsilon: 0.08\n",
"episode: 2195, score: 2.0, epsilon: 0.08\n",
"marking, episode: 2200, score: 0.0, mean_score: 1.01, std_score: 1.40\n",
"episode: 2200, score: 0.0, epsilon: 0.08\n",
"episode: 2205, score: 0.0, epsilon: 0.08\n",
"episode: 2210, score: 0.0, epsilon: 0.08\n",
"episode: 2215, score: 0.0, epsilon: 0.08\n",
"episode: 2220, score: 2.0, epsilon: 0.08\n",
"episode: 2225, score: 2.0, epsilon: 0.08\n",
"episode: 2230, score: 1.0, epsilon: 0.08\n",
"episode: 2235, score: 0.0, epsilon: 0.08\n",
"episode: 2240, score: 1.0, epsilon: 0.08\n",
"episode: 2245, score: 1.0, epsilon: 0.08\n",
"episode: 2250, score: 3.0, epsilon: 0.08\n",
"episode: 2255, score: 0.0, epsilon: 0.08\n",
"episode: 2260, score: 2.0, epsilon: 0.08\n",
"episode: 2265, score: 0.0, epsilon: 0.08\n",
"episode: 2270, score: 0.0, epsilon: 0.08\n",
"episode: 2275, score: 1.0, epsilon: 0.08\n",
"episode: 2280, score: 0.0, epsilon: 0.08\n",
"episode: 2285, score: 0.0, epsilon: 0.08\n",
"episode: 2290, score: 0.0, epsilon: 0.08\n",
"episode: 2295, score: 2.0, epsilon: 0.08\n",
"marking, episode: 2300, score: 0.0, mean_score: 0.93, std_score: 1.25\n",
"episode: 2300, score: 0.0, epsilon: 0.08\n",
"episode: 2305, score: 2.0, epsilon: 0.08\n",
"episode: 2310, score: 3.0, epsilon: 0.08\n",
"episode: 2315, score: 0.0, epsilon: 0.08\n",
"episode: 2320, score: 0.0, epsilon: 0.08\n",
"episode: 2325, score: 1.0, epsilon: 0.08\n",
"episode: 2330, score: 0.0, epsilon: 0.08\n",
"episode: 2335, score: 3.0, epsilon: 0.08\n",
"episode: 2340, score: 1.0, epsilon: 0.08\n",
"episode: 2345, score: 2.0, epsilon: 0.08\n",
"episode: 2350, score: 1.0, epsilon: 0.08\n",
"episode: 2355, score: 0.0, epsilon: 0.08\n",
"episode: 2360, score: 0.0, epsilon: 0.08\n",
"episode: 2365, score: 2.0, epsilon: 0.08\n",
"episode: 2370, score: 0.0, epsilon: 0.08\n",
"episode: 2375, score: 0.0, epsilon: 0.08\n",
"episode: 2380, score: 0.0, epsilon: 0.08\n",
"episode: 2385, score: 0.0, epsilon: 0.08\n",
"episode: 2390, score: 1.0, epsilon: 0.08\n",
"episode: 2395, score: 0.0, epsilon: 0.08\n",
"marking, episode: 2400, score: 0.0, mean_score: 0.85, std_score: 1.24\n",
"episode: 2400, score: 0.0, epsilon: 0.08\n",
"episode: 2405, score: 0.0, epsilon: 0.08\n",
"episode: 2410, score: 1.0, epsilon: 0.08\n",
"episode: 2415, score: 3.0, epsilon: 0.08\n",
"episode: 2420, score: 0.0, epsilon: 0.08\n",
"episode: 2425, score: 0.0, epsilon: 0.08\n",
"episode: 2430, score: 0.0, epsilon: 0.08\n",
"episode: 2435, score: 2.0, epsilon: 0.08\n",
"episode: 2440, score: 1.0, epsilon: 0.08\n",
"episode: 2445, score: 3.0, epsilon: 0.08\n",
"episode: 2450, score: 0.0, epsilon: 0.08\n",
"episode: 2455, score: 0.0, epsilon: 0.08\n",
"episode: 2460, score: 1.0, epsilon: 0.08\n",
"episode: 2465, score: 0.0, epsilon: 0.08\n",
"episode: 2470, score: 2.0, epsilon: 0.08\n",
"episode: 2475, score: 4.0, epsilon: 0.08\n",
"episode: 2480, score: 0.0, epsilon: 0.08\n",
"episode: 2485, score: 0.0, epsilon: 0.08\n",
"episode: 2490, score: 3.0, epsilon: 0.08\n",
"episode: 2495, score: 1.0, epsilon: 0.08\n",
"marking, episode: 2500, score: 0.0, mean_score: 0.94, std_score: 1.20\n",
"episode: 2500, score: 0.0, epsilon: 0.08\n",
"episode: 2505, score: 0.0, epsilon: 0.08\n",
"episode: 2510, score: 0.0, epsilon: 0.08\n",
"episode: 2515, score: 1.0, epsilon: 0.08\n",
"episode: 2520, score: 1.0, epsilon: 0.08\n",
"episode: 2525, score: 1.0, epsilon: 0.08\n",
"episode: 2530, score: 2.0, epsilon: 0.08\n",
"episode: 2535, score: 0.0, epsilon: 0.08\n",
"episode: 2540, score: 0.0, epsilon: 0.08\n",
"episode: 2545, score: 4.0, epsilon: 0.08\n",
"episode: 2550, score: 0.0, epsilon: 0.08\n",
"episode: 2555, score: 1.0, epsilon: 0.08\n",
"episode: 2560, score: 3.0, epsilon: 0.08\n",
"episode: 2565, score: 0.0, epsilon: 0.08\n",
"episode: 2570, score: 0.0, epsilon: 0.08\n",
"episode: 2575, score: 4.0, epsilon: 0.08\n",
"episode: 2580, score: 4.0, epsilon: 0.08\n",
"episode: 2585, score: 3.0, epsilon: 0.08\n",
"episode: 2590, score: 0.0, epsilon: 0.08\n",
"episode: 2595, score: 1.0, epsilon: 0.08\n",
"marking, episode: 2600, score: 1.0, mean_score: 1.18, std_score: 1.58\n",
"episode: 2600, score: 1.0, epsilon: 0.08\n",
"episode: 2605, score: 1.0, epsilon: 0.08\n",
"episode: 2610, score: 0.0, epsilon: 0.08\n",
"episode: 2615, score: 1.0, epsilon: 0.08\n",
"episode: 2620, score: 1.0, epsilon: 0.08\n",
"episode: 2625, score: 0.0, epsilon: 0.08\n",
"episode: 2630, score: 0.0, epsilon: 0.08\n",
"episode: 2635, score: 1.0, epsilon: 0.08\n",
"episode: 2640, score: 0.0, epsilon: 0.08\n",
"episode: 2645, score: 0.0, epsilon: 0.08\n",
"episode: 2650, score: 1.0, epsilon: 0.08\n",
"episode: 2655, score: 3.0, epsilon: 0.08\n",
"episode: 2660, score: 0.0, epsilon: 0.08\n",
"episode: 2665, score: 3.0, epsilon: 0.08\n",
"episode: 2670, score: 1.0, epsilon: 0.08\n",
"episode: 2675, score: 1.0, epsilon: 0.08\n",
"episode: 2680, score: 0.0, epsilon: 0.08\n",
"episode: 2685, score: 0.0, epsilon: 0.08\n",
"episode: 2690, score: 0.0, epsilon: 0.08\n",
"episode: 2695, score: 0.0, epsilon: 0.08\n",
"marking, episode: 2700, score: 0.0, mean_score: 0.94, std_score: 1.31\n",
"episode: 2700, score: 0.0, epsilon: 0.08\n",
"episode: 2705, score: 1.0, epsilon: 0.08\n",
"episode: 2710, score: 0.0, epsilon: 0.08\n",
"episode: 2715, score: 1.0, epsilon: 0.08\n",
"episode: 2720, score: 0.0, epsilon: 0.08\n",
"episode: 2725, score: 0.0, epsilon: 0.08\n",
"episode: 2730, score: 0.0, epsilon: 0.08\n",
"episode: 2735, score: 1.0, epsilon: 0.08\n",
"episode: 2740, score: 0.0, epsilon: 0.08\n",
"episode: 2745, score: 0.0, epsilon: 0.08\n",
"episode: 2750, score: 0.0, epsilon: 0.08\n",
"episode: 2755, score: 0.0, epsilon: 0.08\n",
"episode: 2760, score: 0.0, epsilon: 0.08\n",
"episode: 2765, score: 0.0, epsilon: 0.08\n",
"episode: 2770, score: 0.0, epsilon: 0.08\n",
"episode: 2775, score: 3.0, epsilon: 0.08\n",
"episode: 2780, score: 0.0, epsilon: 0.08\n",
"episode: 2785, score: 0.0, epsilon: 0.08\n",
"episode: 2790, score: 0.0, epsilon: 0.08\n",
"episode: 2795, score: 0.0, epsilon: 0.08\n",
"marking, episode: 2800, score: 1.0, mean_score: 0.76, std_score: 1.18\n",
"episode: 2800, score: 1.0, epsilon: 0.08\n",
"episode: 2805, score: 3.0, epsilon: 0.08\n",
"episode: 2810, score: 0.0, epsilon: 0.08\n",
"episode: 2815, score: 2.0, epsilon: 0.08\n",
"episode: 2820, score: 1.0, epsilon: 0.08\n",
"episode: 2825, score: 0.0, epsilon: 0.08\n",
"episode: 2830, score: 0.0, epsilon: 0.08\n",
"episode: 2835, score: 0.0, epsilon: 0.08\n",
"episode: 2840, score: 1.0, epsilon: 0.08\n",
"episode: 2845, score: 0.0, epsilon: 0.08\n",
"episode: 2850, score: 0.0, epsilon: 0.08\n",
"episode: 2855, score: 2.0, epsilon: 0.08\n",
"episode: 2860, score: 0.0, epsilon: 0.08\n",
"episode: 2865, score: 1.0, epsilon: 0.08\n",
"episode: 2870, score: 2.0, epsilon: 0.08\n",
"episode: 2875, score: 2.0, epsilon: 0.08\n",
"episode: 2880, score: 0.0, epsilon: 0.08\n",
"episode: 2885, score: 2.0, epsilon: 0.08\n",
"episode: 2890, score: 4.0, epsilon: 0.08\n",
"episode: 2895, score: 4.0, epsilon: 0.08\n",
"marking, episode: 2900, score: 1.0, mean_score: 0.89, std_score: 1.22\n",
"episode: 2900, score: 1.0, epsilon: 0.08\n",
"episode: 2905, score: 0.0, epsilon: 0.08\n",
"episode: 2910, score: 4.0, epsilon: 0.08\n",
"episode: 2915, score: 1.0, epsilon: 0.08\n",
"episode: 2920, score: 5.0, epsilon: 0.08\n",
"episode: 2925, score: 0.0, epsilon: 0.08\n",
"episode: 2930, score: 0.0, epsilon: 0.08\n",
"episode: 2935, score: 0.0, epsilon: 0.08\n",
"episode: 2940, score: 0.0, epsilon: 0.08\n",
"episode: 2945, score: 3.0, epsilon: 0.08\n",
"episode: 2950, score: 0.0, epsilon: 0.08\n",
"episode: 2955, score: 0.0, epsilon: 0.08\n",
"episode: 2960, score: 2.0, epsilon: 0.08\n",
"episode: 2965, score: 0.0, epsilon: 0.08\n",
"episode: 2970, score: 0.0, epsilon: 0.08\n",
"episode: 2975, score: 2.0, epsilon: 0.08\n",
"episode: 2980, score: 1.0, epsilon: 0.08\n",
"episode: 2985, score: 0.0, epsilon: 0.08\n",
"episode: 2990, score: 0.0, epsilon: 0.08\n",
"episode: 2995, score: 0.0, epsilon: 0.08\n",
"marking, episode: 3000, score: 0.0, mean_score: 0.94, std_score: 1.50\n",
"episode: 3000, score: 0.0, epsilon: 0.08\n",
"episode: 3005, score: 0.0, epsilon: 0.08\n",
"episode: 3010, score: 0.0, epsilon: 0.08\n",
"episode: 3015, score: 4.0, epsilon: 0.08\n",
"episode: 3020, score: 0.0, epsilon: 0.08\n",
"episode: 3025, score: 1.0, epsilon: 0.08\n",
"episode: 3030, score: 0.0, epsilon: 0.08\n",
"episode: 3035, score: 1.0, epsilon: 0.08\n",
"episode: 3040, score: 0.0, epsilon: 0.08\n",
"episode: 3045, score: 0.0, epsilon: 0.08\n",
"episode: 3050, score: 0.0, epsilon: 0.08\n",
"episode: 3055, score: 0.0, epsilon: 0.08\n",
"episode: 3060, score: 1.0, epsilon: 0.08\n",
"episode: 3065, score: 1.0, epsilon: 0.08\n",
"episode: 3070, score: 2.0, epsilon: 0.08\n",
"episode: 3075, score: 0.0, epsilon: 0.08\n",
"episode: 3080, score: 0.0, epsilon: 0.08\n",
"episode: 3085, score: 0.0, epsilon: 0.08\n",
"episode: 3090, score: 2.0, epsilon: 0.08\n",
"episode: 3095, score: 1.0, epsilon: 0.08\n",
"marking, episode: 3100, score: 0.0, mean_score: 1.19, std_score: 1.59\n",
"episode: 3100, score: 0.0, epsilon: 0.08\n",
"episode: 3105, score: 0.0, epsilon: 0.08\n",
"episode: 3110, score: 1.0, epsilon: 0.08\n",
"episode: 3115, score: 0.0, epsilon: 0.08\n",
"episode: 3120, score: 2.0, epsilon: 0.08\n",
"episode: 3125, score: 4.0, epsilon: 0.08\n",
"episode: 3130, score: 0.0, epsilon: 0.08\n",
"episode: 3135, score: 2.0, epsilon: 0.08\n",
"episode: 3140, score: 0.0, epsilon: 0.08\n",
"episode: 3145, score: 0.0, epsilon: 0.08\n",
"episode: 3150, score: 1.0, epsilon: 0.08\n",
"episode: 3155, score: 2.0, epsilon: 0.08\n",
"episode: 3160, score: 4.0, epsilon: 0.08\n",
"episode: 3165, score: 0.0, epsilon: 0.08\n",
"episode: 3170, score: 0.0, epsilon: 0.08\n",
"episode: 3175, score: 1.0, epsilon: 0.08\n",
"episode: 3180, score: 0.0, epsilon: 0.08\n",
"episode: 3185, score: 0.0, epsilon: 0.08\n",
"episode: 3190, score: 1.0, epsilon: 0.08\n",
"episode: 3195, score: 3.0, epsilon: 0.08\n",
"marking, episode: 3200, score: 0.0, mean_score: 0.72, std_score: 1.18\n",
"episode: 3200, score: 0.0, epsilon: 0.08\n",
"episode: 3205, score: 1.0, epsilon: 0.08\n",
"episode: 3210, score: 0.0, epsilon: 0.08\n",
"episode: 3215, score: 0.0, epsilon: 0.08\n",
"episode: 3220, score: 1.0, epsilon: 0.08\n",
"episode: 3225, score: 2.0, epsilon: 0.08\n",
"episode: 3230, score: 1.0, epsilon: 0.08\n",
"episode: 3235, score: 1.0, epsilon: 0.08\n",
"episode: 3240, score: 2.0, epsilon: 0.08\n",
"episode: 3245, score: 1.0, epsilon: 0.08\n",
"episode: 3250, score: 2.0, epsilon: 0.08\n",
"episode: 3255, score: 6.0, epsilon: 0.08\n",
"episode: 3260, score: 0.0, epsilon: 0.08\n",
"episode: 3265, score: 3.0, epsilon: 0.08\n",
"episode: 3270, score: 1.0, epsilon: 0.08\n",
"episode: 3275, score: 0.0, epsilon: 0.08\n",
"episode: 3280, score: 1.0, epsilon: 0.08\n",
"episode: 3285, score: 5.0, epsilon: 0.08\n",
"episode: 3290, score: 1.0, epsilon: 0.08\n",
"episode: 3295, score: 0.0, epsilon: 0.08\n",
"marking, episode: 3300, score: 1.0, mean_score: 1.03, std_score: 1.31\n",
"episode: 3300, score: 1.0, epsilon: 0.08\n",
"episode: 3305, score: 0.0, epsilon: 0.08\n",
"episode: 3310, score: 1.0, epsilon: 0.08\n",
"episode: 3315, score: 0.0, epsilon: 0.08\n",
"episode: 3320, score: 0.0, epsilon: 0.08\n",
"episode: 3325, score: 0.0, epsilon: 0.08\n",
"episode: 3330, score: 3.0, epsilon: 0.08\n",
"episode: 3335, score: 1.0, epsilon: 0.08\n",
"episode: 3340, score: 1.0, epsilon: 0.08\n",
"episode: 3345, score: 2.0, epsilon: 0.08\n",
"episode: 3350, score: 1.0, epsilon: 0.08\n",
"episode: 3355, score: 1.0, epsilon: 0.08\n",
"episode: 3360, score: 0.0, epsilon: 0.08\n",
"episode: 3365, score: 0.0, epsilon: 0.08\n",
"episode: 3370, score: 0.0, epsilon: 0.08\n",
"episode: 3375, score: 0.0, epsilon: 0.08\n",
"episode: 3380, score: 1.0, epsilon: 0.08\n",
"episode: 3385, score: 0.0, epsilon: 0.08\n",
"episode: 3390, score: 0.0, epsilon: 0.08\n",
"episode: 3395, score: 3.0, epsilon: 0.08\n",
"marking, episode: 3400, score: 1.0, mean_score: 0.85, std_score: 1.29\n",
"episode: 3400, score: 1.0, epsilon: 0.08\n",
"episode: 3405, score: 3.0, epsilon: 0.08\n",
"episode: 3410, score: 0.0, epsilon: 0.08\n",
"episode: 3415, score: 1.0, epsilon: 0.08\n",
"episode: 3420, score: 6.0, epsilon: 0.08\n",
"episode: 3425, score: 1.0, epsilon: 0.08\n",
"episode: 3430, score: 3.0, epsilon: 0.08\n",
"episode: 3435, score: 1.0, epsilon: 0.08\n",
"episode: 3440, score: 0.0, epsilon: 0.08\n",
"episode: 3445, score: 0.0, epsilon: 0.08\n",
"episode: 3450, score: 0.0, epsilon: 0.08\n",
"episode: 3455, score: 1.0, epsilon: 0.08\n",
"episode: 3460, score: 1.0, epsilon: 0.08\n",
"episode: 3465, score: 1.0, epsilon: 0.08\n",
"episode: 3470, score: 1.0, epsilon: 0.08\n",
"episode: 3475, score: 0.0, epsilon: 0.08\n",
"episode: 3480, score: 1.0, epsilon: 0.08\n",
"episode: 3485, score: 2.0, epsilon: 0.08\n",
"episode: 3490, score: 0.0, epsilon: 0.08\n",
"episode: 3495, score: 0.0, epsilon: 0.08\n",
"marking, episode: 3500, score: 3.0, mean_score: 1.07, std_score: 1.41\n",
"episode: 3500, score: 3.0, epsilon: 0.08\n",
"episode: 3505, score: 0.0, epsilon: 0.08\n",
"episode: 3510, score: 3.0, epsilon: 0.08\n",
"episode: 3515, score: 1.0, epsilon: 0.08\n",
"episode: 3520, score: 3.0, epsilon: 0.08\n",
"episode: 3525, score: 0.0, epsilon: 0.08\n",
"episode: 3530, score: 0.0, epsilon: 0.08\n",
"episode: 3535, score: 0.0, epsilon: 0.08\n",
"episode: 3540, score: 0.0, epsilon: 0.08\n",
"episode: 3545, score: 4.0, epsilon: 0.08\n",
"episode: 3550, score: 2.0, epsilon: 0.08\n",
"episode: 3555, score: 0.0, epsilon: 0.08\n",
"episode: 3560, score: 0.0, epsilon: 0.08\n",
"episode: 3565, score: 1.0, epsilon: 0.08\n",
"episode: 3570, score: 0.0, epsilon: 0.08\n",
"episode: 3575, score: 2.0, epsilon: 0.08\n",
"episode: 3580, score: 0.0, epsilon: 0.08\n",
"episode: 3585, score: 0.0, epsilon: 0.08\n",
"episode: 3590, score: 0.0, epsilon: 0.08\n",
"episode: 3595, score: 1.0, epsilon: 0.08\n",
"marking, episode: 3600, score: 0.0, mean_score: 0.97, std_score: 1.30\n",
"episode: 3600, score: 0.0, epsilon: 0.08\n",
"episode: 3605, score: 0.0, epsilon: 0.08\n",
"episode: 3610, score: 1.0, epsilon: 0.08\n",
"episode: 3615, score: 3.0, epsilon: 0.08\n",
"episode: 3620, score: 0.0, epsilon: 0.08\n",
"episode: 3625, score: 0.0, epsilon: 0.08\n",
"episode: 3630, score: 1.0, epsilon: 0.08\n",
"episode: 3635, score: 0.0, epsilon: 0.08\n",
"episode: 3640, score: 2.0, epsilon: 0.08\n",
"episode: 3645, score: 0.0, epsilon: 0.08\n",
"episode: 3650, score: 0.0, epsilon: 0.08\n",
"episode: 3655, score: 0.0, epsilon: 0.08\n",
"episode: 3660, score: 0.0, epsilon: 0.08\n",
"episode: 3665, score: 1.0, epsilon: 0.08\n",
"episode: 3670, score: 4.0, epsilon: 0.08\n",
"episode: 3675, score: 2.0, epsilon: 0.08\n",
"episode: 3680, score: 5.0, epsilon: 0.08\n",
"episode: 3685, score: 3.0, epsilon: 0.08\n",
"episode: 3690, score: 0.0, epsilon: 0.08\n",
"episode: 3695, score: 0.0, epsilon: 0.08\n",
"marking, episode: 3700, score: 1.0, mean_score: 0.88, std_score: 1.15\n",
"episode: 3700, score: 1.0, epsilon: 0.08\n",
"episode: 3705, score: 0.0, epsilon: 0.08\n",
"episode: 3710, score: 1.0, epsilon: 0.08\n",
"episode: 3715, score: 0.0, epsilon: 0.08\n",
"episode: 3720, score: 2.0, epsilon: 0.08\n",
"episode: 3725, score: 2.0, epsilon: 0.08\n",
"episode: 3730, score: 3.0, epsilon: 0.08\n",
"episode: 3735, score: 0.0, epsilon: 0.08\n",
"episode: 3740, score: 0.0, epsilon: 0.08\n",
"episode: 3745, score: 2.0, epsilon: 0.08\n",
"episode: 3750, score: 1.0, epsilon: 0.08\n",
"episode: 3755, score: 0.0, epsilon: 0.08\n",
"episode: 3760, score: 0.0, epsilon: 0.08\n",
"episode: 3765, score: 0.0, epsilon: 0.08\n",
"episode: 3770, score: 0.0, epsilon: 0.08\n",
"episode: 3775, score: 3.0, epsilon: 0.08\n",
"episode: 3780, score: 4.0, epsilon: 0.08\n",
"episode: 3785, score: 4.0, epsilon: 0.08\n",
"episode: 3790, score: 0.0, epsilon: 0.08\n",
"episode: 3795, score: 5.0, epsilon: 0.08\n",
"marking, episode: 3800, score: 4.0, mean_score: 1.25, std_score: 1.56\n",
"episode: 3800, score: 4.0, epsilon: 0.08\n",
"episode: 3805, score: 0.0, epsilon: 0.08\n",
"episode: 3810, score: 1.0, epsilon: 0.08\n",
"episode: 3815, score: 2.0, epsilon: 0.08\n",
"episode: 3820, score: 2.0, epsilon: 0.08\n",
"episode: 3825, score: 0.0, epsilon: 0.08\n",
"episode: 3830, score: 0.0, epsilon: 0.08\n",
"episode: 3835, score: 2.0, epsilon: 0.08\n",
"episode: 3840, score: 2.0, epsilon: 0.08\n",
"episode: 3845, score: 3.0, epsilon: 0.08\n",
"episode: 3850, score: 1.0, epsilon: 0.08\n",
"episode: 3855, score: 1.0, epsilon: 0.08\n",
"episode: 3860, score: 2.0, epsilon: 0.08\n",
"episode: 3865, score: 0.0, epsilon: 0.08\n",
"episode: 3870, score: 0.0, epsilon: 0.08\n",
"episode: 3875, score: 0.0, epsilon: 0.08\n",
"episode: 3880, score: 1.0, epsilon: 0.08\n",
"episode: 3885, score: 0.0, epsilon: 0.08\n",
"episode: 3890, score: 6.0, epsilon: 0.08\n",
"episode: 3895, score: 0.0, epsilon: 0.08\n",
"marking, episode: 3900, score: 4.0, mean_score: 1.24, std_score: 1.48\n",
"episode: 3900, score: 4.0, epsilon: 0.08\n",
"episode: 3905, score: 1.0, epsilon: 0.08\n",
"episode: 3910, score: 1.0, epsilon: 0.08\n",
"episode: 3915, score: 1.0, epsilon: 0.08\n",
"episode: 3920, score: 0.0, epsilon: 0.08\n",
"episode: 3925, score: 3.0, epsilon: 0.08\n",
"episode: 3930, score: 1.0, epsilon: 0.08\n",
"episode: 3935, score: 2.0, epsilon: 0.08\n",
"episode: 3940, score: 0.0, epsilon: 0.08\n",
"episode: 3945, score: 3.0, epsilon: 0.08\n",
"episode: 3950, score: 0.0, epsilon: 0.08\n",
"episode: 3955, score: 0.0, epsilon: 0.08\n",
"episode: 3960, score: 0.0, epsilon: 0.08\n",
"episode: 3965, score: 1.0, epsilon: 0.08\n",
"episode: 3970, score: 0.0, epsilon: 0.08\n",
"episode: 3975, score: 2.0, epsilon: 0.08\n",
"episode: 3980, score: 0.0, epsilon: 0.08\n",
"episode: 3985, score: 0.0, epsilon: 0.08\n",
"episode: 3990, score: 0.0, epsilon: 0.08\n",
"episode: 3995, score: 0.0, epsilon: 0.08\n",
"marking, episode: 4000, score: 0.0, mean_score: 0.78, std_score: 1.03\n",
"episode: 4000, score: 0.0, epsilon: 0.08\n",
"episode: 4005, score: 1.0, epsilon: 0.08\n",
"episode: 4010, score: 0.0, epsilon: 0.08\n",
"episode: 4015, score: 0.0, epsilon: 0.08\n",
"episode: 4020, score: 0.0, epsilon: 0.08\n",
"episode: 4025, score: 0.0, epsilon: 0.08\n",
"episode: 4030, score: 1.0, epsilon: 0.08\n",
"episode: 4035, score: 2.0, epsilon: 0.08\n",
"episode: 4040, score: 4.0, epsilon: 0.08\n",
"episode: 4045, score: 3.0, epsilon: 0.08\n",
"episode: 4050, score: 2.0, epsilon: 0.08\n",
"episode: 4055, score: 1.0, epsilon: 0.08\n",
"episode: 4060, score: 0.0, epsilon: 0.08\n",
"episode: 4065, score: 0.0, epsilon: 0.08\n",
"episode: 4070, score: 0.0, epsilon: 0.08\n",
"episode: 4075, score: 0.0, epsilon: 0.08\n",
"episode: 4080, score: 3.0, epsilon: 0.08\n",
"episode: 4085, score: 1.0, epsilon: 0.08\n",
"episode: 4090, score: 0.0, epsilon: 0.08\n",
"episode: 4095, score: 0.0, epsilon: 0.08\n",
"marking, episode: 4100, score: 4.0, mean_score: 1.02, std_score: 1.34\n",
"episode: 4100, score: 4.0, epsilon: 0.08\n",
"episode: 4105, score: 0.0, epsilon: 0.08\n",
"episode: 4110, score: 2.0, epsilon: 0.08\n",
"episode: 4115, score: 1.0, epsilon: 0.08\n",
"episode: 4120, score: 0.0, epsilon: 0.08\n",
"episode: 4125, score: 2.0, epsilon: 0.08\n",
"episode: 4130, score: 1.0, epsilon: 0.08\n",
"episode: 4135, score: 1.0, epsilon: 0.08\n",
"episode: 4140, score: 1.0, epsilon: 0.08\n",
"episode: 4145, score: 0.0, epsilon: 0.08\n",
"episode: 4150, score: 2.0, epsilon: 0.08\n",
"episode: 4155, score: 0.0, epsilon: 0.08\n",
"episode: 4160, score: 2.0, epsilon: 0.08\n",
"episode: 4165, score: 1.0, epsilon: 0.08\n",
"episode: 4170, score: 1.0, epsilon: 0.08\n",
"episode: 4175, score: 1.0, epsilon: 0.08\n",
"episode: 4180, score: 1.0, epsilon: 0.08\n",
"episode: 4185, score: 0.0, epsilon: 0.08\n",
"episode: 4190, score: 0.0, epsilon: 0.08\n",
"episode: 4195, score: 0.0, epsilon: 0.08\n",
"marking, episode: 4200, score: 0.0, mean_score: 0.93, std_score: 1.23\n",
"episode: 4200, score: 0.0, epsilon: 0.08\n",
"episode: 4205, score: 0.0, epsilon: 0.08\n",
"episode: 4210, score: 5.0, epsilon: 0.08\n",
"episode: 4215, score: 0.0, epsilon: 0.08\n",
"episode: 4220, score: 1.0, epsilon: 0.08\n",
"episode: 4225, score: 0.0, epsilon: 0.08\n",
"episode: 4230, score: 2.0, epsilon: 0.08\n",
"episode: 4235, score: 4.0, epsilon: 0.08\n",
"episode: 4240, score: 2.0, epsilon: 0.08\n",
"episode: 4245, score: 0.0, epsilon: 0.08\n",
"episode: 4250, score: 0.0, epsilon: 0.08\n",
"episode: 4255, score: 1.0, epsilon: 0.08\n",
"episode: 4260, score: 0.0, epsilon: 0.08\n",
"episode: 4265, score: 0.0, epsilon: 0.08\n",
"episode: 4270, score: 1.0, epsilon: 0.08\n",
"episode: 4275, score: 1.0, epsilon: 0.08\n",
"episode: 4280, score: 1.0, epsilon: 0.08\n",
"episode: 4285, score: 0.0, epsilon: 0.08\n",
"episode: 4290, score: 0.0, epsilon: 0.08\n",
"episode: 4295, score: 1.0, epsilon: 0.08\n",
"marking, episode: 4300, score: 0.0, mean_score: 1.02, std_score: 1.27\n",
"episode: 4300, score: 0.0, epsilon: 0.08\n",
"episode: 4305, score: 1.0, epsilon: 0.08\n",
"episode: 4310, score: 0.0, epsilon: 0.08\n",
"episode: 4315, score: 0.0, epsilon: 0.08\n",
"episode: 4320, score: 2.0, epsilon: 0.08\n",
"episode: 4325, score: 0.0, epsilon: 0.08\n",
"episode: 4330, score: 0.0, epsilon: 0.08\n",
"episode: 4335, score: 6.0, epsilon: 0.08\n",
"episode: 4340, score: 2.0, epsilon: 0.08\n",
"episode: 4345, score: 1.0, epsilon: 0.08\n",
"episode: 4350, score: 1.0, epsilon: 0.08\n",
"episode: 4355, score: 1.0, epsilon: 0.08\n",
"episode: 4360, score: 0.0, epsilon: 0.08\n",
"episode: 4365, score: 5.0, epsilon: 0.08\n",
"episode: 4370, score: 0.0, epsilon: 0.08\n",
"episode: 4375, score: 1.0, epsilon: 0.08\n",
"episode: 4380, score: 1.0, epsilon: 0.08\n",
"episode: 4385, score: 2.0, epsilon: 0.08\n",
"episode: 4390, score: 1.0, epsilon: 0.08\n",
"episode: 4395, score: 0.0, epsilon: 0.08\n",
"marking, episode: 4400, score: 0.0, mean_score: 1.03, std_score: 1.28\n",
"episode: 4400, score: 0.0, epsilon: 0.08\n",
"episode: 4405, score: 0.0, epsilon: 0.08\n",
"episode: 4410, score: 6.0, epsilon: 0.08\n",
"episode: 4415, score: 0.0, epsilon: 0.08\n",
"episode: 4420, score: 1.0, epsilon: 0.08\n",
"episode: 4425, score: 0.0, epsilon: 0.08\n",
"episode: 4430, score: 1.0, epsilon: 0.08\n",
"episode: 4435, score: 0.0, epsilon: 0.08\n",
"episode: 4440, score: 0.0, epsilon: 0.08\n",
"episode: 4445, score: 0.0, epsilon: 0.08\n",
"episode: 4450, score: 1.0, epsilon: 0.08\n",
"episode: 4455, score: 0.0, epsilon: 0.08\n",
"episode: 4460, score: 0.0, epsilon: 0.08\n",
"episode: 4465, score: 0.0, epsilon: 0.08\n",
"episode: 4470, score: 1.0, epsilon: 0.08\n",
"episode: 4475, score: 1.0, epsilon: 0.08\n",
"episode: 4480, score: 1.0, epsilon: 0.08\n",
"episode: 4485, score: 2.0, epsilon: 0.08\n",
"episode: 4490, score: 4.0, epsilon: 0.08\n",
"episode: 4495, score: 1.0, epsilon: 0.08\n",
"marking, episode: 4500, score: 4.0, mean_score: 1.03, std_score: 1.43\n",
"episode: 4500, score: 4.0, epsilon: 0.08\n",
"episode: 4505, score: 0.0, epsilon: 0.08\n",
"episode: 4510, score: 0.0, epsilon: 0.08\n",
"episode: 4515, score: 0.0, epsilon: 0.08\n",
"episode: 4520, score: 1.0, epsilon: 0.08\n",
"episode: 4525, score: 1.0, epsilon: 0.08\n",
"episode: 4530, score: 3.0, epsilon: 0.08\n",
"episode: 4535, score: 0.0, epsilon: 0.08\n",
"episode: 4540, score: 3.0, epsilon: 0.08\n",
"episode: 4545, score: 0.0, epsilon: 0.08\n",
"episode: 4550, score: 1.0, epsilon: 0.08\n",
"episode: 4555, score: 1.0, epsilon: 0.08\n",
"episode: 4560, score: 3.0, epsilon: 0.08\n",
"episode: 4565, score: 0.0, epsilon: 0.08\n",
"episode: 4570, score: 2.0, epsilon: 0.08\n",
"episode: 4575, score: 0.0, epsilon: 0.08\n",
"episode: 4580, score: 0.0, epsilon: 0.08\n",
"episode: 4585, score: 1.0, epsilon: 0.08\n",
"episode: 4590, score: 0.0, epsilon: 0.08\n",
"episode: 4595, score: 5.0, epsilon: 0.08\n",
"marking, episode: 4600, score: 0.0, mean_score: 1.11, std_score: 1.41\n",
"episode: 4600, score: 0.0, epsilon: 0.08\n",
"episode: 4605, score: 2.0, epsilon: 0.08\n",
"episode: 4610, score: 1.0, epsilon: 0.08\n",
"episode: 4615, score: 6.0, epsilon: 0.08\n",
"episode: 4620, score: 0.0, epsilon: 0.08\n",
"episode: 4625, score: 0.0, epsilon: 0.08\n",
"episode: 4630, score: 0.0, epsilon: 0.08\n",
"episode: 4635, score: 0.0, epsilon: 0.08\n",
"episode: 4640, score: 0.0, epsilon: 0.08\n",
"episode: 4645, score: 1.0, epsilon: 0.08\n",
"episode: 4650, score: 0.0, epsilon: 0.08\n",
"episode: 4655, score: 0.0, epsilon: 0.08\n",
"episode: 4660, score: 2.0, epsilon: 0.08\n",
"episode: 4665, score: 0.0, epsilon: 0.08\n",
"episode: 4670, score: 0.0, epsilon: 0.08\n",
"episode: 4675, score: 2.0, epsilon: 0.08\n",
"episode: 4680, score: 0.0, epsilon: 0.08\n",
"episode: 4685, score: 1.0, epsilon: 0.08\n",
"episode: 4690, score: 1.0, epsilon: 0.08\n",
"episode: 4695, score: 1.0, epsilon: 0.08\n",
"marking, episode: 4700, score: 1.0, mean_score: 0.96, std_score: 1.36\n",
"episode: 4700, score: 1.0, epsilon: 0.08\n",
"episode: 4705, score: 0.0, epsilon: 0.08\n",
"episode: 4710, score: 5.0, epsilon: 0.08\n",
"episode: 4715, score: 1.0, epsilon: 0.08\n",
"episode: 4720, score: 0.0, epsilon: 0.08\n",
"episode: 4725, score: 2.0, epsilon: 0.08\n",
"episode: 4730, score: 0.0, epsilon: 0.08\n",
"episode: 4735, score: 0.0, epsilon: 0.08\n",
"episode: 4740, score: 0.0, epsilon: 0.08\n",
"episode: 4745, score: 1.0, epsilon: 0.08\n",
"episode: 4750, score: 0.0, epsilon: 0.08\n",
"episode: 4755, score: 0.0, epsilon: 0.08\n",
"episode: 4760, score: 1.0, epsilon: 0.08\n",
"episode: 4765, score: 0.0, epsilon: 0.08\n",
"episode: 4770, score: 1.0, epsilon: 0.08\n",
"episode: 4775, score: 0.0, epsilon: 0.08\n",
"episode: 4780, score: 1.0, epsilon: 0.08\n",
"episode: 4785, score: 0.0, epsilon: 0.08\n",
"episode: 4790, score: 1.0, epsilon: 0.08\n",
"episode: 4795, score: 0.0, epsilon: 0.08\n",
"marking, episode: 4800, score: 0.0, mean_score: 0.93, std_score: 1.31\n",
"episode: 4800, score: 0.0, epsilon: 0.08\n",
"episode: 4805, score: 1.0, epsilon: 0.08\n",
"episode: 4810, score: 0.0, epsilon: 0.08\n",
"episode: 4815, score: 0.0, epsilon: 0.08\n",
"episode: 4820, score: 0.0, epsilon: 0.08\n",
"episode: 4825, score: 1.0, epsilon: 0.08\n",
"episode: 4830, score: 3.0, epsilon: 0.08\n",
"episode: 4835, score: 0.0, epsilon: 0.08\n",
"episode: 4840, score: 1.0, epsilon: 0.08\n",
"episode: 4845, score: 2.0, epsilon: 0.08\n",
"episode: 4850, score: 0.0, epsilon: 0.08\n",
"episode: 4855, score: 1.0, epsilon: 0.08\n",
"episode: 4860, score: 0.0, epsilon: 0.08\n",
"episode: 4865, score: 2.0, epsilon: 0.08\n",
"episode: 4870, score: 0.0, epsilon: 0.08\n",
"episode: 4875, score: 1.0, epsilon: 0.08\n",
"episode: 4880, score: 0.0, epsilon: 0.08\n",
"episode: 4885, score: 0.0, epsilon: 0.08\n",
"episode: 4890, score: 0.0, epsilon: 0.08\n",
"episode: 4895, score: 0.0, epsilon: 0.08\n",
"marking, episode: 4900, score: 4.0, mean_score: 1.04, std_score: 1.50\n",
"episode: 4900, score: 4.0, epsilon: 0.08\n",
"episode: 4905, score: 0.0, epsilon: 0.08\n",
"episode: 4910, score: 0.0, epsilon: 0.08\n",
"episode: 4915, score: 0.0, epsilon: 0.08\n",
"episode: 4920, score: 1.0, epsilon: 0.08\n",
"episode: 4925, score: 0.0, epsilon: 0.08\n",
"episode: 4930, score: 0.0, epsilon: 0.08\n",
"episode: 4935, score: 0.0, epsilon: 0.08\n",
"episode: 4940, score: 2.0, epsilon: 0.08\n",
"episode: 4945, score: 0.0, epsilon: 0.08\n",
"episode: 4950, score: 0.0, epsilon: 0.08\n",
"episode: 4955, score: 0.0, epsilon: 0.08\n",
"episode: 4960, score: 1.0, epsilon: 0.08\n",
"episode: 4965, score: 1.0, epsilon: 0.08\n",
"episode: 4970, score: 0.0, epsilon: 0.08\n",
"episode: 4975, score: 1.0, epsilon: 0.08\n",
"episode: 4980, score: 1.0, epsilon: 0.08\n",
"episode: 4985, score: 0.0, epsilon: 0.08\n",
"episode: 4990, score: 1.0, epsilon: 0.08\n",
"episode: 4995, score: 0.0, epsilon: 0.08\n",
"marking, episode: 5000, score: 2.0, mean_score: 0.60, std_score: 0.89\n",
"episode: 5000, score: 2.0, epsilon: 0.08\n",
"episode: 5005, score: 3.0, epsilon: 0.08\n",
"episode: 5010, score: 0.0, epsilon: 0.08\n",
"episode: 5015, score: 0.0, epsilon: 0.08\n",
"episode: 5020, score: 2.0, epsilon: 0.08\n",
"episode: 5025, score: 1.0, epsilon: 0.08\n",
"episode: 5030, score: 2.0, epsilon: 0.08\n",
"episode: 5035, score: 1.0, epsilon: 0.08\n",
"episode: 5040, score: 0.0, epsilon: 0.08\n",
"episode: 5045, score: 1.0, epsilon: 0.08\n",
"episode: 5050, score: 0.0, epsilon: 0.08\n",
"episode: 5055, score: 2.0, epsilon: 0.08\n",
"episode: 5060, score: 1.0, epsilon: 0.08\n",
"episode: 5065, score: 0.0, epsilon: 0.08\n",
"episode: 5070, score: 2.0, epsilon: 0.08\n",
"episode: 5075, score: 5.0, epsilon: 0.08\n",
"episode: 5080, score: 0.0, epsilon: 0.08\n",
"episode: 5085, score: 1.0, epsilon: 0.08\n",
"episode: 5090, score: 1.0, epsilon: 0.08\n",
"episode: 5095, score: 1.0, epsilon: 0.08\n",
"marking, episode: 5100, score: 0.0, mean_score: 1.12, std_score: 1.51\n",
"episode: 5100, score: 0.0, epsilon: 0.08\n",
"episode: 5105, score: 2.0, epsilon: 0.08\n",
"episode: 5110, score: 1.0, epsilon: 0.08\n",
"episode: 5115, score: 0.0, epsilon: 0.08\n",
"episode: 5120, score: 0.0, epsilon: 0.08\n",
"episode: 5125, score: 4.0, epsilon: 0.08\n",
"episode: 5130, score: 0.0, epsilon: 0.08\n",
"episode: 5135, score: 0.0, epsilon: 0.08\n",
"episode: 5140, score: 1.0, epsilon: 0.08\n",
"episode: 5145, score: 0.0, epsilon: 0.08\n",
"episode: 5150, score: 1.0, epsilon: 0.08\n",
"episode: 5155, score: 3.0, epsilon: 0.08\n",
"episode: 5160, score: 3.0, epsilon: 0.08\n",
"episode: 5165, score: 6.0, epsilon: 0.08\n",
"episode: 5170, score: 2.0, epsilon: 0.08\n",
"episode: 5175, score: 0.0, epsilon: 0.08\n",
"episode: 5180, score: 0.0, epsilon: 0.08\n",
"episode: 5185, score: 0.0, epsilon: 0.08\n",
"episode: 5190, score: 1.0, epsilon: 0.08\n",
"episode: 5195, score: 0.0, epsilon: 0.08\n",
"marking, episode: 5200, score: 3.0, mean_score: 1.07, std_score: 1.49\n",
"episode: 5200, score: 3.0, epsilon: 0.08\n",
"episode: 5205, score: 1.0, epsilon: 0.08\n",
"episode: 5210, score: 0.0, epsilon: 0.08\n",
"episode: 5215, score: 0.0, epsilon: 0.08\n",
"episode: 5220, score: 0.0, epsilon: 0.08\n",
"episode: 5225, score: 3.0, epsilon: 0.08\n",
"episode: 5230, score: 1.0, epsilon: 0.08\n",
"episode: 5235, score: 0.0, epsilon: 0.08\n",
"episode: 5240, score: 4.0, epsilon: 0.08\n",
"episode: 5245, score: 0.0, epsilon: 0.08\n",
"episode: 5250, score: 0.0, epsilon: 0.08\n",
"episode: 5255, score: 1.0, epsilon: 0.08\n",
"episode: 5260, score: 0.0, epsilon: 0.08\n",
"episode: 5265, score: 0.0, epsilon: 0.08\n",
"episode: 5270, score: 0.0, epsilon: 0.08\n",
"episode: 5275, score: 1.0, epsilon: 0.08\n",
"episode: 5280, score: 0.0, epsilon: 0.08\n",
"episode: 5285, score: 0.0, epsilon: 0.08\n",
"episode: 5290, score: 0.0, epsilon: 0.08\n",
"episode: 5295, score: 1.0, epsilon: 0.08\n",
"marking, episode: 5300, score: 1.0, mean_score: 1.17, std_score: 1.58\n",
"episode: 5300, score: 1.0, epsilon: 0.08\n",
"episode: 5305, score: 3.0, epsilon: 0.08\n",
"episode: 5310, score: 0.0, epsilon: 0.08\n",
"episode: 5315, score: 1.0, epsilon: 0.08\n",
"episode: 5320, score: 0.0, epsilon: 0.08\n",
"episode: 5325, score: 0.0, epsilon: 0.08\n",
"episode: 5330, score: 1.0, epsilon: 0.08\n",
"episode: 5335, score: 2.0, epsilon: 0.08\n",
"episode: 5340, score: 0.0, epsilon: 0.08\n",
"episode: 5345, score: 6.0, epsilon: 0.08\n",
"episode: 5350, score: 0.0, epsilon: 0.08\n",
"episode: 5355, score: 1.0, epsilon: 0.08\n",
"episode: 5360, score: 5.0, epsilon: 0.08\n",
"episode: 5365, score: 1.0, epsilon: 0.08\n",
"episode: 5370, score: 0.0, epsilon: 0.08\n",
"episode: 5375, score: 0.0, epsilon: 0.08\n",
"episode: 5380, score: 0.0, epsilon: 0.08\n",
"episode: 5385, score: 1.0, epsilon: 0.08\n",
"episode: 5390, score: 0.0, epsilon: 0.08\n",
"episode: 5395, score: 0.0, epsilon: 0.08\n",
"marking, episode: 5400, score: 0.0, mean_score: 0.74, std_score: 1.20\n",
"episode: 5400, score: 0.0, epsilon: 0.08\n",
"episode: 5405, score: 1.0, epsilon: 0.08\n",
"episode: 5410, score: 0.0, epsilon: 0.08\n",
"episode: 5415, score: 0.0, epsilon: 0.08\n",
"episode: 5420, score: 0.0, epsilon: 0.08\n",
"episode: 5425, score: 1.0, epsilon: 0.08\n",
"episode: 5430, score: 1.0, epsilon: 0.08\n",
"episode: 5435, score: 2.0, epsilon: 0.08\n",
"episode: 5440, score: 2.0, epsilon: 0.08\n",
"episode: 5445, score: 3.0, epsilon: 0.08\n",
"episode: 5450, score: 0.0, epsilon: 0.08\n",
"episode: 5455, score: 3.0, epsilon: 0.08\n",
"episode: 5460, score: 4.0, epsilon: 0.08\n",
"episode: 5465, score: 0.0, epsilon: 0.08\n",
"episode: 5470, score: 0.0, epsilon: 0.08\n",
"episode: 5475, score: 1.0, epsilon: 0.08\n",
"episode: 5480, score: 2.0, epsilon: 0.08\n",
"episode: 5485, score: 2.0, epsilon: 0.08\n",
"episode: 5490, score: 2.0, epsilon: 0.08\n",
"episode: 5495, score: 0.0, epsilon: 0.08\n",
"marking, episode: 5500, score: 0.0, mean_score: 0.94, std_score: 1.26\n",
"episode: 5500, score: 0.0, epsilon: 0.08\n",
"episode: 5505, score: 0.0, epsilon: 0.08\n",
"episode: 5510, score: 0.0, epsilon: 0.08\n",
"episode: 5515, score: 0.0, epsilon: 0.08\n",
"episode: 5520, score: 0.0, epsilon: 0.08\n",
"episode: 5525, score: 2.0, epsilon: 0.08\n",
"episode: 5530, score: 0.0, epsilon: 0.08\n",
"episode: 5535, score: 0.0, epsilon: 0.08\n",
"episode: 5540, score: 1.0, epsilon: 0.08\n",
"episode: 5545, score: 0.0, epsilon: 0.08\n",
"episode: 5550, score: 2.0, epsilon: 0.08\n",
"episode: 5555, score: 2.0, epsilon: 0.08\n",
"episode: 5560, score: 0.0, epsilon: 0.08\n",
"episode: 5565, score: 2.0, epsilon: 0.08\n",
"episode: 5570, score: 0.0, epsilon: 0.08\n",
"episode: 5575, score: 1.0, epsilon: 0.08\n",
"episode: 5580, score: 3.0, epsilon: 0.08\n",
"episode: 5585, score: 3.0, epsilon: 0.08\n",
"episode: 5590, score: 1.0, epsilon: 0.08\n",
"episode: 5595, score: 0.0, epsilon: 0.08\n",
"marking, episode: 5600, score: 1.0, mean_score: 1.07, std_score: 1.46\n",
"episode: 5600, score: 1.0, epsilon: 0.08\n",
"episode: 5605, score: 1.0, epsilon: 0.08\n",
"episode: 5610, score: 4.0, epsilon: 0.08\n",
"episode: 5615, score: 0.0, epsilon: 0.08\n",
"episode: 5620, score: 0.0, epsilon: 0.08\n",
"episode: 5625, score: 0.0, epsilon: 0.08\n",
"episode: 5630, score: 1.0, epsilon: 0.08\n",
"episode: 5635, score: 1.0, epsilon: 0.08\n",
"episode: 5640, score: 1.0, epsilon: 0.08\n",
"episode: 5645, score: 0.0, epsilon: 0.08\n",
"episode: 5650, score: 1.0, epsilon: 0.08\n",
"episode: 5655, score: 0.0, epsilon: 0.08\n",
"episode: 5660, score: 0.0, epsilon: 0.08\n",
"episode: 5665, score: 2.0, epsilon: 0.08\n",
"episode: 5670, score: 1.0, epsilon: 0.08\n",
"episode: 5675, score: 2.0, epsilon: 0.08\n",
"episode: 5680, score: 0.0, epsilon: 0.08\n",
"episode: 5685, score: 0.0, epsilon: 0.08\n",
"episode: 5690, score: 0.0, epsilon: 0.08\n",
"episode: 5695, score: 1.0, epsilon: 0.08\n",
"marking, episode: 5700, score: 1.0, mean_score: 0.91, std_score: 1.38\n",
"episode: 5700, score: 1.0, epsilon: 0.08\n",
"episode: 5705, score: 0.0, epsilon: 0.08\n",
"episode: 5710, score: 2.0, epsilon: 0.08\n",
"episode: 5715, score: 0.0, epsilon: 0.08\n",
"episode: 5720, score: 1.0, epsilon: 0.08\n",
"episode: 5725, score: 0.0, epsilon: 0.08\n",
"episode: 5730, score: 1.0, epsilon: 0.08\n",
"episode: 5735, score: 2.0, epsilon: 0.08\n",
"episode: 5740, score: 0.0, epsilon: 0.08\n",
"episode: 5745, score: 0.0, epsilon: 0.08\n",
"episode: 5750, score: 0.0, epsilon: 0.08\n",
"episode: 5755, score: 0.0, epsilon: 0.08\n",
"episode: 5760, score: 0.0, epsilon: 0.08\n",
"episode: 5765, score: 2.0, epsilon: 0.08\n",
"episode: 5770, score: 2.0, epsilon: 0.08\n",
"episode: 5775, score: 0.0, epsilon: 0.08\n",
"episode: 5780, score: 0.0, epsilon: 0.08\n",
"episode: 5785, score: 0.0, epsilon: 0.08\n",
"episode: 5790, score: 0.0, epsilon: 0.08\n",
"episode: 5795, score: 0.0, epsilon: 0.08\n",
"marking, episode: 5800, score: 0.0, mean_score: 0.77, std_score: 1.02\n",
"episode: 5800, score: 0.0, epsilon: 0.08\n",
"episode: 5805, score: 2.0, epsilon: 0.08\n",
"episode: 5810, score: 0.0, epsilon: 0.08\n",
"episode: 5815, score: 0.0, epsilon: 0.08\n",
"episode: 5820, score: 0.0, epsilon: 0.08\n",
"episode: 5825, score: 0.0, epsilon: 0.08\n",
"episode: 5830, score: 0.0, epsilon: 0.08\n",
"episode: 5835, score: 0.0, epsilon: 0.08\n",
"episode: 5840, score: 2.0, epsilon: 0.08\n",
"episode: 5845, score: 1.0, epsilon: 0.08\n",
"episode: 5850, score: 0.0, epsilon: 0.08\n",
"episode: 5855, score: 1.0, epsilon: 0.08\n",
"episode: 5860, score: 2.0, epsilon: 0.08\n",
"episode: 5865, score: 2.0, epsilon: 0.08\n",
"episode: 5870, score: 0.0, epsilon: 0.08\n",
"episode: 5875, score: 3.0, epsilon: 0.08\n",
"episode: 5880, score: 1.0, epsilon: 0.08\n",
"episode: 5885, score: 0.0, epsilon: 0.08\n",
"episode: 5890, score: 0.0, epsilon: 0.08\n",
"episode: 5895, score: 1.0, epsilon: 0.08\n",
"marking, episode: 5900, score: 2.0, mean_score: 1.04, std_score: 1.21\n",
"episode: 5900, score: 2.0, epsilon: 0.08\n",
"episode: 5905, score: 1.0, epsilon: 0.08\n",
"episode: 5910, score: 0.0, epsilon: 0.08\n",
"episode: 5915, score: 0.0, epsilon: 0.08\n",
"episode: 5920, score: 0.0, epsilon: 0.08\n",
"episode: 5925, score: 1.0, epsilon: 0.08\n",
"episode: 5930, score: 0.0, epsilon: 0.08\n",
"episode: 5935, score: 2.0, epsilon: 0.08\n",
"episode: 5940, score: 1.0, epsilon: 0.08\n",
"episode: 5945, score: 0.0, epsilon: 0.08\n",
"episode: 5950, score: 1.0, epsilon: 0.08\n",
"episode: 5955, score: 0.0, epsilon: 0.08\n",
"episode: 5960, score: 0.0, epsilon: 0.08\n",
"episode: 5965, score: 1.0, epsilon: 0.08\n",
"episode: 5970, score: 2.0, epsilon: 0.08\n",
"episode: 5975, score: 1.0, epsilon: 0.08\n",
"episode: 5980, score: 4.0, epsilon: 0.08\n",
"episode: 5985, score: 1.0, epsilon: 0.08\n",
"episode: 5990, score: 2.0, epsilon: 0.08\n",
"episode: 5995, score: 0.0, epsilon: 0.08\n",
"marking, episode: 6000, score: 4.0, mean_score: 0.90, std_score: 1.17\n",
"episode: 6000, score: 4.0, epsilon: 0.08\n",
"episode: 6005, score: 0.0, epsilon: 0.08\n",
"episode: 6010, score: 0.0, epsilon: 0.08\n",
"episode: 6015, score: 2.0, epsilon: 0.08\n",
"episode: 6020, score: 0.0, epsilon: 0.08\n",
"episode: 6025, score: 3.0, epsilon: 0.08\n",
"episode: 6030, score: 1.0, epsilon: 0.08\n",
"episode: 6035, score: 0.0, epsilon: 0.08\n",
"episode: 6040, score: 0.0, epsilon: 0.08\n",
"episode: 6045, score: 3.0, epsilon: 0.08\n",
"episode: 6050, score: 0.0, epsilon: 0.08\n",
"episode: 6055, score: 2.0, epsilon: 0.08\n",
"episode: 6060, score: 2.0, epsilon: 0.08\n",
"episode: 6065, score: 0.0, epsilon: 0.08\n",
"episode: 6070, score: 1.0, epsilon: 0.08\n",
"episode: 6075, score: 1.0, epsilon: 0.08\n",
"episode: 6080, score: 0.0, epsilon: 0.08\n",
"episode: 6085, score: 0.0, epsilon: 0.08\n",
"episode: 6090, score: 1.0, epsilon: 0.08\n",
"episode: 6095, score: 0.0, epsilon: 0.08\n",
"marking, episode: 6100, score: 0.0, mean_score: 0.93, std_score: 1.15\n",
"episode: 6100, score: 0.0, epsilon: 0.08\n",
"episode: 6105, score: 3.0, epsilon: 0.08\n",
"episode: 6110, score: 3.0, epsilon: 0.08\n",
"episode: 6115, score: 3.0, epsilon: 0.08\n",
"episode: 6120, score: 2.0, epsilon: 0.08\n",
"episode: 6125, score: 3.0, epsilon: 0.08\n",
"episode: 6130, score: 0.0, epsilon: 0.08\n",
"episode: 6135, score: 1.0, epsilon: 0.08\n",
"episode: 6140, score: 2.0, epsilon: 0.08\n",
"episode: 6145, score: 1.0, epsilon: 0.08\n",
"episode: 6150, score: 0.0, epsilon: 0.08\n",
"episode: 6155, score: 0.0, epsilon: 0.08\n",
"episode: 6160, score: 1.0, epsilon: 0.08\n",
"episode: 6165, score: 2.0, epsilon: 0.08\n",
"episode: 6170, score: 2.0, epsilon: 0.08\n",
"episode: 6175, score: 1.0, epsilon: 0.08\n",
"episode: 6180, score: 1.0, epsilon: 0.08\n",
"episode: 6185, score: 0.0, epsilon: 0.08\n",
"episode: 6190, score: 2.0, epsilon: 0.08\n",
"episode: 6195, score: 0.0, epsilon: 0.08\n",
"marking, episode: 6200, score: 0.0, mean_score: 1.13, std_score: 1.32\n",
"episode: 6200, score: 0.0, epsilon: 0.08\n",
"episode: 6205, score: 1.0, epsilon: 0.08\n",
"episode: 6210, score: 0.0, epsilon: 0.08\n",
"episode: 6215, score: 1.0, epsilon: 0.08\n",
"episode: 6220, score: 1.0, epsilon: 0.08\n",
"episode: 6225, score: 1.0, epsilon: 0.08\n",
"episode: 6230, score: 2.0, epsilon: 0.08\n",
"episode: 6235, score: 2.0, epsilon: 0.08\n",
"episode: 6240, score: 0.0, epsilon: 0.08\n",
"episode: 6245, score: 1.0, epsilon: 0.08\n",
"episode: 6250, score: 2.0, epsilon: 0.08\n",
"episode: 6255, score: 1.0, epsilon: 0.08\n",
"episode: 6260, score: 3.0, epsilon: 0.08\n",
"episode: 6265, score: 0.0, epsilon: 0.08\n",
"episode: 6270, score: 0.0, epsilon: 0.08\n",
"episode: 6275, score: 0.0, epsilon: 0.08\n",
"episode: 6280, score: 2.0, epsilon: 0.08\n",
"episode: 6285, score: 0.0, epsilon: 0.08\n",
"episode: 6290, score: 0.0, epsilon: 0.08\n",
"episode: 6295, score: 0.0, epsilon: 0.08\n",
"marking, episode: 6300, score: 3.0, mean_score: 1.02, std_score: 1.36\n",
"episode: 6300, score: 3.0, epsilon: 0.08\n",
"episode: 6305, score: 0.0, epsilon: 0.08\n",
"episode: 6310, score: 0.0, epsilon: 0.08\n",
"episode: 6315, score: 2.0, epsilon: 0.08\n",
"episode: 6320, score: 0.0, epsilon: 0.08\n",
"episode: 6325, score: 2.0, epsilon: 0.08\n",
"episode: 6330, score: 2.0, epsilon: 0.08\n",
"episode: 6335, score: 1.0, epsilon: 0.08\n",
"episode: 6340, score: 1.0, epsilon: 0.08\n",
"episode: 6345, score: 5.0, epsilon: 0.08\n",
"episode: 6350, score: 0.0, epsilon: 0.08\n",
"episode: 6355, score: 0.0, epsilon: 0.08\n",
"episode: 6360, score: 1.0, epsilon: 0.08\n",
"episode: 6365, score: 1.0, epsilon: 0.08\n",
"episode: 6370, score: 4.0, epsilon: 0.08\n",
"episode: 6375, score: 2.0, epsilon: 0.08\n",
"episode: 6380, score: 2.0, epsilon: 0.08\n",
"episode: 6385, score: 2.0, epsilon: 0.08\n",
"episode: 6390, score: 3.0, epsilon: 0.08\n",
"episode: 6395, score: 1.0, epsilon: 0.08\n",
"marking, episode: 6400, score: 2.0, mean_score: 1.29, std_score: 1.46\n",
"episode: 6400, score: 2.0, epsilon: 0.08\n",
"episode: 6405, score: 4.0, epsilon: 0.08\n",
"episode: 6410, score: 0.0, epsilon: 0.08\n",
"episode: 6415, score: 0.0, epsilon: 0.08\n",
"episode: 6420, score: 1.0, epsilon: 0.08\n",
"episode: 6425, score: 1.0, epsilon: 0.08\n",
"episode: 6430, score: 2.0, epsilon: 0.08\n",
"episode: 6435, score: 0.0, epsilon: 0.08\n",
"episode: 6440, score: 2.0, epsilon: 0.08\n",
"episode: 6445, score: 0.0, epsilon: 0.08\n",
"episode: 6450, score: 0.0, epsilon: 0.08\n",
"episode: 6455, score: 2.0, epsilon: 0.08\n",
"episode: 6460, score: 0.0, epsilon: 0.08\n",
"episode: 6465, score: 0.0, epsilon: 0.08\n",
"episode: 6470, score: 0.0, epsilon: 0.08\n",
"episode: 6475, score: 1.0, epsilon: 0.08\n",
"episode: 6480, score: 0.0, epsilon: 0.08\n",
"episode: 6485, score: 1.0, epsilon: 0.08\n",
"episode: 6490, score: 4.0, epsilon: 0.08\n",
"episode: 6495, score: 1.0, epsilon: 0.08\n",
"marking, episode: 6500, score: 3.0, mean_score: 0.95, std_score: 1.29\n",
"episode: 6500, score: 3.0, epsilon: 0.08\n",
"episode: 6505, score: 0.0, epsilon: 0.08\n",
"episode: 6510, score: 1.0, epsilon: 0.08\n",
"episode: 6515, score: 3.0, epsilon: 0.08\n",
"episode: 6520, score: 2.0, epsilon: 0.08\n",
"episode: 6525, score: 0.0, epsilon: 0.08\n",
"episode: 6530, score: 1.0, epsilon: 0.08\n",
"episode: 6535, score: 0.0, epsilon: 0.08\n",
"episode: 6540, score: 0.0, epsilon: 0.08\n",
"episode: 6545, score: 0.0, epsilon: 0.08\n",
"episode: 6550, score: 3.0, epsilon: 0.08\n",
"episode: 6555, score: 0.0, epsilon: 0.08\n",
"episode: 6560, score: 1.0, epsilon: 0.08\n",
"episode: 6565, score: 2.0, epsilon: 0.08\n",
"episode: 6570, score: 0.0, epsilon: 0.08\n",
"episode: 6575, score: 0.0, epsilon: 0.08\n",
"episode: 6580, score: 0.0, epsilon: 0.08\n",
"episode: 6585, score: 0.0, epsilon: 0.08\n",
"episode: 6590, score: 2.0, epsilon: 0.08\n",
"episode: 6595, score: 5.0, epsilon: 0.08\n",
"marking, episode: 6600, score: 1.0, mean_score: 1.01, std_score: 1.36\n",
"episode: 6600, score: 1.0, epsilon: 0.08\n",
"episode: 6605, score: 2.0, epsilon: 0.08\n",
"episode: 6610, score: 0.0, epsilon: 0.08\n",
"episode: 6615, score: 0.0, epsilon: 0.08\n",
"episode: 6620, score: 0.0, epsilon: 0.08\n",
"episode: 6625, score: 0.0, epsilon: 0.08\n",
"episode: 6630, score: 0.0, epsilon: 0.08\n",
"episode: 6635, score: 0.0, epsilon: 0.08\n",
"episode: 6640, score: 0.0, epsilon: 0.08\n",
"episode: 6645, score: 0.0, epsilon: 0.08\n",
"episode: 6650, score: 6.0, epsilon: 0.08\n",
"episode: 6655, score: 1.0, epsilon: 0.08\n",
"episode: 6660, score: 0.0, epsilon: 0.08\n",
"episode: 6665, score: 2.0, epsilon: 0.08\n",
"episode: 6670, score: 2.0, epsilon: 0.08\n",
"episode: 6675, score: 2.0, epsilon: 0.08\n",
"episode: 6680, score: 1.0, epsilon: 0.08\n",
"episode: 6685, score: 0.0, epsilon: 0.08\n",
"episode: 6690, score: 0.0, epsilon: 0.08\n",
"episode: 6695, score: 0.0, epsilon: 0.08\n",
"marking, episode: 6700, score: 0.0, mean_score: 0.96, std_score: 1.37\n",
"episode: 6700, score: 0.0, epsilon: 0.08\n",
"episode: 6705, score: 1.0, epsilon: 0.08\n",
"episode: 6710, score: 1.0, epsilon: 0.08\n",
"episode: 6715, score: 1.0, epsilon: 0.08\n",
"episode: 6720, score: 2.0, epsilon: 0.08\n",
"episode: 6725, score: 0.0, epsilon: 0.08\n",
"episode: 6730, score: 0.0, epsilon: 0.08\n",
"episode: 6735, score: 0.0, epsilon: 0.08\n",
"episode: 6740, score: 1.0, epsilon: 0.08\n",
"episode: 6745, score: 1.0, epsilon: 0.08\n",
"episode: 6750, score: 2.0, epsilon: 0.08\n",
"episode: 6755, score: 0.0, epsilon: 0.08\n",
"episode: 6760, score: 0.0, epsilon: 0.08\n",
"episode: 6765, score: 3.0, epsilon: 0.08\n",
"episode: 6770, score: 0.0, epsilon: 0.08\n",
"episode: 6775, score: 3.0, epsilon: 0.08\n",
"episode: 6780, score: 3.0, epsilon: 0.08\n",
"episode: 6785, score: 2.0, epsilon: 0.08\n",
"episode: 6790, score: 0.0, epsilon: 0.08\n",
"episode: 6795, score: 2.0, epsilon: 0.08\n",
"marking, episode: 6800, score: 1.0, mean_score: 1.23, std_score: 1.37\n",
"episode: 6800, score: 1.0, epsilon: 0.08\n",
"episode: 6805, score: 1.0, epsilon: 0.08\n",
"episode: 6810, score: 0.0, epsilon: 0.08\n",
"episode: 6815, score: 6.0, epsilon: 0.08\n",
"episode: 6820, score: 0.0, epsilon: 0.08\n",
"episode: 6825, score: 0.0, epsilon: 0.08\n",
"episode: 6830, score: 1.0, epsilon: 0.08\n",
"episode: 6835, score: 1.0, epsilon: 0.08\n",
"episode: 6840, score: 2.0, epsilon: 0.08\n",
"episode: 6845, score: 0.0, epsilon: 0.08\n",
"episode: 6850, score: 1.0, epsilon: 0.08\n",
"episode: 6855, score: 4.0, epsilon: 0.08\n",
"episode: 6860, score: 0.0, epsilon: 0.08\n",
"episode: 6865, score: 0.0, epsilon: 0.08\n",
"episode: 6870, score: 0.0, epsilon: 0.08\n",
"episode: 6875, score: 0.0, epsilon: 0.08\n",
"episode: 6880, score: 0.0, epsilon: 0.08\n",
"episode: 6885, score: 0.0, epsilon: 0.08\n",
"episode: 6890, score: 3.0, epsilon: 0.08\n",
"episode: 6895, score: 1.0, epsilon: 0.08\n",
"marking, episode: 6900, score: 0.0, mean_score: 0.77, std_score: 1.17\n",
"episode: 6900, score: 0.0, epsilon: 0.08\n",
"episode: 6905, score: 0.0, epsilon: 0.08\n",
"episode: 6910, score: 3.0, epsilon: 0.08\n",
"episode: 6915, score: 0.0, epsilon: 0.08\n",
"episode: 6920, score: 6.0, epsilon: 0.08\n",
"episode: 6925, score: 1.0, epsilon: 0.08\n",
"episode: 6930, score: 3.0, epsilon: 0.08\n",
"episode: 6935, score: 0.0, epsilon: 0.08\n",
"episode: 6940, score: 1.0, epsilon: 0.08\n",
"episode: 6945, score: 0.0, epsilon: 0.08\n",
"episode: 6950, score: 0.0, epsilon: 0.08\n",
"episode: 6955, score: 1.0, epsilon: 0.08\n",
"episode: 6960, score: 0.0, epsilon: 0.08\n",
"episode: 6965, score: 1.0, epsilon: 0.08\n",
"episode: 6970, score: 0.0, epsilon: 0.08\n",
"episode: 6975, score: 1.0, epsilon: 0.08\n",
"episode: 6980, score: 0.0, epsilon: 0.08\n",
"episode: 6985, score: 6.0, epsilon: 0.08\n",
"episode: 6990, score: 2.0, epsilon: 0.08\n",
"episode: 6995, score: 0.0, epsilon: 0.08\n",
"marking, episode: 7000, score: 0.0, mean_score: 1.04, std_score: 1.54\n",
"episode: 7000, score: 0.0, epsilon: 0.08\n",
"episode: 7005, score: 0.0, epsilon: 0.08\n",
"episode: 7010, score: 3.0, epsilon: 0.08\n",
"episode: 7015, score: 2.0, epsilon: 0.08\n",
"episode: 7020, score: 3.0, epsilon: 0.08\n",
"episode: 7025, score: 0.0, epsilon: 0.08\n",
"episode: 7030, score: 2.0, epsilon: 0.08\n",
"episode: 7035, score: 0.0, epsilon: 0.08\n",
"episode: 7040, score: 1.0, epsilon: 0.08\n",
"episode: 7045, score: 0.0, epsilon: 0.08\n",
"episode: 7050, score: 0.0, epsilon: 0.08\n",
"episode: 7055, score: 0.0, epsilon: 0.08\n",
"episode: 7060, score: 1.0, epsilon: 0.08\n",
"episode: 7065, score: 0.0, epsilon: 0.08\n",
"episode: 7070, score: 1.0, epsilon: 0.08\n",
"episode: 7075, score: 0.0, epsilon: 0.08\n",
"episode: 7080, score: 2.0, epsilon: 0.08\n",
"episode: 7085, score: 1.0, epsilon: 0.08\n",
"episode: 7090, score: 2.0, epsilon: 0.08\n",
"episode: 7095, score: 0.0, epsilon: 0.08\n",
"marking, episode: 7100, score: 0.0, mean_score: 1.06, std_score: 1.42\n",
"episode: 7100, score: 0.0, epsilon: 0.08\n",
"episode: 7105, score: 2.0, epsilon: 0.08\n",
"episode: 7110, score: 0.0, epsilon: 0.08\n",
"episode: 7115, score: 0.0, epsilon: 0.08\n",
"episode: 7120, score: 1.0, epsilon: 0.08\n",
"episode: 7125, score: 0.0, epsilon: 0.08\n",
"episode: 7130, score: 1.0, epsilon: 0.08\n",
"episode: 7135, score: 3.0, epsilon: 0.08\n",
"episode: 7140, score: 0.0, epsilon: 0.08\n",
"episode: 7145, score: 2.0, epsilon: 0.08\n",
"episode: 7150, score: 6.0, epsilon: 0.08\n",
"episode: 7155, score: 5.0, epsilon: 0.08\n",
"episode: 7160, score: 3.0, epsilon: 0.08\n",
"episode: 7165, score: 0.0, epsilon: 0.08\n",
"episode: 7170, score: 1.0, epsilon: 0.08\n",
"episode: 7175, score: 3.0, epsilon: 0.08\n",
"episode: 7180, score: 0.0, epsilon: 0.08\n",
"episode: 7185, score: 0.0, epsilon: 0.08\n",
"episode: 7190, score: 0.0, epsilon: 0.08\n",
"episode: 7195, score: 0.0, epsilon: 0.08\n",
"marking, episode: 7200, score: 1.0, mean_score: 0.96, std_score: 1.30\n",
"episode: 7200, score: 1.0, epsilon: 0.08\n",
"episode: 7205, score: 0.0, epsilon: 0.08\n",
"episode: 7210, score: 0.0, epsilon: 0.08\n",
"episode: 7215, score: 1.0, epsilon: 0.08\n",
"episode: 7220, score: 0.0, epsilon: 0.08\n",
"episode: 7225, score: 0.0, epsilon: 0.08\n",
"episode: 7230, score: 0.0, epsilon: 0.08\n",
"episode: 7235, score: 5.0, epsilon: 0.08\n",
"episode: 7240, score: 0.0, epsilon: 0.08\n",
"episode: 7245, score: 0.0, epsilon: 0.08\n",
"episode: 7250, score: 0.0, epsilon: 0.08\n",
"episode: 7255, score: 0.0, epsilon: 0.08\n",
"episode: 7260, score: 0.0, epsilon: 0.08\n",
"episode: 7265, score: 3.0, epsilon: 0.08\n",
"episode: 7270, score: 1.0, epsilon: 0.08\n",
"episode: 7275, score: 0.0, epsilon: 0.08\n",
"episode: 7280, score: 0.0, epsilon: 0.08\n",
"episode: 7285, score: 1.0, epsilon: 0.08\n",
"episode: 7290, score: 0.0, epsilon: 0.08\n",
"episode: 7295, score: 2.0, epsilon: 0.08\n",
"marking, episode: 7300, score: 4.0, mean_score: 1.00, std_score: 1.36\n",
"episode: 7300, score: 4.0, epsilon: 0.08\n",
"episode: 7305, score: 4.0, epsilon: 0.08\n",
"episode: 7310, score: 0.0, epsilon: 0.08\n",
"episode: 7315, score: 2.0, epsilon: 0.08\n",
"episode: 7320, score: 1.0, epsilon: 0.08\n",
"episode: 7325, score: 0.0, epsilon: 0.08\n",
"episode: 7330, score: 0.0, epsilon: 0.08\n",
"episode: 7335, score: 0.0, epsilon: 0.08\n",
"episode: 7340, score: 0.0, epsilon: 0.08\n",
"episode: 7345, score: 2.0, epsilon: 0.08\n",
"episode: 7350, score: 0.0, epsilon: 0.08\n",
"episode: 7355, score: 0.0, epsilon: 0.08\n",
"episode: 7360, score: 1.0, epsilon: 0.08\n",
"episode: 7365, score: 0.0, epsilon: 0.08\n",
"episode: 7370, score: 0.0, epsilon: 0.08\n",
"episode: 7375, score: 0.0, epsilon: 0.08\n",
"episode: 7380, score: 0.0, epsilon: 0.08\n",
"episode: 7385, score: 2.0, epsilon: 0.08\n",
"episode: 7390, score: 3.0, epsilon: 0.08\n",
"episode: 7395, score: 0.0, epsilon: 0.08\n",
"marking, episode: 7400, score: 1.0, mean_score: 0.92, std_score: 1.18\n",
"episode: 7400, score: 1.0, epsilon: 0.08\n",
"episode: 7405, score: 0.0, epsilon: 0.08\n",
"episode: 7410, score: 0.0, epsilon: 0.08\n",
"episode: 7415, score: 0.0, epsilon: 0.08\n",
"episode: 7420, score: 0.0, epsilon: 0.08\n",
"episode: 7425, score: 0.0, epsilon: 0.08\n",
"episode: 7430, score: 1.0, epsilon: 0.08\n",
"episode: 7435, score: 0.0, epsilon: 0.08\n",
"episode: 7440, score: 0.0, epsilon: 0.08\n",
"episode: 7445, score: 0.0, epsilon: 0.08\n",
"episode: 7450, score: 0.0, epsilon: 0.08\n",
"episode: 7455, score: 2.0, epsilon: 0.08\n",
"episode: 7460, score: 0.0, epsilon: 0.08\n",
"episode: 7465, score: 1.0, epsilon: 0.08\n",
"episode: 7470, score: 0.0, epsilon: 0.08\n",
"episode: 7475, score: 5.0, epsilon: 0.08\n",
"episode: 7480, score: 3.0, epsilon: 0.08\n",
"episode: 7485, score: 3.0, epsilon: 0.08\n",
"episode: 7490, score: 3.0, epsilon: 0.08\n",
"episode: 7495, score: 0.0, epsilon: 0.08\n",
"marking, episode: 7500, score: 3.0, mean_score: 0.97, std_score: 1.31\n",
"episode: 7500, score: 3.0, epsilon: 0.08\n",
"episode: 7505, score: 0.0, epsilon: 0.08\n",
"episode: 7510, score: 0.0, epsilon: 0.08\n",
"episode: 7515, score: 1.0, epsilon: 0.08\n",
"episode: 7520, score: 0.0, epsilon: 0.08\n",
"episode: 7525, score: 0.0, epsilon: 0.08\n",
"episode: 7530, score: 0.0, epsilon: 0.08\n",
"episode: 7535, score: 0.0, epsilon: 0.08\n",
"episode: 7540, score: 0.0, epsilon: 0.08\n",
"episode: 7545, score: 0.0, epsilon: 0.08\n",
"episode: 7550, score: 1.0, epsilon: 0.08\n",
"episode: 7555, score: 0.0, epsilon: 0.08\n",
"episode: 7560, score: 0.0, epsilon: 0.08\n",
"episode: 7565, score: 1.0, epsilon: 0.08\n",
"episode: 7570, score: 0.0, epsilon: 0.08\n",
"episode: 7575, score: 4.0, epsilon: 0.08\n",
"episode: 7580, score: 0.0, epsilon: 0.08\n",
"episode: 7585, score: 4.0, epsilon: 0.08\n",
"episode: 7590, score: 0.0, epsilon: 0.08\n",
"episode: 7595, score: 1.0, epsilon: 0.08\n",
"marking, episode: 7600, score: 0.0, mean_score: 1.08, std_score: 1.44\n",
"episode: 7600, score: 0.0, epsilon: 0.08\n",
"episode: 7605, score: 3.0, epsilon: 0.08\n",
"episode: 7610, score: 0.0, epsilon: 0.08\n",
"episode: 7615, score: 0.0, epsilon: 0.08\n",
"episode: 7620, score: 1.0, epsilon: 0.08\n",
"episode: 7625, score: 0.0, epsilon: 0.08\n",
"episode: 7630, score: 2.0, epsilon: 0.08\n",
"episode: 7635, score: 2.0, epsilon: 0.08\n",
"episode: 7640, score: 1.0, epsilon: 0.08\n",
"episode: 7645, score: 2.0, epsilon: 0.08\n",
"episode: 7650, score: 0.0, epsilon: 0.08\n",
"episode: 7655, score: 0.0, epsilon: 0.08\n",
"episode: 7660, score: 0.0, epsilon: 0.08\n",
"episode: 7665, score: 3.0, epsilon: 0.08\n",
"episode: 7670, score: 1.0, epsilon: 0.08\n",
"episode: 7675, score: 0.0, epsilon: 0.08\n",
"episode: 7680, score: 1.0, epsilon: 0.08\n",
"episode: 7685, score: 0.0, epsilon: 0.08\n",
"episode: 7690, score: 0.0, epsilon: 0.08\n",
"episode: 7695, score: 0.0, epsilon: 0.08\n",
"marking, episode: 7700, score: 2.0, mean_score: 0.91, std_score: 1.07\n",
"episode: 7700, score: 2.0, epsilon: 0.08\n",
"episode: 7705, score: 0.0, epsilon: 0.08\n",
"episode: 7710, score: 2.0, epsilon: 0.08\n",
"episode: 7715, score: 1.0, epsilon: 0.08\n",
"episode: 7720, score: 0.0, epsilon: 0.08\n",
"episode: 7725, score: 0.0, epsilon: 0.08\n",
"episode: 7730, score: 0.0, epsilon: 0.08\n",
"episode: 7735, score: 2.0, epsilon: 0.08\n",
"episode: 7740, score: 1.0, epsilon: 0.08\n",
"episode: 7745, score: 0.0, epsilon: 0.08\n",
"episode: 7750, score: 1.0, epsilon: 0.08\n",
"episode: 7755, score: 0.0, epsilon: 0.08\n",
"episode: 7760, score: 0.0, epsilon: 0.08\n",
"episode: 7765, score: 4.0, epsilon: 0.08\n",
"episode: 7770, score: 1.0, epsilon: 0.08\n",
"episode: 7775, score: 1.0, epsilon: 0.08\n",
"episode: 7780, score: 3.0, epsilon: 0.08\n",
"episode: 7785, score: 0.0, epsilon: 0.08\n",
"episode: 7790, score: 0.0, epsilon: 0.08\n",
"episode: 7795, score: 2.0, epsilon: 0.08\n",
"marking, episode: 7800, score: 0.0, mean_score: 1.08, std_score: 1.35\n",
"episode: 7800, score: 0.0, epsilon: 0.08\n",
"episode: 7805, score: 2.0, epsilon: 0.08\n",
"episode: 7810, score: 0.0, epsilon: 0.08\n",
"episode: 7815, score: 0.0, epsilon: 0.08\n",
"episode: 7820, score: 0.0, epsilon: 0.08\n",
"episode: 7825, score: 1.0, epsilon: 0.08\n",
"episode: 7830, score: 6.0, epsilon: 0.08\n",
"episode: 7835, score: 1.0, epsilon: 0.08\n",
"episode: 7840, score: 0.0, epsilon: 0.08\n",
"episode: 7845, score: 0.0, epsilon: 0.08\n",
"episode: 7850, score: 0.0, epsilon: 0.08\n",
"episode: 7855, score: 1.0, epsilon: 0.08\n",
"episode: 7860, score: 0.0, epsilon: 0.08\n",
"episode: 7865, score: 2.0, epsilon: 0.08\n",
"episode: 7870, score: 3.0, epsilon: 0.08\n",
"episode: 7875, score: 1.0, epsilon: 0.08\n",
"episode: 7880, score: 0.0, epsilon: 0.08\n",
"episode: 7885, score: 1.0, epsilon: 0.08\n",
"episode: 7890, score: 0.0, epsilon: 0.08\n",
"episode: 7895, score: 0.0, epsilon: 0.08\n",
"marking, episode: 7900, score: 0.0, mean_score: 1.27, std_score: 1.62\n",
"episode: 7900, score: 0.0, epsilon: 0.08\n",
"episode: 7905, score: 6.0, epsilon: 0.08\n",
"episode: 7910, score: 0.0, epsilon: 0.08\n",
"episode: 7915, score: 1.0, epsilon: 0.08\n",
"episode: 7920, score: 0.0, epsilon: 0.08\n",
"episode: 7925, score: 0.0, epsilon: 0.08\n",
"episode: 7930, score: 2.0, epsilon: 0.08\n",
"episode: 7935, score: 0.0, epsilon: 0.08\n",
"episode: 7940, score: 1.0, epsilon: 0.08\n",
"episode: 7945, score: 0.0, epsilon: 0.08\n",
"episode: 7950, score: 0.0, epsilon: 0.08\n",
"episode: 7955, score: 3.0, epsilon: 0.08\n",
"episode: 7960, score: 0.0, epsilon: 0.08\n",
"episode: 7965, score: 0.0, epsilon: 0.08\n",
"episode: 7970, score: 4.0, epsilon: 0.08\n",
"episode: 7975, score: 0.0, epsilon: 0.08\n",
"episode: 7980, score: 4.0, epsilon: 0.08\n",
"episode: 7985, score: 0.0, epsilon: 0.08\n",
"episode: 7990, score: 2.0, epsilon: 0.08\n",
"episode: 7995, score: 6.0, epsilon: 0.08\n",
"marking, episode: 8000, score: 1.0, mean_score: 1.10, std_score: 1.43\n",
"episode: 8000, score: 1.0, epsilon: 0.08\n",
"episode: 8005, score: 2.0, epsilon: 0.08\n",
"episode: 8010, score: 1.0, epsilon: 0.08\n",
"episode: 8015, score: 3.0, epsilon: 0.08\n",
"episode: 8020, score: 3.0, epsilon: 0.08\n",
"episode: 8025, score: 0.0, epsilon: 0.08\n",
"episode: 8030, score: 0.0, epsilon: 0.08\n",
"episode: 8035, score: 3.0, epsilon: 0.08\n",
"episode: 8040, score: 0.0, epsilon: 0.08\n",
"episode: 8045, score: 0.0, epsilon: 0.08\n",
"episode: 8050, score: 1.0, epsilon: 0.08\n",
"episode: 8055, score: 0.0, epsilon: 0.08\n",
"episode: 8060, score: 0.0, epsilon: 0.08\n",
"episode: 8065, score: 0.0, epsilon: 0.08\n",
"episode: 8070, score: 0.0, epsilon: 0.08\n",
"episode: 8075, score: 0.0, epsilon: 0.08\n",
"episode: 8080, score: 2.0, epsilon: 0.08\n",
"episode: 8085, score: 0.0, epsilon: 0.08\n",
"episode: 8090, score: 1.0, epsilon: 0.08\n",
"episode: 8095, score: 1.0, epsilon: 0.08\n",
"marking, episode: 8100, score: 2.0, mean_score: 1.13, std_score: 1.30\n",
"episode: 8100, score: 2.0, epsilon: 0.08\n",
"episode: 8105, score: 1.0, epsilon: 0.08\n",
"episode: 8110, score: 0.0, epsilon: 0.08\n",
"episode: 8115, score: 2.0, epsilon: 0.08\n",
"episode: 8120, score: 2.0, epsilon: 0.08\n",
"episode: 8125, score: 0.0, epsilon: 0.08\n",
"episode: 8130, score: 0.0, epsilon: 0.08\n",
"episode: 8135, score: 3.0, epsilon: 0.08\n",
"episode: 8140, score: 0.0, epsilon: 0.08\n",
"episode: 8145, score: 2.0, epsilon: 0.08\n",
"episode: 8150, score: 0.0, epsilon: 0.08\n",
"episode: 8155, score: 0.0, epsilon: 0.08\n",
"episode: 8160, score: 1.0, epsilon: 0.08\n",
"episode: 8165, score: 2.0, epsilon: 0.08\n",
"episode: 8170, score: 0.0, epsilon: 0.08\n",
"episode: 8175, score: 0.0, epsilon: 0.08\n",
"episode: 8180, score: 0.0, epsilon: 0.08\n",
"episode: 8185, score: 0.0, epsilon: 0.08\n",
"episode: 8190, score: 0.0, epsilon: 0.08\n",
"episode: 8195, score: 2.0, epsilon: 0.08\n",
"marking, episode: 8200, score: 0.0, mean_score: 1.29, std_score: 1.52\n",
"episode: 8200, score: 0.0, epsilon: 0.08\n",
"episode: 8205, score: 0.0, epsilon: 0.08\n",
"episode: 8210, score: 4.0, epsilon: 0.08\n",
"episode: 8215, score: 0.0, epsilon: 0.08\n",
"episode: 8220, score: 4.0, epsilon: 0.08\n",
"episode: 8225, score: 0.0, epsilon: 0.08\n",
"episode: 8230, score: 3.0, epsilon: 0.08\n",
"episode: 8235, score: 4.0, epsilon: 0.08\n",
"episode: 8240, score: 0.0, epsilon: 0.08\n",
"episode: 8245, score: 2.0, epsilon: 0.08\n",
"episode: 8250, score: 1.0, epsilon: 0.08\n",
"episode: 8255, score: 0.0, epsilon: 0.08\n",
"episode: 8260, score: 0.0, epsilon: 0.08\n",
"episode: 8265, score: 0.0, epsilon: 0.08\n",
"episode: 8270, score: 0.0, epsilon: 0.08\n",
"episode: 8275, score: 0.0, epsilon: 0.08\n",
"episode: 8280, score: 2.0, epsilon: 0.08\n",
"episode: 8285, score: 0.0, epsilon: 0.08\n",
"episode: 8290, score: 1.0, epsilon: 0.08\n",
"episode: 8295, score: 1.0, epsilon: 0.08\n",
"marking, episode: 8300, score: 0.0, mean_score: 1.15, std_score: 1.65\n",
"episode: 8300, score: 0.0, epsilon: 0.08\n",
"episode: 8305, score: 6.0, epsilon: 0.08\n",
"episode: 8310, score: 3.0, epsilon: 0.08\n",
"episode: 8315, score: 0.0, epsilon: 0.08\n",
"episode: 8320, score: 0.0, epsilon: 0.08\n",
"episode: 8325, score: 0.0, epsilon: 0.08\n",
"episode: 8330, score: 0.0, epsilon: 0.08\n",
"episode: 8335, score: 1.0, epsilon: 0.08\n",
"episode: 8340, score: 0.0, epsilon: 0.08\n",
"episode: 8345, score: 0.0, epsilon: 0.08\n",
"episode: 8350, score: 1.0, epsilon: 0.08\n",
"episode: 8355, score: 6.0, epsilon: 0.08\n",
"episode: 8360, score: 0.0, epsilon: 0.08\n",
"episode: 8365, score: 0.0, epsilon: 0.08\n",
"episode: 8370, score: 5.0, epsilon: 0.08\n",
"episode: 8375, score: 0.0, epsilon: 0.08\n",
"episode: 8380, score: 1.0, epsilon: 0.08\n",
"episode: 8385, score: 1.0, epsilon: 0.08\n",
"episode: 8390, score: 1.0, epsilon: 0.08\n",
"episode: 8395, score: 0.0, epsilon: 0.08\n",
"marking, episode: 8400, score: 1.0, mean_score: 1.00, std_score: 1.69\n",
"episode: 8400, score: 1.0, epsilon: 0.08\n",
"episode: 8405, score: 0.0, epsilon: 0.08\n",
"episode: 8410, score: 4.0, epsilon: 0.08\n",
"episode: 8415, score: 2.0, epsilon: 0.08\n",
"episode: 8420, score: 2.0, epsilon: 0.08\n",
"episode: 8425, score: 0.0, epsilon: 0.08\n",
"episode: 8430, score: 1.0, epsilon: 0.08\n",
"episode: 8435, score: 3.0, epsilon: 0.08\n",
"episode: 8440, score: 4.0, epsilon: 0.08\n",
"episode: 8445, score: 0.0, epsilon: 0.08\n",
"episode: 8450, score: 1.0, epsilon: 0.08\n",
"episode: 8455, score: 4.0, epsilon: 0.08\n",
"episode: 8460, score: 1.0, epsilon: 0.08\n",
"episode: 8465, score: 0.0, epsilon: 0.08\n",
"episode: 8470, score: 1.0, epsilon: 0.08\n",
"episode: 8475, score: 0.0, epsilon: 0.08\n",
"episode: 8480, score: 0.0, epsilon: 0.08\n",
"episode: 8485, score: 0.0, epsilon: 0.08\n",
"episode: 8490, score: 3.0, epsilon: 0.08\n",
"episode: 8495, score: 4.0, epsilon: 0.08\n",
"marking, episode: 8500, score: 1.0, mean_score: 0.92, std_score: 1.17\n",
"episode: 8500, score: 1.0, epsilon: 0.08\n",
"episode: 8505, score: 0.0, epsilon: 0.08\n",
"episode: 8510, score: 0.0, epsilon: 0.08\n",
"episode: 8515, score: 2.0, epsilon: 0.08\n",
"episode: 8520, score: 0.0, epsilon: 0.08\n",
"episode: 8525, score: 2.0, epsilon: 0.08\n",
"episode: 8530, score: 0.0, epsilon: 0.08\n",
"episode: 8535, score: 0.0, epsilon: 0.08\n",
"episode: 8540, score: 0.0, epsilon: 0.08\n",
"episode: 8545, score: 1.0, epsilon: 0.08\n",
"episode: 8550, score: 0.0, epsilon: 0.08\n",
"episode: 8555, score: 0.0, epsilon: 0.08\n",
"episode: 8560, score: 1.0, epsilon: 0.08\n",
"episode: 8565, score: 2.0, epsilon: 0.08\n",
"episode: 8570, score: 1.0, epsilon: 0.08\n",
"episode: 8575, score: 0.0, epsilon: 0.08\n",
"episode: 8580, score: 6.0, epsilon: 0.08\n",
"episode: 8585, score: 0.0, epsilon: 0.08\n",
"episode: 8590, score: 0.0, epsilon: 0.08\n",
"episode: 8595, score: 0.0, epsilon: 0.08\n",
"marking, episode: 8600, score: 2.0, mean_score: 0.89, std_score: 1.30\n",
"episode: 8600, score: 2.0, epsilon: 0.08\n",
"episode: 8605, score: 0.0, epsilon: 0.08\n",
"episode: 8610, score: 1.0, epsilon: 0.08\n",
"episode: 8615, score: 5.0, epsilon: 0.08\n",
"episode: 8620, score: 1.0, epsilon: 0.08\n",
"episode: 8625, score: 0.0, epsilon: 0.08\n",
"episode: 8630, score: 1.0, epsilon: 0.08\n",
"episode: 8635, score: 1.0, epsilon: 0.08\n",
"episode: 8640, score: 0.0, epsilon: 0.08\n",
"episode: 8645, score: 0.0, epsilon: 0.08\n",
"episode: 8650, score: 0.0, epsilon: 0.08\n",
"episode: 8655, score: 0.0, epsilon: 0.08\n",
"episode: 8660, score: 1.0, epsilon: 0.08\n",
"episode: 8665, score: 1.0, epsilon: 0.08\n",
"episode: 8670, score: 3.0, epsilon: 0.08\n",
"episode: 8675, score: 3.0, epsilon: 0.08\n",
"episode: 8680, score: 0.0, epsilon: 0.08\n",
"episode: 8685, score: 0.0, epsilon: 0.08\n",
"episode: 8690, score: 0.0, epsilon: 0.08\n",
"episode: 8695, score: 1.0, epsilon: 0.08\n",
"marking, episode: 8700, score: 1.0, mean_score: 1.16, std_score: 1.43\n",
"episode: 8700, score: 1.0, epsilon: 0.08\n",
"episode: 8705, score: 2.0, epsilon: 0.08\n",
"episode: 8710, score: 2.0, epsilon: 0.08\n",
"episode: 8715, score: 3.0, epsilon: 0.08\n",
"episode: 8720, score: 0.0, epsilon: 0.08\n",
"episode: 8725, score: 0.0, epsilon: 0.08\n",
"episode: 8730, score: 1.0, epsilon: 0.08\n",
"episode: 8735, score: 0.0, epsilon: 0.08\n",
"episode: 8740, score: 1.0, epsilon: 0.08\n",
"episode: 8745, score: 2.0, epsilon: 0.08\n",
"episode: 8750, score: 0.0, epsilon: 0.08\n",
"episode: 8755, score: 0.0, epsilon: 0.08\n",
"episode: 8760, score: 1.0, epsilon: 0.08\n",
"episode: 8765, score: 0.0, epsilon: 0.08\n",
"episode: 8770, score: 0.0, epsilon: 0.08\n",
"episode: 8775, score: 2.0, epsilon: 0.08\n",
"episode: 8780, score: 1.0, epsilon: 0.08\n",
"episode: 8785, score: 4.0, epsilon: 0.08\n",
"episode: 8790, score: 3.0, epsilon: 0.08\n",
"episode: 8795, score: 6.0, epsilon: 0.08\n",
"marking, episode: 8800, score: 1.0, mean_score: 0.99, std_score: 1.32\n",
"episode: 8800, score: 1.0, epsilon: 0.08\n",
"episode: 8805, score: 0.0, epsilon: 0.08\n",
"episode: 8810, score: 1.0, epsilon: 0.08\n",
"episode: 8815, score: 0.0, epsilon: 0.08\n",
"episode: 8820, score: 3.0, epsilon: 0.08\n",
"episode: 8825, score: 2.0, epsilon: 0.08\n",
"episode: 8830, score: 0.0, epsilon: 0.08\n",
"episode: 8835, score: 1.0, epsilon: 0.08\n",
"episode: 8840, score: 1.0, epsilon: 0.08\n",
"episode: 8845, score: 1.0, epsilon: 0.08\n",
"episode: 8850, score: 1.0, epsilon: 0.08\n",
"episode: 8855, score: 3.0, epsilon: 0.08\n",
"episode: 8860, score: 0.0, epsilon: 0.08\n",
"episode: 8865, score: 0.0, epsilon: 0.08\n",
"episode: 8870, score: 0.0, epsilon: 0.08\n",
"episode: 8875, score: 0.0, epsilon: 0.08\n",
"episode: 8880, score: 0.0, epsilon: 0.08\n",
"episode: 8885, score: 0.0, epsilon: 0.08\n",
"episode: 8890, score: 0.0, epsilon: 0.08\n",
"episode: 8895, score: 1.0, epsilon: 0.08\n",
"marking, episode: 8900, score: 1.0, mean_score: 0.84, std_score: 1.24\n",
"episode: 8900, score: 1.0, epsilon: 0.08\n",
"episode: 8905, score: 0.0, epsilon: 0.08\n",
"episode: 8910, score: 0.0, epsilon: 0.08\n",
"episode: 8915, score: 1.0, epsilon: 0.08\n",
"episode: 8920, score: 0.0, epsilon: 0.08\n",
"episode: 8925, score: 0.0, epsilon: 0.08\n",
"episode: 8930, score: 1.0, epsilon: 0.08\n",
"episode: 8935, score: 0.0, epsilon: 0.08\n",
"episode: 8940, score: 0.0, epsilon: 0.08\n",
"episode: 8945, score: 1.0, epsilon: 0.08\n",
"episode: 8950, score: 4.0, epsilon: 0.08\n",
"episode: 8955, score: 0.0, epsilon: 0.08\n",
"episode: 8960, score: 3.0, epsilon: 0.08\n",
"episode: 8965, score: 0.0, epsilon: 0.08\n",
"episode: 8970, score: 2.0, epsilon: 0.08\n",
"episode: 8975, score: 4.0, epsilon: 0.08\n",
"episode: 8980, score: 2.0, epsilon: 0.08\n",
"episode: 8985, score: 4.0, epsilon: 0.08\n",
"episode: 8990, score: 2.0, epsilon: 0.08\n",
"episode: 8995, score: 0.0, epsilon: 0.08\n",
"marking, episode: 9000, score: 0.0, mean_score: 1.15, std_score: 1.44\n",
"episode: 9000, score: 0.0, epsilon: 0.08\n",
"episode: 9005, score: 4.0, epsilon: 0.08\n",
"episode: 9010, score: 0.0, epsilon: 0.08\n",
"episode: 9015, score: 1.0, epsilon: 0.08\n",
"episode: 9020, score: 0.0, epsilon: 0.08\n",
"episode: 9025, score: 1.0, epsilon: 0.08\n",
"episode: 9030, score: 0.0, epsilon: 0.08\n",
"episode: 9035, score: 4.0, epsilon: 0.08\n",
"episode: 9040, score: 3.0, epsilon: 0.08\n",
"episode: 9045, score: 0.0, epsilon: 0.08\n",
"episode: 9050, score: 0.0, epsilon: 0.08\n",
"episode: 9055, score: 0.0, epsilon: 0.08\n",
"episode: 9060, score: 0.0, epsilon: 0.08\n",
"episode: 9065, score: 6.0, epsilon: 0.08\n",
"episode: 9070, score: 1.0, epsilon: 0.08\n",
"episode: 9075, score: 0.0, epsilon: 0.08\n",
"episode: 9080, score: 0.0, epsilon: 0.08\n",
"episode: 9085, score: 2.0, epsilon: 0.08\n",
"episode: 9090, score: 0.0, epsilon: 0.08\n",
"episode: 9095, score: 0.0, epsilon: 0.08\n",
"marking, episode: 9100, score: 0.0, mean_score: 1.04, std_score: 1.41\n",
"episode: 9100, score: 0.0, epsilon: 0.08\n",
"episode: 9105, score: 0.0, epsilon: 0.08\n",
"episode: 9110, score: 0.0, epsilon: 0.08\n",
"episode: 9115, score: 0.0, epsilon: 0.08\n",
"episode: 9120, score: 0.0, epsilon: 0.08\n",
"episode: 9125, score: 1.0, epsilon: 0.08\n",
"episode: 9130, score: 1.0, epsilon: 0.08\n",
"episode: 9135, score: 1.0, epsilon: 0.08\n",
"episode: 9140, score: 4.0, epsilon: 0.08\n",
"episode: 9145, score: 0.0, epsilon: 0.08\n",
"episode: 9150, score: 0.0, epsilon: 0.08\n",
"episode: 9155, score: 2.0, epsilon: 0.08\n",
"episode: 9160, score: 0.0, epsilon: 0.08\n",
"episode: 9165, score: 1.0, epsilon: 0.08\n",
"episode: 9170, score: 0.0, epsilon: 0.08\n",
"episode: 9175, score: 0.0, epsilon: 0.08\n",
"episode: 9180, score: 0.0, epsilon: 0.08\n",
"episode: 9185, score: 0.0, epsilon: 0.08\n",
"episode: 9190, score: 6.0, epsilon: 0.08\n",
"episode: 9195, score: 1.0, epsilon: 0.08\n",
"marking, episode: 9200, score: 1.0, mean_score: 0.99, std_score: 1.31\n",
"episode: 9200, score: 1.0, epsilon: 0.08\n",
"episode: 9205, score: 0.0, epsilon: 0.08\n",
"episode: 9210, score: 0.0, epsilon: 0.08\n",
"episode: 9215, score: 1.0, epsilon: 0.08\n",
"episode: 9220, score: 0.0, epsilon: 0.08\n",
"episode: 9225, score: 0.0, epsilon: 0.08\n",
"episode: 9230, score: 0.0, epsilon: 0.08\n",
"episode: 9235, score: 1.0, epsilon: 0.08\n",
"episode: 9240, score: 0.0, epsilon: 0.08\n",
"episode: 9245, score: 4.0, epsilon: 0.08\n",
"episode: 9250, score: 0.0, epsilon: 0.08\n",
"episode: 9255, score: 2.0, epsilon: 0.08\n",
"episode: 9260, score: 0.0, epsilon: 0.08\n",
"episode: 9265, score: 2.0, epsilon: 0.08\n",
"episode: 9270, score: 1.0, epsilon: 0.08\n",
"episode: 9275, score: 0.0, epsilon: 0.08\n",
"episode: 9280, score: 0.0, epsilon: 0.08\n",
"episode: 9285, score: 0.0, epsilon: 0.08\n",
"episode: 9290, score: 0.0, epsilon: 0.08\n",
"episode: 9295, score: 4.0, epsilon: 0.08\n",
"marking, episode: 9300, score: 3.0, mean_score: 0.99, std_score: 1.33\n",
"episode: 9300, score: 3.0, epsilon: 0.08\n",
"episode: 9305, score: 0.0, epsilon: 0.08\n",
"episode: 9310, score: 0.0, epsilon: 0.08\n",
"episode: 9315, score: 1.0, epsilon: 0.08\n",
"episode: 9320, score: 0.0, epsilon: 0.08\n",
"episode: 9325, score: 1.0, epsilon: 0.08\n",
"episode: 9330, score: 1.0, epsilon: 0.08\n",
"episode: 9335, score: 2.0, epsilon: 0.08\n",
"episode: 9340, score: 1.0, epsilon: 0.08\n",
"episode: 9345, score: 1.0, epsilon: 0.08\n",
"episode: 9350, score: 3.0, epsilon: 0.08\n",
"episode: 9355, score: 1.0, epsilon: 0.08\n",
"episode: 9360, score: 2.0, epsilon: 0.08\n",
"episode: 9365, score: 1.0, epsilon: 0.08\n",
"episode: 9370, score: 0.0, epsilon: 0.08\n",
"episode: 9375, score: 0.0, epsilon: 0.08\n",
"episode: 9380, score: 0.0, epsilon: 0.08\n",
"episode: 9385, score: 0.0, epsilon: 0.08\n",
"episode: 9390, score: 0.0, epsilon: 0.08\n",
"episode: 9395, score: 0.0, epsilon: 0.08\n",
"marking, episode: 9400, score: 1.0, mean_score: 0.87, std_score: 1.11\n",
"episode: 9400, score: 1.0, epsilon: 0.08\n",
"episode: 9405, score: 1.0, epsilon: 0.08\n",
"episode: 9410, score: 3.0, epsilon: 0.08\n",
"episode: 9415, score: 0.0, epsilon: 0.08\n",
"episode: 9420, score: 0.0, epsilon: 0.08\n",
"episode: 9425, score: 1.0, epsilon: 0.08\n",
"episode: 9430, score: 5.0, epsilon: 0.08\n",
"episode: 9435, score: 2.0, epsilon: 0.08\n",
"episode: 9440, score: 1.0, epsilon: 0.08\n",
"episode: 9445, score: 0.0, epsilon: 0.08\n",
"episode: 9450, score: 3.0, epsilon: 0.08\n",
"episode: 9455, score: 0.0, epsilon: 0.08\n",
"episode: 9460, score: 0.0, epsilon: 0.08\n",
"episode: 9465, score: 1.0, epsilon: 0.08\n",
"episode: 9470, score: 5.0, epsilon: 0.08\n",
"episode: 9475, score: 0.0, epsilon: 0.08\n",
"episode: 9480, score: 1.0, epsilon: 0.08\n",
"episode: 9485, score: 0.0, epsilon: 0.08\n",
"episode: 9490, score: 0.0, epsilon: 0.08\n",
"episode: 9495, score: 0.0, epsilon: 0.08\n",
"marking, episode: 9500, score: 2.0, mean_score: 1.12, std_score: 1.56\n",
"episode: 9500, score: 2.0, epsilon: 0.08\n",
"episode: 9505, score: 0.0, epsilon: 0.08\n",
"episode: 9510, score: 0.0, epsilon: 0.08\n",
"episode: 9515, score: 1.0, epsilon: 0.08\n",
"episode: 9520, score: 0.0, epsilon: 0.08\n",
"episode: 9525, score: 1.0, epsilon: 0.08\n",
"episode: 9530, score: 2.0, epsilon: 0.08\n",
"episode: 9535, score: 1.0, epsilon: 0.08\n",
"episode: 9540, score: 0.0, epsilon: 0.08\n",
"episode: 9545, score: 2.0, epsilon: 0.08\n",
"episode: 9550, score: 0.0, epsilon: 0.08\n",
"episode: 9555, score: 0.0, epsilon: 0.08\n",
"episode: 9560, score: 2.0, epsilon: 0.08\n",
"episode: 9565, score: 0.0, epsilon: 0.08\n",
"episode: 9570, score: 4.0, epsilon: 0.08\n",
"episode: 9575, score: 0.0, epsilon: 0.08\n",
"episode: 9580, score: 1.0, epsilon: 0.08\n",
"episode: 9585, score: 1.0, epsilon: 0.08\n",
"episode: 9590, score: 0.0, epsilon: 0.08\n",
"episode: 9595, score: 1.0, epsilon: 0.08\n",
"marking, episode: 9600, score: 0.0, mean_score: 0.93, std_score: 1.46\n",
"episode: 9600, score: 0.0, epsilon: 0.08\n",
"episode: 9605, score: 2.0, epsilon: 0.08\n",
"episode: 9610, score: 2.0, epsilon: 0.08\n",
"episode: 9615, score: 1.0, epsilon: 0.08\n",
"episode: 9620, score: 2.0, epsilon: 0.08\n",
"episode: 9625, score: 0.0, epsilon: 0.08\n",
"episode: 9630, score: 0.0, epsilon: 0.08\n",
"episode: 9635, score: 0.0, epsilon: 0.08\n",
"episode: 9640, score: 2.0, epsilon: 0.08\n",
"episode: 9645, score: 0.0, epsilon: 0.08\n",
"episode: 9650, score: 2.0, epsilon: 0.08\n",
"episode: 9655, score: 0.0, epsilon: 0.08\n",
"episode: 9660, score: 0.0, epsilon: 0.08\n",
"episode: 9665, score: 0.0, epsilon: 0.08\n",
"episode: 9670, score: 1.0, epsilon: 0.08\n",
"episode: 9675, score: 0.0, epsilon: 0.08\n",
"episode: 9680, score: 0.0, epsilon: 0.08\n",
"episode: 9685, score: 1.0, epsilon: 0.08\n",
"episode: 9690, score: 0.0, epsilon: 0.08\n",
"episode: 9695, score: 0.0, epsilon: 0.08\n",
"marking, episode: 9700, score: 1.0, mean_score: 1.07, std_score: 1.42\n",
"episode: 9700, score: 1.0, epsilon: 0.08\n",
"episode: 9705, score: 0.0, epsilon: 0.08\n",
"episode: 9710, score: 1.0, epsilon: 0.08\n",
"episode: 9715, score: 2.0, epsilon: 0.08\n",
"episode: 9720, score: 4.0, epsilon: 0.08\n",
"episode: 9725, score: 3.0, epsilon: 0.08\n",
"episode: 9730, score: 0.0, epsilon: 0.08\n",
"episode: 9735, score: 4.0, epsilon: 0.08\n",
"episode: 9740, score: 0.0, epsilon: 0.08\n",
"episode: 9745, score: 1.0, epsilon: 0.08\n",
"episode: 9750, score: 1.0, epsilon: 0.08\n",
"episode: 9755, score: 1.0, epsilon: 0.08\n",
"episode: 9760, score: 1.0, epsilon: 0.08\n",
"episode: 9765, score: 0.0, epsilon: 0.08\n",
"episode: 9770, score: 0.0, epsilon: 0.08\n",
"episode: 9775, score: 1.0, epsilon: 0.08\n",
"episode: 9780, score: 0.0, epsilon: 0.08\n",
"episode: 9785, score: 0.0, epsilon: 0.08\n",
"episode: 9790, score: 1.0, epsilon: 0.08\n",
"episode: 9795, score: 3.0, epsilon: 0.08\n",
"marking, episode: 9800, score: 0.0, mean_score: 1.05, std_score: 1.41\n",
"episode: 9800, score: 0.0, epsilon: 0.08\n",
"episode: 9805, score: 0.0, epsilon: 0.08\n",
"episode: 9810, score: 0.0, epsilon: 0.08\n",
"episode: 9815, score: 2.0, epsilon: 0.08\n",
"episode: 9820, score: 2.0, epsilon: 0.08\n",
"episode: 9825, score: 0.0, epsilon: 0.08\n",
"episode: 9830, score: 0.0, epsilon: 0.08\n",
"episode: 9835, score: 1.0, epsilon: 0.08\n",
"episode: 9840, score: 1.0, epsilon: 0.08\n",
"episode: 9845, score: 0.0, epsilon: 0.08\n",
"episode: 9850, score: 0.0, epsilon: 0.08\n",
"episode: 9855, score: 0.0, epsilon: 0.08\n",
"episode: 9860, score: 1.0, epsilon: 0.08\n",
"episode: 9865, score: 2.0, epsilon: 0.08\n",
"episode: 9870, score: 4.0, epsilon: 0.08\n",
"episode: 9875, score: 0.0, epsilon: 0.08\n",
"episode: 9880, score: 1.0, epsilon: 0.08\n",
"episode: 9885, score: 1.0, epsilon: 0.08\n",
"episode: 9890, score: 0.0, epsilon: 0.08\n",
"episode: 9895, score: 0.0, epsilon: 0.08\n",
"marking, episode: 9900, score: 3.0, mean_score: 0.89, std_score: 1.36\n",
"episode: 9900, score: 3.0, epsilon: 0.08\n",
"episode: 9905, score: 0.0, epsilon: 0.08\n",
"episode: 9910, score: 1.0, epsilon: 0.08\n",
"episode: 9915, score: 0.0, epsilon: 0.08\n",
"episode: 9920, score: 0.0, epsilon: 0.08\n",
"episode: 9925, score: 0.0, epsilon: 0.08\n",
"episode: 9930, score: 2.0, epsilon: 0.08\n",
"episode: 9935, score: 0.0, epsilon: 0.08\n",
"episode: 9940, score: 0.0, epsilon: 0.08\n",
"episode: 9945, score: 1.0, epsilon: 0.08\n",
"episode: 9950, score: 0.0, epsilon: 0.08\n",
"episode: 9955, score: 6.0, epsilon: 0.08\n",
"episode: 9960, score: 0.0, epsilon: 0.08\n",
"episode: 9965, score: 1.0, epsilon: 0.08\n",
"episode: 9970, score: 0.0, epsilon: 0.08\n",
"episode: 9975, score: 0.0, epsilon: 0.08\n",
"episode: 9980, score: 3.0, epsilon: 0.08\n",
"episode: 9985, score: 0.0, epsilon: 0.08\n",
"episode: 9990, score: 1.0, epsilon: 0.08\n",
"episode: 9995, score: 0.0, epsilon: 0.08\n",
"marking, episode: 10000, score: 0.0, mean_score: 0.99, std_score: 1.52\n",
"episode: 10000, score: 0.0, epsilon: 0.08\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "0dlW1KVhsLMR"
},
"source": [
"scores = np.array(scores)\n",
"opt_scores = np.array(opt_scores)\n",
"rand_scores = np.array(rand_scores)"
],
"execution_count": 25,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "MNOKaBOD524W",
"outputId": "f2340d74-2d1f-45a9-bd9c-7c657666219c",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
}
},
"source": [
"[x.mean() for x in [scores, opt_scores, rand_scores]]"
],
"execution_count": 26,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"[2.085991400859914, 2.423157684231577, 0.9922007799220078]"
]
},
"metadata": {
"tags": []
},
"execution_count": 26
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "7tAF0aGR3Aw5"
},
"source": [
""
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "9-sKhOL_7I38"
},
"source": [
""
],
"execution_count": 26,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "9JpMHLRz7Lmo"
},
"source": [
"def moving_average(a, n=3) :\n",
" ret = np.cumsum(a, dtype=float)\n",
" ret[n:] = ret[n:] - ret[:-n]\n",
" return ret[n - 1:] / n"
],
"execution_count": 27,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "FG0hhRwG7hpe"
},
"source": [
""
],
"execution_count": 27,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "b3WnEUt58PzD",
"outputId": "1d21a587-2a65-4ca6-c048-f6ff583d3987",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 265
}
},
"source": [
"plt.plot(moving_average(scores, n=500))\n",
"plt.plot(moving_average(opt_scores, n=500))\n",
"plt.plot(moving_average(rand_scores, n=500))\n",
"plt.show()"
],
"execution_count": 28,
"outputs": [
{
"output_type": "display_data",
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOyddXgU19eA3xshIQme4BKcAkFSilO0FKhQKlSAUkHar+5QpdRoqf+oG1AKNUpLcW8p7gQtFlyCJYEQ3fv9MSszu7OSZDcb0vs+T56duXNn5u5u9sy55x4RUkoUCoVCUXIJCfYAFAqFQhFYlKBXKBSKEo4S9AqFQlHCUYJeoVAoSjhK0CsUCkUJJyzYAzAjNjZWxsfHB3sYCoVCcdmwYcOG01LKOLNjxVLQx8fHs379+mAPQ6FQKC4bhBAH3R1TphuFQqEo4ShBr1AoFCUcJegVCoWihKMEvUKhUJRwlKBXKBSKEo5XrxshRC1gMlAFkMCXUsqPnPo8AwzSXfMKIE5KeVYIkQykA3lArpSyjf+Gr1AoFApv+OJemQs8JaXcKIQoA2wQQiyUUu6wdZBSjgfGAwghbgCekFKe1V2ju5TytD8HrlAoFArf8Gq6kVIel1JutG6nAzuBGh5OuROY5p/hKRQKRQH5dz6cPxzsURQL8mWjF0LEA62BNW6ORwF9gOm6ZgksEEJsEEKM8HDtEUKI9UKI9SkpKfkZlkKhUBg5uBKmDoRP2gV7JMUCnwW9ECIGTYA/LqVMc9PtBmCFk9mms5QyEegLPCSEuNrsRCnll1LKNlLKNnFxplG8CoVC4Rvf9dVecy6675OZWjRjKQb4JOiFEOFoQv4HKeVvHrregZPZRkp51Pp6CpgBtC3YUBUKhaIA5FyCbCeBfy4ZxtWG9d8FZUhFjVdBL4QQwDfATinl+x76lQO6An/o2qKtC7gIIaKB3sC2wg5aoVAoPBJR1rH9/hXwZnXj8RNJ2uvuuUU3piDii9dNJ2AIkCSE2Gxtex6oDSCl/NzaNgBYIKXUPzqrADO0ZwVhwFQp5Tx/DFyhUCjcEtcEjqzVti+d016zLkBEjLa98GXtdc/8oh9bEPAq6KWU/wDCh34TgYlObfuBlgUcm0KhUBQMM/v7WzVgjLW9bA04u1/b3rsIGvQqurEFARUZq1AoShaWPM0GX6u9++PJyx37U26B+S8UydCChRL0wWTnnzCmHEy6MdgjUShKBhYLjK0IeVnQ/GbzPhdOubatmhDYcQUZJeiDydK3tNcDf8EJtUatUBSa5L91OyYW5/3LYPUnRTWaYoMS9MGkUj3H9t6FMONBSPo1eONRlEyyLsCl8zDnWZjxQLBHE1giyzu2S1eA2h2Mxyf318w6AH3fMR6b+WhAhxZMlKAPJjv/dGwf2wRbpsL0+4M3HkXJ5K0a8HYdWPsFbJkGGWe9n3O5cXqvpiTlZDjamg2AwdNh+BJo2NvRfum89fjN8KIuCn/jJPi8S9GMt4hRgj6Y1O/p2N7xh2M7N7vox6L47/BOXTi+Jdij8C9fXK0pSTZPmvsWQGgYlIqGGlfCHbo4TttCbEwchJWCXq86jp3YWnRjLkKUoA8m+xabt4+vX7TjUPz3+POxYI/Av9hSHSx8RXuNcUqjEurBk7xpyXeGUII+WNimj2ZkuUslpChRfNVT87oKBsc2wZl9wbm3v7FYHNsZ1mzoUZV8P79iPeO+LcCqBKEEfbDIvuDYdvb3TRxatGNRFD0nkuDoem372GbPfQuDlO6PlQSFwpIHYyu4tpcqU/Brnt5T8HOLKUrQB4usdMf2Fdcbj22cBN/0RlGC+V7n451zKXD30Qvz0Uegy1OO/dWfBe6+RYU77TvERLSNOmy+DfDsAej8hLb9zTXGYzmZBR9fMUEJ+mCRaf0B9h0PbUe6Hj9smvI/uKz4WMvzrSg8F3VBOxdOBO4+NhPhtW9CRBno+TKUqaa1bf0pcPctKvSLyu0e9Nw3sqwm0O+ZrW3riaoIdToZ27LSNdPaG1U8m1r1WPIC++AuIErQBwtbLo4aV2or/xHloOcrIHRfiZSwYaLjoRBMpISFLznyfCsKzrbpxv1/PgzcvWwzx7K6onBPbNderxoeuPsWBVLCzEcc+219eD9RFSG+s/mx+j2M+3pXy/3LfBvT2IrwRlVN4BcjlKAPFjZBH2ldjBt9CLo8Cc/oFsjGN9C8I2Y9UfTjc0bve/3TkOCNoyTw633G/eOb4beRgXmg20wbtqyNACGh2uu6r/x/v6LkRBKkHdW2R/4NlerDsCUw+mjBrmf7XGycO+DY/mWoeeoEPfpF4aVvFmwMAUIJ+mCg10Qinbwuoio6tm0eBCeStCnkholFMjxT9KaGnTPhwN/u+yryz9YfYf03/rvexdPaw3mSdf0n7Zj/rl1c0BcTKVdLe615pfGhll8Sh4IINT/2bkPPmv26rx3by98t+BgCgBL0wWDHH5BrteM52wrNOL1bew2m77OzYJ90Q3DGcbmj1/oGfm88Fhrhn3sc3ajFYrxT19HWtL+xT7zvEaCn0jLZcthHG3VRkJejKUrf9XG0lTbxvCkIGyeBzIPUI+bHJ/eH5e8Zv0cbllz/jCEA+FJhqpYQYqkQYocQYrsQwkXaCCG6CSFShRCbrX8v6471EULsFkLsFUKM8vcbuCz5Rec+GWby4+47vujG4itznw32CEoGtsCebqO1QJ0+bzuOSRPhURDMAvGcZ461rBU9nUvsmXDDhH/o/8kKUi/l+GFwfuC1WNg42dgmvJbM8I3wKO3VVoEq7grXPovHwoFlru1608+V9/pnPH7CF40+F3hKStkUaI9W4LupSb/lUspW1r+xAEKIUOATtMLgTYE73Zz732HHTO992tzn/ph+ehgMHtX5fBezBafLAtviaExl7bW9LsnYX++49s8vF8/Akte99zu2SXvds9Br15NpWQC0fHVBYUYWOPp/6r9r3fg/7XXaHdrrNWOhUR/XftPucm3Tm8c2FK9atF4FvZTyuJRyo3U7HdgJ1PB8lp22wF4p5X4pZTbwI9DfyzmBJ+14cBI7pZ+En3ULmSOWmfcLDdNcxULC4X6nH+Lsp+BCivl5geKsdVGqXC2oqDMHrP2yaMdREji5Q3s9d9DR9orVLJJlUhUpv6z9wrd+0dYUAWvc9z9w+iJbN60lBD/NNPyB2YPJOQ6lMOizXwJYcqCfib0918SFckUAvacKSb5s9EKIeKA1YObk3UEIsUUIMVcI0czaVgPQRyYcwfeHhH84tgmOrIfD6yDFaut+vwl86qb6jD/49X5t8dQ5KtGWHhXgmtegemv31+g7Dl4+7Zhi69nxu1+G6TMft9JeU61fZcJt2utqP2pS/xVsEdH1ujra9GYHM9tvfvjrbde2hze4tlVtob0ech8XMfjdX2jxxzXsjxxsb9uXcsFt/4CTlws/3GpsazvS1SxVGJxt/Y36Qvla3s8zS/+cFcTPygmfBb0QIgaYDjwupXT2A9sI1JFStgT+B+RbEgkhRggh1gsh1qekFFJjPbpB+8FkpsGX3eDrnvBNL/ikrSO3yIWThbtH2jE4YvIDSj8B26w55f91qoNeKsqx3ewm3+81+giMOpT/MQaK2taH5PlDJTPlbSCxpT2okmB+/NAq/9ynfB14eL1WIzW2gevx9v/n9RIxwlVrnbBkrz9GVzDG6QRumeraa4uB/r1HaSeN3izCFiDCyYliiy47Zq122qtZ0GNmmiaD1hata6tPgl4IEY4m5H+QUv7mfFxKmSalvGDdngOECyFigaOA/nFY09rmgpTySyllGyllm7i4OLMuvnFkPXzVQ3NvGufDk7igvH8FfN3DoYGdSNJ8499r7Ogz7Q44tUvLk33+EHxuDdRoPRjK1/b9XhFlNK3FNq28mKJpC2u/cp015OXCO/Vh688Ff2/u6GgtzNBGlzP/nbpwaqdr39zsYhkhGFRS/oWVVhtwdKzx2I3WUnZmi/MF4dZvIbah++N6AbZ7rsvhMxey6BbiiDqtKTTla8amAvqo+wN9rvl7ZmkPsZpt/HsPZ9ONjUG/wn3ztcXzxKGQlUZWZgZZuXnkHkty9KvdwfE7mWJSynC7VXzOeRo+TNCizYsAX7xuBPANsFNK+b6bPlWt/RBCtLVe9wywDmgohKgrhCgF3AH4sBpZCCZa7XVL3wjobexs+1VbAPu8M4wzEd6fttPyZH+o0+CuKGBa1MesC6F/va0Vk5jzNOxbYuzz6z2a//1vfop61OfG7/2a9urs4bDwZVx4PU6LEPwPsT/lAr9ucOOWBzBZ9707f4bVrKaUr3sWPHBqkTWveq12vgnA6602ZdvCo46zF7N5MuwX+/4rHUoBEE4uJK8o2Pj8SaUApfLWa/T6PPUNr9Fmsu0fgEraDKnfq1No/OI8wr7URdoeWqXlwHfHv/Md2+cPadHmB5a77+8nfNHoOwFDgB4698l+QogHhBA2w9StwDYhxBbgY+AOqZELPAzMR1vE/VlKuT0A70Pj9B7zRRIzQsI8Z/bzhF5T/W14/oVqblbB7us8XQRNG5NSE7YXUoxVq/zBJatp5rr33PfZ4+SNsWycf8dgRl6O0Wx08Yw2mwHvEYwB4q6v1vD0L1tIz3Tjhph+XHutbOJ4pp/hfdEFfhuRv5sfXgv/WPUwZ595d+hjOPb/pfnffz8A8nJJ+u1tIoTDL7xjAy2Q7/mwH2BiP21ROf1kYDNvuuP+RYG7dkioNlMYkwqdH7c3X8jKJSPb+nlYHzKlySIKY8KzU7I8SeEt3F9fmIjc7MDb8j1k49eQUv6DaZVdQ58JgGkZdaspZ06BRpdf9IudZsRUdSSQsuRqAjc8Mv/3cfZU0IdKeyIkXFvFb1jAzJTOIdqghbFfTNEWaLcHYJHWFkLvvEj14Cr4rINrf4Blb+X/Pnm5WvRt2eq+9f/iaji1A+76RctdMr4etB3h8AS68ydobOIWFyAuZedxIi2TuuI45w8mUaZxovvOek3Rhv7zPZes/V3/gWftUI9e4PoaPJSnC/DRzTY+emU4j4UZLbTReReA0jQLSdYa9N/9GC/eQkm/wq5ZcNtE38aVmaopL3rteucs7bVKAtS6yrfr+JHmr8wnJiKMba9ea/e1fy7sR57PdZgxL8hIuma9z6VPV5Ncv422HnP+kPEhbqZcFsE6V8mKjDWLTOs6SrOtVW4KT+2CprpFUGezhyekhPOHtUCNRa8Yj9nKl9mIKAtDZpiMLwcq1i/Yw8VG6YqubTYvnPMHje2F9eAAmD5Me3W2XVZp6vB2cJ5pNO7n2M7zMVpwwYvauoevRR9OWd0Up97G9o3Wqa/e3XPJ6/BmDe26RYDNdr004ilqTevu2iHtuGO7kZsH/c1OMRI+BDPZ0X/3G793309Hrpuc7c5CHoDMVEb1bUI1TISSzZvNHdPvh+0zfBNoWemaCfTteGP7T4O015NJLqcUFReycpFS2s1uXUK3sTzCkYeqeda3XML6206wegd9mAB7dQFsu2drr9UTYfhSbbsI3JRLlqDPdNIsrhoG3UdrtrX/W6V9QQMnQbWW2nF95jtP5OXAq+Xhw+bGc+qZ/KBBWwir3wMqxLseO1vIqj7PHYA7pvrW9+chmhljbwGnuu81gZPbtG2zRapRh6B2Ry3nuV6g79ZN4E57EQI21lhzo896Mt/DTJ09xrXxZJI2JV75P807Kr9mukvnNe+Iv8b7lKI2OzePSHQmudxsSP5Hu8b/rtRcer3R4jbj/pl8eLjYvifwbGazYrFInt1azffr713M8Jyp1Aox8Yj7pC3sW+r9Gsn/eO9j/31Jc0XFOW1EEbN6/1nSLa4L5qNyhhn25bpvHTu2RVm9LX7EUod79fHAm79KlqCf4ZTXvfsL5v2GWu3YGaeNiyPucC5EYMOdJtPQ2v/2H1yP1XZj7sgPTa7TTCfe2DVLS8Q05RbNhp1f0nVaaFxj8z42P2yb26Azn3V0eBv5wvbfWLzDS372lH8Nux1Dd3ju/3UP7UHtK1t/dgS/LH0d3q4Dpz0L3V82HKF9iG4cm6fAxOu07fwIbD3f9fXtASWlI9nWK+e12ZbbrpKcPAsPTNnAb5uO8U2uj2mnd88m9B8PibqcZ7Vn9mmKl/5z+3mI92jqoxsd2zYlQb+mFVGIylEFZM9JR5GgO79azQPLXC3Zv+UZcwcdrOykBI4p50gwZ8NfaRt8oGQJehv3ztOCety5SukDLKYO9O4GaAsXdyZdF/JcuZn1frovL64JNL8Vhs5ytDXxUxSfhx8z/7fate1UIdfA3WUE7Pa89prnIQ/KiSTPs4pvjbb0Ryabe3WkZ+YgT++BTwpoo5032ihI3PHbcPjnA2Pbcs9a8vZjafQP1QUfuUst3eFhz/ce+qfxf8TZTGhGns4zyovwqDt6Dg1fmMuCHVocyQ95PZmX5+HzHDzd/TE9m6dqQjwvVzO//S9RM8FMuNLRp8n1Wr52d6al7AyjCWrDJO1Vv9hftytFzRtzjO7DK/a6Kk3ZhBv2/6g4zKWPnUd0/4PNbynU2HylZAr6Oh3glq/dBzs480ZVz5pTTBXz9vo9Hdt93oInd2jBTTZCw+DWb6BOR0dbey9VcPLDg1bBct8CR6QjQGWTREz5NV34ukBkW/BM2aW9Zlt9nUOc1vmn3GJucz602iVIqJ44xsUshylo2KR1xI+aTasxcxETHG6Dj2cbg36uyvyE27JMXD1trP4UvuoOqz7x/J7M2OLdXDYg1Ivb4fClcK0Xt9+6V8Mdupngio889089Cq9b8+Z0zn/dgv2yOg/kPEFi5ucux3qGTYQGvUzPeyT7YXhR5910dD28XkUT7hdPm99sl1XhmWnysMtKd32vEWW0Mn4/WSNzb/nG99+0H1m229Vc1STTcy6bDxbvYU1vN84RepNuRc2DJysrk+zcwKWaKFmCPrpywQtrp3swF5hF0Q5fAkN0i1Z1r9Y8JMw0X723jJnnTEGp0kzzeKjdzpF1r6U12ZJzTczfvURC7pqjudbZ+KCZY/thN2YZcEyl5zytvdpy6F//gWvfN008ar691r6ZIrVF3XYhu5i21hEJvGinJlCmhBu9ebJ1TmNDskeRQgXWySacj9Duc13WG3CTSV3U+c+7fz+ecPOwPHsx27TdQNwVUMODJ44ztgAqbxqfPvr6kMlMzge+GdqGs5QlIfNrumVpM5f1lkY8O8DczPh/2Y/yp6Ujyedz4XHd4qglR8vOaRKA5ZFzyfBWTfjLyS3373fgxzsd+2ZrXgHg3MVsvl+VrC28Ane107xm7u5Qx94nkwiyXjzrcMU04faZGcjwKNcDehkQVQmAtq/8QbfxPqxzFJCSI+il1D5Ad9q3N/S1J82IjoOyNTUXyTGpWglA0LSaF1N8s7fpvVH8zT2zoM84uMmaf8Y5z32ah0Ae0H5Q+5Y4UkTYZgW1O3iOsKxQ17hvCwwLi4Q7prn298A2i3atR8N+48+tx12Od3CyxecRwrF712G5ajj/WJrb21uljqdZ5jdsl3Vp+KMb98Qx5VwXBzNTPReCdmPn33ToHHWEl3WFK/KZvz/RmvzOueygHksezNYtXte40n1fKxFhxp/86tE96XlFFd4Y0Jx0okiW1RhleZCR2U/QoLKr0vJ36Z7MsWgpMLq9u0z7TTiz8CWv4zDwUUvj/tXPOLb1nnH+ihr2wi2fr+SlP7azyZqDf+oaTekY2785cWUcY4gIcwjs8bea+85/Vd0pk+g1rxn3D2sP5wfDZnIsNZPDZzMIBCVH0AuhuU/2cLMA68zLZ6GU7h952u2ufTZMcmjCTa6Dx7bAC04CKCxCq/nqjRdTzBdn/UVouGYW8vTAyc7HP1FNq93Wm5eD/n56O33Z6tCkn1aM2h06T52+WW/xbq6Wt2R+3lU+Fbo4J8tQvU4jQq57l04NKusHxUVKA5BDGB0y/2d+AdtiadpxbSzjamuFoG30elX73vQc2QB/6xYl/3iIqxbewl8RDoHbJPM7HrOZlapbtfiK9by+H7dsM3F3BKMG36gv9Brj9VJlIh225O/uuYqq5TR3wEHtNG21cpkIxo0dx4rXbqd+nPX3cd98GDQdxqRyvIdThsaQELjXiwYfHg23TTI/NsYkIVkPNy6xFQMUDQukXsphwpI95Fkk+1M0E+MHC//l0Bnjb+ZCprm78G1tarFyVA/WvdCLVrUcCsGbu6vySo7OylDXqeCLdb3wgTDNrNXlncBo9V4DpkosIaHw/FE4vlWLRDTjz0cd2zmZms29oPjyMPA3z+zXptI2LfvNaubTTOdo2u+ug4NWbdc6tfQJfdBOFauG3eUpaHG7Zgqq5JRcSxcRuFPWobQ1ynBg2F88mzuS2z5fyaM9XWcTI7OfIJIsbhngcEecMqwdA79YxdoDrmsLx6mk/aCc3W9Bm8l9cbVre/9PobXVd7vbaEcQ2NfWAtJZaXabsnO8ciYR/GHpzEcvv65FQp7c5vg8CsKv92p/sY201Na2ICp93Mid08iTUH/UbGJjSnH6QjYH3uqH0D2I0zJzOH1B82D5+u42dG+ifzjCylE9iC6l/Y9HhuvMC7UdmV5vv6o2tSpGcddXuoRdcW5cRwf9Cr/cC4N+dvU22z5Dezi5o9NjRpv90D+NSQH9jC3XfsVoh8a+fM9pBn1jNIddynHvNVS9vKZcOOtak/KupUfD8nQN3+masbbvO0VSIrTkaPQFxZZjJCRMc1GzBTc4B+5s/bFIh+UXoiu5Jk8zc2+zLXbZOKgzafiy+FXfKvxsWTvB6NlUrqbmcaFzM/xhzUE+/v0vAJ7L0VJIZOJ4GAosrEs+x5Bv1trb9lhqsNtSk/mWq/jozbe4va3DZgow8d6reOZaN26gI93UuHWXt0UfldnhIdfj7hZJhy8hIiyEitGlNGVCCKiaUDBXOuf0wqf/NaZGyLLmxLniBhCC+s9r8QunL2hrBs5Cqce7f9m3ezV1NXFWL1+aclHhLu3OdKjnePjHj5qt1Tke/BsWJ4170aky8PwRzRlBCE3JqGZNef3LPVya4VCkki1VWBfTw2G26eFk/ikbmOzmFotk0Q7HGtzzM4wBWYfPah55Njv9V3drzgCxMe7NSNc2c83xdPSKYTDoF6SUxI+azcjvtXUvGer4n08U/7qc5y+UoLdhydXqQU65WZvGO0fmdQpivdbCop8Kj62oTZfTTxQ8148ztiIa+hQUzoLN6nGRmpbK0fOXeGHGNh7dfTcAGVL70Sx9uoe9e2WcTTeShiFHaRxyhHKlzYVRVKkwHuregLbxFWlRsxyLn+pKaatmeia8Gow2ybx40o3bqT6NQEQZeMB7sI+MuwJqXMng9nXI9KD5+YxZemF9LVNbEZM+5rmF0jO1SM4Zm46QmZNn1+YbVSlE8WwwzBIAur+7jFOVOzG76yy7X/6yvJYcDzFJaicdn0vpHT/Zt6fl9WCUeMzxvxoaDjd/BYN/00ymAUpi1vq1hQyb7MHZwEqTqprTwTVNq7D82e6sHt3Dbd9hneuy+KmulIlwWABKWddGMnM0z5r527WHS1auhVNSUyruDM1HpH4+UYLejCyT7IFm+UkuF/SLWzbea+y6uPjsAc/77mhnzW1n8wB53CRMvafmD37nW1PoNM74Dx0tMunSMJb42Gj75+ycC71ftKbtnJTl6dzQKcWvEz8/0IGZD3emflwMLWtpM4u7vlqjeUT9n1OO8M1TzC/ivPBXNcE8IZUOcbu2nlEpphQZ2XlcyvaDsB/plNnw+GbHAuWyN7XXyPJMN8ma+d6C3SzbncITP22hyUsO7xwzc1h+6akz+xw4fZG35+3mkWmbeCN3EFdlfsI9Oc+RZeYuGOMq/E/K8nyd1499KRc5n6HzYGoxEBr0DKi3ja91cGN0QrtWxSjCQt3/L4SFhlA/LobNr/Tm2T7aDDPpyHmycy1c8bLje8jJs/D9qoO8lKPVl701ahPJ464ryNvwihL04BBUNn50qgd544QijWIrUk7tcmxHVdRy5YNmyooyyatjhnP2PbMsm1YT0tUhW7UuOH7QkWTz7T3WxV+rt08MDkHfsmY5Ps3THgBHZSxhIb5/F6v3azb73bboxspN4Jl93hcuzRb+vBXvtnonVYrWpuNnLhYwS6meaibeHH8+Brt1bpWlonnqF1evsZ/XH7Fr8Xr0ppeC8tlgo4fP9I3ag8ZCCClUsN7f4eK7dNcp4kfNZnNN11qrA7NfJg9t5tVqrPcatkWBswmwVFj+RWVoiGBEF20RftKqg5xMM3p0NXxhLm/M2cl8i/a/L7LTtXoaAUAJenAV9OV1tt/RRx2ubpczI/4yb/+0nXH/hv9p9nUf8qXYaTbAuO9cpQe0OAPgEpqmXB7Hw+GHvF6E2zSkNM288nv7PfbjOXlSy0wJDMp+ntuv8r2gzOT7HKUY7/7Wau+PjnUNLhp1SMtR5Ok9mJArQxiTczepjzjsq7YFPZ/86wtCqTJGLzEPSsicJKOX2Cd3JVLJg33Z5yGEhbh1KbTx78kLfPX3fvp8+Df3TlwHwE3zIuiY6Si2kSLLMuWZO91dIqA4z7ha1nSsK93TMZ7drzsitvOjXOjRa/5fLd/voaeVr3t671MAlKAHreB1S52moV949TVNbHGneivPx22CPSREE3pX3uP7tX35jKw272fCfiKCbG4L1R48T2Q/yH1dGzn6xWrbQjcr2HE8TdOmS1dk7Zgb6Vjfs+lGz9WNHNXK/v43he3H3KTUjSirBSeVijH3DdcxPsdRvq5p1ndMzOtDuUqOxc1KMZpGf+B0PrJP+sLT1oefLp1FXrzRY+il642pMZY6RXXWqljab8NZY+Lh9NkgY1DYG3N2sutEuqHtGLH0y3qTKbk96ZA1gcplI6igWwS2BSr540GZmZNn98Rate8Ms7Yes7vurks2jn/LEcf/RlSpUCLCQnX/P4Wf0U9eddB7pwChBL2N/p+45oh54WTJMtk8uVP7MyPOJG2Cr8RU9t4nVPshx4hMdkfew9PhWvWic5RhcDvdDCrWOmXeu4jqnGZHxL1Ec0kLHAqLMPiBF4TrPv7HLkgM2L7n55K1xT8zbtA8bRZZEvkg5xZuyhrrkuMEoKx1jI/96DnMJRIAACAASURBVKeshP0/0fKwm3zOe/YfMBQ6ub9zXXaOdZ+Hv26s/xSXm1sbPWE6N4ilb0I1pg5vR8tanmdEO2Q8L+beTy5hRISFGlw9646eQ/yo2SS+tpBhk8xNGRaL5r1y3ceeqzPd9MkKBn6xilHTt3LnV6t5eOom+n+ygtw8i90cU6tiafq3qs49HeMBmHRfW/uC87u3tmBQu9p0b1Lw8qYVo42u1W/dbKwXbPPoCSS+lBKsJYRYKoTYIYTYLoRwcT8RQgwSQmwVQiQJIVYKIVrqjiVb2zcLIQJjgPIHISGu/sCFyRtfHClbXfsbk6olW9Njlsu/IDy0Ll/dL8kIalXU+UfbUiqc3s3KyEeJEllsj7xfc3dNd42WLQg2jwe6WNM26CsyhYa7j5e48h7qZ37PblmbGx77mN/feox1L/Ri7fPG6XY9qzDVzyYKRevB8KC518+N2a/bXQBtlC4Vyqs3NjO0rRrdgxWjehT6QaknsY6xwMmuE5oTQ8f6sVQpkz/z0Nj+5nEGi3aeZMNB15lDPasr6fZjnssu2mYTP64zpgTZl3KRdGvw0yd3JfLRHa0Zc2Mz9r7Rl666761y2UjeGJBgiILNL/88Z8xkeWfb2oZF1yd6NXJE9EeaBJD5AV80+lzgKSllU6A98JAQwjl14gGgq5QyAXgNcM6k311K2UpK6edKvn6mBGnvZy9mk5rhwaPg6qc1E0ULa71Q52Cm/PJcshb5G9fI0PzvyXTiR83W/K1N+HKYUzbC8MjCzS5MmDa8vWHfJpDo+ZI2axs42edr2RYN68dpwjyuTASVyxoVgpAQQd3YaMpGBjYe8YHsx8km3O6p8s1Qx8/rJp22fW2zKlQrV5oa5f1ntgEtqCp53HV210N7JC2eA4vMiIkIY+K95lk035qzy7S9MFz74d92X3b9w8+TN01BiSoVRmS463W3vNybL4dcqaVVeNQ6+yvs79ANXt+VlPK4lHKjdTsdrfZrDac+K6WUtgij1YBnI+flwFM+FswoJpy7mE38qNkM+WYNeRZJ4msLaTl2gfsTKl8BT26Hm7/QNPxyhQxIKV0BrnBNwTw3yZEDJj7TKQVE9xcoV9/kx53ixrxUQDrUr8SeNxxRmNX1Aq+AszZnX3JnMrJzmbX1uD1PSkE5fDaDXSfSyMrNY07ScdKrOSJM90utcIjNtqx/X/pYg1KF0EZ94U2rKeKKao51lVF93Rda2Tm2DzvGXsvIrvXY/qojqV2dSuZmpfUHzxnMbRsOGoMZDS6Z+cBivaTedTJQ7HqtLwfe6mfQ5MtFhdPbFlxVKkpbI/KhyE1ByNc7FELEA62BNR663Q/ok19IYIEQQgJfSClN62YJIUYAIwBq1w68zcotow5rOVuiC++CVpS0fk1zS1u+5zR3f2v+9czccoxujePsNmTQ8nmcvpDFGwMcdsNNh85RJjLcNKlVfildSq9LCOIzfyD5nhCtOEtoPs0I9boVeBzhoSFsebk3Lccu4Nj5S1gskl0n0mla3cQV1A3zt3tJXKbjZJrm1vj8jCTualeb2VuPk5NnMWja3sjJs7jkPnkqrAqPWH+1Z6zZPm0ulNXKmT+0Hu9VeL95TyTWrsDXd7cxmKpqVnBNV7B1TG9yci2ULqU9eEb3Nc7coksZH0hlI8NIs5pXVuw9Q6MqMdz0yQqOpRrdFM9l5FA+yjXFyOKdJllnTYiNKZr0JN6UA0pXgMzACHqf5ylCiBhgOvC4lNLUMCaE6I4m6J/TNXeWUiYCfdHMPiaJRUBK+aWUso2Usk1cnJ9smwUhsuxlJ+SdEy+ZFUbYdSKNR6dtosWYBWTm5PHPntM0fXkeHy3eww9rDpGda2HjoXPEj5rNgE9X0uv9v0jLzOGhHzaycu9pEl6ZT3IBvEjedJl2Cy3ZmSchH+3m+x/sJrmXj8RYTSkfLtrDtHWH6Pfxclbuc5M73YStR7QfoW3RzhN675Y8i+ShqRt5/Kf8Lc4+9+tWl7ZPcx1FvMcN0Wy/tu/Fnf09qlRgNXrQUirofc31M4pPByUytEMdykaGe3TtjHbSrPWfc47FwrDJ6w1C3jaDSHMT9HS/yULuLYmuxgavArio6P0GPF3AamRe8EnQCyHC0YT8D1JK01+bEKIF8DXQX0pplzRSyqPW11PADKCt2fmKgnO1hzzWOXlakM/FLIfNtPXYhQz+Zg0ZOj/iRi/O5eZPVxrOHf1bErOTjnPX12tIz8pldlL+FkPzLEbvluFd6rLlFTeFsfWMXK6VgazkpIkWMpd/qM4X+oUZWo3Vu75aw5tzfDMVfbJUq/c7uH0dLz3h/7o5bK22HDTg6rvtid82uaZsuEQkSZZ4cq8aSQWrv/7iXVq+/lAnX+9Zj3Tmrna1qVImOE4Fbw5IoErZCPo0q8qrbhZb9djSVSTWLs/Gl67hyd6Nef0m7bx7v1vH1iNG11hbioFzJqYb/f/enW0dFoL3BrZk3uNukhgGm/DIgBVW8cXrRgDfADullO+76VMb+A0YIqX8V9ceLYQoY9sGegPbzK6hCAw2reaWzxxCvF+Cb0WhZzvlhM/Izp9nztvzjNp8Yu0KbvPUGChbDbo+Cw+u0DJHBpgv/zYPZDl9IYvvVx9ESmmIMPXFpHWHm6CuK16ex6k0DznvdbStax6ZfFPuW4Rd946puUJP8xrleHNAAiEFDPYpLHe1q82a53v5fP+QEMHsRzsz8b62dpdE5wybetKtVcjeW+CaDOwJ3ezpmWsbs+u1PvYAqCZVy7LNujZQXJT5QOPL46MTMAToYXWR3CyE6CeEeEAIYQspfRmoBHzq5EZZBfhHCLEFWAvMllLOc7mDosBYLJ4Tk+07dYEPFhp/CLZw9fxiC2Dp8s4SFyFuhrMANbhR+kJYBFz9LAxb7LaKT37p1MB3s1yb1xfx0u/bWLXvDG1e91Dz1gQhBH8/0930WNs3F3s9Xx/oo/e7nv/41ex7UytgE2ivnmDQrHo5wxqSJ8XglkRtvUOi/QaOnMug70fLOXsxm5lbtHrOjarEUDG6FJHhoQYXyZiIMD4dlMjSp7oF4F0UP7z+p0gp/8FLWJiUchjgUg1XSrkfaOl6hsJfnNFFD64e3ZMFO07w8h+OyEkh4KPFe8xOzTfT1h5maMd4Dp+9xGfL9tG9cWUaVI5xCQhxZu8bfdl9Mp1m1QvgIxwSAjX955Vrtn4B2gNTr3nqTSx3fe1Y3HZ21fRE7UpRvHNrC541sbV74yvdQ7JPs6p8sPBfvh7ahsZWV0bQXDubVC3DrhPp7Bh7rdllLnucPWLa1q3I2gNn+eOhTiTUKMeEpXtpXr2cwX038TVHvpwuDd2v9/k6sy0JlDyV4D9G6iVN0P/vztZULRfJ3R3imbrmEPGVopm3/QRHzjmCaXpdUYVFTp4I+97sx/6UCyzdfYor61QgsXYFLBK2HU3lhd+T2HbUuO7e50NHJOLAL1bRtFpZ5jzmavO0ucONuLoeYaEhBRPyASBEONzqWtUqT4WocJbuTiE9M9eQi93ua+9Em/gKpu3uGNimFm/P3WV4IPvCe7pZWIXoUqx9wbVItxCCeY+b+jaUWDo3iOXnkQ4X0/MZOS7BUHpevM6/MRmXKyoFwmXI3lNaQrC0zBye/kXTFqMjHNPSeY9fzedDXOuHvjHAuCD23b1XERoiaFilDCOurs+VdSoihCA0RNCyVnlmPeJ90WrHcXOBaEv/WqVs8YounqRLcvb7Q5242eqFsTfFkWRt94l0BjgtTAPEV4pyJF/LBxteusbgLw6amcHGF3/tY9gkY0Rxryu0SMkNL7oK+P8aW8c4FvCHdoj3+bzE2uWLj0dNkFGCPogMn7yezm8vMc+9YmXPyXS75wzAq39up9f7fxE/ajY/rT3MZmuCpqhS3idnVcpG8u09mhlk6vB2dG/sPUfNC/00jahNnfxpsqfStcXLuHyGwgeaLg3j+HlkB/59XQugqlFBc4M8nuqY+Qz41Lzq1PQHOxb4vtERYYZgGX0e9Lfm7mLRzlP2/azcPPvMyx+ZJi93ykaG2yORyzitS5Ty8OB9sFtgokwvR5SgDxJHz19i4Y6THDl3yXTqaUvadM0Hfxvc/75bkWzf1uf7NsOWdOrZPo3tQqZHkyokj7vO5wyQw7rU5a9nuvHjCPe26dw8Y572PIu0u2rmWbzkcA8CbetWtPt817RGk57XpYvIcOMC6Q+h+501zN9WaUjvBmgT/tuO+mfhuSSx+KluJI+7zsWDp3p59zNGnzy8/iMoQR8kBusW+Eb/ZqzIFD9qtsE33ibcncvT7TnlMDeYmRRsGQQbVynjcsxXhBDUqRRNWGgI39+vmT2cMyA+PHUTWbmOsQ3+eg0XrK5v7sLaiwu2ICrbeMHhzw2w7oVevHx9UxfTS0GJsl7bttir97G3FaYIsZobWnnJAKlw1HC1UVvn2aX/Hv/rKEEfJNzlKrfZ3/WLqKAJ+R/c5E15vl8TEmu7CoXB7eswbXh7el7hWgi6IHSqH8tL1zdl5sOduL2Nw0983vYTNH5xnj3kfNV+h2dLs3ykGAgGpcNDCRFwwRpqn5NnsSfkSh53HXFlIrivc12XqM0C388apbpo50mDSQ4cEZ42zf6l69VCojcaVinD3jf68uJ1V7D2hZ789Uw3wkO1B2V5Hwqd/1dQXjdBYIFJzpTsXC0/9u8m0ZAAvd7/y0X4A1wVX4ERV5sXTg4NEXSo7790DiEhgvs71wXg7VtbMPzquvR6/2/78fsnrWfqcGPFqsKkdy0KhBBElwoj+Yz24H0in2kK8kuu1VQzcWWy3dfbRpo1r7wtfW5ZP6YULsmEhYYwzFqyD2Dbq9ey+0R6/uM2SjBKoy9CbHb3Ed9vAIw+wo1enMubc3a6BDPZzC56Ib/2BUf+83XJxkx+RUlMhKsgmrfN8RALVKFjf2ORkotW082srf7Jee+O1jpzjHMFpdRLOVgskkembQJcc78ofCMiLJQWNZXZS48S9EXI/U4udO/e1oJ+CVXt+1/+vZ/jTpn5fnmwA85U1uUu8Ueh54JS1SRbYjDLpRWU+pVjXEruVSkbGG8XM3e/T+7Syu898dMWuxIAStAr/IcS9EWIszDp1CCW8bd6Dhx2nr7bUs7abOTTPHjDBJPPByd671RMsCXL2q/zpV81KjBFmgFinbx3WuvWV/QBbSUxxYEiOChBX0SY+cpHlwozTSHbqIoxaVadSg5b4yM9NEH/9q0tioVpZNYjnZn1SGe+cArQurZZVTdnFD+6NdbC5Ad+sYoQAdc0rRLQRGD67xOgrBs3QBXso/AXSmUoImwBRHpswuS6hGqGFMCj+jahevnSRFoXMpc+1Y1FO0/S64rACqCC0LxGOcOrjctJSH0wsBWtX1tI98aV+WXDERbu8K1gRUH5dFAi0zce4Z15u6kUXcq0wpGnuAWFIr8ojb6I2G0tUmxjwl2t7dufDErkwW6a50zzGmXp3rgyTaqWJd7qrx4SIujdrGqxE/LOvHNrCwCXYtnFnfJR4YSGCLtbpd51NBBUKRvJ/3VrQPK469jw0jUALHrSmLOmfRDXXhQlD6XRFxH/nrRWox/R3vRH/HTvxjxwdX1DYq3LjYFtajEwwEIyEAghyLNIu8fN0fOubqyBpkHlMoSHCnLyPKedVigKgtLoi4BdJ9J4fbaWxsCdphYaIi5rIV+S6NzQt/QQ/mb16J5ElwplxageQbm/ouTiS4WpWkKIpUKIHUKI7UKIx0z6CCHEx0KIvUKIrUKIRN2xoUKIPda/of5+A5cD+tS+iuJP3+bBWUiuFBPB9rF9qFG+tPfOCkU+8EWjzwWeklI2BdqjFfhu6tSnL9DQ+jcC+AxACFEReAVoh1Yr9hUhRP7SIF7mfLosMMV+Ff7l2T6N7dvFPT+PQpFfvAp6KeVxKeVG63Y6sBOo4dStPzBZaqwGygshqgHXAgullGellOeAhUAfv76DYs4783bbt50X3BTFhyHt61A3NtqwSK5QlBTytRgrhIgHWgNrnA7VAPQ5c49Y29y1m117BNpsgNq1a5t1uaz5dFAiDSoXPIukIrCUiQxn6dPdgj0MhSIg+LwYK4SIAaYDj0spzcsKFQIp5ZdSyjZSyjZxce7rPF5OLN3lKCbxX6pPqVAoihc+CXohRDiakP9BSvmbSZejgN6vrqa1zV37f4J7J67z3kmhUCgCjC9eNwL4BtgppXzfTbeZwN1W75v2QKqU8jgwH+gthKhgXYTtbW37TzF1WDvvnRQKhSJA+GKj7wQMAZKEELZk3c8DtQGklJ8Dc4B+wF4gA7jXeuysEOI1wKbajpVSnvXf8Is3TaqW4dj5S3RsEBy/bIVCoQAfBL2U8h/AY+y91DJ2PeTm2LfAtwUa3WVOdq6FqxuVjPUGhUJx+aIiYwPI6QtZLilpFQqFoqhRgj5AZOdaSMvMpVJ0qWAPRaFQ/MdRgj5ApFzQ0hJXUhq9QqEIMkrQB4h352sRsecysr30VCgUisCiBH2AmLFJCxfo36p6kEeiUCj+6yhBH2BqVojy3kmhUCgCiBL0CoVCUcJRFaYCgMUiEQIe7t4g2ENRKBQKpdEHgoycPKSEMpHqOapQKIKPEvQB4GJWLgAxEao0oEKhCD5K0AeAyauSAci1WII6DoVCoQAl6APCdyuSAagQpaJiFQpF8FGCPgDcnKgV0bq+hSo2olAogo8S9AEg9VIu8ZWi0FL5KxQKRXBRbiEB4M8txyhXWi3EKhSK4oHS6P1MemYOAKmXcoI8EoVCodDwqtELIb4FrgdOSSmbmxx/Bhiku94VQJy1ulQykA7kAblSyjb+Gnhx5doP/gbg+X5NgjwShUKh0PBFo58I9HF3UEo5XkrZSkrZChgN/OVULrC79XiJF/IAx1IzAWharVyQR6JQKBQaXgW9lPJvwNc6r3cC0wo1ohJC27oVgz0EhUKhAPxooxdCRKFp/tN1zRJYIITYIIQY4eX8EUKI9UKI9SkpKf4aVpGilc7VKBWmlj8UCkXxwJ/S6AZghZPZprOUMhHoCzwkhLja3clSyi+llG2klG3i4i7Pgtppl7TUB4/2bBjkkSgUCoUDfwr6O3Ay20gpj1pfTwEzgLZ+vF+x4/RFrXxgvdjoII9EoVAoHPhF0AshygFdgT90bdFCiDK2baA3sM0f9yuuHEi5CECsqhOrUCiKEb64V04DugGxQogjwCtAOICU8nNrtwHAAinlRd2pVYAZ1ujQMGCqlHKe/4Ze/MjIyQNQwVIKhaJY4VXQSynv9KHPRDQ3TH3bfqBlQQd2OXIhU7PRx5VRGr1CoSg+KNcQP/LvyXQAYlTBEYVCUYxQgt6PTFyZDEBUeGhwB6JQKBQ6lKAPACEhKmulQqEoPigbgx9pUDmGBnExwR6GQqFQGFAavR85fSGL2DKqqpRCoSheKEHvJ3LyLJzPyFE+9AqFotihBL2fOGHNWql86BUKRXFDCXo/8efWYwDkWaSXngqFQlG0KEHvJ0pbXSr7NK8a5JEoFAqFESXo/US6NSq2StnIII9EoVAojChB7yfSLuVQOjyU8FD1kSoUiuKFkkp+Ij0zlzIq9YFCoSiGKMnkJ35afzjYQ1AoFApTlEbvB/aeSg/2EBQKhcItStD7gUvZFgBeuaFpkEeiUCgUrngV9EKIb4UQp4QQptWhhBDdhBCpQojN1r+Xdcf6CCF2CyH2CiFG+XPgxYlzGdkANKteLsgjUSgUCld80egnAn289FkupWxl/RsLIIQIBT5BKwzeFLhTCFEiVd4Zm44CUK2ccq1UKBTFD6+CXkr5N3C2ANduC+yVUu6XUmYDPwL9C3CdYo9N0Fcuq/LcKBSK4oe/bPQdhBBbhBBzhRDNrG01AL0ryhFrmylCiBFCiPVCiPUpKSl+GlbREhGmCo4oFIrihz8E/UagjpSyJfA/4PeCXERK+aWUso2Usk1cXJwfhqVQKBQK8IOgl1KmSSkvWLfnAOFCiFjgKFBL17Wmta3EERkewrDOdYM9DIVCoTCl0IJeCFFVCCGs222t1zwDrAMaCiHqCiFKAXcAMwt7v+JGZk4emTkWKkSrgiMKhaJ44jUyVggxDegGxAohjgCvAOEAUsrPgVuBB4UQucAl4A4ppQRyhRAPA/OBUOBbKeX2gLyLILL9WBoA0aWUfV6hUBRPvAp6KeWdXo5PACa4OTYHmFOwoV0evPqn9uw6lZ4V5JEoFAqFOSoytpB0b1wZgJFd6wd5JAqFQmGOEvSF5Oj5S5SJDFMlBBUKRbFFCfpCsi/lAmUiVBJQhUJRfFGCvpBk5VioXr50sIehUCgUblGCvpCcy8gmPjY62MNQKBQKtyhBX0hSL+VQXtnnFQpFMUYJ+kKQlZtHRnYe5aOUoFcoFMUXJegLQeqlHADKRamoWIVCUXxRgr4QpGZogl6ZbhQKRXFGCfpCcM4m6JXpRqFQFGOUoC8Eq/efASBa+dErFIpijBL0heD9hf8GewgKhULhFSXo/UBi7QrBHoJCoVC4RQl6hUKhKOEoQa9QKBQlHLWKWAjKRIZxS2LNYA9DoVAoPOJVoxdCfCuEOCWE2Obm+CAhxFYhRJIQYqUQoqXuWLK1fbMQYr0/Bx5s8iyS9MxcykaqZ6VCoSje+GK6mQj08XD8ANBVSpkAvAZ86XS8u5SylZSyTcGGWDwZ+f0GAFbsOxPkkSgUCoVnfCkl+LcQIt7D8ZW63dXAf8KWcTz1EgAvX980yCNRKBQKz/h7MfZ+YK5uXwILhBAbhBAjPJ0ohBghhFgvhFifkpLi52H5nxY1yxMbU4qWtcoHeygKhULhEb8ZmIUQ3dEEfWddc2cp5VEhRGVgoRBil5Tyb7PzpZRfYjX7tGnTRvprXIHiVFomlctEBnsYCoVC4RW/aPRCiBbA10B/KaXdaC2lPGp9PQXMANr6437FgVPpWcSViQj2MBSK/xxSSizSEuxhXFYUWtALIWoDvwFDpJT/6tqjhRBlbNtAb8DUc+dy5MyFLCrFqPTECkVR02JyC1pObkmeJS/YQ7ls8Gq6EUJMA7oBsUKII8ArQDiAlPJz4GWgEvCpEAIg1+phUwWYYW0LA6ZKKecF4D0UOeczsjmWmkny6YvBHopC8Z8hJy+H/236n31/wcEF9K3bN4gjunzwxevmTi/HhwHDTNr3Ay1dz7j8+ffkBQAaVi4T5JEoFCWfjJwMpu+Zzjvr3jG051pygzSiyw+VAqEAjPhei/26sVX1II9EoSj5TNg8wUXIAzz/z/NBGM3liRL0BeC8teBIbIxajFUoAklGTgbf7/g+2MO47FGCvgDUi42me+M4GldVphuFIlBIKWk3tZ1L+6Yhm4IwmssbJejzSZ5Fsv/0RaJUVSmFwu/M2DODdSfWAXA+67zL8Rvq3UBYSBhDmg4B4EK2tl6WY8kpukFehihplU+SjqYCMDfpeJBHolCUPF5e+TIAW+7ewsUczastvmw8I1qM4Pl/nmd0u9EArD6+GoAO0zoQWzqW05dOA5A0NCkIoy7+KI0+n1zM0lb6vxhSonK0KRRBZ9/5ffbtr5O+ZsmhJQB0rN6RG+rfQNLQJMqU0sylj7Z+1N7XJuQBTl48WUSjvbxQGn0+OXMxG4C6sVFBHolCUbLYc26PfVvvL9+rTi+Xvh2rdzS9xr7z+6gSXcX/g7vMURp9PjlzIQuAitHK40ahKAzbz2xnzMox9v0jF46Y9mse29ylrVSoMSp9QIMBAIxcNNJ/AyxBKI0+n5y5kE2IgPKlw4M9FIUi35zNPMuQOUPIk3n8cdMfRIQGT2G5Y9YdAFxX7zquqnoVH238CIC40nGkXNIy2Haq0YnSYaVNz980ZBPZedlEhUchpWTG3hlFM/DLEKXR55MzF7OpGF2KkBAR7KFc9mTkZNDtp25sTdka7KGUOCZvn8wNM26w73+44UMSJiXQ9aeuHEo/xNELR5m4bWLwBqjjvvn3sfDgQvu+TcgDHE0/6va8sJAwosI1E6o11QoA+8/vByA1K5Vvkr5RCdBQgj7fnLmQRcVolcyssFzMuUi7qe04k3mGQXMGcf/8+9UPsgBk5mbywYYPOJVxij/3/cmTy55kw8kNjF8/nuS0ZNYeXwvAN9u+cTl3wuYJQXvI5uQZ3SGfXPakfXvhrQ6h/12f73y+5ttd3gbg2MVj9Py5J1f/dDUfbvzQ7q75X0aZbvLJ+oPnaFQlJtjDuOxJTks27K89sZaDaQepW64uAHmWPFp938p+fNaAWUzYNIFFBxexcchGgwb3X2bKzil8u+1bvt32rb1Nrx3fv+B+j+cPmjMoKC6JEzZPMG2vX64+VaOrFmhMNcrUAODBRQ8a2lVOHKXR54vUjBzOXsxm9f6zwR7KZc8vu39xabvx9xvtP8p5ycZEp9fPuJ55yfPIlbn0+sXVC0OPRVoMHhxSSiZsmsDig4v9MPLiQ2pWqt2u7SvtqrZjzV1r6FxDqw9Up2ydQAzNK7YHU5gw6pof9cjf+9HTtJJ5Wc8HFj1A15+6svPMzgJf+3JHCfp8kGL1uOnaKC7II7n8mb5nOoDLYmBKRgrzDszjUPoht+eeunTK7TEpJS0nt+TmmTez++xuANacWMMXW7/g8WWPsyVlCwfTDpKalWo/frkyft34fPVvUrEJX1/7NVHhUXzW6zMSKycSV9r1f/lS7iUs0kJ2XjYWaWHHmR3+GrILG4dsZEIPh3ZfmAdPeIh7B4mzmWcZOGsgxy/4Fug4Y88MEiYl2IO2LneU6SYfHDyjfenDu9QL8kgub9Ky0wAt4vHPAX9y8uJJev2qaem9p/c29F1++3K6/NTF52u3mNzCvv3qqleZ0m8KwxcMt7eduXSGwXMG2/fHdhzLgIYDCvQ+vGELy/ckgArDH/v+AGBqv6nsOreLmPAYetfpTavvWzG241h61unJsPnDiC8bz9zkubzY/kXD+WUjyrLx5EZACzqaRJccbgAAGl9JREFUunMqiw8tZn/qfpd7vd3lbfrV6+f39yCEoF21dnSq0YmHWj5U6OslDU1i0cFFCAQxpWIYtsCYQX3P+T1Ui6nm8RrHLxy3R+i2n9reYEbKycsBYfxORy8fzaz9s9h699Zia1L0SaMXQnwrhDglhDCtECU0PhZC7BVCbBVCJOqODRVC7LH+DfXXwAPNDzt/YNHBRYa2NQc0k02jqspGXxjeX/++Yb9KdBUebvWwad/ykeWJLxtv3+9UoxOAT9WFkk4n0XKysSSCcybE77Z/x8qjK30Zdr7p91s/evzcw+/XzbXkkjApwb6fEJfAbY1uo2/dvoSGhJI0NIkBDQdQtlRZfr7hZ97p+g5JQ5NoGdfS5TpSSv7c9yfdf+7OV0lfmQp5gOeWP+e38UspiQyN5O6mdwMQGRbJ570+JyEuwcuZvtGrTi961ulJu2rtmNpvKgMbDbQf+33v717Pd1Y2bORYckickkji94lk5GRwPvM8X239iln7ZwFwLuucT+P7fe/v9mjefef3kTApgYRJCby55k2fzi8IvppuJgJ9PBzvCzS0/o0APgMQQlREq0jVDq1e7CtCiAoFHWxRMm7tOJ5Y9oRher/7RDr14qJNi4KnZaep0mY+kGPJsZttro2/1t4+osUIt+f8OeBPkoYmkTQ0iY7VtIhIvQueDZspIyzE/UR1/cn1hv0DqQcYuWgk2XnZvr8JEyzSQt/pfUmYlMDCgwtZdWwVJy6e4HzWeW6YcYPf/jc2n9pM6+9b++VazSo140LOBZ/zuqdkuH7mBSEtO43MvExiwgOvMCXEJTCy5UhaV9Y+M/1CtUvfSQmG2Z8N27rRY0ses7e1m9qOLj914eNNH9vbuv7Ulf9b9H8u50spmbR9Eo8teYyESQm8tOIlbv7jZg6mHeSmP26y95u2a1r+36CP+CTopZR/A55WIPsDk6XGaqC8EKIacC2wUEp5Vkp5DliI5wdGsWDbacfE5ZElj9i3T6ZlUi822r7/1+G/7E/jTtM60er7VhxIPcCZS2dw5kj6ERYfXIyU0u19957by8x9M/30LjSWHV5mOh53XPvrtQZt0d9sOLnBvj28heNHJYTgi15fGPq+0uEVl/NtGuc1v17Db3t+s7dLKZm8YzKg/TCfuvIpw3nX17ve47j6/uZbSbpHFj9CwqQE1p8wPjCm75luj+x8ctmTjFjoeHAlpyXz1F/G8RSEizkXGTJ3iKEtsXKim97eCQ8JR+L6/3h749sBiAmP4eUOL9OjljYrGbZgGBk5GYa+adlpZOZmGtqy87L5Oulr04dnniWPhxY/ZD+3KKgcVZnJfSfb9++eezeH0w4b+kzerh23JUvT88GGD0iYlMDyo8u93mv50eU8+9ezSClZcXQFa46vYdXxVby7/l2WHF5i73cu6xzXz3D9n0zNSvX5feUHf9noawD6T+6Itc1duwtCiBFoswFq167tp2Hln5y8HO6c7aiemJGbYW23sOtEOlfWqUBmbiZbU7by8BJXc8ONv98IwIJbFthtgQfTDhq+1EqRlVh2+zLDeVl5WQyYqdmKr693PSHC93XyXEsu5zLPERdlXFjTC+wNgze4hI07k5KRwrGLxwDtodOgQgOfx+CN7LxsrpxypX3/pgY3uSzEdqzRkYW3LiQnL4clh5dwc8ObXa5Tv3x9+/YrK19ha8pWXmr/EiuOrbC3N67QmMFNB/PehvcAuK/5fTxx5ROcuHjCrtFvvXurwZ5/KsP9Aq+Nzac2s+zIMgDunX8v066bRvPY5py5dIaxq8Z6PHflscKZhzJyMmg/tb2h7Z87/rEn+SoIB9MO2rfvbXYve8/v1QTVVc8a7PltqrRhyeEl7E/dT7up7Qw2607TNFPa6rtWk5GTwZ7ze9h9djcfbfyIH3b+wJLbliCEYPq/0xmzaozh/l1q+r724k82ndrEc8uf47NenxEeEk5UeBTj1xsXtj/s9iF7z+9lwuYJdgXCV+Ymz+W2xrfxwKIH8j224QuG8/MNP+f7PG8UG68bKeWXUso2Uso2cXHB82pJnGLUkJpUaAJAwxfmAlC2dDgD/hjg1T+59/TebDqlFUiYnzzfcOxMpquG/eOuH+3bL614yb49/d/pbEnZAmi2PbPp3T3z7qHHLz3s2lZyarKLVu4sJEzH/KvDNjlg5gCPs4/88MDCBwxCHmBoU/PlmqrRValVthZDmw01fdjd0eQOw/70PdNZe2KtXUsE+OKaLwgLCeODbh8QWzqWxxMfB+CTnp/QvVZ3nmnzDEII+sQbJ5eevEuklC7a9J2z7yRhUoJH04eteHWNGFP9xmcm7Zhk2E8amkS5iHL5UgicebXTq/bt5rHNea/be8waMMtFIageYyyZOWTOEAbPGcx32xzBTO2ntqfHLz0YuXAk72/Q1mBOXzrNqmOrOJJ+xEXIg/YAKUqiwhyJCJNOJ9H5x860m9qOrLwsQ7+uNbvSs05Pe857PUsHLuW+5vcx86aZPHnlk0y7bhpJQ5P446Y/DP3um3+f6RgqRlbkqqpX2ferRRsXhneeDYwLqL80+qNALd1+TWvbUaCbU/syP93TgJSSAX8MoEVcC8Z2cmhXqVmpfLPtGx5u9TCvrnqVmftmsvXureRacgkPNXpDOE9LQZteZuU67Kt1K0Uz5aQx+dJDrR5CIvl086eG9rvn3u1xvPoV+qWHl9q3Z+6bSY2YGny25TN7W+mw0lzKvQRoQSVtq7UFYMzKMfYHwZlLZ4gKj+KG3x2h7zbcFWaQUjJh8wS+3/E9udIYWJKalUr5yPJu34Mem8Z+fb3reavLW/b2XEuuQdu2UbtswWZt4SHhJA1NosuPXeyFKfRmEr3nQ686vQyZD6PCo/i4h8OmOr7reMZ3HW9/KN4+63a3gTp6M0OTik3YdXaXfd9muvjp+p9oWqkpH2/8mK+SvuKuJncxut1oqkZV5bvt37E/dT/1yuXfYysnL8fwvzXvlnkeevtOeEg419e7nln7Z3FNnWsQQpi6N0aERjCuyzhGLR8FwOaUzQD2/ztPeEoy5m2G6W/GdRnHo0sfdWlvM8XxwJl500z7ZxAVHsUTVz7BBxs+ABy57p+48gkAe3AfQL1y9UgamuTR7PlY4mMMS9C8gGz9jl88zvxb5rPv/D7+b7Grfd9f+EujnwncbfW+aQ+kSimPA/OB3kKICtZF2N7WNr8jhGBf6j5m7J1hX0CduG0inX/szHfbvmPW/ll2+3eLyS1InJLo4kftHCrdrFIzdp7dyTWff0J4+dWI0AvUrO6wRNUqU4utd2/lgZYP8GDLB+0h2L4wdN5QQ8RelagqBp9mvZAH7EIetGjHq6ZcRcKkBPvCJpjbPOff4vi4B/wxgENphzhx8QQjF44kIyeDyTsm8+XWLw3Xt7HupGvo+OZTm3n2r2d5d9279vUJgBVHNWFu80Cwcfyiud9yYX/k0eHRpu0FcW/7vNfn9m3n0HzQBNqL/zhMGT/0+8EwI9l4SnNRtAXsPNz6Yd7v9j7PtdU8VWyLw/1/7+9y7YycDPae22toc04FYROsoAmbws4O9LzV5S2ShiZ5/dyuq3cdU/tNLfT9+tfvb19YL2psypE7bqh3A3XL1TXMku5rrmnmDcr7ZsZcOnCpYb9LDc08JRB2IQ+aQgLQu05vqsdUD7gZy1f3ymnAKqCxEOKIEOJ+IcQDQgibEWoOsB/YC3wF/B+AlPIs8Bqwzvo31toWUH75V4u6tNloQbPnOnPrn7caouVsNvcZN84gaWiSXUidK/MVkdV+J6bR6wYN5fHExw0/kH71+jH9xukMvsLhp23j816fG7wMNp3aROvvWzNoziBOXjzJnANzSLmUwkOtfPMlzszLdGnbdGqTYeF145CNVI+pzvAEbdFz7/m93Dn7Tq759RpWHlvJssPLeHf9uy7XmdRHMxPo84/YGDJ3CHOT5xpMCQmTEgyakv6Bc9PvNxnOf/aqZ5l781yf3qMnHm7tuj6y/Hbvi2VmdKzekRaxmr1en0oAtFnN4DmD7bb5z3p9RqnQUtzb/F761TX3Kw8RIVxT5xq7wNB7FDmbw9pNbceAmQPsD8yOUzvaA75Ac/O1mQGebvN0gd6fv0iIS+DD7h8CGIKcbHza0zHrWDpwKS+1d5gg72l2D0lDk3i98+uBH6gbosOj7QLWjFc7vmravvz25fzQ7wef7hFbOpZ7mt1j3x/fdTyNKzQ2KFygKSRJQ5N4r5tDRv14/Y+GPD/+xFevmzullNWklOFSyppSym+klJ9LKT+3HpdSyoeklPWllAlSyvW6c7+VUjaw/vmeoagA9K+vaUw/7f7J53MGzhro4jZmy5kRKkLdnlcjpobpU7hRhUZ2Tc7GS+1folONTvze/3d+veFXw7GtKVvtwUKgacw2Ztw4g3e7OgTxh90+9Phe3l73Ni/88wIAnap3sgd1DG7qePDohfDhdMfsRG+/bBHnWKTUewGsOrbK4/1t3DPvHgAWH1xMtkXzvPj79r9JGprEkKZDqFmmpk/X8US7qu0QaA/Zu5rcRdLQJJ/NTM4IIfiguzY9n7B5AvvP72f18dUkTEqg84+dDX2bVNTWbCqVrsS4LuPs7baUAmZEhjnccT35cd/0+02k56TD/7d378FVVfcCx78/3gmFkBADSHgkDISCYoPhNaiggEYor1suYh0NWsbbW9pCuZ2WXi7TaXUovdfbXm07IEMpXh8l3IDFxtoKoradIhWocGJ5GMAGKIhUQHBaEPjdP/Y6J+dJXkdOzj6/z0wme6+zd7LX3ie/7LPWb6+FNwnHo288yvI/1v+Oewffm2jXa2ZC3wkEKgKM6zOOQEWAPQ/sYU7JHB4b9xi3Ft5K1dQqtszaQn5WPrNL6vPXv3hT0zsmPwkiwtZ/3srA3IFsmrGJ7p26A/D03U/HNOUGdevULTRKZmMsGF6fhtm5fWeqplU1+JAWeC0IPTv3bPTvaQpJVodbMpWVlemOHTsa3jCO4J3RwuELqT5UzbmL57h05VKoA/TZyc9y36/ui9lvavFUfnnol9xTck8o46DmVE1EBk64hj56Bo/jxZkvxrRHJ2rHG5Y/jCfueILx68cn/B1b67aycvfKUKfNqF6jmDZgGkt+v4TinGIKuxTy26O/5bXZr9E9q3tov1fqXmHhqwsTHu+eB/awpmYNw64bxoieI0LH+PWyr1MxtOKqxx3Ur2u/iEyOcOkwl2ewfgVZBQmHWXjr/rdo26ZtzD6bZ22+6h/potcWhXK4101Zx9D8oVQdqOI72+LfRcaTDucw2v4P9lNzqobPDfpcqg/lmjrx0QkuXL5wTccSEpGdqhq3h9t3gX7EMyMimjVKC0pZW76WFbtXMCx/GLcW3krdh3XUnKqJ+7Tf4pGLue/T9f8IZjw/i72HC7i5500EPv4xy25ZRnn/8oT//RtrdWB1zIBUa+5aE9Ejn8jpf5zmtsrbqJ5ZHXojRQfh3Q/sjsnI2PXeLip+HZvtEgw84Y6cO8LkjV7TxMqJKxmcNzj0D2jDtA189PFHlOSWUH2omkfeeASAjdM2hpocwo0vHM+PJvwopry1Kd9QzrHzicc/D3auhqv7sI6sdlkxqa3Rzl44G/HpILzjbuz1YyM6rCs/W8k91fdE7D99wPSUNnuY1u9qgb7VpFcmy1N3R6ahjek1hjbShvmfmR9qaunbtW/CcTt6Zkfelc0v+QkXTk7hC6UzCFQEmDpgaouDPMC8G+cRqAhEdPIMyh3UqH1zO+USqAhc9W4hXtrd8B71qaMDcurz0XM65sRsG/xIC97of8EgP7b3WAblDqK0oJTs9tnMLpkdqkPXDl1jfs6oXqPSIsiD9wRuPFtmbSFQEYgJ8uC9lxoK8hB7jsMntF4xcQUP3vAgQ7sPZdWkVXFHYbza077GNMR3757oP5Lwnu5oKyauCI1d/dXSr9I/pz/j+4yP2Ob8BS8Lo3du/OnMWmp0r9HUnvGyLlry8Mv9Q+6PGcclnicnPUlWuywKsgso3+DlkcdrM0/UJhnMrgm3tnwt2/66jR6de9CtYzfOXDgTGrAsnbRv056Hbngo1CG75q41dOnQJWmTTYffxd++/vZQuYiw6ObIju+5Q+dSub+S7Z/fzqaDm5jUb1JSjsFkJt813YDXPgZemlr0wx5NNf+5Xby45zi7lk76RGaWOvOPMzyz9xnm3TgvotOuOYJBJLp9PpEDpw9QnFOc8G4x3tOM3xzxzYjO3WiXrlziyLkjETnG6ebld1/mpcMvhTpokym6iW3u0Ln8W1nLh0cwJqPa6JOt/+IXATi0bHKrnyd23wf72Pnezog+hpYKn+kpHTsDW5tTfz8VcTffmoe2NenlaoHed003yRT+RGxrD/Lgpf4F0/+SJTzDxLRcflY+swfN5rJeZunopRbkzTVhgf4qTp33csAXTBiY4iNJreqZ1TbvZhItHbO04Y2MSSIL9Fex6vWDAAy9PjabJJOkal5RY0xy+C69Mlm27nuPp7Z5D/8U5ccfV8UYY9KBBfoEjp2uH+RrYI/mpz0aY0yqWaBPYOmmtwH43Tdub2BLY4xp3SzQx/G38/UTEfTJa/xgRsYY0xr5KtBfvHSF9W8e4YebD7To59z86JYkHZExxqSer7Ju/uMXAdbv8GZ/+uK4AWR1aHoOePgDZJvmj03asRljTKo0KtCLSDnwONAWWK2qy6Ne/yEQbMzOBgpUtZt77TIQfKSyTlWnJePAo52/cCkU5AF+FTjOkOu7cvfjv2N8yXUsmDCQJc/XMLhnF8r65/GHg6f465m/s/FLY3l1/0n65mUz4b9fp7P75zC4Zxdu6tO88c2NMaY1aXAIBBFpCxwAJgFH8WaKuldV486kLCJfAUpV9SG3fl5VPxVv20SaOwTCxB+8Tu3J803aZ9nMG/n352Mf7a/+yi3c0Dt2VEdjjGmNWjpM8UigVlUPqepFYB0QO/llvXuBnzf9MFtuy6JxHP5e/OGHE3nilXfilg/pldkPSRlj/KMxgb43cCRs/agriyEi/YAiYGtYcScR2SEib4jIjHj7JZOIsPvbd4bWG3rY6cSH9ZOUDO7ZhUdm3MCWRbelxdg2xhjTGMnujJ0DVKnq5bCyfqp6TESKga0iElDVg9E7isjDwMMAffv2jX65SXKy2vPu8ils3HWU20sK+K+X9/Pc9jrW/8sYhhXm0LFdG0QkNDIlQOXDoxlZlGeDTBljfKcxd/THgD5h64WuLJ45RDXbqOox9/0Q8BpQGm9HVV2lqmWqWnbddQ3P2NMY/zS8kNzOHVg280beXT6FkUV5dGrfNhTMKx8eDUC/7tmMKu5uQd4Y40uNCfRvAgNFpEhEOuAF8xeiNxKRwUAusC2sLFdEOrrlfGAsELcTNxVGFuXxtYmDWPvgyFQfijHGfGIabLpR1Usi8mXgN3jplWtU9W0R+S6wQ1WDQX8OsE4j03g+DTwpIlfw/qksT5StkwoiwoKJmT0EsTHG/2yGKWOM8YGWplcaY4xJYxbojTHG5yzQG2OMz1mgN8YYn7NAb4wxPmeB3hhjfM4CvTHG+FyrzKMXkfeBvzRz93zgVBIPJ91Y/a3+Vv/M1E9V444f0yoDfUuIyI5EDw1kAqu/1d/qn7n1T8Sabowxxucs0BtjjM/5MdCvSvUBpJjVP7NZ/U0M37XRG2OMieTHO3pjjDFhLNAbY4zP+SbQi0i5iOwXkVoRWZzq40kWEekjIq+KyJ9F5G0RWeDK80Rks4i8477nunIRkSfcedgjIsPDflaF2/4dEalIVZ2aQ0TaisifRKTarReJyHZXz0o3+xki0tGt17rX+4f9jG+58v0icldqatJ0ItJNRKpEZJ+I7BWRMZl0/UXka+69XyMiPxeRTpl0/ZNCVdP+C2/mq4NAMdAB2A0MSfVxJaluvYDhbrkLcAAYAvwnsNiVLwa+75YnAy8BAowGtrvyPOCQ+57rlnNTXb8mnIdFwHNAtVtfD8xxyyuBf3XLXwJWuuU5QKVbHuLeFx2BIvd+aZvqejWy7k8B89xyB6Bbplx/oDdwGMgKu+5zM+n6J+PLL3f0I4FaVT2kqheBdcD0FB9TUqjqcVXd5ZbPAXvx3vzT8QIA7vsMtzwd+F/1vAF0E5FewF3AZlX9QFVPA5uB8mtYlWYTkUJgCrDarQtwB1DlNomuf/C8VAET3PbT8aa6vKCqh4FavPdNqyYiOcBtwE8BVPWiqp4hg64/3pSnWSLSDsgGjpMh1z9Z/BLoewNHwtaPujJfcR9DS4HtQA9VPe5eOgH0cMuJzkU6n6P/Ab4BXHHr3YEzqnrJrYfXJVRP9/pZt3261r8IeB/4mWu6Wi0incmQ66+qx4DHgDq8AH8W2EnmXP+k8Eug9z0R+RSwAVioqh+Gv6beZ1Nf5smKyGeBk6q6M9XHkiLtgOHAClUtBT7Ca6oJ8fn1z8W7Gy8Crgc6kz6fRFoNvwT6Y0CfsPVCV+YLItIeL8g/q6obXfF77iM57vtJV57oXKTrORoLTBORd/Ga5O4AHsdrkmjntgmvS6ie7vUc4G+kb/2PAkdVdbtbr8IL/Jly/ScCh1X1fVX9GNiI957IlOufFH4J9G8CA11PfAe8TpgXUnxMSeHaF38K7FXVH4S99AIQzJyoADaFlT/gsi9GA2fdR/zfAHeKSK67S7rTlbVqqvotVS1U1f5413Wrqt4HvArMcptF1z94Xma57dWVz3FZGUXAQOCP16gazaaqJ4AjIlLiiiYAfyZDrj9ek81oEcl2fwvB+mfE9U+aVPcGJ+sLL9vgAF5v+pJUH08S63UL3sfyPcBb7msyXrvjK8A7wBYgz20vwE/ceQgAZWE/6yG8Tqha4MFU160Z52I89Vk3xXh/qLXA/wEdXXknt17rXi8O23+JOy/7gbtTXZ8m1PszwA73HvgFXtZMxlx/4DvAPqAGeBovcyZjrn8yvmwIBGOM8Tm/NN0YY4xJwAK9Mcb4nAV6Y4zxOQv0xhjjcxbojTHG5yzQG2OMz1mgN8YYn/t/zERTV/F8zXIAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"tags": [],
"needs_background": "light"
}
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "Nvwec4vk2arI"
},
"source": [
""
],
"execution_count": null,
"outputs": []
}
]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment