Skip to content

Instantly share code, notes, and snippets.

@kweimann

kweimann/selfie-cifar10.ipynb

Last active December 1, 2019 18:40
Show Gist options
  • Save kweimann/842e6fbf473f021140017a1f5c1213f2 to your computer and use it in GitHub Desktop.
Save kweimann/842e6fbf473f021140017a1f5c1213f2 to your computer and use it in GitHub Desktop.
Download ZIP
"Selfie" - pretraining by solving jigsaw puzzles on CIFAR10 data set
Display the source blob
Display the rendered blob
Raw
selfie-cifar10.ipynb
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "Selfie-CIFAR10.ipynb",
"provenance": [],
"collapsed_sections": [],
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"accelerator": "GPU"
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/kweimann/842e6fbf473f021140017a1f5c1213f2/selfie-cifar10.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "gx_mOxCWbCHD",
"colab_type": "text"
},
"source": [
"**\"Selfie\" - pretraining by solving jigsaw puzzles on CIFAR10 data set**\n",
"---\n",
"---\n",
"Reference: [Selfie: Self-supervised Pretraining for Image Embedding \\[arXiv:1906.02940\\]](https://arxiv.org/abs/1906.02940)\n",
"\n",
"![Selfie](https://drive.google.com/uc?id=1rTPRpoa35D6s81Ff43Ro7JMlOtrGjRi4)\n",
"\n",
"Selfie is a pretraining technique which adapts the concept of masked language modeling in [BERT](https://arxiv.org/abs/1810.04805) to continuous data such as images. Given masked-out patches in an image, the method learns to put the correct patch from a pool of masked-out patches into the masked location. This method is analogous to solving jigsaw puzzle where a few patches are knocked out from the image and are required to be put back to their original locations. The architecture of Selfie includes a residual network for processing the patches. During finetuning on a target task, the weights of this residual network found by pretraining are reused.\n",
"\n",
"We will begin by implementing code for splitting an image into a jigsaw puzzle. Next, we will implement each part of the Selfie network: Attention Pooling *A* which consists of [Transformer](https://arxiv.org/abs/1706.03762) blocks, Patch Network ***P*** which is a custom residual network, the encoder-decoder architecture which attempts to solve the puzzle, and finally we will combine all parts into a single network - Selfie. Then, we will compare the classification performance of a residual network that is randomly initialized (no pretraining) with a pretrained residual network on a subset of the CIFAR10 data set.\n",
"\n",
"**Note: implementation deviates from the instructions provided in the reference paper.**"
]
},
{
"cell_type": "code",
"metadata": {
"id": "GywTLI-yg-q7",
"colab_type": "code",
"outputId": "49a6f18b-cf6e-45ce-ee0c-97893f1f311a",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
}
},
"source": [
"%tensorflow_version 2.x\n",
"import tensorflow as tf\n",
"import numpy as np\n",
"import time\n",
"from sklearn.model_selection import train_test_split\n",
"import seaborn as sns\n",
"import matplotlib.pyplot as plt\n",
"import matplotlib.gridspec as gridspec\n",
"import matplotlib.ticker as ticker\n",
"%matplotlib inline\n",
"sns.set()"
],
"execution_count": 1,
"outputs": [
{
"output_type": "stream",
"text": [
"TensorFlow 2.x selected.\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "0upO79Kua5-h",
"colab_type": "text"
},
"source": [
"**Jigsaw Puzzle**\n",
"---\n",
"First we implement a method for splitting an image into a grid of square patches\n",
"of equal dimensions (jigsaw puzzle). Moreover, we want our method to randomly select patches that will be hidden and return them together with the remaining visible patches and the corresponding positions. The position matrices allows us to solve the jigsaw puzzle."
]
},
{
"cell_type": "code",
"metadata": {
"id": "kkz7S0yR6WRi",
"colab_type": "code",
"colab": {}
},
"source": [
"def extract_patches(images, ph, pw):\n",
" b, h, w, c = images.shape\n",
" assert not h % ph or w % pw, 'patches must be evenly laid out'\n",
" patches = images.reshape((b, h // ph, ph, w // pw, pw, c))\n",
" patches = patches.swapaxes(2, 3).reshape((b, -1, ph, pw, c))\n",
" return patches\n",
"\n",
"def jigsaw(patches, num_visible_patches): \n",
" b, num_patches, *_ = patches.shape\n",
" shuffled_idx = np.argsort(np.random.uniform(size=(b, num_patches)))\n",
" visible_idx = shuffled_idx[:, :num_visible_patches]\n",
" hidden_idx = shuffled_idx[:, num_visible_patches:]\n",
" batch_idx = np.arange(b)[:, np.newaxis]\n",
" visible = patches[batch_idx, visible_idx]\n",
" hidden = patches[batch_idx, hidden_idx]\n",
" return visible, visible_idx, hidden, hidden_idx"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "4g70txMzXIkK",
"colab_type": "text"
},
"source": [
"Now we download the CIFAR10 data set. We divide the train set into train (90%) and validation (10%). Furthermore, we simulate the case of having significantly more unlabeled data than labeled by randomly selecting only 20% of train data for later finetuning."
]
},
{
"cell_type": "code",
"metadata": {
"colab_type": "code",
"id": "B0hwidx_vn0D",
"outputId": "60e6391b-db6e-42be-c0df-033be6d6b1a6",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 51
}
},
"source": [
"(x_train, y_train), (x_test, y_test) = tf.keras.datasets.cifar10.load_data()\n",
"\n",
"x_train = x_train.astype('float32') / 255\n",
"x_test = x_test.astype('float32') / 255\n",
"\n",
"x_train, x_val, y_train, y_val = train_test_split(\n",
" x_train, y_train, test_size=0.1, stratify=y_train, random_state=12345)\n",
"\n",
"# we simulate a scenario where additional unlabeled data is available\n",
"# by training networks for image classification on only a subset of labeled data\n",
"subset_size = 0.2\n",
"\n",
"if subset_size < 1:\n",
" x_train_subset, _, y_train_subset, _ = train_test_split(\n",
" x_train, y_train, train_size=subset_size,\n",
" stratify=y_train, random_state=12345)\n",
"else:\n",
" x_train_subset, y_train_subset = x_train, y_train"
],
"execution_count": 3,
"outputs": [
{
"output_type": "stream",
"text": [
"Downloading data from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz\n",
"170500096/170498071 [==============================] - 4s 0us/step\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "5ZfuN6jobt-y",
"colab_type": "text"
},
"source": [
"Let's visualize jigsaw puzzle created from a random CIFAR10 image."
]
},
{
"cell_type": "code",
"metadata": {
"id": "dcVKI3ZIPfUt",
"colab_type": "code",
"colab": {}
},
"source": [
"def plot_patches(patches, hidden_idx=None, hide=False):\n",
" p, h, w, c = patches.shape\n",
" assert h == w, 'patch height and width must be equal'\n",
" rows = cols = int(np.sqrt(p))\n",
" fig = plt.figure(figsize=(rows, cols))\n",
" gs = gridspec.GridSpec(rows, cols) \n",
" gs.update(wspace=0.060, hspace=0.055)\n",
" if hidden_idx is None:\n",
" hidden_idx = [] \n",
" for i, patch in enumerate(patches):\n",
" ax = fig.add_subplot(gs[i])\n",
" ax.set_xticks([])\n",
" ax.set_yticks([])\n",
" if i in hidden_idx and not hide:\n",
" plt.setp(ax.spines.values(), color='red', linewidth=2)\n",
" else:\n",
" plt.setp(ax.spines.values(), color='white')\n",
" if i not in hidden_idx or not hide:\n",
" ax.imshow(patch)"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "JdIumVmgQ00I",
"colab_type": "code",
"outputId": "230c5851-6ffe-46a7-abeb-dbdfb8078695",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 265
}
},
"source": [
"_, h, w, c = x_train.shape\n",
"assert h == w, 'image height and width must be equal'\n",
"b = 512\n",
"hidden_ratio = 0.2\n",
"patch_dim = 8\n",
"# create jigsaw puzzle by randomly picking hidden and visible patches\n",
"num_patches = (h // patch_dim) ** 2\n",
"num_visible_patches = int(np.ceil(num_patches * (1 - hidden_ratio)))\n",
"num_hidden_patches = num_patches - num_visible_patches\n",
"patches = extract_patches(x_train[:b], patch_dim, patch_dim)\n",
"visible, visible_idx, hidden, hidden_idx = jigsaw(patches, num_visible_patches)\n",
"# solve the jigsaw puzzle by sorting patches according to the patch indices\n",
"patches = np.concatenate([visible, hidden], axis=1)\n",
"patch_idx = np.concatenate([visible_idx, hidden_idx], axis=1)\n",
"sorted_patch_idx = np.argsort(patch_idx)\n",
"batch_idx = np.arange(b)[:, np.newaxis]\n",
"patches = patches[batch_idx, sorted_patch_idx]\n",
"# plot random jigsaw puzzle\n",
"i = np.random.randint(b)\n",
"print('hidden patches', hidden_idx[i])\n",
"plot_patches(patches[i], hidden_idx[i], hide=True)"
],
"execution_count": 5,
"outputs": [
{
"output_type": "stream",
"text": [
"hidden patches [14 3 13]\n"
],
"name": "stdout"
},
{
"output_type": "display_data",
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAO0AAADnCAYAAADy1tHpAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAUAElEQVR4nO2dS4wd2VnH//W4j37a7e4e2/NI4gwi\ng4SGRCJgJYKgiGyQWIzEDqEghKJIjISyQkhs2bFCsGOHkGCBECJsIBMFEJJhhHgoMOMkMyGehz3u\nttvt7tv3UVWnWDg945nz/U/1vWnG/Xn+P6k3p6pO1alb/1t9/98535e1bdtCCOGG/FFfgBBiPiRa\nIZwh0QrhDIlWCGdItEI4Q6IVwhll1w4v/dt3MZ5WUfuHFik6pdMsDXr4xc/+uLntGy9/xxzjYnw4\n98U6y9Kghy999lNR+9+/fH3O8fExsC3ZHL3/KB0+GONz72vb2xshhMcncpnnGTY2Vuj2TtGOpxWO\nJrOoPRDRLvLhJW/3h/BZjKcVRsYYF+PRiZYx//jOrmgtQmgfK9F2oX+PhXCGRCuEMyRaIZwh0Qrh\njE4jqpcDvSJ2Bdjv/lM3ok6JsuRXVpYZeub2Ba6sXeAOLHCI5QOWxud03G59hotx6lbUXOcpi1M+\njUP0phXCGRKtEM6QaIVwhkQrhDMkWiGc0ekeAwFom6g1a4O9+yLmacKkPS1PMm/591Pe1sjbeo7e\nEtP8mKueJUaSGD+b4211l5Pv4Aw1Mswxvkc9rzRBZjyLHzX0phXCGRKtEM6QaIVwhkQrhDMkWiGc\nIdEK4YzOkE+RNSjz2GYPgYR8For5JDYtlNYm7jAHu14gQzC3p6I0vLPTnVDfkhCL1crGWCCgSIw/\nIhWDW4DT7C0/5WvziN60QjhDohXCGRKtEM6QaIVwhkQrhDO63WMEFDAWDBA3MlvgeyDtB87vFlpH\nsOsFHriuuTHGhWxPvmJggc44mXFx/DMh4zsD0FvMbiNbqPIRQm9aIZwh0QrhDIlWCGdItEI4Q6IV\nwhkSrRDO6Az5tG2DNljhEGbWzx9ayJKT6RfJ8m+co+Wp6bO2RmbkiJq/4GN60yKwO2O152SMeaiR\nh3lyYHFOu77AvOtB8qASA3rTCuEMiVYIZ0i0QjhDohXCGRKtEM6QaIVwRvcqn7ZBYZRiYDmi0otZ\niL+fKMSc7I5ujDeUqVU+bUBujfGUwk3JdiwYLjEOYitgsjaY5TQWy77FPsMFOgMfO8uNhUTo7qOC\n3rRCOEOiFcIZEq0QzpBohXCGRCuEMzrd44w4qzmx/bJEBvjGWnjw4Ch+AQlnmWFeQuBDzdrGdFfz\nBdxuxmnnxbeugLvH9vi4c5tg3qROHfAFCKSYtopK600rhDckWiGcIdEK4QyJVghnSLRCOKPTPR4M\nBnN1mHKPw2m7x3PYn6lxDMk2Pv/1DLjHxiWwcbB2xllwj9kWayw5C2U4pWs8WdsuVLVZCPGI0L/H\nQjhDohXCGZ2/aV/65rcwHo+j9ldefc3cf+viJdrXJ6+QbU1ilssi2VWNYwaDAa7+zFVz92v/eg3T\n6TTunlVuS1zSo2QwGODqz8ZjvPYv9vgYqd9Lp/1jat7uhoMBPnc1HuM/X7uGyRxjXDBB7kLM099w\nMMTPGeN7mE7RjsdjHB0dRe337u2b+w+W12hfk8nE3pAQbdLzOaUkvNPpFGPj2j480S62HP2ksPGx\n87aJvhcT7f+/QCbTKcaT+OXCT3w2RXsS9O+xEM6QaIVwhkQrhDMkWiGc0V2AK9QITVy86Zsv/ZO5\n/8rqOu3r17/8gtm+vb1Bj7HOfUwOOzOfNZOpJdkjj7e1KQf7BP0/tHF+UhPCqGESH8RmnIXQJO/j\nPDzymThsPW3bAHTGnbH7AiOpKn4PU7ZgWXbK7L1+TjAGvWmFcIZEK4QzJFohnCHRCuEMiVYIZ3Ta\nWsOVZbTG+r5PP/ekuf9z+d/QvlZv23NDDwa/Ro9ZWeZrQZuWuHlZ7BSHRDbG0NgO+YfFAstzTVe5\nCfYYQqgRyDZvtGwcoX7wd+KOuHvMPg4rK2nnQQBIkkzzMwxNj3d0fB2dewghzhQSrRDOkGiFcIZE\nK4QzJFohnCHRCuGMzpBPnj/4+yCfef5j5v4f2+VW+t1b3zbb//3N/6LHfPGLn6bb2nDyzAuh4QsG\nQmjskMgiaRqs3KY/EmyMRhuZbN6GYG5bZNL8aZORWAm7NroooplvUUQ6IYp97iIV1kn111QnuiYA\nJwpb6U0rhDMkWiGcIdEK4QyJVghnSLRCOKM73Uw9RVvHOXOH5+0FAy+9/Ena17dfv2+23w62qwwA\nK+dW6Laf+smPm+1FEX8X0YnmAEKoEOZx+BLe40Le8WkZztRZrdDU8fiyU3W6E/5p6jzzGtjMIW/n\nWxSx0MhT0YTkEMlGqz+2uuAh9KYVwhkSrRDOkGiFcIZEK4QzJFohnCHRCuGM7lKX9+9jNDo0tth6\nf3XvKdrXjX37mIvb3Ep/7ZX/pdu2V/pm+xWjDm6ZqDDQa1vUhtXOrqpNfNdl5KhkdCURSqiMUM3x\nmaJuevbH2cuAvnXJJOdRnYg6tCcISUQkxz5n8IVVgmgCMmNblpF7mywSYV9TBp4jKiTGETL7eTFS\nrynkI8TjiEQrhDMkWiGcIdEK4QyJVghnSLRCOKMz5NPPatRZHHYYjWfm/murdqFnANhct1dhfP4z\nvKj0L3/pc3Tb8tAuGbK6MozaysES7ecTm8uoV428UiwskPiqCzTkww8qcn7PRkdWuA145fr34nOs\nnbP7b2coQ1yS5ZnLT5j75+BhBxZBCakVMIkwBt9k99cjn+PT55cwG8bH3Ny5Z+4/GtklagCg6Nml\nOYZDLpcLF5bptoOJHbbb2z+K2qoTFKDWm1YIZ0i0QjhDohXCGRKtEM6QaIVwRqdVlWX2xOa1dTt3\n0y984fO0r+9e3zTbiVkHAJhMbPcPAM4t225pZuURSszDHhYTNMU4al9etZ3KMuXwWeUYAKDgDnGd\ncF6fetK+Z69877pxbntC+5/9+V9i5/Y7UfvPX33e3P9TP8bzfD118aLZvn35Aj2mXyYWBZAFEStD\nezFIVsaRAQB49pl1oImP2Vi3zz2e2NEPAKjIuoDhEi9w/uTT9n0BgFFtPy//c/2N+BxD7kIfozet\nEM6QaIVwhkQrhDMkWiGcIdEK4YxO93i4bLvEBTm0brhTuHHBdkJJ1hgAwGB5jW4rh+Ta+rEdXfT5\n3GO2Le/ZTmWWco9ZXpmEe5wn3OOMHLeyGjvnSyv2vdrc3DLb18/Zc75ZPwDQI+5mQe4VAOSpwq6Z\nfS+zkoQUCuLgkvaib7vEvTbxGRL3uOzzBzUr+LacnMtyigcDfh/fPVfbLlI5WQjxqNC/x0I4Q6IV\nwhkSrRDO6DSi/uHrf4GxsRD7zqH9A38247lhi8yeS3hnj09V/P4PXqfbVogxcPGJeErd+rkN/MaL\nv2Pu/9d/+kcYHexH7UtLtkHVJHLcnjtvGwl5zm/1/oiX2RwQs+2Nm3eitrW1dfzql38zav/933sR\ne3d3ovaGJDjeOLdNr+fZj9t5rZ977hP0mAFJVgAAF7dWzfbDQ3vxf9lfxjPP/1LUfuM//hb1LF5U\nXhplTwEgL7kxyBbm5wWfC9tPGKb5sm347eyOorZeb4Bnf+Iq7Qs4SbLyo0McHcZ1ZQ8P7JX/0+n8\nor23t0eP2d25TbdNBvbDMOzN562NDvZxuB9fQ6jtMdYJ0fZ7truaEu2IfAECQNPY9+z+Pv+i+yB7\nd3ewu3Mr7puItk2Ueb143h7fdGxHBgAgA/9SCpUtqnpqi5ZRz47sY4j7XgQuWpbXvkiItiGJ4h9s\ntO9ZNYvnu58E/XsshDMkWiGcIdEK4YzO37R3br2Gg/3Y9Hjz7djYAIC378VmwDEHR/ZvN/JTFwBQ\nJr5Wvr8bm0cAcOOt3ajt4qW4KNcxX//Wf+KdW/Fvvhu37d9VDSsCBWDJrHTFf5sCwLTm/T39pG0K\nZcZv5EuXLuKrL8b7hpAhhPi63to9MPueVfZnCwDVxL4nr994ix5z4dw63Xblij2+/371O2b7+Qvb\n+NpP/0rU/nff+Efc34s/9xnJgjkeTeg1DYgq+mR2IAAMevxBrYh3sHs3fn43Np/A7z7/BdoXoDet\nEO6QaIVwhkQrhDMkWiGcIdEK4YxO9/j6mz/A3TtxJr9Bz55RsrbKa6TkhT0zpkzUsmlq/r1yd2S7\n0YWxRjMv+TSf5aUGK8uxg7t1nhyQmhFFMg+GxDrj6YyP8col23m8+U7s8PZbew3qWzd3cetm/Bke\nHpFMiH0+7fCNMXHUb3LH+eIGn+J34217mup4an+2R5V9Hw9GB9g/jN1YMkTc3uGz8JgTzNZXA2Cl\nhwAAdeAzwj5IYOuFH76OE/cmhDgTSLRCOEOiFcIZEq0QzpBohXCGRCuEMzpDPgfTPexP4gUD+dQO\n04SGL+gGbLt+2vDvjiZwL33jgj3R3jpmeY2HHfpLBxisxOGCZ+ykCsitimQ/hCVEaBPjSKwlQJ3b\ni937K/GE996SPQl+eK7B8iQOeS2dt8exup4I25HwXCqp5wT2wg4AmJBT5aSQWZXbC1Levn8Hu3tx\nwgR2b9sBDzOOSQ7VliRFAOwFHO9ttO9NZrwzZ1l3eEhvWiGcIdEK4QyJVghnSLRCOEOiFcIZne7x\naHaEAyM1ZR3sCfhtYjK9lfIE4Ok4AGAJ3OXrEYfxsI1d1ME0zjH77v6zI9y3xtgQR5S4gQBQEIc8\nZ4W5kM6nWxKnOjdSPud9290shxXKpdiVZJd00Nyl19PW9tjJR/Hg/Cm3nbjRDbn348bORT2q9nBQ\nxVEONpM/8ZjyjYmDWp4xCC1JpGxFOfrT7veo3rRCOEOiFcIZEq0QzpBohXCGRCuEMyRaIZzRGfI5\nmkwxGschlIJkbp8kquZluW2/h0SCnTy38x4BQEauoarjcFTV8BxRs7rBtIqvu0cc/iwVLyBDmSZK\ngCYKFtD+rAn6w9IOLRzca7B/Nx4/K/SWWhDBtpWJsNU0cbsmmb3AhIWQZjNSrXFSYTqO++LXyy8q\ngz2WGauBCaBK5AALZLGIValiMuGVD47Rm1YIZ0i0QjhDohXCGRKtEM6QaIVwhkQrhDM6Qz7LvRKr\nvTjswha65Iniuj1SMiNVU6FJhFfYmTaX4iUwGwMeOjq/VKBaiW9FQZbBMAsfACqyqd9PhYkS4QLS\nbh2xum6vmDl/vodqFt8TVrCbROYA8FVciUVMCMlQCck5RUIoZWMsbwKAqgcYYxyTSF+bGOSAPaep\nQbIVYeDh0cKQX68l43sIvWmFcIZEK4QzJFohnCHRCuEMiVYIZ3S6x5PQ4ijE7h9zBEPCYGPOasJc\nRCJxPQI5WV7H7SR9EgBgf9Ji7yi+iMYY94Nr4oOsyTGJutlJt5aNsTXs++HUtkrvTSrcMSbTMzM0\ntWCgx3JWJT73lOnKFpGwY2qyKKJcaVGuGtuIq5ta9FGQwfQK/o7LiUMMADVZEFIbudF6bJXKw+fq\n3EMIcaaQaIVwhkQrhDMkWiGcIdEK4YxO93hr85LZTt3jhBPMUogs7B4TBzA3ZuZubdnjSG0Lztzj\n7a3L5r7bZHzMoc0Sdu8i7jEpuvDDczF3195/kzyPrJ1ViUi7x3Z7mfigUu4xq5FrucdsHA+Ttalq\nwEKIM4f+PRbCGRKtEM6QaIVwRqcR9ZXffgE7uzejdrqcPWEqMVcpNW0usd7cLBX44Dxxf09sX8af\n/PFfRe17e6PkonZv5HmGjY2V97U9TmO0xgcAv/W1F7Cze8s4ghlR/H2VEdMwz/jDnTIn2Z23Hvut\nrUv4wz+In9OH6RTtzu5N3Hrnzah9EdEyz4vN9QTSoqX1Y5PFRz/Qf2gfmwea8VEY487uLfM5PU3R\nFvliomW3PjGVOYn+PRbCGRKtEM6QaIVwRudv2rWyxFFp7Eb+T0/VkmrID95UZrxUcSqWTdCqrbSc\nmJEk/FOGHL1w8ndQ6hd+VrNUo7z/lJfDfu0WxoNanGAMetMK4QyJVghnSLRCOEOiFcIZEq0Qzuh0\nj1d6AWv92BqrKlLnJjEzpCR1fnr9xdYphta2hK21mFtrA9qP8M8wD1gyZiy15L3UJtKGNqT+T2pN\ndJFQElv8OpvF19AabdF1dO4hhDhTSLRCOEOiFcIZEq0QzpBohXCGRCuEMzpDPmUvQ9k3JjaT8E0w\n0kK+u40sWq+rxILkkoeDysL25q0spiG5lEF4JysKZGUck8nIwvU+eX4BoG3ssEudWL2SJ9LOlqW9\nrTC6G8ZJOeJzde8ihDhLSLRCOEOiFcIZEq0QzpBohXBGp3uM0Jo5IAel7aTlqbyQrJhUythNpVcl\naWpmhoOdJxMyC+/0ioB+YS0YIM9cqGhfLXnmUsXlmsAf4oYtAjBSy7SJ6MsxetMK4QyJVghnSLRC\nOEOiFcIZEq0QzpBohXBG94IBZOgZtnnLsrAbtvsxfZKhvWClApCuVBaInR+MhD3nhiox8Dgz7OVY\n6sfPUUUWqVQNfx5KEkosE+HHLLGtIY+wlaeqpPUI3kNvWiGcIdEK4QyJVghnSLRCOEOiFcIZEq0Q\nzugM+eS5XUu3Ijb2dMpDNDXJ99QSWx4AmkRensHA/s6xrq0/VY6ox5lJ3WBcxZ/xjJT4mPJFPsis\nujLguZ4AoEhFFEkIKRhFpatEiPPd7jr3EEKcKSRaIZwh0QrhDIlWCGdItEI4o9M9vrvfYHcvduVY\ngV1WQBcAZmRSdZ4oRD1JVR84Yjmi4va8pxxRjzO394Fbe3F7VdtRg9Rz2pKNLfhBWWKif2a4xHTf\nQXeUQ29aIZwh0QrhDIlWCGdItEI4Q6IVwhmd7vHm1mWzfRH3uCTpZrKWf3dM2SRn8DmiVoqRrW0y\njkSaEI9Y43mcxsjGsr19yWyvmXucOAd3jzmpO5xKRfNBniDP6fv6a9kVCiHOJPr3WAhnSLRCOEOi\nFcIZEq0QzpBohXCGRCuEM/4PlRslyyQL6xIAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 288x288 with 16 Axes>"
]
},
"metadata": {
"tags": []
}
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "iVn6ds30978H",
"colab_type": "text"
},
"source": [
"**Transformer** \n",
"---\n",
"Here we have the implementation of the encoder part of the Transformer as proposed in [Attention Is All You Need \\[arXiv:1706.03762\\]](https://arxiv.org/abs/1706.03762). We will use these parts of the Transformer in our Attention Pooling layer. Refer to https://www.tensorflow.org/tutorials/text/transformer for a complete Tensorflow guide on implementing and training Transformers.\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "H4LLEIzauWgN",
"colab_type": "code",
"colab": {}
},
"source": [
"class MultiHeadAttention(tf.keras.layers.Layer):\n",
" def __init__(self, d_model, num_heads, **kwargs):\n",
" super().__init__(**kwargs)\n",
" assert d_model % num_heads == 0\n",
" self.d_model = d_model\n",
" self.num_heads = num_heads\n",
" self.depth = d_model // num_heads\n",
" self.W_v = tf.keras.layers.Dense(d_model)\n",
" self.W_k = tf.keras.layers.Dense(d_model)\n",
" self.W_q = tf.keras.layers.Dense(d_model)\n",
" self.W_o = tf.keras.layers.Dense(d_model)\n",
"\n",
" def scaled_dot_product_attention(self, v, k, q, mask=None):\n",
" dk = tf.cast(self.d_model, tf.float32)\n",
" attn_logits = tf.matmul(q, k, transpose_b=True) / tf.sqrt(dk)\n",
" if mask is not None:\n",
" attn_logits += mask * -1e9\n",
" attn_weights = tf.nn.softmax(attn_logits, axis=-1)\n",
" scaled_attn = tf.matmul(attn_weights, v)\n",
" return scaled_attn, attn_weights\n",
"\n",
" def split_heads(self, x):\n",
" batch_size, n_sequences, d_model = x.shape\n",
" x = tf.reshape(x, (-1, n_sequences, self.num_heads, self.depth))\n",
" x = tf.transpose(x, (0, 2, 1, 3))\n",
" return x\n",
"\n",
" def concat_heads(self, x):\n",
" batch_size, num_heads, n_sequences, depth = x.shape\n",
" x = tf.transpose(x, (0, 2, 1, 3))\n",
" x = tf.reshape(x, (-1, n_sequences, self.d_model))\n",
" return x\n",
"\n",
" def call(self, v, k, q, mask=None):\n",
" v = self.split_heads(self.W_v(v))\n",
" k = self.split_heads(self.W_k(k))\n",
" q = self.split_heads(self.W_q(q))\n",
" scaled_attn, attn_weights = self.scaled_dot_product_attention(v, k, q, mask)\n",
" scaled_attn = self.concat_heads(scaled_attn)\n",
" mh_attn = self.W_o(scaled_attn)\n",
" return mh_attn, attn_weights\n",
"\n",
"class EncoderLayer(tf.keras.layers.Layer):\n",
" def __init__(self, d_model, num_heads, dff, dropout=0.1, **kwargs):\n",
" super().__init__(**kwargs)\n",
" self.mha = MultiHeadAttention(d_model, num_heads)\n",
" self.dropout1 = tf.keras.layers.Dropout(dropout)\n",
" self.layernorm1 = tf.keras.layers.LayerNormalization()\n",
" self.ff1 = tf.keras.layers.Dense(dff)\n",
" self.relu1 = tf.keras.layers.ReLU()\n",
" self.ff2 = tf.keras.layers.Dense(d_model)\n",
" self.dropout2 = tf.keras.layers.Dropout(dropout)\n",
" self.layernorm2 = tf.keras.layers.LayerNormalization()\n",
"\n",
" def call(self, x, training=None, mask=None):\n",
" shortcut = x\n",
" x, _ = self.mha(x, x, x, mask)\n",
" x = self.dropout1(x, training=training)\n",
" x = self.layernorm1(x + shortcut)\n",
" shortcut = x\n",
" x = self.ff1(x)\n",
" x = self.relu1(x)\n",
" x = self.ff2(x)\n",
" x = self.dropout2(x)\n",
" x = self.layernorm2(x + shortcut)\n",
" return x"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "L9b5NgImOZEE",
"colab_type": "text"
},
"source": [
"**Attention Pooling**\n",
"---\n",
"We implement the attention pooling mechanism as the mean vector of the output of the encoder (Transformer)."
]
},
{
"cell_type": "code",
"metadata": {
"id": "ovHCCeEbh_1u",
"colab_type": "code",
"colab": {}
},
"source": [
"class AttentionPooling(tf.keras.layers.Layer):\n",
" def __init__(self, num_layers, d_model, num_heads,\n",
" dff, dropout=0.1, **kwargs):\n",
" super().__init__(**kwargs)\n",
" self.d_model = d_model\n",
" self.enc_layers = [EncoderLayer(d_model, num_heads, dff, dropout) \n",
" for _ in range(num_layers)]\n",
"\n",
" def call(self, x, training=None, mask=None):\n",
" dk = tf.cast(self.d_model, tf.float32)\n",
" x *= tf.sqrt(dk)\n",
" for enc_layer in self.enc_layers:\n",
" x = enc_layer(x, training, mask)\n",
" x = tf.reduce_mean(x, axis=1)\n",
" return x"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "bygXeJQWOgKg",
"colab_type": "text"
},
"source": [
"**Patch Network**\n",
"--\n",
"Below you can find the implementation of residual networks (ResNet) as proposed in [Deep Residual Learning for Image Recognition \\[arXiv:1512.03385\\]](https://arxiv.org/abs/1512.03385)). The ResNet used for image classification has $(2, 3, 4, 2)$ regular residual blocks and uses a stride of $1$ in the first convolutional layer as well as no max pooling after the first convolution. \n",
"\n",
"We refrain from reducing the spatial dimensions in the first layers of the model because of the already small image resolution. Furthermore, our patch network has significantly less convolutional layers compared to the reference paper due to the increased computational effort required to train a deeper network. Note that we use the regular residual blocks instead of bottleneck blocks. \n",
"\n",
"The patch network consists of the first 3 blocks of the aforementioned ResNet. After pretraining, the last block is appended to the patch network and the whole model is finetuned on a subset of CIFAR10 data set."
]
},
{
"cell_type": "code",
"metadata": {
"id": "T5gPrly853eL",
"colab_type": "code",
"colab": {}
},
"source": [
"BATCH_NORM_DECAY = 0.9\n",
"BATCH_NORM_EPSILON = 1e-5\n",
"\n",
"def batch_norm():\n",
" return tf.keras.layers.BatchNormalization(\n",
" momentum=BATCH_NORM_DECAY, epsilon=BATCH_NORM_EPSILON)\n",
" \n",
"def relu():\n",
" return tf.keras.layers.ReLU()\n",
"\n",
"def conv2d(filters, kernel_size=3, strides=1):\n",
" return tf.keras.layers.Conv2D(\n",
" filters, kernel_size, strides=strides, padding='same', use_bias=False,\n",
" kernel_initializer=tf.keras.initializers.VarianceScaling())\n",
"\n",
"class ResidualBlock(tf.keras.layers.Layer):\n",
" def __init__(self, filters, kernel_size=3, strides=1, **kwargs):\n",
" super().__init__(**kwargs)\n",
" self.filters = filters\n",
" self.kernel_size = kernel_size\n",
" self.strides = strides\n",
"\n",
" def build(self, input_shape):\n",
" num_chan = input_shape[-1]\n",
" self.conv1 = conv2d(self.filters, self.kernel_size, self.strides)\n",
" self.bn1 = batch_norm()\n",
" self.relu1 = relu()\n",
" self.conv2 = conv2d(self.filters, self.kernel_size, 1)\n",
" self.bn2 = batch_norm()\n",
" self.relu2 = relu()\n",
" if num_chan != self.filters or self.strides > 1:\n",
" self.proj_conv = conv2d(self.filters, 1, self.strides)\n",
" self.proj_bn = batch_norm()\n",
" self.projection = True\n",
" else:\n",
" self.projection = False\n",
" super().build(input_shape)\n",
"\n",
" def call(self, x):\n",
" shortcut = x\n",
" if self.projection:\n",
" shortcut = self.proj_conv(shortcut)\n",
" shortcut = self.proj_bn(shortcut)\n",
" x = self.conv1(x)\n",
" x = self.bn1(x)\n",
" x = self.relu1(x)\n",
" x = self.conv2(x)\n",
" x = self.bn2(x)\n",
" x = self.relu2(x + shortcut)\n",
" return x\n",
"\n",
"class BottleneckBlock(tf.keras.layers.Layer):\n",
" def __init__(self, filters, kernel_size=3, strides=1, expansion=4, **kwargs):\n",
" super().__init__(**kwargs)\n",
" self.filters = filters\n",
" self.kernel_size = kernel_size\n",
" self.strides = strides\n",
" self.expansion = expansion\n",
"\n",
" def build(self, input_shape):\n",
" num_chan = input_shape[-1]\n",
" self.conv1 = conv2d(self.filters, 1, 1)\n",
" self.bn1 = batch_norm()\n",
" self.relu1 = relu()\n",
" self.conv2 = conv2d(self.filters, self.kernel_size, self.strides)\n",
" self.bn2 = batch_norm()\n",
" self.relu2 = relu()\n",
" self.conv3 = conv2d(self.filters * self.expansion, 1, 1)\n",
" self.bn3 = batch_norm()\n",
" self.relu3 = relu()\n",
" if num_chan != self.filters * self.expansion or self.strides > 1:\n",
" self.proj_conv = conv2d(self.filters * self.expansion, 1, self.strides)\n",
" self.proj_bn = batch_norm()\n",
" self.projection = True\n",
" else:\n",
" self.projection = False\n",
" super().build(input_shape)\n",
"\n",
" def call(self, x):\n",
" shortcut = x\n",
" if self.projection:\n",
" shortcut = self.proj_conv(shortcut)\n",
" shortcut = self.proj_bn(shortcut)\n",
" x = self.conv1(x)\n",
" x = self.bn1(x)\n",
" x = self.relu1(x)\n",
" x = self.conv2(x)\n",
" x = self.bn2(x)\n",
" x = self.relu2(x)\n",
" x = self.conv3(x)\n",
" x = self.bn3(x)\n",
" x = self.relu3(x + shortcut)\n",
" return x\n",
"\n",
"class ResNet(tf.keras.Model):\n",
" def __init__(self, num_classes=1, blocks=(2, 2, 2, 2),\n",
" filters=(64, 128, 256, 512), kernel_size=(3, 3, 3, 3),\n",
" block_fn=ResidualBlock, include_top=True, **kwargs):\n",
" super().__init__(**kwargs)\n",
" self.conv1 = conv2d(filters[0], 7, 1)\n",
" self.bn1 = batch_norm()\n",
" self.relu1 = relu()\n",
" self.blocks = []\n",
" for stage, num_blocks in enumerate(blocks):\n",
" for block in range(num_blocks):\n",
" strides = 2 if block == 0 and stage > 1 else 1\n",
" res_block = block_fn(filters[stage], kernel_size[stage], strides)\n",
" self.blocks.append(res_block)\n",
" self.include_top = include_top\n",
" if include_top:\n",
" self.global_pool = tf.keras.layers.GlobalAveragePooling2D()\n",
" out_act = 'sigmoid' if num_classes == 1 else 'softmax'\n",
" self.classifier = tf.keras.layers.Dense(num_classes, out_act)\n",
"\n",
" def call(self, x):\n",
" x = self.conv1(x)\n",
" x = self.bn1(x)\n",
" x = self.relu1(x)\n",
" for res_block in self.blocks:\n",
" x = res_block(x)\n",
" if self.include_top:\n",
" x = self.global_pool(x)\n",
" x = self.classifier(x)\n",
" return x"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "hwKz5SJpXqYC",
"colab_type": "text"
},
"source": [
"**Selfie - Encoder**\n",
"--\n",
"Let's now move on to the encoder part of the Selfie model. The encoder extracts features from each visible image patch using the patch network, then pools the extracted feature vectors into a single context vector of an image. "
]
},
{
"cell_type": "code",
"metadata": {
"id": "TmoWsBHVSFhy",
"colab_type": "code",
"colab": {}
},
"source": [
"class Encoder(tf.keras.layers.Layer):\n",
" def __init__(self, patch_net, d_model, num_attn_layers=3,\n",
" num_heads=8, dff=256, dropout=0.1, **kwargs):\n",
" super().__init__(**kwargs)\n",
" self.patch_net = patch_net\n",
" self.d_model = d_model\n",
" self.attn_pool = AttentionPooling(num_attn_layers, d_model,\n",
" num_heads, dff, dropout)\n",
"\n",
" def call(self, visible_patches, training=None):\n",
" \"\"\" for each image compute a context vector from the visible patches \n",
"\n",
" visible_patches.shape == (b, num_visible_patches, patch_h, patch_w, c)\n",
" \n",
" context_vec.shape == (b, d_model)\n",
" \"\"\"\n",
" b, p, ph, pw, c = visible_patches.shape\n",
" # extract feature vectors from visible patches using the patch network\n",
" visible_patches = tf.reshape(visible_patches, (-1, ph, pw, c))\n",
" feature_vec = self.patch_net(visible_patches)\n",
" feature_vec = tf.reshape(feature_vec, (-1, p, self.d_model))\n",
" # compute context vectors by attention pooling over patches within images\n",
" context_vec = self.attn_pool(feature_vec, training)\n",
" return context_vec"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "PU_x-t6VYXqx",
"colab_type": "text"
},
"source": [
"**Selfie - Decoder**\n",
"--\n",
"On the other hand, the decoder extracts feature vectors from the hidden patches using the same patch network, then computes the similarity between these feature vectors and the context vector of an image to find the most fitting patch for a given position. \n",
"\n",
"Note that the context vector passed to the decoder differs from the one returned by the encoder. This is because before the similarity can be computed for a chosen position in the image, the position must be embedded into the context vector returned by the encoder."
]
},
{
"cell_type": "code",
"metadata": {
"id": "xHb6BaVhYbVp",
"colab_type": "code",
"colab": {}
},
"source": [
"class Decoder(tf.keras.layers.Layer):\n",
" def __init__(self, patch_net, d_model, **kwargs):\n",
" super().__init__(**kwargs)\n",
" self.patch_net = patch_net\n",
" self.d_model = d_model\n",
"\n",
" def call(self, hidden_patches, context_vec):\n",
" \"\"\" for each context vector at given position in the image\n",
" compute the most fitting hidden patch at this position \n",
"\n",
" hidden_patches.shape == (b, num_hidden_patches, patch_h, patch_w, c)\n",
" context_vec.shape == (b, num_hidden_idx, d_model)\n",
"\n",
" returns similarity matrix for each image in the batch where each row\n",
" represents a position of a masked patch in the image and each value\n",
" in the row corresponds to the score of a hidden patch at this\n",
" position. The hidden patch with a highest score is the best fit for\n",
" this position.\n",
"\n",
" similarity_vec.shape = (b, num_hidden_idx, num_hidden_patches) \n",
" \"\"\"\n",
" b, p, ph, pw, c = hidden_patches.shape\n",
" # extract feature vectors from hidden patches using the patch network\n",
" hidden_patches = tf.reshape(hidden_patches, (-1, ph, pw, c))\n",
" feature_vec = self.patch_net(hidden_patches)\n",
" feature_vec = tf.reshape(feature_vec, (-1, p, self.d_model))\n",
" # transpose to allow matrix broadcasting\n",
" feature_vec = tf.transpose(feature_vec, (0, 2, 1))\n",
" # compute similarity between hidden patches and context vectors\n",
" # to find the most fitting patch at every position in each image\n",
" similarity_vec = tf.matmul(context_vec, feature_vec)\n",
" return similarity_vec"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "M9Q78KBTkSm9",
"colab_type": "text"
},
"source": [
"**Selfie**\n",
"--\n",
"Now we bring all components of the network together. First, Selfie computes the context vector for each image using the visible patches. Next, for each position where a patch is missing (i.e. a position we want to find a patch for) a new context vector is produced which is a point-wise sum of the context vector returned by the encoder and the embedding for this positon. Finally, Selfie returns similarity of all hidden patches to every position in the image where a patch is missing. At each such position, the most fitting hidden patch is the one with the highest similarity value."
]
},
{
"cell_type": "code",
"metadata": {
"id": "XuKoQ9kWovmf",
"colab_type": "code",
"colab": {}
},
"source": [
"class Selfie(tf.keras.Model):\n",
" def __init__(self, patch_net, d_model, num_patches, **kwargs):\n",
" super().__init__(**kwargs)\n",
" self.patch_net = patch_net\n",
" self.d_model = d_model\n",
" self.pos_embed = tf.keras.layers.Embedding(num_patches, d_model)\n",
" self.encoder = Encoder(patch_net, d_model)\n",
" self.decoder = Decoder(patch_net, d_model)\n",
"\n",
" def call(self, x, training=None):\n",
" \"\"\" for each image represented by an array of visible patches compute\n",
" the most fitting hidden patch at the selected positions \n",
" where a patch is missing\n",
"\n",
" visible_patches.shape == (b, num_visible_patches, patch_h, patch_w, c)\n",
" hidden_patches.shape == (b, num_hidden_patches, patch_h, patch_w, c)\n",
" hidden_idx.shape == (b, num_hidden_idx)\n",
"\n",
" returns similarity matrix for each image in the batch where each row\n",
" represents a position of a masked patch in the image and each value\n",
" in the row corresponds to the score of a hidden patch at this\n",
" position. The hidden patch with a highest score is the best fit for\n",
" this position.\n",
"\n",
" similarity_vec.shape = (b, num_hidden_idx, num_hidden_patches)\n",
" \"\"\"\n",
" visible_patches, hidden_patches, hidden_idx = x\n",
" b, nh = hidden_idx.shape\n",
" # compute context vector for each image\n",
" context_vec = self.encoder(visible_patches, training)\n",
" # repeat context vector for each hidden patch in the image\n",
" context_vec = tf.tile(context_vec, (1, nh))\n",
" context_vec = tf.reshape(context_vec, (-1, nh, self.d_model))\n",
" # add positional embedding to the context vector\n",
" context_vec = context_vec + self.pos_embed(hidden_idx)\n",
" # compute similarity of hidden patches to each position in an image\n",
" similarity_vec = self.decoder(hidden_patches, context_vec)\n",
" return similarity_vec"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "e68dr6CqzNjC",
"colab_type": "text"
},
"source": [
"Let's perform a quick sanity check before moving on to training and evaluation."
]
},
{
"cell_type": "code",
"metadata": {
"id": "18VB3jiozMkg",
"colab_type": "code",
"outputId": "5ce5af00-c0ae-44c4-9d45-b6a93975f1a7",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 51
}
},
"source": [
"# batch_size, patch_height, patch_width, num_channels\n",
"b, ph, pw, c = 128, 8, 8, 3\n",
"# num_patches, num_visible_patches, num_hidden_patches\n",
"n, nv, nh = 25, 15, 10\n",
"assert n == nv + nh\n",
"# number of patches that will be matched with missing patches in the image\n",
"# note that this number may be smaller than the number of all missing patches\n",
"nh_idx = 5\n",
"assert nh_idx <= nh\n",
"\n",
"visible = tf.random.uniform((b, nv, ph, pw, c))\n",
"hidden = tf.random.uniform((b, nh, ph, pw, c))\n",
"idx = tf.argsort(tf.random.uniform((b, n)))\n",
"hidden_idx = idx[:, :nh_idx]\n",
"print('visible', visible.shape,\n",
" 'hidden', hidden.shape,\n",
" 'hidden_idx', hidden_idx.shape)\n",
"\n",
"d_model = 512\n",
"patch_net = tf.keras.Sequential([\n",
" ResNet(include_top=False),\n",
" tf.keras.layers.GlobalAveragePooling2D()\n",
"])\n",
"\n",
"selfie = Selfie(patch_net, d_model, n)\n",
"out = selfie((visible, hidden, hidden_idx), training=False)\n",
"\n",
"print('output', out.shape)\n",
"assert out.shape == (b, nh_idx, nh)"
],
"execution_count": 12,
"outputs": [
{
"output_type": "stream",
"text": [
"visible (128, 15, 8, 8, 3) hidden (128, 10, 8, 8, 3) hidden_idx (128, 5)\n",
"output (128, 5, 10)\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "820-cZNjkINl",
"colab_type": "text"
},
"source": [
"**Pretraining**\n",
"--\n",
"We will start with the implementation of data generators that allow iteration over the entire CIFAR10 data set and transform each image into a completely new jigsaw puzzle every time the image is encountered."
]
},
{
"cell_type": "code",
"metadata": {
"id": "6qwZCMiil6f9",
"colab_type": "code",
"colab": {}
},
"source": [
"class NumpyArrayIterator(tf.keras.utils.Sequence):\n",
" def __init__(self, x, y=None, batch_size=32, shuffle=True):\n",
" self.x = x\n",
" self.y = y\n",
" self.batch_size = batch_size\n",
" self.shuffle = shuffle\n",
" self.index = np.arange(len(x))\n",
" self.on_epoch_end()\n",
" \n",
" def __len__(self):\n",
" return (len(self.x) + self.batch_size - 1) // self.batch_size\n",
" \n",
" def __getitem__(self, index):\n",
" index_array = self.index[index * self.batch_size:(index + 1) * self.batch_size]\n",
" batch_x = self.x[index_array]\n",
" if self.y is not None:\n",
" batch_y = self.y[index_array]\n",
" return batch_x, batch_y\n",
" else:\n",
" return batch_x\n",
" \n",
" def on_epoch_end(self):\n",
" if self.shuffle:\n",
" np.random.shuffle(self.index)\n",
"\n",
"class JigsawDataGenerator(tf.keras.utils.Sequence):\n",
" def __init__(self, image_data_generator, patch_dim,\n",
" num_visible_patches, num_labels=None, pad=False):\n",
" self.image_data_generator = image_data_generator\n",
" self.patch_dim = patch_dim\n",
" self.num_visible_patches = num_visible_patches\n",
" self.num_labels = num_labels\n",
" self.pad = pad\n",
"\n",
" def __len__(self):\n",
" return len(self.image_data_generator)\n",
"\n",
" def __getitem__(self, index):\n",
" batch_x = self.image_data_generator[index] # (b, h, w, c)\n",
" if self.pad:\n",
" batch_x = np.pad(batch_x, ((0,), (self.pad,), (self.pad,), (0,)))\n",
" # create jigsaw puzzle\n",
" patches = extract_patches(batch_x, self.patch_dim, self.patch_dim)\n",
" vis, vis_idx, hid, hid_idx = jigsaw(patches, self.num_visible_patches)\n",
" b, nh = hid_idx.shape\n",
" num_labels = self.num_labels or nh\n",
" # sort the visible patches by their position in the image\n",
" batch_idx = np.arange(b)[:, np.newaxis]\n",
" sorted_vis_idx = np.argsort(vis_idx)\n",
" sorted_vis = vis[batch_idx, sorted_vis_idx]\n",
" # select hidden positions to solve\n",
" labels = np.argsort(np.random.uniform(size=(b, nh)))[:, :num_labels]\n",
" hid_idx = hid_idx[batch_idx, labels]\n",
" return (vis, hid, hid_idx), labels\n",
"\n",
" def on_epoch_end(self):\n",
" self.image_data_generator.on_epoch_end()"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "vW89Hex7cVDx",
"colab_type": "text"
},
"source": [
"Finally, we train the Selfie network to solve jigsaw puzzles. For this task we create puzzles from our 45,000 train images ($32x32x3$) by splitting them into $5x5$ grids containing patches of size $8x8x3$ (we employ zero-padding of size $4$). We hide 50% of patches.\n",
"\n",
"The model is trained with Adam optimizer for 20 epochs. The accuracy describes the percentage of correctly matched patches."
]
},
{
"cell_type": "code",
"metadata": {
"id": "OoJqWrp6lQt7",
"colab_type": "code",
"outputId": "0cd3d7cf-39d4-4623-fa8f-6ee86fa92124",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 697
}
},
"source": [
"_, h, w, c = x_train.shape\n",
"assert h == w, 'image height and width must be equal'\n",
"b = 128\n",
"hidden_ratio = 0.5\n",
"patch_dim = 8\n",
"pad = 4\n",
"\n",
"num_patches = ((h + 2 * pad) // patch_dim) ** 2\n",
"num_hidden_patches = int(np.floor(num_patches * hidden_ratio))\n",
"num_visible_patches = num_patches - num_hidden_patches\n",
"cifar10_data_gen = NumpyArrayIterator(x_train, batch_size=b)\n",
"jigsaw_data_gen = JigsawDataGenerator(cifar10_data_gen, patch_dim,\n",
" num_visible_patches, pad=pad)\n",
"\n",
"loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n",
"accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name='accuracy')\n",
"\n",
"learning_rate = 0.01\n",
"optimizer = tf.keras.optimizers.Adam(\n",
" learning_rate, beta_1=0.9, beta_2=0.98, epsilon=1e-9)\n",
"\n",
"d_model = 256\n",
"patch_net = ResNet(blocks=(2, 3, 4), include_top=False)\n",
"selfie_patch_net = tf.keras.Sequential([\n",
" patch_net,\n",
" tf.keras.layers.GlobalAveragePooling2D()\n",
"])\n",
"selfie = Selfie(selfie_patch_net, d_model, num_patches)\n",
"\n",
"selfie.compile(optimizer, loss, metrics=[accuracy])\n",
"\n",
"selfie.fit_generator(jigsaw_data_gen, epochs=20, workers=4,\n",
" use_multiprocessing=True)\n",
"\n",
"patch_net.save_weights('pretrained.weights', save_format='tf')"
],
"execution_count": 16,
"outputs": [
{
"output_type": "stream",
"text": [
"Epoch 1/20\n",
"352/352 [==============================] - 205s 582ms/step - loss: 2.2205 - accuracy: 0.3507\n",
"Epoch 2/20\n",
"352/352 [==============================] - 203s 577ms/step - loss: 0.8398 - accuracy: 0.5664\n",
"Epoch 3/20\n",
"352/352 [==============================] - 202s 574ms/step - loss: 0.7852 - accuracy: 0.6022\n",
"Epoch 4/20\n",
"352/352 [==============================] - 201s 571ms/step - loss: 0.7548 - accuracy: 0.6266\n",
"Epoch 5/20\n",
"352/352 [==============================] - 202s 574ms/step - loss: 0.7383 - accuracy: 0.6380\n",
"Epoch 6/20\n",
"352/352 [==============================] - 202s 573ms/step - loss: 0.7292 - accuracy: 0.6466\n",
"Epoch 7/20\n",
"352/352 [==============================] - 202s 572ms/step - loss: 0.7191 - accuracy: 0.6517\n",
"Epoch 8/20\n",
"352/352 [==============================] - 202s 574ms/step - loss: 0.7152 - accuracy: 0.6569\n",
"Epoch 9/20\n",
"352/352 [==============================] - 202s 574ms/step - loss: 0.7091 - accuracy: 0.6587\n",
"Epoch 10/20\n",
"352/352 [==============================] - 202s 574ms/step - loss: 0.7063 - accuracy: 0.6619\n",
"Epoch 11/20\n",
"352/352 [==============================] - 203s 578ms/step - loss: 0.6995 - accuracy: 0.6666\n",
"Epoch 12/20\n",
"352/352 [==============================] - 203s 578ms/step - loss: 0.6976 - accuracy: 0.6675\n",
"Epoch 13/20\n",
"352/352 [==============================] - 202s 574ms/step - loss: 0.6936 - accuracy: 0.6700\n",
"Epoch 14/20\n",
"352/352 [==============================] - 202s 575ms/step - loss: 0.6905 - accuracy: 0.6733\n",
"Epoch 15/20\n",
"352/352 [==============================] - 202s 575ms/step - loss: 0.6848 - accuracy: 0.6742\n",
"Epoch 16/20\n",
"352/352 [==============================] - 202s 575ms/step - loss: 0.6813 - accuracy: 0.6778\n",
"Epoch 17/20\n",
"352/352 [==============================] - 202s 575ms/step - loss: 0.6793 - accuracy: 0.6791\n",
"Epoch 18/20\n",
"352/352 [==============================] - 202s 573ms/step - loss: 0.6741 - accuracy: 0.6827\n",
"Epoch 19/20\n",
"352/352 [==============================] - 202s 575ms/step - loss: 0.6736 - accuracy: 0.6837\n",
"Epoch 20/20\n",
"352/352 [==============================] - 202s 574ms/step - loss: 0.6689 - accuracy: 0.6855\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4vmZeQQ3ZjqJ",
"colab_type": "text"
},
"source": [
"**Finetuning**\n",
"--\n",
"In the finetuning phase, we reuse the weights obtained from pretraining the first 3 residual blocks of the network. We train the resulting network for image classification on the 20% of the CIFAR10 train data (9,000 images). The validation (5,000 images) and test set (10,000 images) remain the same."
]
},
{
"cell_type": "code",
"metadata": {
"id": "W6_sXyadEd_R",
"colab_type": "code",
"outputId": "b8b34c72-6084-4ee9-cf80-5d442346a32a",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 1000
}
},
"source": [
"num_classes = np.unique(y_train).shape[0]\n",
"\n",
"patch_net.load_weights('pretrained.weights')\n",
"\n",
"pretrained_classifier = tf.keras.Sequential([\n",
" patch_net,\n",
" ResidualBlock(512, 3, 2),\n",
" ResidualBlock(512, 3, 1),\n",
" tf.keras.layers.GlobalAveragePooling2D(),\n",
" tf.keras.layers.Dense(num_classes, activation='softmax')\n",
"])\n",
"\n",
"learning_rate = 0.01\n",
"optimizer = tf.keras.optimizers.Adam(learning_rate, beta_1=0.9, beta_2=0.98,\n",
" epsilon=1e-9)\n",
"accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name='accuracy')\n",
"\n",
"pretrained_classifier.compile(optimizer=optimizer,\n",
" loss='sparse_categorical_crossentropy',\n",
" metrics=[accuracy])\n",
"\n",
"chkpt = tf.keras.callbacks.ModelCheckpoint(\n",
" 'finetuned.weights', monitor='val_accuracy', verbose=1, save_best_only=True,\n",
" save_weights_only=True, mode='max')\n",
"\n",
"finetune_history = pretrained_classifier.fit(\n",
" x_train_subset, y_train_subset, batch_size=128, epochs=100,\n",
" validation_data=(x_val, y_val), callbacks=[chkpt])\n",
"\n",
"pretrained_classifier.load_weights('finetuned.weights')"
],
"execution_count": 21,
"outputs": [
{
"output_type": "stream",
"text": [
"Train on 9000 samples, validate on 5000 samples\n",
"Epoch 1/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 2.0186 - accuracy: 0.3262\n",
"Epoch 00001: val_accuracy improved from -inf to 0.42980, saving model to finetuned.weights\n",
"9000/9000 [==============================] - 18s 2ms/sample - loss: 2.0166 - accuracy: 0.3269 - val_loss: 1.5974 - val_accuracy: 0.4298\n",
"Epoch 2/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 1.4165 - accuracy: 0.4935\n",
"Epoch 00002: val_accuracy improved from 0.42980 to 0.49300, saving model to finetuned.weights\n",
"9000/9000 [==============================] - 15s 2ms/sample - loss: 1.4160 - accuracy: 0.4939 - val_loss: 1.5273 - val_accuracy: 0.4930\n",
"Epoch 3/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 1.1781 - accuracy: 0.5807\n",
"Epoch 00003: val_accuracy improved from 0.49300 to 0.54980, saving model to finetuned.weights\n",
"9000/9000 [==============================] - 15s 2ms/sample - loss: 1.1779 - accuracy: 0.5810 - val_loss: 1.6571 - val_accuracy: 0.5498\n",
"Epoch 4/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 1.0373 - accuracy: 0.6295\n",
"Epoch 00004: val_accuracy improved from 0.54980 to 0.56480, saving model to finetuned.weights\n",
"9000/9000 [==============================] - 15s 2ms/sample - loss: 1.0382 - accuracy: 0.6293 - val_loss: 1.4487 - val_accuracy: 0.5648\n",
"Epoch 5/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.8918 - accuracy: 0.6797\n",
"Epoch 00005: val_accuracy improved from 0.56480 to 0.60160, saving model to finetuned.weights\n",
"9000/9000 [==============================] - 15s 2ms/sample - loss: 0.8919 - accuracy: 0.6801 - val_loss: 1.2033 - val_accuracy: 0.6016\n",
"Epoch 6/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.7747 - accuracy: 0.7208\n",
"Epoch 00006: val_accuracy did not improve from 0.60160\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.7740 - accuracy: 0.7209 - val_loss: 1.2473 - val_accuracy: 0.5858\n",
"Epoch 7/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.6837 - accuracy: 0.7513\n",
"Epoch 00007: val_accuracy improved from 0.60160 to 0.63240, saving model to finetuned.weights\n",
"9000/9000 [==============================] - 15s 2ms/sample - loss: 0.6853 - accuracy: 0.7509 - val_loss: 1.1106 - val_accuracy: 0.6324\n",
"Epoch 8/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.5688 - accuracy: 0.8002\n",
"Epoch 00008: val_accuracy did not improve from 0.63240\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.5678 - accuracy: 0.8004 - val_loss: 1.3221 - val_accuracy: 0.6200\n",
"Epoch 9/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.4923 - accuracy: 0.8157\n",
"Epoch 00009: val_accuracy did not improve from 0.63240\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.4929 - accuracy: 0.8156 - val_loss: 1.6109 - val_accuracy: 0.5878\n",
"Epoch 10/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.4130 - accuracy: 0.8501\n",
"Epoch 00010: val_accuracy did not improve from 0.63240\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.4150 - accuracy: 0.8497 - val_loss: 1.6968 - val_accuracy: 0.5894\n",
"Epoch 11/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.3479 - accuracy: 0.8763\n",
"Epoch 00011: val_accuracy did not improve from 0.63240\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.3471 - accuracy: 0.8767 - val_loss: 1.5736 - val_accuracy: 0.5988\n",
"Epoch 12/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.2837 - accuracy: 0.8977\n",
"Epoch 00012: val_accuracy did not improve from 0.63240\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.2844 - accuracy: 0.8974 - val_loss: 2.6163 - val_accuracy: 0.5184\n",
"Epoch 13/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.2438 - accuracy: 0.9150\n",
"Epoch 00013: val_accuracy did not improve from 0.63240\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.2437 - accuracy: 0.9150 - val_loss: 1.6032 - val_accuracy: 0.6256\n",
"Epoch 14/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.2046 - accuracy: 0.9237\n",
"Epoch 00014: val_accuracy did not improve from 0.63240\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.2048 - accuracy: 0.9233 - val_loss: 1.8819 - val_accuracy: 0.6278\n",
"Epoch 15/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.1788 - accuracy: 0.9376\n",
"Epoch 00015: val_accuracy did not improve from 0.63240\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.1788 - accuracy: 0.9376 - val_loss: 2.0077 - val_accuracy: 0.6144\n",
"Epoch 16/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.1620 - accuracy: 0.9429\n",
"Epoch 00016: val_accuracy improved from 0.63240 to 0.64200, saving model to finetuned.weights\n",
"9000/9000 [==============================] - 15s 2ms/sample - loss: 0.1624 - accuracy: 0.9429 - val_loss: 1.7066 - val_accuracy: 0.6420\n",
"Epoch 17/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.1534 - accuracy: 0.9455\n",
"Epoch 00017: val_accuracy improved from 0.64200 to 0.66340, saving model to finetuned.weights\n",
"9000/9000 [==============================] - 15s 2ms/sample - loss: 0.1532 - accuracy: 0.9457 - val_loss: 1.7775 - val_accuracy: 0.6634\n",
"Epoch 18/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.1326 - accuracy: 0.9554\n",
"Epoch 00018: val_accuracy did not improve from 0.66340\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.1326 - accuracy: 0.9554 - val_loss: 1.7652 - val_accuracy: 0.6454\n",
"Epoch 19/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.1197 - accuracy: 0.9567\n",
"Epoch 00019: val_accuracy did not improve from 0.66340\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.1210 - accuracy: 0.9564 - val_loss: 1.9555 - val_accuracy: 0.6448\n",
"Epoch 20/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.1340 - accuracy: 0.9550\n",
"Epoch 00020: val_accuracy did not improve from 0.66340\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.1340 - accuracy: 0.9549 - val_loss: 2.0985 - val_accuracy: 0.6524\n",
"Epoch 21/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.1155 - accuracy: 0.9628\n",
"Epoch 00021: val_accuracy did not improve from 0.66340\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.1153 - accuracy: 0.9629 - val_loss: 2.0044 - val_accuracy: 0.6472\n",
"Epoch 22/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0920 - accuracy: 0.9686\n",
"Epoch 00022: val_accuracy did not improve from 0.66340\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0924 - accuracy: 0.9683 - val_loss: 2.5958 - val_accuracy: 0.5996\n",
"Epoch 23/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.1177 - accuracy: 0.9613\n",
"Epoch 00023: val_accuracy did not improve from 0.66340\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.1180 - accuracy: 0.9610 - val_loss: 2.1603 - val_accuracy: 0.6298\n",
"Epoch 24/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0886 - accuracy: 0.9679\n",
"Epoch 00024: val_accuracy improved from 0.66340 to 0.67540, saving model to finetuned.weights\n",
"9000/9000 [==============================] - 15s 2ms/sample - loss: 0.0886 - accuracy: 0.9678 - val_loss: 1.7614 - val_accuracy: 0.6754\n",
"Epoch 25/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0882 - accuracy: 0.9699\n",
"Epoch 00025: val_accuracy did not improve from 0.67540\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0892 - accuracy: 0.9696 - val_loss: 2.1676 - val_accuracy: 0.6514\n",
"Epoch 26/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0956 - accuracy: 0.9666\n",
"Epoch 00026: val_accuracy did not improve from 0.67540\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0961 - accuracy: 0.9663 - val_loss: 2.1972 - val_accuracy: 0.6432\n",
"Epoch 27/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0813 - accuracy: 0.9729\n",
"Epoch 00027: val_accuracy did not improve from 0.67540\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0816 - accuracy: 0.9728 - val_loss: 2.9436 - val_accuracy: 0.5770\n",
"Epoch 28/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0942 - accuracy: 0.9683\n",
"Epoch 00028: val_accuracy did not improve from 0.67540\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0943 - accuracy: 0.9681 - val_loss: 2.3351 - val_accuracy: 0.6220\n",
"Epoch 29/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0720 - accuracy: 0.9741\n",
"Epoch 00029: val_accuracy did not improve from 0.67540\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0719 - accuracy: 0.9741 - val_loss: 2.2645 - val_accuracy: 0.6638\n",
"Epoch 30/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0951 - accuracy: 0.9720\n",
"Epoch 00030: val_accuracy did not improve from 0.67540\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0952 - accuracy: 0.9719 - val_loss: 2.7367 - val_accuracy: 0.5584\n",
"Epoch 31/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0900 - accuracy: 0.9686\n",
"Epoch 00031: val_accuracy did not improve from 0.67540\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0897 - accuracy: 0.9688 - val_loss: 2.2215 - val_accuracy: 0.6322\n",
"Epoch 32/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0360 - accuracy: 0.9865\n",
"Epoch 00032: val_accuracy did not improve from 0.67540\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0361 - accuracy: 0.9863 - val_loss: 2.2220 - val_accuracy: 0.6416\n",
"Epoch 33/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0696 - accuracy: 0.9750\n",
"Epoch 00033: val_accuracy did not improve from 0.67540\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0694 - accuracy: 0.9751 - val_loss: 2.0110 - val_accuracy: 0.6700\n",
"Epoch 34/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0676 - accuracy: 0.9782\n",
"Epoch 00034: val_accuracy did not improve from 0.67540\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0676 - accuracy: 0.9782 - val_loss: 2.5047 - val_accuracy: 0.6182\n",
"Epoch 35/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0669 - accuracy: 0.9767\n",
"Epoch 00035: val_accuracy improved from 0.67540 to 0.68700, saving model to finetuned.weights\n",
"9000/9000 [==============================] - 15s 2ms/sample - loss: 0.0672 - accuracy: 0.9764 - val_loss: 1.8377 - val_accuracy: 0.6870\n",
"Epoch 36/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0600 - accuracy: 0.9811\n",
"Epoch 00036: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0599 - accuracy: 0.9811 - val_loss: 1.9627 - val_accuracy: 0.6706\n",
"Epoch 37/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0449 - accuracy: 0.9847\n",
"Epoch 00037: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0448 - accuracy: 0.9848 - val_loss: 2.5476 - val_accuracy: 0.6356\n",
"Epoch 38/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0667 - accuracy: 0.9766\n",
"Epoch 00038: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0665 - accuracy: 0.9767 - val_loss: 2.1200 - val_accuracy: 0.6584\n",
"Epoch 39/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0625 - accuracy: 0.9821\n",
"Epoch 00039: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0624 - accuracy: 0.9821 - val_loss: 2.3840 - val_accuracy: 0.6374\n",
"Epoch 40/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0552 - accuracy: 0.9802\n",
"Epoch 00040: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0551 - accuracy: 0.9803 - val_loss: 2.6027 - val_accuracy: 0.6072\n",
"Epoch 41/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0388 - accuracy: 0.9872\n",
"Epoch 00041: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0406 - accuracy: 0.9867 - val_loss: 2.2553 - val_accuracy: 0.6650\n",
"Epoch 42/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0718 - accuracy: 0.9756\n",
"Epoch 00042: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0717 - accuracy: 0.9757 - val_loss: 2.2548 - val_accuracy: 0.6354\n",
"Epoch 43/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0481 - accuracy: 0.9855\n",
"Epoch 00043: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0487 - accuracy: 0.9852 - val_loss: 2.4585 - val_accuracy: 0.6352\n",
"Epoch 44/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0594 - accuracy: 0.9782\n",
"Epoch 00044: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0595 - accuracy: 0.9780 - val_loss: 2.2168 - val_accuracy: 0.6582\n",
"Epoch 45/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0544 - accuracy: 0.9829\n",
"Epoch 00045: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0543 - accuracy: 0.9830 - val_loss: 2.1248 - val_accuracy: 0.6702\n",
"Epoch 46/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0404 - accuracy: 0.9860\n",
"Epoch 00046: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0403 - accuracy: 0.9860 - val_loss: 2.1676 - val_accuracy: 0.6554\n",
"Epoch 47/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0585 - accuracy: 0.9800\n",
"Epoch 00047: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0589 - accuracy: 0.9800 - val_loss: 2.1488 - val_accuracy: 0.6472\n",
"Epoch 48/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0446 - accuracy: 0.9835\n",
"Epoch 00048: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0458 - accuracy: 0.9832 - val_loss: 2.2123 - val_accuracy: 0.6564\n",
"Epoch 49/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0530 - accuracy: 0.9829\n",
"Epoch 00049: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0539 - accuracy: 0.9828 - val_loss: 2.0525 - val_accuracy: 0.6592\n",
"Epoch 50/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0460 - accuracy: 0.9843\n",
"Epoch 00050: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0459 - accuracy: 0.9842 - val_loss: 2.1059 - val_accuracy: 0.6662\n",
"Epoch 51/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0475 - accuracy: 0.9842\n",
"Epoch 00051: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0473 - accuracy: 0.9842 - val_loss: 2.2118 - val_accuracy: 0.6638\n",
"Epoch 52/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0333 - accuracy: 0.9885\n",
"Epoch 00052: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0334 - accuracy: 0.9884 - val_loss: 2.0536 - val_accuracy: 0.6784\n",
"Epoch 53/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0569 - accuracy: 0.9798\n",
"Epoch 00053: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0573 - accuracy: 0.9798 - val_loss: 2.3707 - val_accuracy: 0.6194\n",
"Epoch 54/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0569 - accuracy: 0.9824\n",
"Epoch 00054: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0568 - accuracy: 0.9824 - val_loss: 2.3419 - val_accuracy: 0.6342\n",
"Epoch 55/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0221 - accuracy: 0.9933\n",
"Epoch 00055: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0220 - accuracy: 0.9933 - val_loss: 2.2632 - val_accuracy: 0.6632\n",
"Epoch 56/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0286 - accuracy: 0.9896\n",
"Epoch 00056: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0288 - accuracy: 0.9896 - val_loss: 2.6523 - val_accuracy: 0.6196\n",
"Epoch 57/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0556 - accuracy: 0.9819\n",
"Epoch 00057: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0567 - accuracy: 0.9817 - val_loss: 2.2249 - val_accuracy: 0.6620\n",
"Epoch 58/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0443 - accuracy: 0.9865\n",
"Epoch 00058: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0441 - accuracy: 0.9866 - val_loss: 2.1686 - val_accuracy: 0.6548\n",
"Epoch 59/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0330 - accuracy: 0.9892\n",
"Epoch 00059: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0329 - accuracy: 0.9892 - val_loss: 2.2418 - val_accuracy: 0.6824\n",
"Epoch 60/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0378 - accuracy: 0.9874\n",
"Epoch 00060: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0378 - accuracy: 0.9874 - val_loss: 3.0160 - val_accuracy: 0.6236\n",
"Epoch 61/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0386 - accuracy: 0.9879\n",
"Epoch 00061: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0385 - accuracy: 0.9880 - val_loss: 2.2303 - val_accuracy: 0.6796\n",
"Epoch 62/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0280 - accuracy: 0.9902\n",
"Epoch 00062: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0279 - accuracy: 0.9902 - val_loss: 2.1716 - val_accuracy: 0.6654\n",
"Epoch 63/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0511 - accuracy: 0.9837\n",
"Epoch 00063: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0520 - accuracy: 0.9834 - val_loss: 2.4909 - val_accuracy: 0.6280\n",
"Epoch 64/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0488 - accuracy: 0.9831\n",
"Epoch 00064: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0486 - accuracy: 0.9832 - val_loss: 2.2237 - val_accuracy: 0.6670\n",
"Epoch 65/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0179 - accuracy: 0.9948\n",
"Epoch 00065: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0182 - accuracy: 0.9947 - val_loss: 2.3885 - val_accuracy: 0.6688\n",
"Epoch 66/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0403 - accuracy: 0.9878\n",
"Epoch 00066: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0403 - accuracy: 0.9878 - val_loss: 2.8600 - val_accuracy: 0.6300\n",
"Epoch 67/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0453 - accuracy: 0.9843\n",
"Epoch 00067: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0458 - accuracy: 0.9841 - val_loss: 2.0384 - val_accuracy: 0.6858\n",
"Epoch 68/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0313 - accuracy: 0.9906\n",
"Epoch 00068: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0312 - accuracy: 0.9907 - val_loss: 2.0592 - val_accuracy: 0.6784\n",
"Epoch 69/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0252 - accuracy: 0.9922\n",
"Epoch 00069: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0252 - accuracy: 0.9921 - val_loss: 2.2172 - val_accuracy: 0.6464\n",
"Epoch 70/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0386 - accuracy: 0.9869\n",
"Epoch 00070: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0385 - accuracy: 0.9870 - val_loss: 2.3661 - val_accuracy: 0.6494\n",
"Epoch 71/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0393 - accuracy: 0.9882\n",
"Epoch 00071: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0400 - accuracy: 0.9881 - val_loss: 2.2410 - val_accuracy: 0.6554\n",
"Epoch 72/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0268 - accuracy: 0.9912\n",
"Epoch 00072: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0270 - accuracy: 0.9911 - val_loss: 2.2106 - val_accuracy: 0.6734\n",
"Epoch 73/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0398 - accuracy: 0.9863\n",
"Epoch 00073: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0398 - accuracy: 0.9862 - val_loss: 2.2922 - val_accuracy: 0.6606\n",
"Epoch 74/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0244 - accuracy: 0.9920\n",
"Epoch 00074: val_accuracy did not improve from 0.68700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0247 - accuracy: 0.9919 - val_loss: 2.3550 - val_accuracy: 0.6634\n",
"Epoch 75/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0284 - accuracy: 0.9910\n",
"Epoch 00075: val_accuracy improved from 0.68700 to 0.69140, saving model to finetuned.weights\n",
"9000/9000 [==============================] - 15s 2ms/sample - loss: 0.0285 - accuracy: 0.9909 - val_loss: 2.2196 - val_accuracy: 0.6914\n",
"Epoch 76/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0323 - accuracy: 0.9903\n",
"Epoch 00076: val_accuracy did not improve from 0.69140\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0325 - accuracy: 0.9902 - val_loss: 2.1914 - val_accuracy: 0.6812\n",
"Epoch 77/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0337 - accuracy: 0.9891\n",
"Epoch 00077: val_accuracy did not improve from 0.69140\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0336 - accuracy: 0.9891 - val_loss: 2.3250 - val_accuracy: 0.6856\n",
"Epoch 78/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0180 - accuracy: 0.9937\n",
"Epoch 00078: val_accuracy did not improve from 0.69140\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0180 - accuracy: 0.9937 - val_loss: 2.5604 - val_accuracy: 0.6566\n",
"Epoch 79/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0342 - accuracy: 0.9883\n",
"Epoch 00079: val_accuracy did not improve from 0.69140\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0342 - accuracy: 0.9883 - val_loss: 2.9006 - val_accuracy: 0.6298\n",
"Epoch 80/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0280 - accuracy: 0.9904\n",
"Epoch 00080: val_accuracy did not improve from 0.69140\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0291 - accuracy: 0.9902 - val_loss: 2.6783 - val_accuracy: 0.6668\n",
"Epoch 81/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0366 - accuracy: 0.9875\n",
"Epoch 00081: val_accuracy improved from 0.69140 to 0.69200, saving model to finetuned.weights\n",
"9000/9000 [==============================] - 15s 2ms/sample - loss: 0.0365 - accuracy: 0.9876 - val_loss: 2.2443 - val_accuracy: 0.6920\n",
"Epoch 82/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0218 - accuracy: 0.9920\n",
"Epoch 00082: val_accuracy did not improve from 0.69200\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0217 - accuracy: 0.9920 - val_loss: 2.4268 - val_accuracy: 0.6834\n",
"Epoch 83/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0230 - accuracy: 0.9930\n",
"Epoch 00083: val_accuracy did not improve from 0.69200\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0229 - accuracy: 0.9930 - val_loss: 2.7737 - val_accuracy: 0.6404\n",
"Epoch 84/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0385 - accuracy: 0.9869\n",
"Epoch 00084: val_accuracy did not improve from 0.69200\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0383 - accuracy: 0.9870 - val_loss: 2.6137 - val_accuracy: 0.6324\n",
"Epoch 85/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0256 - accuracy: 0.9910\n",
"Epoch 00085: val_accuracy did not improve from 0.69200\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0255 - accuracy: 0.9910 - val_loss: 2.4326 - val_accuracy: 0.6724\n",
"Epoch 86/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0251 - accuracy: 0.9915\n",
"Epoch 00086: val_accuracy did not improve from 0.69200\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0255 - accuracy: 0.9913 - val_loss: 3.3225 - val_accuracy: 0.6472\n",
"Epoch 87/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0440 - accuracy: 0.9849\n",
"Epoch 00087: val_accuracy did not improve from 0.69200\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0440 - accuracy: 0.9849 - val_loss: 2.7542 - val_accuracy: 0.6664\n",
"Epoch 88/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0311 - accuracy: 0.9892\n",
"Epoch 00088: val_accuracy improved from 0.69200 to 0.70320, saving model to finetuned.weights\n",
"9000/9000 [==============================] - 15s 2ms/sample - loss: 0.0309 - accuracy: 0.9892 - val_loss: 2.0983 - val_accuracy: 0.7032\n",
"Epoch 89/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0168 - accuracy: 0.9945\n",
"Epoch 00089: val_accuracy did not improve from 0.70320\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0168 - accuracy: 0.9946 - val_loss: 2.1198 - val_accuracy: 0.6980\n",
"Epoch 90/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0240 - accuracy: 0.9911\n",
"Epoch 00090: val_accuracy did not improve from 0.70320\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0239 - accuracy: 0.9911 - val_loss: 2.2857 - val_accuracy: 0.6992\n",
"Epoch 91/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0300 - accuracy: 0.9891\n",
"Epoch 00091: val_accuracy did not improve from 0.70320\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0301 - accuracy: 0.9890 - val_loss: 2.6250 - val_accuracy: 0.6422\n",
"Epoch 92/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0322 - accuracy: 0.9896\n",
"Epoch 00092: val_accuracy did not improve from 0.70320\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0321 - accuracy: 0.9897 - val_loss: 2.3787 - val_accuracy: 0.6840\n",
"Epoch 93/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0249 - accuracy: 0.9923\n",
"Epoch 00093: val_accuracy did not improve from 0.70320\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0248 - accuracy: 0.9923 - val_loss: 2.7284 - val_accuracy: 0.6634\n",
"Epoch 94/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0208 - accuracy: 0.9931\n",
"Epoch 00094: val_accuracy did not improve from 0.70320\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0208 - accuracy: 0.9931 - val_loss: 2.7575 - val_accuracy: 0.6720\n",
"Epoch 95/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0242 - accuracy: 0.9914\n",
"Epoch 00095: val_accuracy did not improve from 0.70320\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0241 - accuracy: 0.9914 - val_loss: 2.4495 - val_accuracy: 0.6824\n",
"Epoch 96/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0241 - accuracy: 0.9921\n",
"Epoch 00096: val_accuracy did not improve from 0.70320\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0241 - accuracy: 0.9921 - val_loss: 2.2430 - val_accuracy: 0.6850\n",
"Epoch 97/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0181 - accuracy: 0.9942\n",
"Epoch 00097: val_accuracy did not improve from 0.70320\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0182 - accuracy: 0.9941 - val_loss: 2.5996 - val_accuracy: 0.6642\n",
"Epoch 98/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0342 - accuracy: 0.9894\n",
"Epoch 00098: val_accuracy did not improve from 0.70320\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0341 - accuracy: 0.9894 - val_loss: 3.0167 - val_accuracy: 0.6432\n",
"Epoch 99/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0213 - accuracy: 0.9926\n",
"Epoch 00099: val_accuracy did not improve from 0.70320\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0212 - accuracy: 0.9927 - val_loss: 2.6029 - val_accuracy: 0.6626\n",
"Epoch 100/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0210 - accuracy: 0.9932\n",
"Epoch 00100: val_accuracy did not improve from 0.70320\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0220 - accuracy: 0.9928 - val_loss: 2.5853 - val_accuracy: 0.6750\n"
],
"name": "stdout"
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
"<tensorflow.python.training.tracking.util.CheckpointLoadStatus at 0x7f252d81a470>"
]
},
"metadata": {
"tags": []
},
"execution_count": 21
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "b9MPCA6VXHXm",
"colab_type": "text"
},
"source": [
"**Purely Supervised Training**\n",
"--\n",
"Finally, we want to compare the performance of the pretrained network with the performance of a randomly initialized network (i.e. without any pretraining). Both networks have the same architecture."
]
},
{
"cell_type": "code",
"metadata": {
"id": "JFMDo6lMXFuC",
"colab_type": "code",
"outputId": "ddf13431-7c3d-4f6f-fe6a-2691272e80c1",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 1000
}
},
"source": [
"num_classes = np.unique(y_train).shape[0]\n",
"\n",
"random_classifier = ResNet(num_classes, blocks=(2, 3, 4, 2))\n",
"\n",
"learning_rate = 0.01\n",
"optimizer = tf.keras.optimizers.Adam(learning_rate, beta_1=0.9, beta_2=0.98,\n",
" epsilon=1e-9)\n",
"accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name='accuracy')\n",
"\n",
"random_classifier.compile(optimizer=optimizer,\n",
" loss='sparse_categorical_crossentropy',\n",
" metrics=[accuracy])\n",
"\n",
"chkpt = tf.keras.callbacks.ModelCheckpoint(\n",
" 'random_init.weights', monitor='val_accuracy', verbose=1, save_best_only=True,\n",
" save_weights_only=True, mode='max')\n",
"\n",
"random_history = random_classifier.fit(\n",
" x_train_subset, y_train_subset, batch_size=128, epochs=100,\n",
" validation_data=(x_val, y_val), callbacks=[chkpt])\n",
"\n",
"random_classifier.load_weights('random_init.weights')"
],
"execution_count": 22,
"outputs": [
{
"output_type": "stream",
"text": [
"Train on 9000 samples, validate on 5000 samples\n",
"Epoch 1/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 2.4774 - accuracy: 0.2095\n",
"Epoch 00001: val_accuracy improved from -inf to 0.27420, saving model to random_init.weights\n",
"9000/9000 [==============================] - 18s 2ms/sample - loss: 2.4749 - accuracy: 0.2092 - val_loss: 2.0568 - val_accuracy: 0.2742\n",
"Epoch 2/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 1.9038 - accuracy: 0.2972\n",
"Epoch 00002: val_accuracy did not improve from 0.27420\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 1.9039 - accuracy: 0.2972 - val_loss: 2.0938 - val_accuracy: 0.2714\n",
"Epoch 3/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 1.7724 - accuracy: 0.3438\n",
"Epoch 00003: val_accuracy improved from 0.27420 to 0.31380, saving model to random_init.weights\n",
"9000/9000 [==============================] - 15s 2ms/sample - loss: 1.7723 - accuracy: 0.3438 - val_loss: 2.0708 - val_accuracy: 0.3138\n",
"Epoch 4/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 1.6631 - accuracy: 0.3874\n",
"Epoch 00004: val_accuracy did not improve from 0.31380\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 1.6630 - accuracy: 0.3868 - val_loss: 2.4078 - val_accuracy: 0.2566\n",
"Epoch 5/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 1.5902 - accuracy: 0.4171\n",
"Epoch 00005: val_accuracy improved from 0.31380 to 0.35820, saving model to random_init.weights\n",
"9000/9000 [==============================] - 15s 2ms/sample - loss: 1.5890 - accuracy: 0.4177 - val_loss: 1.7630 - val_accuracy: 0.3582\n",
"Epoch 6/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 1.5396 - accuracy: 0.4378\n",
"Epoch 00006: val_accuracy did not improve from 0.35820\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 1.5393 - accuracy: 0.4374 - val_loss: 1.9787 - val_accuracy: 0.3256\n",
"Epoch 7/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 1.4516 - accuracy: 0.4664\n",
"Epoch 00007: val_accuracy did not improve from 0.35820\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 1.4514 - accuracy: 0.4661 - val_loss: 2.2933 - val_accuracy: 0.2976\n",
"Epoch 8/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 1.3818 - accuracy: 0.4867\n",
"Epoch 00008: val_accuracy improved from 0.35820 to 0.46760, saving model to random_init.weights\n",
"9000/9000 [==============================] - 15s 2ms/sample - loss: 1.3824 - accuracy: 0.4866 - val_loss: 1.4441 - val_accuracy: 0.4676\n",
"Epoch 9/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 1.2955 - accuracy: 0.5317\n",
"Epoch 00009: val_accuracy did not improve from 0.46760\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 1.2948 - accuracy: 0.5326 - val_loss: 1.7752 - val_accuracy: 0.4078\n",
"Epoch 10/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 1.2087 - accuracy: 0.5602\n",
"Epoch 00010: val_accuracy improved from 0.46760 to 0.53460, saving model to random_init.weights\n",
"9000/9000 [==============================] - 15s 2ms/sample - loss: 1.2081 - accuracy: 0.5607 - val_loss: 1.3049 - val_accuracy: 0.5346\n",
"Epoch 11/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 1.0618 - accuracy: 0.6167\n",
"Epoch 00011: val_accuracy did not improve from 0.53460\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 1.0630 - accuracy: 0.6160 - val_loss: 1.4416 - val_accuracy: 0.4728\n",
"Epoch 12/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.9649 - accuracy: 0.6497\n",
"Epoch 00012: val_accuracy improved from 0.53460 to 0.55640, saving model to random_init.weights\n",
"9000/9000 [==============================] - 15s 2ms/sample - loss: 0.9659 - accuracy: 0.6490 - val_loss: 1.2666 - val_accuracy: 0.5564\n",
"Epoch 13/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.8173 - accuracy: 0.7097\n",
"Epoch 00013: val_accuracy improved from 0.55640 to 0.57320, saving model to random_init.weights\n",
"9000/9000 [==============================] - 15s 2ms/sample - loss: 0.8183 - accuracy: 0.7092 - val_loss: 1.2677 - val_accuracy: 0.5732\n",
"Epoch 14/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.6876 - accuracy: 0.7563\n",
"Epoch 00014: val_accuracy improved from 0.57320 to 0.57820, saving model to random_init.weights\n",
"9000/9000 [==============================] - 15s 2ms/sample - loss: 0.6882 - accuracy: 0.7560 - val_loss: 1.2994 - val_accuracy: 0.5782\n",
"Epoch 15/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.5317 - accuracy: 0.8087\n",
"Epoch 00015: val_accuracy did not improve from 0.57820\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.5321 - accuracy: 0.8084 - val_loss: 1.6263 - val_accuracy: 0.5672\n",
"Epoch 16/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.4099 - accuracy: 0.8512\n",
"Epoch 00016: val_accuracy improved from 0.57820 to 0.61700, saving model to random_init.weights\n",
"9000/9000 [==============================] - 15s 2ms/sample - loss: 0.4106 - accuracy: 0.8510 - val_loss: 1.2800 - val_accuracy: 0.6170\n",
"Epoch 17/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.2799 - accuracy: 0.9033\n",
"Epoch 00017: val_accuracy did not improve from 0.61700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.2812 - accuracy: 0.9026 - val_loss: 1.6297 - val_accuracy: 0.6062\n",
"Epoch 18/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.1863 - accuracy: 0.9366\n",
"Epoch 00018: val_accuracy did not improve from 0.61700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.1880 - accuracy: 0.9361 - val_loss: 1.8839 - val_accuracy: 0.6086\n",
"Epoch 19/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.1560 - accuracy: 0.9453\n",
"Epoch 00019: val_accuracy did not improve from 0.61700\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.1565 - accuracy: 0.9452 - val_loss: 2.3432 - val_accuracy: 0.5980\n",
"Epoch 20/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.1456 - accuracy: 0.9471\n",
"Epoch 00020: val_accuracy improved from 0.61700 to 0.61820, saving model to random_init.weights\n",
"9000/9000 [==============================] - 15s 2ms/sample - loss: 0.1456 - accuracy: 0.9470 - val_loss: 1.8856 - val_accuracy: 0.6182\n",
"Epoch 21/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.1093 - accuracy: 0.9621\n",
"Epoch 00021: val_accuracy improved from 0.61820 to 0.63540, saving model to random_init.weights\n",
"9000/9000 [==============================] - 15s 2ms/sample - loss: 0.1097 - accuracy: 0.9620 - val_loss: 1.8972 - val_accuracy: 0.6354\n",
"Epoch 22/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.1000 - accuracy: 0.9658\n",
"Epoch 00022: val_accuracy did not improve from 0.63540\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.1001 - accuracy: 0.9659 - val_loss: 2.2323 - val_accuracy: 0.6098\n",
"Epoch 23/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.1055 - accuracy: 0.9632\n",
"Epoch 00023: val_accuracy did not improve from 0.63540\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.1055 - accuracy: 0.9631 - val_loss: 2.1665 - val_accuracy: 0.6144\n",
"Epoch 24/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0958 - accuracy: 0.9683\n",
"Epoch 00024: val_accuracy did not improve from 0.63540\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0961 - accuracy: 0.9681 - val_loss: 2.7539 - val_accuracy: 0.5566\n",
"Epoch 25/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0646 - accuracy: 0.9773\n",
"Epoch 00025: val_accuracy did not improve from 0.63540\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0645 - accuracy: 0.9773 - val_loss: 2.3505 - val_accuracy: 0.6210\n",
"Epoch 26/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0927 - accuracy: 0.9693\n",
"Epoch 00026: val_accuracy did not improve from 0.63540\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0926 - accuracy: 0.9693 - val_loss: 3.5558 - val_accuracy: 0.5610\n",
"Epoch 27/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0735 - accuracy: 0.9753\n",
"Epoch 00027: val_accuracy did not improve from 0.63540\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0732 - accuracy: 0.9754 - val_loss: 2.4894 - val_accuracy: 0.6254\n",
"Epoch 28/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0551 - accuracy: 0.9810\n",
"Epoch 00028: val_accuracy did not improve from 0.63540\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0551 - accuracy: 0.9809 - val_loss: 2.5030 - val_accuracy: 0.6254\n",
"Epoch 29/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0803 - accuracy: 0.9729\n",
"Epoch 00029: val_accuracy did not improve from 0.63540\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0808 - accuracy: 0.9729 - val_loss: 2.2859 - val_accuracy: 0.6230\n",
"Epoch 30/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0542 - accuracy: 0.9811\n",
"Epoch 00030: val_accuracy did not improve from 0.63540\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0541 - accuracy: 0.9812 - val_loss: 2.2874 - val_accuracy: 0.6256\n",
"Epoch 31/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0568 - accuracy: 0.9811\n",
"Epoch 00031: val_accuracy did not improve from 0.63540\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0566 - accuracy: 0.9812 - val_loss: 2.7428 - val_accuracy: 0.6172\n",
"Epoch 32/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0609 - accuracy: 0.9787\n",
"Epoch 00032: val_accuracy did not improve from 0.63540\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0612 - accuracy: 0.9786 - val_loss: 2.9249 - val_accuracy: 0.5828\n",
"Epoch 33/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0471 - accuracy: 0.9839\n",
"Epoch 00033: val_accuracy did not improve from 0.63540\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0470 - accuracy: 0.9840 - val_loss: 2.1659 - val_accuracy: 0.6300\n",
"Epoch 34/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0523 - accuracy: 0.9823\n",
"Epoch 00034: val_accuracy did not improve from 0.63540\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0524 - accuracy: 0.9822 - val_loss: 2.4052 - val_accuracy: 0.6304\n",
"Epoch 35/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0476 - accuracy: 0.9839\n",
"Epoch 00035: val_accuracy did not improve from 0.63540\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0484 - accuracy: 0.9837 - val_loss: 2.5983 - val_accuracy: 0.6192\n",
"Epoch 36/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0563 - accuracy: 0.9817\n",
"Epoch 00036: val_accuracy did not improve from 0.63540\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0567 - accuracy: 0.9817 - val_loss: 2.2805 - val_accuracy: 0.6156\n",
"Epoch 37/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0420 - accuracy: 0.9857\n",
"Epoch 00037: val_accuracy did not improve from 0.63540\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0423 - accuracy: 0.9857 - val_loss: 2.5878 - val_accuracy: 0.6088\n",
"Epoch 38/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0400 - accuracy: 0.9868\n",
"Epoch 00038: val_accuracy improved from 0.63540 to 0.65960, saving model to random_init.weights\n",
"9000/9000 [==============================] - 15s 2ms/sample - loss: 0.0401 - accuracy: 0.9868 - val_loss: 2.0463 - val_accuracy: 0.6596\n",
"Epoch 39/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0604 - accuracy: 0.9815\n",
"Epoch 00039: val_accuracy did not improve from 0.65960\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0602 - accuracy: 0.9816 - val_loss: 2.3079 - val_accuracy: 0.6276\n",
"Epoch 40/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0355 - accuracy: 0.9875\n",
"Epoch 00040: val_accuracy did not improve from 0.65960\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0356 - accuracy: 0.9874 - val_loss: 2.7469 - val_accuracy: 0.6260\n",
"Epoch 41/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0354 - accuracy: 0.9884\n",
"Epoch 00041: val_accuracy did not improve from 0.65960\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0355 - accuracy: 0.9883 - val_loss: 2.7061 - val_accuracy: 0.6452\n",
"Epoch 42/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0355 - accuracy: 0.9879\n",
"Epoch 00042: val_accuracy did not improve from 0.65960\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0354 - accuracy: 0.9880 - val_loss: 2.3771 - val_accuracy: 0.6342\n",
"Epoch 43/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0424 - accuracy: 0.9854\n",
"Epoch 00043: val_accuracy did not improve from 0.65960\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0424 - accuracy: 0.9853 - val_loss: 2.4901 - val_accuracy: 0.6390\n",
"Epoch 44/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0399 - accuracy: 0.9865\n",
"Epoch 00044: val_accuracy did not improve from 0.65960\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0398 - accuracy: 0.9866 - val_loss: 2.6482 - val_accuracy: 0.6244\n",
"Epoch 45/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0453 - accuracy: 0.9856\n",
"Epoch 00045: val_accuracy did not improve from 0.65960\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0452 - accuracy: 0.9856 - val_loss: 2.4591 - val_accuracy: 0.6328\n",
"Epoch 46/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0358 - accuracy: 0.9879\n",
"Epoch 00046: val_accuracy did not improve from 0.65960\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0358 - accuracy: 0.9879 - val_loss: 2.7266 - val_accuracy: 0.6212\n",
"Epoch 47/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0418 - accuracy: 0.9860\n",
"Epoch 00047: val_accuracy did not improve from 0.65960\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0418 - accuracy: 0.9860 - val_loss: 3.0078 - val_accuracy: 0.6028\n",
"Epoch 48/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0319 - accuracy: 0.9902\n",
"Epoch 00048: val_accuracy did not improve from 0.65960\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0318 - accuracy: 0.9902 - val_loss: 2.6264 - val_accuracy: 0.6400\n",
"Epoch 49/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0361 - accuracy: 0.9892\n",
"Epoch 00049: val_accuracy did not improve from 0.65960\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0361 - accuracy: 0.9891 - val_loss: 2.5864 - val_accuracy: 0.6290\n",
"Epoch 50/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0237 - accuracy: 0.9916\n",
"Epoch 00050: val_accuracy did not improve from 0.65960\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0236 - accuracy: 0.9917 - val_loss: 2.7915 - val_accuracy: 0.6256\n",
"Epoch 51/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0159 - accuracy: 0.9950\n",
"Epoch 00051: val_accuracy did not improve from 0.65960\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0162 - accuracy: 0.9949 - val_loss: 2.9309 - val_accuracy: 0.6448\n",
"Epoch 52/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0464 - accuracy: 0.9863\n",
"Epoch 00052: val_accuracy did not improve from 0.65960\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0471 - accuracy: 0.9861 - val_loss: 3.0083 - val_accuracy: 0.6084\n",
"Epoch 53/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0339 - accuracy: 0.9879\n",
"Epoch 00053: val_accuracy did not improve from 0.65960\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0345 - accuracy: 0.9879 - val_loss: 2.8766 - val_accuracy: 0.6272\n",
"Epoch 54/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0257 - accuracy: 0.9913\n",
"Epoch 00054: val_accuracy did not improve from 0.65960\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0261 - accuracy: 0.9912 - val_loss: 2.4057 - val_accuracy: 0.6488\n",
"Epoch 55/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0281 - accuracy: 0.9915\n",
"Epoch 00055: val_accuracy did not improve from 0.65960\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0280 - accuracy: 0.9916 - val_loss: 2.4706 - val_accuracy: 0.6544\n",
"Epoch 56/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0185 - accuracy: 0.9932\n",
"Epoch 00056: val_accuracy did not improve from 0.65960\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0185 - accuracy: 0.9932 - val_loss: 2.8189 - val_accuracy: 0.6376\n",
"Epoch 57/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0356 - accuracy: 0.9885\n",
"Epoch 00057: val_accuracy did not improve from 0.65960\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0355 - accuracy: 0.9886 - val_loss: 2.6251 - val_accuracy: 0.6290\n",
"Epoch 58/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0344 - accuracy: 0.9888\n",
"Epoch 00058: val_accuracy did not improve from 0.65960\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0352 - accuracy: 0.9888 - val_loss: 2.6452 - val_accuracy: 0.6396\n",
"Epoch 59/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0225 - accuracy: 0.9932\n",
"Epoch 00059: val_accuracy did not improve from 0.65960\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0224 - accuracy: 0.9932 - val_loss: 2.9172 - val_accuracy: 0.6344\n",
"Epoch 60/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0258 - accuracy: 0.9903\n",
"Epoch 00060: val_accuracy did not improve from 0.65960\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0259 - accuracy: 0.9901 - val_loss: 2.8934 - val_accuracy: 0.6344\n",
"Epoch 61/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0214 - accuracy: 0.9927\n",
"Epoch 00061: val_accuracy did not improve from 0.65960\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0213 - accuracy: 0.9928 - val_loss: 2.6919 - val_accuracy: 0.6362\n",
"Epoch 62/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0218 - accuracy: 0.9924\n",
"Epoch 00062: val_accuracy did not improve from 0.65960\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0218 - accuracy: 0.9923 - val_loss: 2.5939 - val_accuracy: 0.6552\n",
"Epoch 63/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0233 - accuracy: 0.9915\n",
"Epoch 00063: val_accuracy did not improve from 0.65960\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0233 - accuracy: 0.9916 - val_loss: 2.8022 - val_accuracy: 0.6398\n",
"Epoch 64/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0163 - accuracy: 0.9945\n",
"Epoch 00064: val_accuracy did not improve from 0.65960\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0166 - accuracy: 0.9944 - val_loss: 2.6505 - val_accuracy: 0.6590\n",
"Epoch 65/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0303 - accuracy: 0.9908\n",
"Epoch 00065: val_accuracy did not improve from 0.65960\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0302 - accuracy: 0.9909 - val_loss: 3.1847 - val_accuracy: 0.6016\n",
"Epoch 66/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0308 - accuracy: 0.9901\n",
"Epoch 00066: val_accuracy did not improve from 0.65960\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0307 - accuracy: 0.9901 - val_loss: 3.0070 - val_accuracy: 0.6306\n",
"Epoch 67/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0204 - accuracy: 0.9948\n",
"Epoch 00067: val_accuracy improved from 0.65960 to 0.66840, saving model to random_init.weights\n",
"9000/9000 [==============================] - 15s 2ms/sample - loss: 0.0205 - accuracy: 0.9947 - val_loss: 2.7048 - val_accuracy: 0.6684\n",
"Epoch 68/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0206 - accuracy: 0.9933\n",
"Epoch 00068: val_accuracy did not improve from 0.66840\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0207 - accuracy: 0.9932 - val_loss: 2.7552 - val_accuracy: 0.6526\n",
"Epoch 69/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0252 - accuracy: 0.9910\n",
"Epoch 00069: val_accuracy did not improve from 0.66840\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0251 - accuracy: 0.9910 - val_loss: 3.7162 - val_accuracy: 0.6130\n",
"Epoch 70/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0206 - accuracy: 0.9924\n",
"Epoch 00070: val_accuracy did not improve from 0.66840\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0206 - accuracy: 0.9924 - val_loss: 2.7575 - val_accuracy: 0.6532\n",
"Epoch 71/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0216 - accuracy: 0.9923\n",
"Epoch 00071: val_accuracy did not improve from 0.66840\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0217 - accuracy: 0.9923 - val_loss: 3.6271 - val_accuracy: 0.6042\n",
"Epoch 72/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0264 - accuracy: 0.9910\n",
"Epoch 00072: val_accuracy did not improve from 0.66840\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0264 - accuracy: 0.9910 - val_loss: 2.5890 - val_accuracy: 0.6610\n",
"Epoch 73/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0130 - accuracy: 0.9949\n",
"Epoch 00073: val_accuracy did not improve from 0.66840\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0131 - accuracy: 0.9948 - val_loss: 3.5514 - val_accuracy: 0.5978\n",
"Epoch 74/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0236 - accuracy: 0.9927\n",
"Epoch 00074: val_accuracy did not improve from 0.66840\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0242 - accuracy: 0.9927 - val_loss: 3.3748 - val_accuracy: 0.6356\n",
"Epoch 75/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0252 - accuracy: 0.9922\n",
"Epoch 00075: val_accuracy did not improve from 0.66840\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0251 - accuracy: 0.9922 - val_loss: 3.6968 - val_accuracy: 0.5950\n",
"Epoch 76/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0159 - accuracy: 0.9942\n",
"Epoch 00076: val_accuracy did not improve from 0.66840\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0158 - accuracy: 0.9942 - val_loss: 2.8805 - val_accuracy: 0.6568\n",
"Epoch 77/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0190 - accuracy: 0.9941\n",
"Epoch 00077: val_accuracy did not improve from 0.66840\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0190 - accuracy: 0.9941 - val_loss: 2.9855 - val_accuracy: 0.6550\n",
"Epoch 78/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0256 - accuracy: 0.9926\n",
"Epoch 00078: val_accuracy did not improve from 0.66840\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0255 - accuracy: 0.9927 - val_loss: 3.0335 - val_accuracy: 0.6430\n",
"Epoch 79/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0225 - accuracy: 0.9931\n",
"Epoch 00079: val_accuracy did not improve from 0.66840\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0227 - accuracy: 0.9930 - val_loss: 2.6969 - val_accuracy: 0.6572\n",
"Epoch 80/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0169 - accuracy: 0.9944\n",
"Epoch 00080: val_accuracy improved from 0.66840 to 0.67160, saving model to random_init.weights\n",
"9000/9000 [==============================] - 15s 2ms/sample - loss: 0.0168 - accuracy: 0.9944 - val_loss: 2.5987 - val_accuracy: 0.6716\n",
"Epoch 81/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0082 - accuracy: 0.9973\n",
"Epoch 00081: val_accuracy did not improve from 0.67160\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0082 - accuracy: 0.9973 - val_loss: 3.4963 - val_accuracy: 0.6062\n",
"Epoch 82/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0132 - accuracy: 0.9952\n",
"Epoch 00082: val_accuracy did not improve from 0.67160\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0132 - accuracy: 0.9952 - val_loss: 3.0904 - val_accuracy: 0.6474\n",
"Epoch 83/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0341 - accuracy: 0.9891\n",
"Epoch 00083: val_accuracy did not improve from 0.67160\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0339 - accuracy: 0.9891 - val_loss: 3.0512 - val_accuracy: 0.6458\n",
"Epoch 84/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0134 - accuracy: 0.9958\n",
"Epoch 00084: val_accuracy did not improve from 0.67160\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0134 - accuracy: 0.9958 - val_loss: 3.1474 - val_accuracy: 0.6590\n",
"Epoch 85/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0160 - accuracy: 0.9950\n",
"Epoch 00085: val_accuracy did not improve from 0.67160\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0169 - accuracy: 0.9949 - val_loss: 3.2088 - val_accuracy: 0.6434\n",
"Epoch 86/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0271 - accuracy: 0.9920\n",
"Epoch 00086: val_accuracy did not improve from 0.67160\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0270 - accuracy: 0.9920 - val_loss: 3.0850 - val_accuracy: 0.6368\n",
"Epoch 87/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0134 - accuracy: 0.9955\n",
"Epoch 00087: val_accuracy did not improve from 0.67160\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0133 - accuracy: 0.9956 - val_loss: 2.7749 - val_accuracy: 0.6512\n",
"Epoch 88/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0101 - accuracy: 0.9969\n",
"Epoch 00088: val_accuracy did not improve from 0.67160\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0101 - accuracy: 0.9969 - val_loss: 3.7593 - val_accuracy: 0.6276\n",
"Epoch 89/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0174 - accuracy: 0.9937\n",
"Epoch 00089: val_accuracy did not improve from 0.67160\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0174 - accuracy: 0.9938 - val_loss: 2.8371 - val_accuracy: 0.6504\n",
"Epoch 90/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0186 - accuracy: 0.9937\n",
"Epoch 00090: val_accuracy did not improve from 0.67160\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0186 - accuracy: 0.9938 - val_loss: 3.1484 - val_accuracy: 0.6118\n",
"Epoch 91/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0204 - accuracy: 0.9939\n",
"Epoch 00091: val_accuracy did not improve from 0.67160\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0205 - accuracy: 0.9937 - val_loss: 3.3726 - val_accuracy: 0.6318\n",
"Epoch 92/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0170 - accuracy: 0.9943\n",
"Epoch 00092: val_accuracy did not improve from 0.67160\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0170 - accuracy: 0.9943 - val_loss: 3.2478 - val_accuracy: 0.6362\n",
"Epoch 93/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0196 - accuracy: 0.9941\n",
"Epoch 00093: val_accuracy did not improve from 0.67160\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0195 - accuracy: 0.9941 - val_loss: 2.7975 - val_accuracy: 0.6516\n",
"Epoch 94/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0254 - accuracy: 0.9917\n",
"Epoch 00094: val_accuracy did not improve from 0.67160\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0273 - accuracy: 0.9917 - val_loss: 3.0455 - val_accuracy: 0.6490\n",
"Epoch 95/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0148 - accuracy: 0.9954\n",
"Epoch 00095: val_accuracy did not improve from 0.67160\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0147 - accuracy: 0.9954 - val_loss: 3.1745 - val_accuracy: 0.6538\n",
"Epoch 96/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0081 - accuracy: 0.9968\n",
"Epoch 00096: val_accuracy did not improve from 0.67160\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0081 - accuracy: 0.9968 - val_loss: 3.2104 - val_accuracy: 0.6436\n",
"Epoch 97/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0142 - accuracy: 0.9956\n",
"Epoch 00097: val_accuracy did not improve from 0.67160\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0142 - accuracy: 0.9957 - val_loss: 3.5052 - val_accuracy: 0.6276\n",
"Epoch 98/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0208 - accuracy: 0.9934\n",
"Epoch 00098: val_accuracy did not improve from 0.67160\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0207 - accuracy: 0.9934 - val_loss: 3.0822 - val_accuracy: 0.6376\n",
"Epoch 99/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0147 - accuracy: 0.9953\n",
"Epoch 00099: val_accuracy did not improve from 0.67160\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0147 - accuracy: 0.9953 - val_loss: 3.1447 - val_accuracy: 0.6382\n",
"Epoch 100/100\n",
"8960/9000 [============================>.] - ETA: 0s - loss: 0.0163 - accuracy: 0.9948\n",
"Epoch 00100: val_accuracy did not improve from 0.67160\n",
"9000/9000 [==============================] - 14s 2ms/sample - loss: 0.0162 - accuracy: 0.9948 - val_loss: 3.0231 - val_accuracy: 0.6440\n"
],
"name": "stdout"
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
"<tensorflow.python.training.tracking.util.CheckpointLoadStatus at 0x7f2528b8c080>"
]
},
"metadata": {
"tags": []
},
"execution_count": 22
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "AMrwjiUw9Ox4",
"colab_type": "text"
},
"source": [
"**Results**\n",
"--\n",
"The results show that this pretraining method has a positive impact on the classification performance. While the training loss and accuracy of the pretrained and the randomly initialized model equalize at certain point, the validation and test accuracy of the pretrained model remains significantly better than the accuracy of the randomly initialized model."
]
},
{
"cell_type": "code",
"metadata": {
"id": "EAhDflR0aQLq",
"colab_type": "code",
"outputId": "d8fce025-c519-4c31-b6fc-c89af530a25f",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 319
}
},
"source": [
"finetune_hist = finetune_history.history\n",
"random_hist = random_history.history\n",
"num_epochs = max(len(finetune_hist['loss']), len(random_hist['loss']))\n",
"epochs = range(1, num_epochs + 1)\n",
"fig, ax = plt.subplots(2)\n",
"\n",
"ax[0].plot(epochs, random_hist['loss'], 'r--')\n",
"ax[0].plot(epochs, finetune_hist['loss'], 'b--')\n",
"ax[0].plot(epochs, random_hist['val_loss'], 'g-')\n",
"ax[0].plot(epochs, finetune_hist['val_loss'], 'm-')\n",
"ax[0].legend(['[Train] Random Init', '[Train] Finetuned',\n",
" '[Validation] Random Init', '[Validation] Finetuned'],\n",
" loc='center left', bbox_to_anchor=(1, 0.5))\n",
"ax[0].set_ylabel('Loss')\n",
"ax[0].set_xlim((0.5, num_epochs + 0.5))\n",
"ax[0].set_ylim(bottom=-0.05)\n",
"ax[0].xaxis.set_major_locator(ticker.MaxNLocator(integer=True))\n",
"\n",
"ax[1].plot(epochs, random_hist['accuracy'], 'r--')\n",
"ax[1].plot(epochs, finetune_hist['accuracy'], 'b--')\n",
"ax[1].plot(epochs, random_hist['val_accuracy'], 'g-')\n",
"ax[1].plot(epochs, finetune_hist['val_accuracy'], 'm-')\n",
"ax[1].legend(['[Train] Random Init', '[Train] Finetuned',\n",
" '[Validation] Random Init', '[Validation] Finetuned'],\n",
" loc='center left', bbox_to_anchor=(1, 0.5))\n",
"ax[1].set_xlabel('Epoch')\n",
"ax[1].set_ylabel('Accuracy')\n",
"ax[1].set_xlim((0.5, num_epochs + 0.5))\n",
"ax[1].set_ylim((-0.05, 1.05))\n",
"ax[1].xaxis.set_major_locator(ticker.MaxNLocator(integer=True))\n",
"plt.show()\n",
"\n",
"random_init_test_pred = random_classifier.predict(x_test, batch_size=512)\n",
"finetune_test_pred = pretrained_classifier.predict(x_test, batch_size=512)\n",
"random_init_test_acc = tf.keras.metrics.SparseCategoricalAccuracy()(y_test, random_init_test_pred)\n",
"finetune_test_acc = tf.keras.metrics.SparseCategoricalAccuracy()(y_test, finetune_test_pred)\n",
"\n",
"print('Random init test accuracy: {:.2%}'.format(random_init_test_acc))\n",
"print('Finetuned test accuracy: {:.2%}'.format(finetune_test_acc))"
],
"execution_count": 23,
"outputs": [
{
"output_type": "display_data",
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAj0AAAEMCAYAAAAvXweUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOydZ2Ac5bm2r9mmbapb1GV1Wbbkjgu4\ngzk0xxCSQCDOgUASSE4IKYd8ISGck0AIOUASEgKE0O2AaY7jhNBtcO+yJav3XrZod6Ut2vb9WGml\n9UqyjG1sYK4/0sy8M/PM7khzz9NeIRgMBhERERERERER+YwjOdcGiIiIiIiIiIh8EoiiR0RERERE\nRORzgSh6RERERERERD4XiKJHRERERERE5HOBKHpEREREREREPheIokdERERERETkc4EoekRERERE\nREQ+F8jOtQFnGqt1kEDg3LQe0um0mM0D5+Tc55MN54sd54MN54sd54MN54sd54MN54sdOp0Wq3WQ\nxETNObVDROST4jMnegKB4DkTPSPnP9ecDzbA+WHH+WADnB92nA82wPlhx/lgA5wfdpwPNoiIfFKI\n4S0RERGRs4DT62TAO3iuzRARERmDKHpERETOCsFgEMfQuQ8jnSteqHqFpys2nmszRERExiCKHhER\nkbNClaWWu3fdh9llOSPHGxga5G/Vr+H0Os/I8c42HQNddDg6z7UZIiIiYxBFj4iIyFmh1dFBIBig\nc7D7jBxve/tOdnXup8pSd0aOdzbxBXxY3f0M+sQQl4jI+YQoekRERM4KfS5T6KfTFLXtneZtPF2x\nYcrH8gZ87OzYB0DXYM+ZMfAsYnZbCRJKEO4d5/pFRETODaLoEZkUq7v/XJsg8imlz2kGoNdljtpW\nbq6iwlRFMBhdOeT1e6PWl/WW4/AOIBEkdH8KRI9pzDWPJ/pERETODaLoEZmQWmsD9+x+gO7B3nNt\nisinkMk8PT3OXoYCXhzeyETnYDDIr/Y9xItVr0QIn486dmNQ6Zipm/6p8PSMCD6AXmffObRERERk\nLKLoEZmQrsEeggQ/FQ8ZkfMLt8+DfcgBQN8Jnp6BoUEGh5ORT0xyHvQ5Mbut7Os+xI6OvQC0Otpp\ntLWwPH0JaZoUel0mfAHfads45PfyQuUmTGco0XosJpcZhVSBQaWjx/XJenread7GH488Na4XTUTk\n844oekQmZCS0ZXaf+YeCyGebkfBOstqIxW2NECndzt4x4yLvrREPSbwijtfr/kGLvY2P2vegkMhZ\nnLqAFI2RQDBwRvJkWuyt7Os+xEftu0/7WCfS5zKjVyZhVBs+0fCWy+fi7ZYPqLbWiS8rIiLjIIoe\nkQmxeoZFj8t6ji0R+bQx4t2ZoSskSBCze/Qe6hkjek4U1CMhsVtKvkasIpanyl/kYM8RLkiZh1qu\nJlWTDEQKp49Lz3DY6Uhf+Rn3iphcZgwqHUa1nl6X6RPzuuzs2Ifb7wGgrK/8EzmniMinCVH0iExI\nv8cGiJ4ekVNnRLwUJxWFlsd4O3oG+5BLZMTKtVHhrT6XGQGBrLgMbi39GvYhB96AjxUZFwIhz5GA\ncEa8GCOix+K20uJoO+3jjRAIBjC5LehVOowqPUP+IWxD9gnHH+49xjst2wgEA6d1Xm/Ax7a2nRQl\n5pMXn01ZX8VpHU9E5LOIKHpEJsTqHhY9ZyHnQeSzTZ/TTKxcS1Zsemh5TF5Pj7MPo9qAXqUbN7yV\nqExALpGRHZfFTTO/yuXZl5CuTQVAIZWjUyWdEdHT6zShUyYiFaQc7jn2sY7hcftorosMX9k8dnwB\nHwa1DqPaED7XeJhdVl6o3MSWhn/zQuUr+AP+j2UHwMHuI9iG7KzJWskcQwkdA10RCdUiIiKi6BGZ\ngEAwMMbTYxWTIkVOiT6XCYNah1auQSlVniB6eodFT1KUF9HkMmFQ6cLL84yzuCr30ogxqZrkM1K2\n3uvsIys2g+KkAg73HvtY93jV0S7+/fpx+i2jXaJHrlU/HN4aOdd4vF6/FQFYnbmMAz2H+Uv58wz5\nh07ZjkAwwHutH5KuTWV6UgGzDSUAHDWJ3h4RkbGIokdkXBxDg/iDfpLVBrzjlBaLiExGn8uMQaVH\nEAQMal04vOX1ezG5LKSoDehUSVjc/RHejd4TRM94pGqS6XWaTssr4g/4MbktJKsNzDXOwurpp9l+\n6iEuqykkdrrabOF1I0ncBpWOhJh45BIZveNUcB3pquBoXwWXZ1/CtQVrub7oGo6ba/hT2V/xnmJ1\n2nFzNd3OXi7JWoEgCOhUSWRq0yjrFUWPiMhYZOfaAJHzk/7hJOb8hBx6nH2YXVbiFLHn2CqRc8WI\nF0QQhJOOHfJ76ffYwuLFoNLR5ugAoHugjyBBktVGvAEfQYJYPf3oVTqcXieDXieGYe/IRKRqkvEH\n/fS5TKQMJzafKiaXmUAwgFFtoFQ/A6kg5UjvMXLisybdzzE0wK7OfcQp4rgw7YKwh6erzUbx7FAI\nrs9lRiJISIxJQCJIMKj0UeEtr9/LM4dfIVltYHXWMgCWpS9BKkjZWP0aleYaZhtmnvQ6bB47NdZ6\n3mnZRmJMAvONs8PbZhtK+WfT2/R7bCTExJ/S5wMQCAQwmUxYLFb8/o8vMEVEPklUKiWZmZnI5fJx\nt4uiR2RcrMOhrbz4HHZ17sfstpz0gSDy2eXFqlcwuy18b843kUkm/7cx1tMR+qmnrK8Cf8BPhz00\nD1eyxoDb5xkeH0r67Tthv4lI0RgB6Bzs+diiZySJ2ag2oJarwiGua/KvHFfY9Tr7eL9tB/u6Doa9\nMCaXmX5L6EWgqz3S05OkTEQqkYauR62n54QGn++2bqdnoC/q81yYMo/X6/5JhalyUtHTZGvlb9Wv\nhec108jVfLXo2vA5AeYYS/hn09sc6zvO8uFE8EAwgIAwJfHa1taGzxdAp0tGKpVNaR8RkXNJMBjE\n4bDR1tZGbm7uuGNE0SMyLiM9evIScgCwiGXrn1v8AT9H+ypw+z1srv8XXy5cN+n4sHgZ9tgYVDoC\nwQAWdz+dw7k4RpUhqkHhqOiZ3NOTMlzBdTp5PSPhpuRhG+cZZ1NhrqbZ3hYh7v0BP2+3fMC/m99H\nIkhYlDKPlRlL2d6+i/fqd1DsuhRNbAz2fjeDDg+a2Jjh0N6ocDOq9FSYqggEA0gECY6hAd5p2cbi\nzHlMTyqIsEsmkVGsK6TcPDr+RGqtDTx+7Fli5VquzruC6UkFpGtTo8amqI0kqw2U9VUw1ziLD9t3\n8VH7HrLjs/jGzBtRymIm/YwGBwdJSclEGMcGEZHzEUEQiI2Np6urdcIx553o+c53vkN7ezsSiQS1\nWs0999xDcXHxuTbrc4fV049MIkOnTEQr12ASy9Y/t7QNdOD2e0jXprK9fRd5CTnMM86acPxIuXrY\n0zMsLPpcJjrs3STGJKCUxaCQypEIkvC9NVJppFclTWqPQqpAp0w8relRegb70Mo1qOVqgHCIa2vj\nW6zOXEZhYh72oQGer3yJRlsLFyTP5YsFV4VDvF8t+iIKeyx9gDOlFxzxdLXbyJtuwOQykxM3KpyM\nagP+oB+L24pepWNnx168AR/XlawFT7RtpbpijvQeo83RwbS4zIhtFaYq/lrxIjqVjjvmfJP4mLgJ\nr1EQBGYbSni3ZTv37P41voCfgsQ8Ks01PHrkL9w++2YMTByyDgYRBY/Ip46TeSTPO9Hz4IMPEhsb\n+kN87733uPvuu9m8efM5turzR787lAcgCAI6ZRIWt+jp+bxSa20A4PZZN/N0xQY2Vr1KujaV5OFy\n7BPpc5rQyNRhQTHiuelzmel09IT3kwgSkpSJYzw9JhJi4lFIFSe1KUWTPGHZ+pB/iPaBTnLipk34\nD7DX1Rdhv1qu4rLs1bzbsp3HrfXIJTIEQYIECTfN+CoXpMyN2F8QBEqUs9hGDbXKMopkK+lqs5GS\nr8Hlc6Mf6+kJV3CFru+jjj3MSCoiPS6Fvj5HlG0z9dMRECg3VUaInnJTJU+Vv0iaNoX/mn0rWoXm\npJ/TopR57Os6xEzddC7JWk6yxki5qZKnKzby8KHH+EX895GhOulxREQ+K5x3Mn5E8AAMDAyIceRz\nhNVjI3E4+VGnShR79YwhGAzyr8Z3aPkY1T4jNPQ387fq1z6Rz7W92crLTx8gEPh4bQdqrQ2kaJJJ\nVCZwS8nXkEqk/LX8xQnnv+pzmdGrRx/6cQotCqmCPqeJTnsPyZpRsaFXJo16ek4IC01GqiaZHmdf\nuIIrEAxQZa7lhcpN/L+dv+ThQ39mX/ehCfcf6RU0lity1vDb5f/Lf825laXpi5ljKOHuhXdGCZ4R\n+i1OJBIBZZwEX/wAXW22cD7TRKLncO8x7EMOVmYundA2rVxDbvw0yk1V4XUe/xAvVb9OqiaZO+Z8\na0qCB0DtjWd+wxWsTb2S5OFcqFL9DO6Y+y2cXheP7H5qSsc5X1i8eB433vgV9u/fx7PP/pX1669n\n/frrWbXqIr74xbXh5Z6e7lM67muvbWLTppdOOq6s7DDr11/P4sXz8Hii3XRbtmxmzZoVrF9/Pddf\nfy0/+tH3sVjO7N94W1srV1yx5owecyw+n2/C6zuR73//O3R2dgKwdevfaWubOKx0vnDeeXoAfvaz\nn7Fr1y6CwSB//etfT2lfnU57lqyaGgbDua9wOhM22L12ig35GAyxZCSlcMxUSZJOjUSQTFmIflY+\nixMp66rkzeb36PZ08//yvntKdvQNmtl4dDO720IP5Jr+en6x6k5StON7Tc4E+z9sorayhzVfKEZn\nOLW/D1/AT6O9hRXZizAYYjEQy+3Ceh7a9SQ1zmpW5iyJ2sfisVA0fO+MkBprpHmgBZfPTZ4xM7wt\nIzGZ/R1lGAyxmD0W5qeWTOk7KxyYxrut2/Gr3MTIVfxu91NUmxpQyZVcmLWAyr469vYeYO2sVVH7\nOodcOIYGyDVkjHuutOT5LGf+SW1wDgyRpNcwc/oq3ms+jqRDw4A/1NqhKD0LQ3zo2PqgFpVMiSNo\n43BXGemxKSwvmgdMfH8unjaXjcc2I2i86NVJvH78TWxDDn609NtMMxhPatsIlUe66O1y0NZgZeVl\no/sZDCXkpN6N2Wk55/8zT5WnnnoOtVrNwoWLuPnmWwG4/fZvcuON61m6dPm4+/h8PmSyiR93X/rS\ndVM695w583j22Q0sXbpwwjGLFi3hvvt+QyAQ4Oc//38888xT/PjHP5nS8T9t/OEPfw7/vnXrP9Dr\nDWRmnt8FL+el6Ln//vsB+Pvf/85vf/tbnnpq6m8jZvPAx36jPV0Mhthx3dWfNhsCwQAWVz8qNPT1\nOVAHNfj8Pv5w//uUzs1gzqLMkx5jPDucXhfvtX7IJVkrUMvPvkv9bH0fm45uBaCsu5Kmju6TvnWP\n2PFR+x7eqN8KCFyRfQnTkwp5svw57nn3Ib4/91vhN/EzQZOtFbPbQpomhfbWUFJ6fU0vAU7tb6PR\n1ozH5yFTmRX+LLMVuaRqktlS+S4zNDMjRLA34MPktHKBEBfx2SfKE8NzQWkCo9s0xGL3DFDf3oHN\nbSdWiJ/Sd6YNhLyQWys+4EDPETw+DzdMv5aFyfOQS+V8IP2I1+v/SVlTbbib8wg2Scgbowme3v3R\n2+UgPknFnITp/CtuD3RATWUXABKXkr6h0WMbVDr2tB7CNuTgusJrMJsGJ70/c1WhypMPaw4y11jK\nlqp3mG0oQYfxlGyuOBxqFVB+pIOZ89MitgkoKDYUYDYPTFn4ND/w66h1cQsXknTxJQQ8HlofeThq\ne8LSZSQsW4bP4aD9T3+M2p64ejXxixZP6fynwtq1l3H55Vdy8OB+CgoKueWWb3PvvT/D6RzE4/Gw\nYsVKbr/9ewA88cRj+P1+vvvdO9iyZTPbtr2PWq2mqamRuLg4HnjgIZKSJs81OxGJRMKcOfM4cGBf\neN3vf/8QR4+W4fV6SUxM4uc/v5fk5BTa2lr59rdv4cor17Jv3x48Hg8/+9m9zJoVakGwadNLvPrq\nS2g0WpYsuSjiPP/85xZeemkjAJmZWfzkJz8jMTFx+DreQ6VS09jYgNGYzA9+8GMeffR3dHS0U1JS\nyi9+8cuTvsSuXXsZX/jCOvbu3YPZbGb9+pu49tovh7f98Y+Pc/RoGXV1NTz00IOo1WruvPNHzJ9/\nwSl9Xp8U5114ayxXX301+/btw2oV80k+SexDDgLBAIkxCQAkKZOQD6kY6B+ipf7jtbUPBoNsrH6V\nt1s+oNxUeSbN/USpszbSYGvmwtSFBIIBDvdObfqCj9r3sKl2MwWJedy7+L+5MvdS8hKyuXPubQSC\nAX535Ikz0mUYQuLysaN/5dnjf+P+fY/Q2R36znp7Jp7/aSJqrY0AFCbkhdcJgsDqzOV0DHRRY62P\nGG92WQgSjOq1MzZslTJG3OmGk5arLXUAEWGxyRgRiB+07UAlU/LfC77HRWmLkEtDvTkWpS5AJpGx\no30vf99Yxodv14b37XSEPueJcpKmQiAQxGZ1kZCkRivXUJKXTVAI0NsxQLwiDoU0skeIQaXHNuRA\nJVOxKPXkXqRktRG9Ske5uZJ/N7/HUMDLutzLTsnGfosTc98giXo1VpMTq2nwlPb/LOByuXjmmRf5\n6U/vIS4ujkceeZTnn/8bL774MkePlrF//75x96usPM6dd/6Yl156jYyMTF5/fdMpn3toaIg9e3Zx\nySWjHcVvuukWnn12Axs2bGLVqov5859HRaDFYmbevPm88MJLfP3rN/P446FtNTXVbNjwPH/5y3M8\n//zfsFpHw2V1dbU8+eTjPPron9m48RWysqbxu989NOY6KvnhD+9i06Y3kEol/M//3MP99/+Gl156\nlerqKg4fnjgEHHktXp5++gUee+xJ/vSn3+N2uyO2r1t3DQUFRfz4xz/hxRdfPm8FD5xnnp7BwUHs\ndjupqaE3sw8++ID4+HgSEhLOsWWfL0bm3EpUht6m9cpElM6QG76320EgEEQiObVcqw87docnQBzp\nLfJp5O2WD4iVa/ly4Tqa7C0c6DnC8ozoEM9YPmjczabazZToivlm6fqIvixp2hTunHcbjxz6M1sb\n3+abpV8/bRu3t+/E5XPzzZL1DNg9lB8IPeyOtzSymLyT7B1JnbWBNE1KhDerrclC/Zs+4mcm8H7r\nRxFl1ydWbo1gGBYzSlkM8YrRiqORSq2qYdFzsnL1EWKkCkr1M1BKY7iu6BpUMmXEdo1czVzDLMrr\nG8hoS6KrzUZBsZG0rAQ6HT0ICBF5N6eKw+YmEAiSkBTyWK7KXspL6r0Ee4Po86KPO5I/dFHaQmKm\nkKgtCAKl+mJ2tO+hmjouTFt4yp7AxprQd7Hy8iI2v3iExhoT8/VTywWaiOyf3j3hNklMzKTbZbGx\nk24/G1xxxVXh3wOBAI8++gjl5aEXFbPZRF1dDQsXLorab86cuRiNoc+7pKSUsrIjUz7nvn17WL/+\nejo7O8nNzWPVqovD23bt2snrr7+K2+2KCrlptaNenJKSUp544jEADh06yNKly8OepquvvpYdOz4C\n4ODB/SxdugydTh/edssto/9D5syZi8EQuvcKC6fj8bjRaEJevfz8Atrb25g/f8FJr2nNmv8AID09\nA7VaQ19f73kfxpqI88rT43K5+P73v8/atWtZt24dzz33HE888YSYzPwJYx3uxpwQ9vQkonSFHlQ+\nbwBz76lNSdHqaGdz3T8p0U0nXZtKx0DXmTX4E6LF3kaVpZaLs5ajkMq5IHkujbbmSZOR93cf5skD\nGyhOKuTWkq+N29gvRWNkjrGEakvdpFMrBINBHj/6DK/V/mPCMS6fiw/adjJLP5M5xlIyCfVZkiiD\n2M0eaiz1E+5rdll4r/XDsA3egI8GWzOFiZFCqbail0HHEAuUi6m01NA5MCpiJ+q1YxxeTotNjvh7\nHvH01FhHRM/UQwi3zbqJm2Z+NUrwjLA0fRGa7hQkMoiNi+Gjd+rw+wN0OXrRqZJO2mRxMkY6MSfo\nQhVqadoUlIYgqsEE9Ipo0ZOXkI1Grg7PFj8VSnUz8AX9yCQyrsg+9cTVxhoTxtRYUtLjSEmPC4ug\nzxMq1WgYfePG53G5XDz77AY2bnyFpUuX4/GMP8+ZQjEqTCUS6Sl1pF60aAkvvvgyW7b8C7/fx9NP\nPwlAR0c7f/zj77jvvt/wt7+9yk9/+nOGhkaThRWK0b5JEonkjHTBjrwOycc+x4nH+TR36D6vRI9e\nr+eVV15h69atbNmyhRdeeIGZM0/eil3k9Gjob8bpHZ0wsX+4MeGIp0culRPrTgJZAICejqmHSVw+\nN09XbESr0LK++DoytGl0fgKiJxAMPdwCwcAZO+ZbzR+glqlYlh7KP1iQHKrqOdBTNu74joEuNlS9\nygxjAd8q/Xo49DIeM3XTcfs9NNiaJxzT6minwlzNtvadE4bVtrftwuVzcUXOJQBYhkMac+ZlEePW\nsqHyNdw+d9R+gWCAZ47/jc31/2Jj9WsEg0Fa7G14A94I0RMMBmltCom8aZIc5BI529p2hI/R5uhA\nJVOiGS5XH2Ek3JUWlxKxXiNTo5TGYB9yEKvQopxAwHwcMmIySLCk4TaaWbomH6vJybGDHRFl8xPR\n1mRh68vHcA6O/1DsN7sASEgavc6SghwkQQmxzmjRU5xUyINL7yVJmThl+/MTcjCodFw+7RICA1Jq\nyrtpa5paJZDD5qav20FuUehzzy0yYOodwGZ1Tfn8nzUcjgH0ej0KhYKenm527vzorJ5Pq43lJz/5\nGa++ugmLxcLAwAAKRQw6nY5AIMDmzW9M6Tjz5y9g164d9PeH0jz+8Y+/h7ctWLCQnTt3hCvEtmx5\nY1zP1SeBRqNhYOD8n6PxvBI9Ip88Lp+L3x95gi2Nb4XXWT025BI5GtnoP3SlM45A0iBqjYLuzslF\nzz83HWPfjiYA3m7+ALPLws0zb0Cr0JCmTcE25GBg6OzmF3zYvpvvv3kvP991P3+rfo1yU+VpzRTf\nMdDFMdNxVmZcFH4w61SJ5MVnc6DnSNSx/QE/G6peQSVT8oMLv3nS3jNFiflIBSnHzdUTjtnVuR+F\nRE5mbDovV79Bv8cWsT3k5dlBqX4GmbHpAJj7BtHExpCTa0BAwNnvY3P9v6KOvb1tJ832VoqTCtnX\nfYitjW9Ta61HQKAgYbSde1+3A7fTC8Bgv49FqfPZ332YzfX/4p7dD7C/+zD5CTlR3tk4RSzp2lRm\nJU+PWD8yOSZMPbQ1VeorexECUloTK5GmeMjO13FwZzMmky1cRj4RB3e20N5s5a03juPzRQvnfouT\nGKUMlXpUyF4w3ETV6MkY95in6rGWCBJW2a6hbbOcl586wAf/quGt14/jdnlPuu+IVye3yDD8Uz+8\nfvzZ3j8PXH/9DRw6dJAbbvgyDzxw3yeSd1JUNJ0VK1axYcPzFBVNZ9my5Xz1q1/illu+TkbG+PfJ\neMe48cavc+utN/Gf/3kD8fGj86gVFBTy7W/fzve+dxs33vgVmpubuPPOH5+ty5mUa665lqeeeoL1\n66/n0KED58SGqXBe5fSIfPI02loJBAOU9ZbzlYJ1SCXSUI8eZXz4n7TX60fiVODQ95KXkDOpp6ff\n4qStyYp3yE9OkY4jfeUU6wrJH57OYqSSpnOwi0JF/lm7rsO9x0jW6ElVp3Kwp4xdnfu5Nv8qVmeN\nX9J6Mv7d9B5KaUxUf5UFyXPZVLuZ9oEuMmNHq2Peb/2IVkcHt5R8jbgYLX1MXnGjlCnJS8ih0lzD\nNflXRm33+Ic41FPGPONsLp22kgcO/IENVa/y3dm3hL+n7W27cfpcXJF9SXg/S+8gOoMGQ2ooJ2u+\n+gJ2dr5PmjaV5elLEAQBk8vMPxrfpkRXzG2zbuKlmjd4u+UDNDI1GdrUcJNBgJaG0BtlfJIKa98g\nq5cuY1fHPt5v/YhiXSFfzL+SWfpo76xEkHD3wh+MW7GkVybRMdA15R49UyEYDHK8rIskoxp/rJPX\n6v7BdSu+TNvzFpKaCkieObGnx2IapLvDTmZOIm1NVra9WcMla6dHiJZ+izMc2hpBrY4hPlGFw3Rq\nM6RPRFe7jcqyLnIK9UzLS0KplvPW68epLOti3pLJ8ykaa/vQGTXEJ4bCO7HxSoypsTRUm5i7+NOZ\nizEZjz8eXeG7detbEctpaek899zGcfe/7bbR1hPr1l3DunXXTLg8GeONvffeX4V//+///mnEtm99\n63YgVHX15pvvhtefuHzddV/luuu+Oq69V121jquuip4a5kRbxu4D8L//e/+41yCTydi793B4+cTP\ncezy2N+XL1/J8uUrxz3m+YTo6fmc0zgcThnwDlLXH6rU6Xf3h/N5gOGqD4H+mD6MqVrs/e4J3f6t\nww/F7k47beYuTC4zpbrRaUTSNCHR0zFw9pKZHUMDNNlaWJ69iG+WrufBZf9DXnw229t3Txrucvnc\nPF/5MrUnVCS1Ozo50lfOqsylUWGbecZZSAQJ+7oPhr093YM9/KvpHeYYSiedruFEZuqK6BzsDs97\nNpbDvcdw+z3E1+bSW+3li/lXUWWpZWvj2+zo2MvrdVt5v+1DSvXFZMWF3iD9/gBWs5MkgwadXoNE\nIpAt5DEjqYhXav/OUxUvMuAdZGP160gFKdcXXYMgCFxXeDWl+mIGfU4KTsjnaW20YEyLJS0zHovZ\nSbLawF0XfI/7L/oZ3519C/OT50waxhuPs+Hp6e1yYOkbpGRuOtcXfZEWexuP1vwZVcEQ8dZU4vwT\nh5mqj3YjkQisvmo6i1bkUF/Zy6FdLRFj+i0uEhKj2y4YUmPp7Tq1MnjvkB+PO9p7U320G7lCysVX\nTad4dio5BXoyshOoONSB3z/xfTzo8NDdbg97eUbILdLT1+3AYYsOb35aSErS8a1v3Txh1dXZpqzs\nMDff/DWSknRirumnFFH0nGWcg0NsfGIffd3ntn+PuXeA7nZb1PrG/mZSNckopIpwnsjYbsyhfUOh\nKJfKhtoQmsV5Im9Pa6MFmUwCQThUHUpOnTlG9MQptGjlmknzerwBHzs79uIaJ/fEG/BFzVh9IuWm\nKhQuDdIaI4FAELlExoqMCzG7LVRZasfdxx/w80zFRvZ3H+bpio3YPKPX92bze6hkSlZnLovaT6vQ\nUKqfwba2nfzP3t/yau0WnmNK+jkAACAASURBVK/cRIwshuuKrp7UzhOZqQuFfsYLce3u3E+qkEFH\n9SBH97WxNG0RM3XTebvlA16ueYMdHXsxqPRcnXdFeB+b1UUgEERn0CCVSUjQqek3ubh99s1ck38l\nFaYq7t39ILXWeq7Jv4JEZUjoSiVSvjHzRlZnLmNp2mh+gMs5RG+ng2m5SSTqNLidXlzOIbJiMyad\nA+pkhEXPFMvVp0JlWRcyuYSCGUYWpy7grgV3oJVrOCzbBYCna3wnt98XoKaih+wCHWqNgrmLMykq\nSebAzhYaqkOhoSGPD+fAUJSnB8CYEsugw4NzYPyXgvF4d0slrz13GP+YMNqQx0dDTR8FM4zIFaMz\np8+6IIPBgaGwLePRVDsS2ooUkSMi6MO3aynb10ZjTd+nTgC9+ea7bNiw6ZzlrcyZM48XX3yZN998\nNyK5V+TTgyh6zjKmngHs/W66xhEcnyQfvVPHW5uPR+Se+AN+mu2tFCXmU6orpqyvHK/fi81jDz8A\nISR6pHKBoRgnxLuRSAR6xsnr8Xr9dLb2Uzw7FUWMjI5mK2maFHSq0bdqQRCGK7gm9vRsbXyLl2re\n4N2W7VHbttT+m4c+/EtE4vWJHO09zrTG+VTs7qW9OZT8N9tQQqxCy0fte6LGB4NBXq37B5WWGi6d\ntgqPf4gXq14hEAzQ6mjnaF8FqzKXRYR5xrK++CtcX3QNKWoDuzr30epo58sF68KTU06VFLWRJGUi\nx801Eeu7B3tptDVT4Ap5jRx2D+beQb4x8wbunPtt7rvwbh5Z8St+csEdpGiSw/tZ+kJiNcmgCf+0\nmJxIBAmXZK3gx/O/S3xMHDOSirgwLbLDrEKq4NqCtRFl0m1Noc8yKy+JRH3os7CaJv4epkqmNh0B\ngQxt2skHTwGP20d9VS8FM4woYkLiJk2bwl0L7mBJwWzQDtHdOH5OWXO9GbfLS/HskEdSEARWXFaI\nMS2W7f+uwd7vot8yksQc7ekxpo62dpgK9n4XLQ0W7P1uyg91hNfXV/Xi8waYPisy8TsrN4mEJBXH\nDnRMmKNWW9lLol5N0gnl6fGJKgpnJtPT4WDPtkbe3lzJxqfOjcdERORcIeb0nGXs/e6In+cCny9A\nb5eDgD9IT6edlPSQF6d9oJOhgJfc+GykEimHeo9yoKeMIMETPD0DxOuUIIDV14/OqKG9zYJjyEis\nYrSTa2dLP35/kGn5SQwOuqlqslGypDjKnnRtKjs69hIIBpAMz+Ls9wdwu7x0eNv4oHUHcomMXZ37\nuDz74nC4xOVzU32ol+z2i6gsaWBBVmnUsYf8Q/RUeUge1CKRClQd7SIrN1SefFHqQt5u2YbZZY0Q\nYtvadrCjYw9rslayLu9ykpSJvFzzBtvbdlLb34BKpmL1JHMlqWRKlqUvYVn6Eob8Q5hcFtK0KROO\nnwhBEJipm87+7kN4Az7kwyXVu7v2IxEk+DtUJOol9JudNNWaWJiSExV+Gou5bxBBgMRhj0SSXk19\nZS9DHh+KGBlZcRncs+hHBAmGv4fJaG2woFTLMQx7MwCsZidpWafXRysvIZsHlt4TcS+dDkf2tuLz\nBpg5N1JEKaRyvlJ4Ncf629m9rQGP20uMMjIUV3W0C21cDBnZo/eHVCZhzRdm8Oqzh3jn71WUzAsd\nd2zl1gj6ZC2CEAqvZeef3HNVdbQbQQBDSiyHdrcyfVYKSpWcqqPdJBk0YRE1giAIlC7IYMc7dXR3\n2EnNiI/YbjU76emws2RVLuNx8dqQN9Hj9mLvd5OWnoD/DFY4ioic74ienrOMwxZ6K7Sfw1JRU3dI\n8AA01Y52VG60hfIU8hKymZFURIxUwQdtoTLOhGHREwwGMfUOYkyOQ0Dg5erXqQqW09Nl4/69jzDg\nHX1jbmm0IJNLSMtMQGYcIsajIT9mtHHdCGmaFLwBb3hyRoBDu1p4+akDPF/xCgaVjltKvsaAd5BD\nvUfDY/Z07ifGloAkKKG6uiPquABlbdXo2/JIyozhgouyaa4zh/OPLkoPucR3dY6+3e7uPMAbdW8y\nx1DKF/JCHW+Xpi1iln4mmxvepNxUxcWZy1HJpjZthkKq+FiCZ4SZuiI8/iEa+kPVb26fm/1dhymJ\nmY3N7Gbm3DRSMuJprD15zxVL7yAJSWqkstCf+YjHx2oe9c4IghAWPO3NVjZvOMKQJzoRNxAI0tZk\nISsnCUEQ0MTGIFdIz4inBzhjgsdmdXH0QDuFM5MxpIzvaSsqSSEYhJb6yPJvh81NW5OVotKUqOab\ncQlKVl9ZRF+3gz3bGhAEwknCY5ErpCTqNfRNIa/H7w9QfaybrDwdKy8vZMjj4/CeVsy9A/R2OZg+\nK2XcvJGikmRilDKOHWiP2lZTHhJRhTOTo7aNJUYZEq/jCTcRkc8ynznR0/vKS5j+/vq5NiNM2NNz\nDmPnXcP5N3qjlua60Ydlg62ZJGUiCTHxKKRySvUz6BqeCmEkvDXo8DDk8WFIjmVd3uVcmLaQ4tws\nJAEZfruMLfVvAsP9WxospE9LRCqT0K0MCSrBHP1PdaSCa2yIq7nOzJDHz5ANbpr5VUp0xaRqkvmw\nPTTxbCAYYHvbLjSDoTdwS8v4ORNlH3UhIOHSy0uZtyiLQCBIbUXompKUiZToi9nduZ8hv5c36v/J\nmx/tY0bZpaxL+UL44S8IAjdO/xKxci0amZqVmReNe66zQWFiPjJByqGeo7zZ9C6/2P0bHN4BMu2F\nCALkTTeQW6jHanKGG+RNhLlvMCx0AHTDv4+EvU7kwM5mutvtlB+MFpS9XQ7cLh9ZeaH8G0EQSNSp\nw32Azhf2bGtEIhFYtDJnwjHpmQmotQqa6iKFY/Wx0P04vXR80ZpTqKd0QTpul4/YeGVYTJ6IMSWU\nzHyyFgm1x3twDg4xY04qOqOW6aUplB/q4MDOFiQSYULhIldImTEnlaZaU8Q9EAgEqanoIStPh1or\n5puIiIzHZ070BOwOBo+O3yxuhGAwyPOVL3Os7/hZt2ckUdDe7z6tPjGnQ0+7jbgEJcVzUui3uLCa\nnQSDQRr7m8mNnxYeN884O/z7yLxbpuEkZp1Ry5ppK7mu6Br+Y06oq+wc2QJ2dx2gztpIv8WFw+Ym\nKzeJQDDAscFjCDF+Oluic5lSNMkICOHOzIMDHszDD+ILlIuZFpcZyqXIuJBWRwdN9haO9h1n0OJH\n8EuRaYJILVr6ByLfppvrTfg7lSgKB0hM0mBICXWjrTrWHf7sl6UvweEd4DcHfs+O2oOkt5UgeKUc\n+rAt4lhahYYfzf8uP5h/+4Qdf88GMVIF+Qm57O7az7+a3iU3YRo/mvcdrM0+MrITUWsU5BSGElSb\nJvH2eIf8OGzusNCBUNmyTC4Jf9Zj6et20N1uRxEjpWx/Ox53pLentdGCIEBmzmjYJ0mvOWOenjNB\nR4uVploT85ZkoY2NmXCcIBHIztfR2mgJ9+AZsIdyajJzEolLmPj7XrIql5SMOFIz4yccY0iNxe3y\n4rB5JhwDcHhvK5rYGLJyQ0LygmXZSASBploTOYX6iB5AJzJrQQZyhZTt/64N39ttjRacA0MTijYR\nEZHPoOhR5uTgaW/H75z4DbSuv5H93YfZ2z21ydZOB3u/G6lUwO8LTFjmfTo4vS4GJ0nqDQaDdHXY\nScmIJzt/9GFpcVuxDdnJi88Oj52RVIhSGkOMVBF+0I9MOTH24amNi0GtVWDwpKMbzn9prg9Vk2Tl\nJtFka2XAO0hSupKOlv4osaeQyjGq9eEKrubG0MM7SBDD0GgexgXJ81DJlGxv28W2th0Y3KGGe8VL\n9EiCUg5V1IXHBgJBtr9Tg1vpYM7C0Vngp89Kod/spHvY21WcVIBepaPPaWZO92oUMhml89NprjfT\n0RI5sa1OlUiqZvIwwZkgGAzy5qvlvPbcYT56p45ZnoVcmLCEuxf+gNtm3YzGmYS9301+cSipODZe\niT5ZO6noGfHAjPX0CIJAkl4zrqen4nAnMrmEy75YwpDHFxE6sfe7OX64k9SMeJSq0Qdxol6Nc3Bo\nSs3yhjw+3t1SydEDbScdO0IwGKS5zkTt8ZNPxBoIBNn1XgOxcTHMvuDkTd9yCvX4vAE6mq0EAkHe\n/UcVgUCQpWsm7x0llUq4+sY5rLqiaMIxI3k4k1Vs2vtdNNT2UTxrNJSmjYth1rDtxbMnFy5qrYKL\nLs6nq83G8SOdAFSXd6NUyZmWf2qzgX+aWLx4Hjfe+BX279/Hs8/+lfXrr2f9+utZteoivvjFteHl\nnp5Ta4nx2mub2LTppZOOKys7zPr117N48Tw8nmhRu2XLZtasWRG2Y/3662lvb6Onp5v/+q/bTsmm\nE7HZbGzc+MJpHeNU8fl8E17rp5XPXCJzzLRsCAZx1dejnTV73DE7O/YCobmUziYetw+P20f6tAQ6\nWvqxW91otBO/gX4cnip/AV/Qz4/mf2fc7fZ+N26nl9SMOLRxMRhSYmmuM+HLGS5rHSN65FI5C1Pm\n0zXYHc4lsPQNEhuvDFfBQOjhmZIeR2uDlYsSLmWr91XK2hpJ1GmJS1CyraEKqSBhekEGuxqbsJqc\nEQ9fgDRtKm2OUBjl0PFafDIvxuQ4TD2jD2SlLIbFqQvY3raLIEGWeK8kGBfDBXOKOPLhNlrqvRCa\nEYKa8m5cdh99hbWUGEcn+MsvNrLr/QaqjnaRmhGPRJBw+6ybqC0zcbzXxKor8skvNtBUZ2L3B418\n6aZ5n3j/jY6WfloaLCQZNNRW9OAd8iMISTRZBtEv9VJX2YtUKoQ9PAC5hXr272hm0OFBM45Xw9wX\nLXog5J1paTRHrHO7QucoKkkmfVoCOYV6jh1sp3RBOlKphLderyAQCLDi8sKI/cIVXGZnVELtWPy+\nAG+9cZyOln7qq/pYvDJn0gZ5fn+AuuO9lO1rC+cf2awuLliaPeE+Rw+0Y+4b5NKrZyCTSyccN0J6\nVgJyhZSmOhPdHXa62+1c8oXiKeW4nOz+0BlCPZF6uxzkTR+/CWLVsW4EiKrOWnDRNFIz4yMSqSei\nqDSZ+qpe9m5vIjktjuY6MyXz0pBKP3PvshE89dRzqNVqFi5cxM033wrA7bd/kxtvXM/SpeM3Hz1x\ncs8T+dKXrpvSuefMmcezz25g6dKFE45ZtGgJ9933m6j1f/rTE1M6x0TY7TY2bnyRG288/UmJP8+c\nFdHz7LPPsnjxYoqLiykrK+POO+9EIpHw8MMPM3fu3LNxyjAxWdNAKsVdXzeu6HEMDVDWV4FGrqbf\nY6PfYwsn7Z5IIBjgkUOPMy951rg9Wk7GSGgrLHr6XZO6xU+VkYaCQYLYhxzjlkiPlMonD1ds5RTo\n2L+jGXePFaVUGZV0++XCLyAw+k/d1DuIzhg9M/OiFTm4nF4a99iYqVqD0w3WjEZ+uXc7Vnc/xYYC\ncjKN7KKJ9hZr1MM3XZPKkd5jdA/2Yuv0Ijf6mJZl5Oi+dny+QKjXD7A8/UK2t+1CKYnBb5GTlhVP\njEyOkOzC3RWLzxdAEODgrhY8WjvpOfERScdyhZT8YiN1lT0sXJ6DXC5F4dJSs7eWzNxEikpDE2Au\nWpHD+1urqano+cTDAxWHOlCq5Fz7n/OQSASsZicVhzooP9hB3fFegsEgWXk6YpSjf645w6KnafhB\ndyKW3kFkcklUqCbJoKa6vBuXcwiVOpT3UXW0C78vED7OBUun0VRromx/G/1mFxbTIFd8uTRKECTq\nhhOjTROLnmAwyAf/qqajpZ8Vlxdi7h5g7/Ym3C4vi1fmRgiIYDBIU62Z3e/X47B70Bk1XLx2Ou1N\nVg7ubBm2LTvqHOWHOti7rZGcAl1Ub5qJkMokTMtLoq4yVBpePDuFghmnNov5ZMfWGbUTNin0uH1U\nH+0mf7qR2Hhl1L4j4a6TMVJOv+npg/zjpaMEAsEoEXWm+fULB6PWLZyRzCULMvF4/Tz8UvRs5Mtm\np7FsdhoO5xB/fC16zrjV8zNYPPPM27127WVcfvmVHDy4n4KCQm655dvce+/PcDoH8Xg8rFixkttv\n/x4ATzzxGH6/n+9+9w62bNnMtm3vo1araWpqJC4ujgceeCg8y/nHoa2tlW9/+xbefPNdfD4fS5cu\n5Pbb/4vt2z/Abrdzxx0/YMWKVQCUlx/lz3/+Ey5XSPB/61vf4cILL+Khh36DzWZj/frrUavVPPnk\nM6xdexl//OPjZGfnhK95ZHnt2sv4whfWsXfvHsxmM+vX38S1134ZgKamRv7wh4fp7+/H5/Nxww3r\nwzPSv//+uzz55J+JiYlh5crVH/uaz1fOiuh57rnn+NKXvgTAww8/zE033YRGo+HXv/41r7766tk4\nZRiJQkHckouQJY5/g+7tOog/6OfqvCvZWP0qLfZ2Egzj/8OuMFXRZG+ZsDHdyRhJYk6flhCxfKao\nNNcQJBQ6Om6qZkla9FwyI3kagzFWWnobyMrPZv+OZjob7eTkZEWVKo9d9nn92CzOcd9WE5LUrLth\nNu3NVnZvb8DicqLNEIjXppIZm85VM1YTKyiJS1DSVGvGkBxLkCAxMTJ0Ri3pw2Lrhb2bifVOZ8HM\nXJK0sQQCQcy9AySnhZrdGdV61kxbicobS8NeL6npofWpuVp626U0NnQzNAgDdg9dhVXcmv2lKFuL\nZ6dQdbSLFx/bG16niJGy8rKi8EO3YIaR8oMd7P+wibwiQ0RDuLOJvd9Nc72ZOYszw0JPZ9Cw4rJC\nZs5NY+e79XS12ygqiQyzJerVxCeqaKzpY+bc1AjxMOjw0NHaT5JeE+WVGBGf3R12cgr0BAJBjh/u\nJC0zHp0xVEGlM2rJm27gyJ6QJ/TC1XnjPohj42OQySXDHbujCQaD7Hy3PuTdWZXLjNmp6FdrQYCy\nfSHPTHaejuT0OGRyKbver6etMSSQr/xKIZk5iQiCEA7rHdzZAkGYd2FW2Jtx7EA7u95vILtAx5qr\nZ5ySly67QE99VR+JejUXXXJmp0QxpsZSV9lDMBiMsCkQCPL+1ircLi9LLz79c8bGK1m8Mocd79Sj\nT9aGv0OREC6Xi2eeeREAt9vNI488ikqlwuv18r3v3cb+/fvGbXRYWXmcDRs2YTQa+dWv7uX11zfx\nzW/ePqVz7tu3h/XrrwcgIyOTBx74v3HHxcbG8eyzGzh8+BC//OU9rFixCpvNxv/932/43e/+hE6n\no7e3l1tu+Tovv/waP/7x/+Pb376FF198ecrXPzTk5emnX6Cjo52vfe06rrxyLTKZlHvv/Tn33fcA\nWVnTGBwc4D//80ZKS2ehUql48MFf89e/Pk9WVhbPPff0lM/1aeGsiB6Hw0FsbCwDAwPU1NTw3HPP\nIZVKefDBB8/G6aJIuekb464PBAPs6txHXnwOC5Ln8FLN67TY25htGH8m9+3toe6t3c7JOwBPxEi5\neqJOjTYu5oyLngpTFbFyLVKJlApz1fiip8NGclocz1b+jV6nCX1MEpnaJQS6VeTOSR33uH5fgL6e\nAVobzASDkfk8YxEEgcycJL6SnciA3RPx1joyx1JWbhIVhzv5+8bR5PJll+aTVhw692B3kFiguDCT\nQCAk4Pq6HWHRA7Au73Jqj/fQQDUpwx6F0qIc3tnZyLFjrQz2BHDH2sjMSaIgMbo/iTE1lku+UDzc\nJTd0jrSsBLRxo2EhQRC48OI8/r6hjH+/XsHqK4vQxk2ewBwIBEMekX0hcbD2+lkRYcCxDHl8bP93\nLfZ+N2uvLw33hzl+JBTiK5kb7a3RJ2tZd+NsbFZXlJdFEAQKZhg5OFzqP3NuGjmFeiqPdnFsfzuB\nQJDl/xHdLsCQEkuMUsZbrx9nWl4S+pRYHHYPS1ZH9vtZMOztKZyZzKwL0se9ptEKrvFzysr2tVFx\nuJPZCzOYuyiUZyVIBJauyUcTG0P5oQ7aGkfzqOQKKRddnEfJ/PSIknGJRGDlcA7NwV0tHNnbii5Z\nS2yckobqPnIK9axZV3zKYZ3sfB3Fs1OYvTAT+RRCYqeCMTWW40c66be4wn2SAPZ/1ERLg4Vll+aT\nmZMUNQ/Zx2Hm3DT6La4pe4hOh7u/vmDCbTFy6aTbY9WKSbefDUa8FwCBQIBHH32E8vKQt8lsNlFX\nVzOu6JkzZy5GY0hsl5SUUlYW7cGaiInCWyeyZs1/hI/f3d2Nz+fl6NEyOjs7uPPO0XmyBEGgo6MD\nlWpqbTPGO0d6egZqtYa+vl7cbjctLc387Gc/CY/z+Xw0Nzfh8/mYMWMmWVmh8PPVV3+RJ5547JTP\nez5zVkRPamoqhw8fpr6+ngULFiCVShkYGEAq/WTengECXi/4fUiUozdKrbWBPpeZK3LWoJDKSdek\nTJjX0znQTY21njhFLBa3FY9/iJiTzJR9IvZ+N4oYKTFKOXHxSuz9Z65Xjz/gp9JSy2z9TGQSKQd6\njuANRFbceNxerCYniTlSep0mlqdfSLO9lS5tI0k908hSRT/Myg91sPuDhnBfn7gE5UlDcoIgRLnp\nR1i0IoecQn34jffwnlb2bm/iK3nzUUpjiLenkKhXoYmNIRgMolTL6esaiDpOV7sNuUIa9lTkJGYx\nmLifvoaQeOiaXsV38r4atd+IfVMJXaRmxLPy8kJ2vlfPpqcPctHF+eQW6WlrClUF9XY50GgVxCWq\n0GgV1Ff3YbO4iEtQ4rC5eXdLFZd/qSSqx0tvt4PXnj+M3epCEATeeuM4V103i0AgSNXRbnIK9RMK\nLEEQJswzmX/RNOISVVQc7mDX+w3ser8BgPwZRhYuyx63j4xSJee6Wxdw/EgXlUc6aWmwhGZhL4wM\nCyXpNaz/7mJUavmk3pNEnYaOVmvU+obqPvZubyK/2BDVKE8QBOYtyWLu4kwG7B66O+w4bG6KSpMn\nzHkbET7ZBXq6O2z0djloa7JQODOZlVcUfqw8FrlCysrLJ05IPh1GJnjt7XKERU9dZS9H9rYxY05q\nVOPE00EQBJaeYU/VZ4WxQmHjxudxuVw8++wGFAoF9933P3g84xeXjJ1iQiKR4vf7z7htI+eQSEYb\ntEKQoqLpPPbYX6LGt7W1Rq2TSqURhSJDQ5HXE3kdkvB16HS6cT1G27a9f+oX8injrIieu+66izvu\nuAOFQsGjjz4KwLZt2ygtje6geyJWq5W77rqL1tZWFAoF06ZN45e//OUpxVMDbhcNP/w+iVetJeGy\ny5ENd7bd2bkPjUzNXEPIjmlxmRzqPRrRGXiED9t3IZfIuCJnDS/XvEGPs5es2JNXhYzFbnMTFx/6\no4tLVIUn4zwTNNpacPlclOiLkUmk7OzcR521gThVSXjMSMVSq7QBtUTFNflXIpfI2KuuoGyrBbkp\nDsaE0oPBIIf3tKI3apm7OIvktNhxk2RPBUWMLCIpMy5ByaanD7LznXquvuBKju/zkLUg1LlWEIRQ\nj5Nxql662+2kpMeFBYVcIkOdEQQzuOIs5OYmkx13+rNHF89OJX1aAh/8q4Ztb9aw/d81BIOgVMlI\nzUzA5RyipcGMa9CLPlnLpVfPIKdQT9XRLj56u47d7zeEK4ACgSC1x3vY+W49MrmEL3x1NgN2D+//\ns5oP36olJT0ej9tH6fzxPSknQyIRKCpJpqgkmb5uB831ZrLzdRM25RtBo41h4bJs5i/JorHWRGx8\nTJRQA1BrTi7ykwxqao/34HH7wjlH3R123v9nNSnpcay6cvqEomlELE8kmE9EIhHILdKH83ZODB2d\nTyTq1MjkEppq+hhy+7CanVSXd5OaEc/SNfnnrd2fZRyOAfR6PQqFgp6ebnbu/IivfOWGc21WBLNm\nzeY3v7mfI0cOMXfufAAqKsopKSlFo9Hidrvw+/1hB0JGRiaVlcfJycll797d2GzRkxWfSE5ODhKJ\nhHfeeYtLLw01ZG1qaiQ5OYXS0ln85jf3097eRkZGJv/4x9/P3sWeI86K6FmxYgU7d+6MWHfZZZdx\n2WWXnXRfQRC49dZbWbQo5HJ88MEHeeihh/j1r389pXM7hgY5aqng4PJEWmQ78WzfiVGtJ1WTzDFT\nJSszLgpPazAtLoudnfvoc5lJVo/mrTi9TvZ1H+aC5Lnhku7uwVMXPY5+d3hSwrgEJc7BIbxe/xlx\npR83VyMVpExPKkAqSJALMva908qHLd3MuiCdRSty6e6wIwhQ4TvKsmkLUQxf9+IZJdRt30tjtZmi\nktEQV1e7DefAEBeuzptyQuipEpegYuGyHHZ/0ECRKplgoCei94shRUtbkyXic/K4fVj6Bsk7cQLF\nfAPlrV30pdfxw9xbz6iN626YTWVZFzari+x8HSkZ8RHCwOcLIJUK4YdXKMTg5NiBDlRqOT5fgOry\nbpwDQ2RkJ7L6yqKwgLRZXRzcFZrAUmfQnJHkdkNK7EnFzolIZZLTTt4d8WJYzU6MqbFY+gZ56/UK\nNFoFl107M5yndDY4n4WDRCJgTI2jqc5MU50ZRYyU1Ix4Ll47/TNfXXW+cv31N3D33Xdxww1fxmhM\nZv786HSAc01CQiK//e0jPPbYH3A4HPh8PtLT03n44UdJSkpi1aqLueGGL5OQkMCTTz7Dbbd9l1/9\n6l42bfobCxYsxGAYv1pwLDKZnIcf/gO/+91DPP/8MwQCAXQ6Hfff/1v0egN33fVTfvjDO1AqlWIi\n81Spr68nISEBvV7P4OAgTz/9NBKJhFtuuQW5fOKGWwAJCQlhwQMwZ84cXnrp5P0TRnjm+AaqzfXE\n6mTktrvIXHYp3a4+Oge6UUpjWJa+ODx2WlxIxLTY2yJEz67O/XgDXlZmLsWo1iMRJPQ4J57VeDyC\nwSB2mzvcwTYuIeTxcfS7oyqZPg4V5iryE3JQyZQEg0GKuhfhblGQmZ3IsQMd4ckhFQlBfJIhlqaN\nXrcgCORNN1BxuDPiDb2hqg+pTDKlOYNOh9IF6dRX9VJT0YNUKkRU/hhSYwkGQ/2BRuYIG5ncNOWE\nCqEiQx5vFrzDfOPscJfnM4UgCJOGIMZ7mC9ZlYfN4mL/jmYEIdSzqPjSVBYsnobZMprsu2DpNGz9\nLuqO91KyIP28fnifPTyxtQAAIABJREFUjMThSS3ffuM4breXgD+IIkbGui+XhqvDPq9cvHY6NouT\nBJ0atUbxqf6ez3cef/ypqHVbt74VsZyWls5zz20cd//bbhvNoVm37hrWrbtmwuXJmGhsZmYWb775\nLgAymYy9ew+Ht524XFJSyuOP/3Xc499zz/9GLJeUlLJp0xvh5Tvu+EH49xOvf+zytGnZ/P73fxr3\nHBdfvIaLL14TXv7GN7457rhPK2dF9Pzwhz/k97//PXq9ngcffJCmpiZiYmL4xS9+wf/93/iZ7OMR\nCAR46aWXWL166mpztnIlpanLmWs107PrKaZdMpOY/PHDHqmaZBRSBc32NhamzANCuTIftu+mICE3\n/CDVq5LoHjy1ZGbn4BB+X4C4Ydf9SOmwvd81rujp7rDx0Vt1LF2Tf9IJHM0uC12DPVyYGnpTObCz\nBVoSMCU38Y2vz6eneohtb9bgHBhiMK2bgoRcUjSRb/T5xUaOHeigqc7E9NKUUIfmWhNZuUlnvXJJ\nIhFYeXkhrz13mNTMhIi+KsZhb0Vf16jo6Wq3IQhEJDfz/9k77/ioqvT/v++dmplMykx6AgkQSEIo\ngVAERJRqoSkWFLCLq2tbdV3Lrq4guujPsuu6q2LhC0tZC7oIroILFpReBSIlgRDSM2lTMvXe3x+B\nSCRAwDTlvF+vvF6ZW875nHPv3PvMOc95HqBreDLju4xt0oG7PZBliTGTenIwp4xOXawNjtLyT37Z\nS5LEJZel0T0jhk5t4HzamoRFGEnNiCbgV4iwhRBhNZGYfPqoxucLoRbDaSNDC84eq9XGzJm3cN99\nDzbpgNza7NixjRdffB6r1SaM2F8orWL0FBYW0rVrV1RVZfXq1axcuRKj0cioUaPOfPIJzJ49G5PJ\nxPTp05t9zq699Y5cEyb1p/Qd0BTnE92/6dVZAN2syRTVFREdXf+yXZP3HVXeam4bcF3Dts4RCZQ4\nyxs+n47jx3ic9VFqk5IjiY62YAqp/9UbDKgnlZO7r5wV//4evy/I9g0F9M3uxOnYeqA+kvRFPQaS\nu72crd/mk9E/hvc1K9lWvJuJg8eQ0Sue/3y6iZXutfwmfepJdUZFhbLG+gMFuZUMH9md/Dw7bqeP\nfoM6NaudzeF05URHW7j+9kGERRgbHRcdbWlY6RYdbSHgD1J0uJq4xHASEk82Bm+MOf0vsJZqy9nQ\nlM6mdMTFt1zMpubQWn1xw+1n9/Jpj2vSETVAx9Bhs/1ylrkfHy1pL7Ky+p/VknFBx6NVjB6DwYDT\n6SQ3N5f4+HisViuBQOCsQlnPnTuX/Px8Xn/99Qbv9uaQFGVi9ZYCqulJ9LXXE0zsetploQnGeL4q\n/I7i0irqAh4Wbv+QbuEpJOu7NJwXqbWy3bGHktJqNPKpR0GOL9MGOJJf77SsolJeXp98UKfXUHy0\nppGe3B/K+WJ5DpE2E527Wdm+oYA9u4oaQtk3xYbDO4gJicKe52f18r106RHFRaPT+W5LPNuLdzMw\nckB94MLIreg1El2NqU32QUr3KHZtPkrBkUq2rs9Ho5WxxphbZBntiX1xKsKs9aMBPz3OFhNKweFK\njhZU8dmHuykpqmXk+PSz1tUcDW1BR9DRETR0FB0dQUNH0REdbcFud/6iDB+B4OfQKkbP+PHjuemm\nm3C5XA2jNHv37iUpqXmOwC+99BK7d+/mzTffbLTkrjkkx1mo8wYptrtJGlsfo0DxepENTQ8zJ4d1\nIlAQoMhZwpqCdXiCXq5Pn9JoNVecOYagGqSizk6suXmOn45jMXmOr0yRJImwiMbL1g8dqGD1f/YS\nmxjG5Vf3RpJgz/Yidm4qYMyknk2W6/bXsb86lwtjh/DF8hzMFgOXXJ6GLEtk2tJZlb+W+8sebzh+\nTOeL0clNX+bUjGh2bCwgb195m01tNYeYuFDyD9pZvngnlRUuRk/MaLFouQKBQCA4f2kVo+fxxx9n\n3bp1aLVaLrig3oFWkiQee+yxM5574MAB3njjDVJSUpg69XhUyyRee615AZKSj/mEHCysISk6FF9p\nKUf/31+IuuY6wgZdcNLxKWH1U0mf569lR/n3XJoy6qQkk8f9YUrc5c02emprPJhC9Y38VcIiQqg+\nlktIVVU2fHmICJuJ8df1aViplNE3nl2bj3LBxZ4ml/J+ceQrAkqAkP1J2GtqmXhD3wZH5BFJQzEY\nNag+DRa9GYveQob15AB1x4mKrc+VtXldPm6n75R5gtqa4zFOqivdXDolk+RuretYLRAIBILzg1ZL\nOHrhhRdSVFTE9u3biY2NbVaMHoDu3buzb9++c67XFmbEYtJReCzhojY8DF10DCXz3oBAkLChwxod\nbzVGEqozs6P8e2JCorg0+WSn6VhTvaFT6iqDE6I313gdhBuanoaqra5rcGI+TliE8ViUY5W8fRVU\n292MmZTRaAl77+xEdm0+yvdbChk6qj5KbiCgYC9z4pM9fHXoO/qpF1CQU0v/oZ1J6PSj/0iEIZwZ\nWVOaPWR+PMT/tvVH2mTVVnNJ6BRBeu84MvrGnbRiSyAQCASCc6VVAkaUlZUxffp0xo4dy7333svY\nsWOZPn06paWlrVFdIyRJYs4dFzBtTH1GaNkYQuL9D2JKz6Dk3bewr1jeKIKlJEkkHxvtmZp2VUMM\nnxMJ0RqJMIQ3SkdxsPoQT3z7DD9UHmhSh6PagyXiZKMnGFRxOXxs/S6fCGsIXdMaj65Ywo10y4hh\n785iPHV+9u4oZvEbm1i2YDsr5ueQumUU/q02YuItDBiWfG6ddAKpGfX1d5SpLaiPlHvJFWnC4BEI\nBI244IL+TJt2LZs2bWT16s+57baTM47Pm/c6s2c/ddpyVqxYzmOP/R6AnJy9PPnkE00eV1RUxLhx\nZ149vH//Pr74YlWjbTNmTMXjOffUQ36/nxkzpnLJJcNYt+7rJrUNGzaQGTOmMm3atdx88zR27dp5\nzvWdismTryA392CLl3ucu+66o8n2/ZQ33/wnq1d/DsDWrVvYuHH9OdXXKkbPn//8Z9LT09m0aRPr\n1q1j06ZNpKen89RTp78RW4rQkMaGi2wwkHDvA1gGXYD942VU/6/xCoBLU0ZyfdpVpFlPHco91hTd\naNn6l0e/RUVlZ/luoD6juqeufsVWMKjgdHibGOmpj9Xz/daj2Mtc9B/SuclIuFmDkvD7gvzrnxv5\n6rP9hFr0DLo0icKuOwnJdJM9LJlxV2W2SJAza7SZQReltIgBJRAIBK3NvHnzGTRoMCNGXEJBQQGH\nDx9q2KeqKp9+uoIJEyY1u7yMjJ7MmjXnZ2nav38f//vJe2XhwqUYjeceukGn07Fw4VLS05v27wQI\nDbWwcOFSFi16jwkTJjNnztOnPPaXzsyZdzXkEtu2bQsbN244wxlN0yrTW1u3buWvf/1rQyBCk8nE\nI488wvDhZ5+p/FyodnpZ+r8DjOibQEZKfRwUSafDeNV0TBkZWAbWL7FV/H5knY6u4Sl0PRZ5+VTE\nmWPYWLy1Puigz8HO8t1ISOwp38fGkjx2bDxKeGQIl13dC0kCVf3RyDnO8dglOzcdxRJuJPUUzrnR\ncRa69IjCUeNh4IXJJKfaeHfPYlwxpVw55JZTTqmdC5IkkT1UGDwCgeD0rC/awneFm1ql7KGJgxiS\ncHbJSPV6PWPHXsqKFcu55577Adi6dTNarZasrP4EAgEeeug+ampq8Hq99OyZyaOP/vGkALlbt27h\n1Vdfbghc+MEH/2bp0kWYTKEMG3Zhw3GnKs/tdjFv3uu4XE5mzJhKVlZ/HnroES64oD9r1qzDZDKx\nd+8eXnrpeerq6ggJCeHBBx+hZ89MioqKuOWW6UyefBXr13+Lx+Ph8cefJCur31n3Yf/+2bz66ssN\nnz///L/8+99LCATqf4zfe+8DDDz27ps8+Qouv3w8mzZtoKKigmnTZnDNNfU+tDt2bOOFF+oTpvbr\n15/jSZqBM7Zj0qQr2bDhO7xeL08//QzLln3Inj3fYzAYeeGFl7DZTh/pf9aspzAY9Bw5kk9paSm9\ne/fhySdnIUkSs2Y9RUZGBv36ZfPRRx+iqgqbN29kzJhx3HjjLc3up1YZ6QkPDyc3N7fRtry8PMLC\nwk5xRstiMmjZuq+cPYfroxIHgwr/++QHls7bQmlkGrLBgBoIkP/0nyj91wICNTVnLDPOFIMn6KXG\nV8v64s0oqsKFxouxbunNtvUFdOkRhafOz0f/2k7evgqAkxyRj39WVeg/pFOjkZrjGeDd/npH50uv\nyuSaW7JJ6R7FUWcRW8t2MrLT8BY1eAQCgeCXzIQJk/jss5UNiTRXrFjO+PETgfpknLNmPcv8+YtY\nvPh9FEXhk0/+c9ryDhzYz/z5b/PGG++yYMFiak54N5yqvPDwCO644zcMHDiYhQuX8tBDjzQq0+/3\n89hjv+fOO+9m0aL3mDnzbh577Pf4/fXGSE1NNb1792HBgiXceusdvPba386pL776am3DSAjABRcM\n4e23/48FC5Ywe/ZzzJrVeKbF4/Hw1lv/xz/+MY9//ONV3G43Pp+PP/3pMR588BEWLXqPrKz+lJSU\nNLsdfftmsWDBEiZMmMQ99/yGq6++lkWL3iM9PYP33/93s9qRm5vLSy+9ypIlH/DDDzls2rSx0f7U\n1O5ceeUULrtsPAsXLj0rgwdaaaTn9ttv5+abb+bqq68mISGBoqIili1bxv33398a1TViy7p8gkGF\n5GgzuYU1BAMKq5fncGh/BaZQPd9+cZBOXSIxyEFMGZnUfP0lteu/JXLUGMKGDUcfG9tkucdXcBU5\nS1hXuJF0elP1tQnJ4CJxJIwd1BNJkVjw+nrWr80DOCkqrUYjExpmQFUhrVdco3177D+w+IcPqair\nZFK3yxrtW1uwDoNGz6jOI1qqmwQCgeCsGJIw4KxHY1qbtLR0rFYbGzZ8R1ZWP77++it++9v694yi\nKCxatJD1679FURRqa2vPON20bdtWhg4djs1Wv6hj8uSrGqatzqU8gPz8w+h02oZRlkGDBqPTacnP\nP4zJZMZkMnHhhRcB9Wkl/va3l09XXCOcTgczZkyluroKr9fHO+8saNh39OhR3njjccrLy9BqtVRW\n2rHbKxpGW44bSAkJCVgsYZSVleL3+zEYjGRn11/n0aPH8pe/PNPsdgwbVj+bk5aWTkxMLD16pAGQ\nnp7Bpk3Nm44aMeJiDMdCzKSlpVNYeLTZ/dEcWmWk59prr+Xll1+mqqqKtWvXUlVVxYsvvthgMbYm\nwaDClm/ziarw4Cys5dMPd3NofwUXjk5lwnV98HmDrPsiF9kYQuy0GaTMmoO5V28q/7uSw0/8Ac+h\nvCbLjTXVG0NrCr6hyltNVFE3jCYdjkH7OKyrX20WFRvKldP7ERlVn2G5qQzlw8d2Z8zEDDQ/yd30\ndWG9U9aG4i0ElWDD9rqAh+1luxgQm4VJ13i6TCAQCM53xo+fxIoVy1m9ehVZWf0akm6uWvVfdu7c\nzuuvv82iRe8xZco1eL2+c66npcs7jk73Yyw6WdY0jFo1h+M+PR9//Cnjxl3Gk08+0bBQ58knH2fK\nlGtYsuQD/u//FqPRaBrpPTEGnizLp6y3uek2ftqO5pb/UxqfpyEYDDTrvObSaul+hwwZwpw5c5g3\nbx5z5swhOzub119/vbWqa2DwiC5cfXN/LFEmEhSJo4equPiyHvQekIg12kz20M4c3FvG4QP1U1D6\n2DgS7rqHLnNfJHrqNAzJKQDUfPtNIwMoTB9KiNZITuV+ovxxVB/102dAIpmxPThYfQhPoD7adGiY\ngatm9GPKTf2bdFJOSbWdlFW73G0nx76fLmHJ1Poc7Lb/0LBva+kOfIqfIfGDWrqrBAKB4BfPuHGX\nsXHjBt5/fynjx//owOxwOImIiMRsNuN0Oli16rPTlFJP//7ZrF+/jsrK+oj6y5d/3Kzy6rc5mywz\nOTkFvz/A1q2bAdiyZROBQIDkY++alkCj0XDPPfdjt1fw9ddfHtPrICEhEYBPPvkPPt+ZDbTk5BS8\nXi87dtQnQF2z5gscDkebtaO5mM2hp+zvM9FqRk9TnLhUvDWJjrNw2ZReVEYayBrZlYy+P2bg7jek\nM9ZoM19/fgCv50cLUme1Ejl6DJIsowYCVK5cwZFnZ1P+3lIUrxdJkog7Fq+nqz0LnV5Dr/4JZNrS\nCKpB9lX9uKRPb9BijWp+JvVvitYjSRK39rqBcL2F74p+dBb8rngz8ebYhiCKAoFAIPiR8PBwhgwZ\nit1ewUUXXdSw/fLLr8DlcnHddVfx8MMP0LfvmZ2Du3fvwU033crMmbdw0003YLH86EN5uvIGDhyE\nx1PH9OnX8eKLzzcqU6fT8dxzL/DPf/6dadOu5fXXX+PZZ184yaH652IwGLjzzrt5++03UVWV3/3u\nIR555EFuvPEGioqOEh5++kTWUD/KMnv2s7zwwl+YNu1atm3bQlxcXJu2ozlcfPEl5OTsYcaMqSxY\n8O5ZnSupbWSJ+Hw++vbtS05OTqvWY7c7UZTTN6msuJZlC7aT2S+B4WObjlgcdLup+PA9ar76Ek14\nONbLxrMi3s62/B9I+/4S+gxMYujIbgSUAH/45mmyY7O4f/jNlJXV8unhL8ix7+euvrdg1plOq8UX\n9PPHb+fQI7Ibt/eewfLcz1iVv5Znhj2O21/HnE0vMSV1PCM7X3Taco7TEfL5dBQdHUFDR9HRETR0\nFB0dQUNH0XG63Ft79uwlIaFjrew8cUXU+cJdd93BtGkzGvx+BGemqCifzMyml/q3qCPz+vWnDhZ0\n3MO7rXG4fWhkCZPxR2s0Jj6MjKx49u4opveARCKsJ3+BNCYTsTNuJmzIMCo++pDyfy9m5OMPY/F3\noVT20ndgfR4xrawl3dqdvfZ9qKrKirzP+Sx/DQAL9i7lzj43N8rj9VO2le3EFXBzUdIQAIbED+Tz\n/DVsKN6C0+9CI2kYFJfdkl0iEAgEv0isVhszZ97Cffc9yKBBg9tbTqvi9/u59dYZ1NbWotc3nTtS\ncPa0qNHzxBNNR7U8Tnx8/Gn3tzRuT4An5m1kSGYc149uPKIz8MIUDuwpY8OXh7j0qsxTlAAhqd3p\n9PtH8RYWEoyIpuzARlIi/OicdrAkANDTlsaO8t38feN8vsnfxND4QSRa4nl//39Ylb+WS1NGnbL8\nrwvXE2uKoXtEfcqJaJONHpGpfFe0CU/QS5/oTEL1zZ8qEwgEgl8rn366+swH/Uo4HpxQ0LK0qNGz\nZs2alizuZ2MyaslOi2bNtqNc3C+BeNuPxoPJrCdrcCc2f3OY4oKak5yLf4ohMZHv/pdLMKAQl7OK\n/K1LsE6YhHXcZWTa0gH4Jn8TQ+IHcn36VUhIHK45woq8VSSHdSLD2qNReZ6Al90Ve8mvLeCa7pMa\necgPix/Iu3uXADA0fmBLdYdAIBAIBOc1rZZwtKNw5fCubNxbyvtrc7nv6j6N9vUdmMSe7UV8tzaX\nq2b0Q5IkVFXF6wlg/Ekqi/ISB7u2HCWjbxyZQx+nbMm/sH/0IY7Nm4i7+VYGxGYRFRbBFUmXNkxn\nXZ8+haPOIt7ds5jetp6oqCiqQqm7jKPOYhRVIVxvYXB8/8a6onth0oZg0BhIP02WdIFAIBAIBM3n\nV2/0hJn1TBiawvtf5rLnUCWZXawN+3R6DYOGp/Dlf/ezfUMBPm+AvH0V1FTVMXRUtwa/HUVR+fK/\n+wkx6RlySVe0Rh0Jv/ktzu3bKF20gKMvvcBNz79IbKeYRo6JBo2eO3rfyLu7F7Gv6iCyJCEhEWmM\nYGzyJXQLT6FreDJGbeMAVzqNjpszb0Ana0/rDyQQCAQCgaD5/OqNHoDRAzrx5Y5C9uY3NnoA0nrH\nsWtLIRu/OoQsSyQmR2AJN/Ld/3KRJYneAxLZufkoFaVOxk7uieEEh+jQfv0JSUvHeyQf2VgfOFBV\nFCT5R0Ml1hTNo4MeOGvNmba0c2ytQCAQCASCpjgvjB6dVuapmwc2WsF1HFmWGHdlT8qKHXTuasUY\noiMYVFj98V7WfXEQl8vLrs2FdOluo2vaycnSNCYTpvQMAAo/+g9lG7eScN/9yCdEpxQIBAKBQND+\ndLi5k7lz5zJy5EjS0tLYv39/i5V73OA5Uupg6f8OoJwQnijCaqJHZmyDH49GIzNmck9SUm1sX1+A\nRiNx4djuZwzHrQsPx52zh+I3X0c9i1DiAoFAIDgzF1zQn2nTrmXTpo2sXv05t91240nHzJv3OrNn\nP9XE2T+yYsVyHnvs9wDk5OzlySebXnlcVFTEuHEjz6hr//59fPHFqkbbZsyYisfjOeO5p8Lv9zNj\nxlQuuWQY69Z93aS2YcMGMmPG1Ia/zz77FIDf/e5ejh4tOOe6AZYuXdQQmbqtuOuuO5psa0vS4Yye\nUaNGsWjRIhITE1ul/O/z7KzaXMCiVftPGyFao5EZe2VPeg9IZOQV6YQ2kUfrp8SMvJjoqdNwbd9G\nybtvobRTbCKBQCD4tTJv3nwGDRrMiBGXUFBQwOHDhxr2qarKp5+uYMKESacpoTEZGT2ZNWvOz9K0\nf/++hsSkx1m4cGmzEpKeiuNL1tPTmw6yBz/m3jr+d+mllwPw8suvkpT086L4L126mKqqtjV62oIO\nN701YEDrZvG9/IJk3N4A/91wBL1O5tpLUk85gqPRyFw4OvWsyo8cPQbFU4f942X4Skro/PifGvn4\nCAQCwS+RnF0l5OwobpWyM7LiyegTd1bn6PV6xo69lBUrlnPPPfWZ1bdu3YxWqyUrqz+BQICHHrqP\nmpoavF4vPXtm8uijfzwpbcLWrVt49dWXmT9/EQAffPBvli5dhMkUyrBhFzYcd6ry3G4X8+a9jsvl\nZMaMqWRl9eehhx5pFD167949vPTS89TV1RESEsKDDz5Cz56ZFBUVccst05k8+SrWr/8Wj8fD448/\nSVbWmVNmnI7Jk6/gxRf/Srduqdx11x307NmT77//noqKckaNGsNvf3sfABUV5bz44vOUlpbg9XoZ\nM2YcN998G++++xYVFeU8/vgj6PV6Zs16loUL/4+MjAyuuWYqALNmPdXwedaspzAY9Bw5kk9paSm9\ne/fhySdnIUkSLpeTV155idzcA3i9XrKzB3L//Q+i0Wg4dCiP2bP/TF2dm27dUvH5vD+r3c3hvHsb\nS5LE1SO6Map/Ep9vKuDTDfktXodt/EQS7n2A8OEXCYNHIBAIWokJEybx2WcrGzJ4r1ixnPHjJwL1\nSThnzXqW+fMXsXjx+yiKwief/Oe05R04sJ/589/mjTfeZcGCxdTU1DTsO1V54eER3HHHbxg4cDAL\nFy7loYceaVSm3+/nscd+z5133s2iRe8xc+bdPPbY7xuyFNTUVNO7dx8WLFjCrbfewWuv/a3Z7Xc6\nHY2mt2pqqps8rqSkhNdff4sFCxazfPnHHDlyBICnn36Sa6+dyjvvLGT+/EWsX/8tGzdu4JZbbicq\nKppnn32ehQuX0qVL1zNqyc3N5aWXXmXJkg/44YccNm3aCMArr7xE//79eeedhSxcuJSqqsqG6/Dn\nP/+xIRP81Kk3kJOzt9ltP1c63EjPz6WpHDJNcd/1/QkCh0qdWG2haJrIiH4uREfXJ6iLHj28YZt9\nw0ZceYfofMPUFqmjuRram46goyNogI6hoyNogI6hoyNogI6ho7nPzIw+cWc9GtPapKWlY7Xa2LDh\nO7Ky+vH111/x29/Wj/ooisKiRQtZv/5bFEWhtrb2jNNN27ZtZejQ4dhsNgAmT76qYdrqXMoDyM8/\njE6nZeDA+rQZgwYNRqfTkp9/GJPJjMlkasir1atXb/72t5eb3f7j01tnYtSoMciyTGiohZSUFAoL\nC4iOjmbbtq1UV1c1HOd2uzl8+BCDB1/QbA3HGTHiYgyGejeQtLR0CguPArBu3Vfs3bubxYv/BYDH\n4yEmJgaXy0leXi6XXXYFAL169aFbt7ObWTkXfnVGT3MSjh7nhlH1HVxpdxJUFDQ/c1TmVAkESzds\noWbtGjyqhsixl/6sOs5VQ1vTEXR0BA0dRUdH0NBRdHQEDR1Fx+kSjv5SGD9+EitWLKe8vJysrH5E\nR0cDsGrVf9m5czuvv/42ZrOZ+fPfbhjhOBdaurzj6E5Y6SvLmoZRq5ZErz+5DkVRkCR4992FaLVn\nzpSu0WgavVt/OhV1ch0BoN7P6vnnXyIxManR8S6X85za8nM5r+detBoZrUamzhvgmf/bytptR1ul\nnpjrpxOaPYDy95ZS+923rVKHQCAQnI+MG3cZGzdu4P33lzJ+/I8OzA6Hk4iISMxmM06ng1WrPjtj\nWf37Z7N+/bqGVUvLl3/crPLqtzX9Ek9OTsHvD7B162YAtmzZRCAQIDk55Vya22KYzWaysvqxYMH8\nhm2lpSXY7RUN+09sU6dOncjJ2QPU+wJt3bqlWfUMHz6CBQvebTDmqqurKCoqxGwOpVu3VD7//L8A\n7Nmzm9zcgy3RtNPS4UZ6nnnmGVatWkVFRQW33HILERERrFy5slXrlCQID9WzcNV+yms8XH1xN+Qz\nLE8/q/Jlmbjb76TI7aZk/tvIJhOhP9NRTSAQCAQQHh7OkCFD2bp1MxdddFHD9ssvv4Kvv/6S6667\nisjISPr27YfXe3pH2e7de3DTTbcyc+YtmM1mhg790ZH5dOUNHDiIxYsXMn36dfTrl93Ir0en0/Hc\ncy80cmR+9tkXTnKobg+efnoOr7zyItOmXQuAyWTiiSeewmaL4tprr+eZZ/6M0Whk1qxnmTTpSh57\n7BGmTp1C587JZGb2alYdDzzwMH//+1+ZMWMqkiSh0+l44IGHSUhI5KmnZjN79p9ZuHA+3bqlkpFx\n6pVqLYWknm7d9i+Qs5neOpGgorB49QHWbi9kYHoMt1yejlF/djbhmYarFU8dBf/veUzpGURffe1Z\na2wJDW1FR9DRETR0FB0dQUNH0dERNHQUHaeb3tqzZy8JCcntoOrUnLgi6nzhrrvuYNq0GQ1+P4Iz\nU1SUT2Zm0wZFJjGpAAAgAElEQVTUeT29dSIaWWb62B5cc0k3tvxQxvz//tDidcjGEDr9/lGiplwD\ngPIzAlcJBALB+YbVamPmzFsaVgb9mjkenLCoqBC9/sxx4gTNo8NNb7UnkiRx2eBkuiWEE2aud8py\nuH1oZBmTsWW6Sj7m3e4vL+fI3DlETbyS8ItGtEjZAoFA8Gvm009Xn/mgXwnHgxMKWhYx0tMEPTpF\nEGetHz7916r9zFm4hfLquhatQxMejiGpE6UL3sWxZXOLli0QCAQ/F0kCVVXaW4ZAcFacyWNHGD1n\nYGT/RGpdPuYs2EJeUW2LlSvr9STcfS/GbqmUvP0mdW3gtS4QCATNxWw2U1lZQSDgP+OLRCDoCKiq\nisNRQ0jIqeMniemtM5DWOZLHZ2Tz8ns7eX7xNm4f35MB6TEtUras15N4z/0cee4Zil79K52e+BP6\n6JYpWyAQCH4OnTp1oqKigsrKUgIBkUBZ8MsgJMRIp06nzjsmjJ5mEG8z88cbB/C3D3ex5H8H6Jsa\nhU7bMoNkGouFxPt/h/0/H6MJbf/orAKBQAAgyzIxMTHExIgfYoJfD8LoaSZhZj1/uKE/pVVudFoZ\nf0Dh42/yGDuwE+GhP8+zXh8bR/zM3wAQdLlQPHXobFEtIVsgEAgEAsExhE/PWaDTyiRF18ezyCuq\nYdXmAh59cwOfbsjHH2gZh7+Sd+Zx5Nln8Bb8/PDmAoFAIBAIfkQYPedIWudInrl9MBmdI/ngy1z+\n+NYGvtlR+LMd/qKuugZJlil4/jncP+S0kFqBQCAQCATC6PkZxFpN3Hd1Hx66Lgu9VsNHX/78FViG\nxEQ6PfYE2kgrha+8iHPXjhZQKhAIBAKBQBg9LUBmFytP3zqIP946GEmScLjrl7h/t7uYQPDsp710\nVhudHnkMfWIS5UuXoAYCraBaIBAIBILzC+HI3ELIsoQ1zEh5uZ/KWi8eX5C3VuSw7Os8xg7oxODM\nOMKPRXluDprQUJIeegTF5ULSalEVBUkWNqpAIBAIBOeKeIu2AslxFp6+bRD3X92HqDAjS9cc5KG/\nf4uzzg+Ay9O8YF8akwlddDQApQvnU7pwvsjXJRAIBALBOSJGeloJWZLomxpF39QoCsqc5BXVEBqi\nA+CN5Xs4WuakV1cbvbpYSYmzEB0RgiRJTZalqioak4mqVZ/j3ruXuNvuICS1e1s2RyAQCASCXzzC\n6GkDOsWE0ikmtOHz0Mw4tukr2LqvnHW7igHISo3ivqv7APD5piOYjToiQvWYQ3SYjVosE6/G3Lcf\nJe/Mo2Dusxi7diP2xpsxJCY1jP5IBpGJVyAQCASCUyGMnnbggsw4LsiMIxBUKChzcqTUQWhIvb9P\nUFH44Mtcgkrj6a+LsxK48dJ0Ov1pFu8s/ApjVRmJh+sId5Th2LoF47rPicbNQVsM2+P7EZ8cR+f+\nvYiPtqDXaRqV5fYEqHJ4iLWa0GrEDKdAIBAIzg86nNFz6NAhHn30Uaqrq4mIiGDu3LmkpKS0t6xW\nQauR6RIfRpf4sIZtGlnmHw+OoLLWQ63bh6sugMvjJyHKDIBT0ZDjC8UhGVC/KQAKACOXDb6c7uYq\niuxuPqu2wm4f7N6GJIFNE2Bs3R66S7V8H4xkmaFXQ/3JsaEk+OyMjnBhTU+lwBjDngIHdW4vXmQC\nQRVDdTmX9wwjIimevDodB4ud1Lh8AMTbTMTqgyRrXBji4tla5GV/QTUanQYlqGDQabCFGbmkXyKy\n3Hj67rhf06mm9VoLRVFRUdEIx3CBQCA4r5DUDpY+98Ybb2TKlClMmjSJ//znP3z44YcsWLCg2efb\n7U4UpX2aFB1tobzc0SZ1BRUFZ12AWpcPCbCFGwkxaImOtnA4v4KivCKqNCaKKlwc3pPHEG8eiUot\nVbpQditWrGFGqjtnkFdUy6GjVdx+ZDkRvlrWR/bma2tfDJKKKcyEVpaoqnRw76H30asB1tr6szGy\nF0ZZBa0Wjy+IVgnwYN4SZFSWJ43kkDEOk9mIV5Hw+gLEqC7u1OxF8fnID5opNkZRGN2NvKoALk+A\n8dEuLjQ7KPNIvFpan+dH1mow6rUYJIWRyiF6h3hwW6xs8UViCDFQaYigpMZHqd3JDck+utSVsrfY\nxapgEiEGLZbYaKy2UIxBPxd2MWML1bGrwMmaH6o4XFFHQIE4m4m4ELimC4ToNXxd4GdXeQBJAlNE\nOHqdjMHr4upUDZpQC0dccKDYSYUzQBUGdBqZWBNcOSAeSa/jy50lFFfW4Q0qmMIshBi0xESGMHlk\nD8rLHXyzqwiPL0iEWU9kmBGTQYtrz/ckRIWiDQtn3vpy/KqEJMuYjTrSOkeQmWLFFv5jxmBFVSmr\nquNwcS15hdVkRMr0TDBT6wmw66iL8IhQwiIthIcakGWJMJMeg16DwWTg+32lDeWEGLRYTDosJh0a\nWabY7uL7XDsVNR40GgmtRkankRneN4FIiwFVUVD9fgJBhQpXgIpaL4GgSv8e9Y72+SUOvP4gkRYD\nFpMOvU6DfIIxW+XwUlnrwRRqoLLKjaqqhOi1dEsMB+oN4DpvgBqXj1qXD71Ogy3cSJip8YpHVVUJ\nBFX8AQVJqm8HQGWtB41GxmLSNar3+DmVtV70Ohm9VkNMjIWComq0sozJqEVVVVSVk4zyQG0tGosF\nn1/haIWTEL2W8FA9JoO2kaGuqiq1Lh9l1XXYazwY9BoiQg3E20wY9af+XXn8eaEo9e0JKirKscex\nQadpsfx+tW4fBq0Gg15z0r7oaAt2uxObLbSJMwWCXx8dyuix2+2MGzeOjRs3otFoCAaDDB48mFWr\nVmG1WptZxvlh9LSkBkVVUT0evIcP4T5wAMXjwZSWRmjfLFRVJehw4C8vw19WiqOknKC9nPCMDMKG\nDqOyto4jX2+gR5dofKUlOA8fIVBwmK43XIfaoxfegiPkvzsfo07GqwvhhUBfAshEmzWkdYsh1F1F\n1Fcf0xkHTm0IG0NSUVUFc/ZAguZwnGUVZOZ9R7KnlDyPnsXxYwAINWhIiLEQWVdJn63Lidb4KInt\nzjopCY9fgeSueAIq9koXNx5ZQayvij2hXdgU0ZNEXwWxY8dQVO6i6HAxt+5bigRsiMjkoDkJZA1y\n5xR8foVAVSW3H3gPgI/iRrAvNBmj4icmPpJgUEVXUczUgx8DsDhhDKUGK3oZAkYTHm+QTrKL29wb\nUHUG/qHJpoyQRn3fzV3INUX/A2BR4lj8khbZEoZTZ6bW7Se7dh9j7Ftwmq3Mix6NDy3qsReuTiMx\nuWA13dxFHA6JY2ni2JOu7dTaTXSrK2S/IZ73Iy44af+Nnq2kBCrZpo1nhbEXejUAWi1BVSKoqDwU\n2EBITQVblSg2hadTpbOgSPUvT70Gfl+6HI05lA8MvdkrRzcqO8oIdx75GEmrY2H4EPI1jb/DCSa4\n9ehKVL+P/4sYzlFD43xzqRaF64u/QPF4+GfESGo0IQSlH1/c/WxwlWcXaiDAHG9fvGjQoGCNMBFm\n1pPmPEL/vHUEtDr+Ej7upLaP6ywzylSJw6sw51AkFk0QgwyyJQxVVRlWspm0sj3Y41J5U5fdcJ5W\nI2HQabjCs4fu5fvYL9t43zbspPJvT6ghVecmpxpWOSIwqX6CZguBkFA83gAz3JsJ9zlYr8bxuabb\nSeffL+0gUnGzMWBjg5yEVgmgtVjAHIrfH+Qu6XtCTHq+qrOxwWlGK6kYTUaMoSFIfj8zataDz8Oy\nYBd2yrHoCRJu1mMwGTEQ5GbPFtLuvA23PlQYPYLzhg5l9OzevZs//OEPrFy5smHb5ZdfzgsvvEBm\nZmY7KhO0BP5AkL2HKkmKCcUWXv/yP9UUl6qqJ29TFNyVNbgqq4hKSULW6/FVV4OqoouIaDhe8fuR\ndfUr5Wpzc/EUl/zk179E1LAhAHjLywnW1aEGFZRAADUYRNbrCe3apX6/vRJ/dTUBp5PicgehZgMR\ntnAsPepXz9V8vxtPaSmK34+k1aEx6NFbrYT3ykRRVHIXLiZYWkzQXUedy0tAb0Dq2RdN/8HUurzE\naX0k6f34qqoJOB0EXG7MKclE9MviSHENZStWEG1Qcbk8rCjWo/N76dQ9iQGXDaNTlImqjZuQJAgo\nKtXuAFWVToLxnfBZrPgcDqw7vsZmlHAENRx1A4EAkQOyITaRiqOlJOxci0Wr4kFDABmLJkjCxAlY\nuqdSvf8ghYuXoI+M5JtADLl1euJN0KNfdzp3iUctLkS75WsCDgcVfg12RU+NoseQPYig0YxaVcEF\nVbtR/X72OzR4/UH0kkrS5IkYwsPwHNhHaM4WZL2etTVmgnUeTHW1ZN50HT5VxrN3D3GHd6AJMfF5\njQW/KqOTIfaCgei0Moa924jfvwlZp2OHFINX1uPQmQn26EW1w0tPTQ2DPIfw+wNsqTXi8QcJ6gzY\nhg/HZNSi/99yzNu+oU5vYrM1E6c2FDXUQkSvTCQJ+hpdpFbnUnO0mJxyPz40BBJT0PTuj9cfJC1/\nM0laL06Nka1lQcLdVXTq0Rnr2HFUO7y4nvsTIYqHI5ZObDClUifrCY2wEJEUh1GvYdDez7FqgxRh\n4oA/FEkJEtatK5b0HrgddaSu+wCjViJHjuL7YCRBWYvOZiMkIgzZ72XYlg/QeVzsVm0c0EUTlDXo\nu6ZCuBXF62Xykc8xGvUUasLJ9YZQG5AIJqciWcLQeesYk7OctIcewNS5c4t+zwWCjsyvzuipqnK1\n20iPzRaK3e5sl7o7koaOoqMjaOgoOjqCho6i40QNTRnX7aHj53Ku7bDZQqmqchEZaW4RHQJBR6dD\nOTLHx8dTWlpKMBhsmN4qKysjPj6+2WW095e3IwwTdwQN0DF0dAQN0DF0dAQN0DF0dAQN0DF0tPcz\nUyBoSzrU8hWbzUZGRgYrVqwAYMWKFWRkZDTbn0cgEAgEAoHgVHSo6S2A3NxcHn30UWprawkLC2Pu\n3Ll07dq1vWUJBAKBQCD4hdPhjB6BQCAQCASC1qBDTW8JBAKBQCAQtBbC6BEIBAKBQHBeIIwegUAg\nEAgE5wXC6BEIBAKBQHBeIIwegUAgEAgE5wXC6BEIBAKBQHBeIIyec2Du3LmMHDmStLQ09u/f37D9\n0KFDXHfddYwbN47rrruOw4cPt5qGqqoq7rjjDsaNG8eECRO45557qKysBGDHjh1MnDiRcePGceut\nt2K321tNB8Ddd9/NxIkTmTx5MjfccAM5OTlA2/bHcf7+9783ui5t3RcjR47k0ksvZdKkSUyaNIlv\nvvmmzXV4vV6eeuopxo4dy4QJE/jTn/4EtO31OHr0aEMfTJo0iZEjRzJo0KA217F27VomT57MpEmT\nmDhxIqtWrWpzDV9++SVXXnklEyZMYPr06RQUFLSJhnN5TrXHd1YgaFNUwVmzefNmtaioSL3kkkvU\nffv2NWyfMWOG+vHHH6uqqqoff/yxOmPGjFbTUFVVpW7YsKHh81/+8hf1scceU4PBoDp69Gh18+bN\nqqqq6muvvaY++uijraZDVVW1tra24f/Vq1erkydPVlW1bftDVVV19+7d6m233dZwXdqjL356T6iq\n2uY6Zs+erc6ZM0dVFEVVVVUtLy9XVbXtr8eJPPPMM+rTTz/dpjoURVEHDBjQcD1ycnLUrKwsNRgM\ntpmG6upqddCgQWpeXl5DXbfeequqqq3fD+fynGrPe0QgaAuE0fMzOPFhUlFRoWZnZ6uBQEBVVVUN\nBAJqdna2arfb20TLZ599pt50003qzp071SuuuKJhu91uV7OystpEg6qq6kcffaReeeWVbd4fXq9X\nvfbaa9WCgoKG69IefdGU0dOWOpxOp5qdna06nc5G29vz/vR6vergwYPV3bt3t6kORVHUQYMGqVu2\nbFFVVVU3bdqkjh07tk017Ny5U7388ssbPldVVak9evRoUw3NfU619zNMIGgLOlTC0V8yxcXFxMbG\notFoANBoNMTExFBcXNzqucMURWHJkiWMHDmS4uJiEhISGvZZrVYURaG6upqIiIhW0/DEE0/w7bff\noqoqb731Vpv3x1//+lcmTpxIUlJSw7b26ouHH34YVVXJzs7mwQcfbFMdBQUFRERE8Pe//52NGzdi\nNpu5//77MRqN7XZ/rlmzhtjYWDIzM9m9e3eb6ZAkiVdeeYW7774bk8mEy+XizTffbNN7s0uXLlRU\nVLBr1y769OnDJ598ArTf8+J09aqq2m73iEDQVgifnl8Bs2fPxmQyMX369HbTMGfOHL788kt+97vf\n8fzzz7dp3du3b2f37t3ccMMNbVpvUyxatIjly5fz4Ycfoqoqs2bNatP6g8EgBQUF9OzZk2XLlvHw\nww9z77334na721THiXz44YdMmTKlzesNBAK88cYb/OMf/2Dt2rX885//5IEHHmjTvrBYLLz88ss8\n99xzXHXVVdjtdsLCwtr1eggE5zPC6Gkh4uPjKS0tJRgMAvUvn7KyMuLj41u13rlz55Kfn88rr7yC\nLMvEx8dTVFTUsL+yshJZllt1ZONEJk+ezMaNG4mLi2uz/ti8eTO5ubmMGjWKkSNHUlJSwm233UZ+\nfn6b98Xx9un1em644Qa2bdvWptckPj4erVbL+PHjAejbty+RkZEYjcZ2uT9LS0vZvHkzEyZMaNDX\nVjpycnIoKysjOzsbgOzsbEJCQjAYDG3aF0OHDmXJkiUsW7aM6dOn4/F4SExMbJfrcbr+b69nmEDQ\nlgijp4Ww2WxkZGSwYsUKAFasWEFGRkarDgu/9NJL7N69m9deew29Xg9Ar1698Hg8bNmyBYClS5dy\n6aWXtpoGl8tFcXFxw+c1a9YQHh7epv0xc+ZM1q1bx5o1a1izZg1xcXG8/fbb3H777W3aF263G4fD\nAYCqqnz66adkZGS06TWxWq0MHjyYb7/9FqhfjWO320lJSWnz+xPgo48+YsSIEURGRgJt+z2Ji4uj\npKSEvLw8AHJzc7Hb7SQnJ7dpX5SXlwP109AvvfQSU6dOJTExsV2ux+n6vz2eYQJBWyOyrJ8Dzzzz\nDKtWraKiooLIyEgiIiJYuXIlubm5PProo9TW1hIWFsbcuXPp2rVrq2g4cOAA48ePJyUlBaPRCEBS\nUhKvvfYa27Zt46mnnsLr9ZKYmMgLL7xAVFRUq+ioqKjg7rvvpq6uDlmWCQ8P5w9/+AOZmZlt2h8n\nMnLkSF5//XV69OjRpn1RUFDAvffeSzAYRFEUunXrxh//+EdiYmLaXMfjjz9OdXU1Wq2WBx54gBEj\nRrTL9Rg3bhxPPPEEF110UcO2ttSxfPly5s2bhyRJANx3332MHj26TTU88cQTbNu2Db/fz7Bhw3j8\n8ccxGAytruFcnlPt9Z0VCNoKYfQIBAKBQCA4LxDTWwKBQCAQCM4L2mTJ+ty5c/n8888pLCzkk08+\noUePHicdEwwGeeaZZ/jmm2+QJImZM2dyzTXXnHVdVVUuFKV9Bq9stlDsdme71N2RNHQUHR1BQ0fR\n0RE0dBQdHUFDR9Fhs4VSVeUiMtLcrjoEgraiTYyeUaNGceONNzJt2rRTHvPJJ59w5MgRVq1aRXV1\nNZMnT2bIkCGN4q40B0VR283oOV5/e9MRNEDH0NERNEDH0NERNEDH0NERNEDH0NERNAgEbUWbTG8N\nGDDgjMseP/30U6655hpkWcZqtTJ69Gg+++yztpAnEAgEAoHgPKDDRGT+adTa+Ph4SkpK2lGRQPDz\nUQOBhv/rcg+ieL0Yu3RFExJy+vMUBRQFRVGRdVokScJvt6N46pCNIWgjI5Hkpn+zKH4/gcpKQEUb\nHoF8bHWfqigodXUE3S5QQZIlZJMZjclUv8/rBVVFcbkIOh0EXS708fHobPWrzArKnNjCjJiM9Y8N\nxVMHsgZZryfocuE9WoCs1yMZDKiGEII6A7JBT4hBh+L3EXS68OLFX+kEJCSNjMYciqTVoioKwdpa\nAtXVBGqqQZLQhIRg6JyMbDAQqHMTdNevEFS9HhSPB6WuDmNqd2SdDl9ZGQF7BapOhzOopTYA4SYt\nUZ0TkDUa/HY7gcpKFJ8X2SBRXVSO6vEQObY+dIA7Zy/OklKCoZFI0TGooWFotBqiw41IkkTtpg0o\nHg+SRouk1SIbDGitVoydkwHwFhTgr6okWFuD4vGi+n3ooqKxDByEqqrs27ATp18i1KAhzChjMWow\ne+PBEFZ/bxw4gBoMoAaDoCgggSbSijExCUVRKTx4BFVnQJJAU+dE46rFEm3DlJxMwO2m8LPV+JDx\nSlp8khZZp6NzRhesXZNRvF48ebl43W4cDi/+oAqyRHT3LkRHp53zvS0Q/BLpMEZPS2GzhbZr/dHR\nlnatv600qMEgajCIfCw+0IkE3G68djthx97JGqMRrflHn4Hc/AoiA07UijLUQABz1y6ExMcT9Hhw\nHsxF8XpRfH5kgx5NSAiqLRpdWBgaVy35X61n21E3STovcbIXgkFix4zC0j2VoNdL5b6DlB0qRC0r\nIb/CTolHJjjkYkZdlIH3+50UfbISWSMjGwwoZgtH5HC86VlMGt2T6h07qdy4GUmWqZN1VCp6qr0q\niaMupn/PeMq+/JoN6/ZgNuqIs2jxu5zU1Ljpce/dxEeFkr9wEVs2/IAqa/B5/Xg8PoKGEIY/8QDp\nKVZy3lrN6pxqanWhuMOj8ZnC6Ww1MG7KcNKSrez5y/+j7HARtY46yjFx0JTI4fBkZt8/ku6dInnn\niRdYFZKBzVdLgs9OZ4OPLtmZDLv1aiRJYsHDL7LHE4I7KOGR9ahIhEaG8eLTV+F3OFj6yPPkhSSg\nIOGTdXhlPfqIMN545kq8FXY+eGQWXlmHrCpIqAQlGduo0Uy6bTyu/CM8N28DHo2B8KCLaH8NFq+D\nYWMHMO7akeQfPMAr/9pIuT4Cp9ZEUKpPZTClTxg333QJef9bx2vvbUOjKgQkDUFJxifruGFyf4aP\nHcDG9z7j1XUVRPidhPudeDV67LowHrrFQq/sHnz05hre2QeyqiCrCppjGuf8JoEe6Z34+L3/8uE+\nH7VaE8qxugH++WAcSXERzH/3E7464scn61AkCWPQh1Hx8vJVBowhep57dT/fucKAymN/oEPhgxcm\nI8sSH3/9A7tqtTg1JhxaE15ZR4xUyNsv9gLg6dlrcHkC2Hw1+GUttVozCVEOHrh8FAC/XVuCR278\nXRls/oE/zqqPoj7z6R34qe+XoCTj0egZbsvngT9m4HJ5eGpZ7knfsyviSvnN73tRfMTN0/t/+p1X\nuNqZy02De7FnWwlPLTuCV9O4/tvr8ujWL63dn5kCQVvSYYye41Fr+/TpA5w88tNc7HZnu81RR0db\nKC93tEvdLaWh8r+fUpd7AGOXroR07YY+Pp6A201tSCQWk57af/8Lx759+CoqkFQFV1IPnN164e3e\nm4HdIqn961+wl1VRqzFhCnowBb14h4/jaJf+TByYQN4Tf+DVyNG4ZSPJdcWEBusIJJWSPSSDobES\nh558ivcSRqFRFZyaEGp0oXg0BqaM6MrImAAHF73HopQpgAGdEiA2UEOg9AemjdeRXHWYr9/9kA8S\nRgHhx/6A1YeJjIokqcrJOnc424lGWxOkUA5DkWRs9iMMy+pM1d6DLNzp4gdjYqMXVLeVe+gUHUrV\n0RKW2q04ZCMUA9QH3Bv80S7unNwbnymMRcb++JFBDxx7D/m3FGAz67BMnMLqit1oJZUw1Yu+xs2a\nOh2R3xdjNekolMN51dwdjtmHJo1KRpQWZ62H8nIHnS+5kF6FfoocJjY7Y9mABHvhlSOVhJn01OpD\nKZOtmPQaIo31o0MBrY7KShdRVhNSem+8biOyRsasAZsGwiLr75dgXZAtaaM46Gw8ehRzVMuQslp8\n7gDXJ3ko9vgo8moo80dTbI4nyqmjf7kDTUwcdfFd6GaWidSp6NQA2oCPXt1jKS93UGm0Uh2djCRJ\nyKhoJTDI4NWG1N+vMQl0jfNQFYjgkFclRCcTZ5Jxo6e83EF8amfGuQsIKDKKrEeRNcf+dJSXOzD1\nSKObr4RIg0SkTsGiUajxqmiA8nIHYd1TSZIrMeo1hJiNVLuD1AWh1uHB4fTSb+QgoourMQY84KiG\n6iq0QR8VFQ4kSaK0az+oriMpREtkiAajrGAxRjV813SpaZSVe8lxB9FqJKwWA/FdbQ37Z45ORuv3\n4vQq1HqDODwKaaldKC93oCgq0ckJaGQJjUZGI0uY9DJpKZGUlzsI+PxMz9Ah+32oqorfYCagN9K1\na1z9flnPDaNTMWg1GLVglMHv9REfG0Z5uQOtOZQLukcSFqonLNSIXqcBRaFr12ig/pkpDB/B+UKb\nxuk5MWjcT1m2bBkrV65k3rx5DY7MixYtolOnTmdVhzB6LJSV1XLwiJ192/ZRXOGk3BmkUhdGn9Qo\nrummwbFtC7s8ZoqrvRS5oUwx4gkJ4+pLUsmy7+Hgl+t5J2QwpqAHnRqgSh+GX9Jy9+RedNr6OTn2\nIAvdnZAABamh7j/fMhD9yqVs1iTyUbGxkS6zUcvT03rjWbGMI4Zo9inh5FRLeIMqYWY9Q/skMqZv\nLDUHDvLa+mr8iorFIGM1QExsJJkZiaREm/C7Xdh9Wg6XOMgrquVImYPQEB2XDu5Mss5LyeFCcv1m\n/PoQQi1GLAYNiVFmIi0GJEliU04pX2w5SlBRSe8cQXpyJMmxoYSZDQD8e80BfAGF6PAQoiNCsFr0\nRFiMRFrq9xfbXZTY3ZRUutHIEmFmPXE2Eylx9dMU+wuqUVUVnVaDXiuj08l0TozE6/aiqioOtx+L\nSdcQLM8fCBIIqoQYtLg9fjbllGEO0WENM9AlLgxZlmgKf0ChoMxJlcNLry5WDHpNk8edeF+c6d50\newKUV9ehnPBISIwy178kT4Gqqg1taQ4d5TvSWhr8geD/Z++8w6Oq0gb+mz6Z1Mmk995IpYTeVUBA\nQJYFQVBEUCyoq599ZRd1rWsv2FBEQASkLiJFEamhhpAACaT3XiaZPvP9kSUYE0KAANG9v+fJ82Tu\nufec9/7tDH4AACAASURBVNwp573veQtSibhT96S73AtB6RH4X+K6KD0Xyww6d+5cFixYQFxcHBaL\nhUWLFrWkz587dy5Tp0697LH+rEqPyWyhsk6Pt6bZDLB4w0nyy7SYLVZEIlDIJAR7O/HIuDCyVqzh\nxXPONEmUKCwGNBYtvuEB9OsVQkhpBjmr1/Ke70RENhsakR4vpQWPsCD6xPsTGaCmul7PD3uzqa2s\noUmrx1NtR1B0ELHBrrg6KSmpaiTlVDlmixVvjQofN3vcXeywk0sRi0XUNBioN1goLKmjvsmIUi6l\nfw9PlPLra1jsDotKd5GjO8jQXeToDjJ0FzkEpUfgf40/XUbmP7LSU17ThIdaBcCOwwWkZVej1Zlo\n1Jmo0RqwV0r594MDsRn0fLvpOOVltdBQh9VgxObuRUBMCPcM8ebog49QFTuAgAG98YyJQKJsbXUx\nma2UVmnxcLVH0cFT/NXQXX7Qb7QM3UWO7iBDd5GjO8jQXeQQlB6B/zW6jU/P/ypanYn96aXsOVFC\nYbmWxU8MQyYVo9WZqG804qCS4e6iJMnRjUh/NeaGBvKee4reOh1IJCgDg5AH+eKQoMEhKQyluyMh\nb71LuOriycZkUjH+nk7XcZYCAgICAgI3HkHpuUGcK6pj9a5zZBfXYbbYCPRyZPrNEUCzlWri4BAm\nDm6/0J/nrNmIVSrswsIRKxRt2iUdKDwCAgICAgL/qwhKz3Wiqk7Pjyn5xIdqiA3RIJdJMJmt3NTL\nn/6xXvh7XNy8bLPZqNn6Awo/P+zj4nHsk3wdJRcQEBAQEPhzICg915jKOh0b9uRwIL0MALWjgtgQ\nDf4eDvz9rt6d6qN2+49Urv0Op0FDsI+Lv5biCggICAgI/GkRlJ5ryJYDeaz/NQeRCIYn+TIqOQCN\ns/LSF/4GfW4OFWtX49CzF56z7r42ggoICAgICPwPICg914DzuUtUCinJ0R7cPiQEV6fLU3agOc1/\nyaeLkTo743nXPRctOyAgICAgICBwaYRVtAtp0pv4dFM6v54oAWBYki/3jou5IoUHoP7gAUwV5Xjd\nex8Se8E5WUBAQEBA4GoQLD1dRE5JPZ99doCy6ib83Lsm54XzkGEog0NaihoKCAgIXC+sViuVlZVU\nV9dgsVhutDgCAp3Czk6Jv78/Mpms3XZB6ekCjmVW8PGGdNROCp6a3pMIf5er6k+fm4NIKkXh5y8o\nPAICAjeEgoICzGYrGo0nEon0ssqNCAjcCGw2Gw0NdRQUFBAS0n7KF2F76yopr2niw3Un8fdw4O1H\nh161wmOsKKfo3bcp/eIz/mTJsgUEBP5ANDY2ola7IZXKBIVH4A+BSCTC0dEZnU5/0XMES88Vcr4o\no4daxbzbYogP1eDsoKBCZ7ziPi1aLUXvvIXNasH7vvnCD42AgMANw2YDkUh4Lhb4Y3GpdVP4RF8m\n9U1GfjiQx7OfHCAjpxqA5OirL6ZpNZko/vA9zFWV+D70CHIv764QV0BAQEBAQOC/CJaey2DnkUJW\n/ZSF2WIjwt8FmbTrdMa6n39Cl5WJ97z52IVHdFm/AgICAgICAs0ISk8nySyoZeWOLGKC1UwdEY6v\nW9eGkLuMGIlUo8GxV+eyNAsICAj8r9GvX09CQ8N45JHHSU9P46efdgBQWFiAWu2K/X9Te7z55jt4\nenp1ut81a1ZhsViZOvWODs87fvwo//7362RlZfLLL/tR/K724YYN6/jgg3fw8vLGZDLh6+vHc88t\nxNXV9TJnenEKCvK57745bNmyvcv6/C1ms5lBg5Lbnd/veeSRB3jqqefx8fFh06b1JCb2xN8/4JrI\n1VUISk8nadKb8fd0YP6EWOwUXXfbLI2NIGouEiooPAICAgId89lnX6FSqUhO7svs2fcCMH/+XGbM\nmMmgQUPavcZsNiOVXvx3+y9/mdqpsRMTe/Lll98waNDF6x/27dufl156FavVyvPPP82SJZ/xxBNP\ndar/PxrvvvtRy/+bNm3Ezc39z6H0LF26lPHjx3eptvpHIzHcjfgwDeIudC622WyUff0lhvx8Ahe9\njPgieQUEBAQEugO5r/yrzTGn5GRcR96E1WAg/61/t2l3GTQYl8GDMTc0UPjB+23a1SNG4Ny3X5fL\nOn78aMaMGcvhwymEh0cwZ859LFz4HE1NjRgMBoYOHcb8+Q8DsHjxh1gsFh58cAEbNqzj5593olKp\nyMnJxsnJiVdeefOy1z+xWExiYk8OHTrYcuydd94kNfU4JpMJtdqV559fiKenV4v1ZuzY8Rw8uB+D\nwcBzzy0kPj4BgFWrVrJ69Urs7R3o339gq3E2b97AypXLAfD3D+Cpp55DrVb/dx47sLNTkZ19Dg8P\nTx577Anee+9tiooKiY2N44UXFl3S8Xf8+NHcdtsEDhzYT1VVFTNn3s3kyVNa2t5//2NSU4+TlXWG\nN998DZVKxaOPPk6vXn0u635dLzrllHLgwAFGjhzJfffdx5YtWzAarzxC6Y/GDwfy2HG4AJvN1qUK\nD0Ddr7+gPXIY5yHDBIVHQEBAoIvR6XQsWbKMZ575O05OTrz11nssXbqCZcu+JTX1OCkpB9u9LiMj\nnUcffYKVK9fg5+fP2rWrLntso9HI/v17uemmW1qO3X33HL788hu++WYVw4eP5KOPLiiB1dVV9OzZ\ni6+/XsmsWbP5+OPmtjNnTvPNN0v59NOvWLp0BTU11S3XZGVl8sknH/Peex+xfPl3BAQE8vbbb/5m\nHhn87W9PsmrV90gkYv7xj7/z8suvsnLlak6fPsXRo0c6ORcTX3zxNR9++AkffPAOen3rkPAJEyYR\nHh7JE088xbJl33ZbhQc6aen5+OOPqampYcuWLSxdupSFCxdyyy23MHHiRPr06b6TuxosViurfz7H\ntkMF9I3xZGSvru1fl32OihXfoIrpgXrU6K7tXEBAQOAaEPTMsxdtEysUHbZLHR07bL8W3HrruJb/\nrVYr7733FmlpJwCoqqokK+sMycl921yXmJiEh4cHALGxcRw/fqzTYx48uJ+ZM6dRXFxMSEgow4eP\nbGnbu3cPa9euRq/Xtdlyc3C4YMWJjY1j8eIPAThy5DCDBg1psTRNnDiZX3/dDcDhwykMGjQYjcat\npW3OnFmt5uHu7g5AREQUBoMee/vmigFhYeEUFhbQqxNuFTffPAoAX18/VCp7KirKu/021sXodPiR\nWq1mxowZrFq1imXLlpGWlsasWbMYMWIEH3/8MY2NjddSzuuKVmfirVWpbDtUwMhefswZG92lOXPM\ndXWUfPwBEhcXvOfNFwqJCggICFwD7OzsWv5fvnwpOp2OL7/8huXLv2PQoCEYDO3vWsjl8pb/xWLJ\nZZXh6Nu3P8uWfcuGDf/BYjHzxRefAFBUVMj777/NSy+9yooVq3nmmecxGg2/GfOC07BYLO6S0h+t\n5yG+4jF+388fuSzJZa22+/fv55lnnmHWrFm4ubnx2muv8frrr3Pq1Cnmzp17rWS8rugMZl5ceois\nwlpm3xrFjJsjkEq6VikRicUo/APweeBhJA5dU6dLQEBAQODiNDRocXNzQy6XU1ZWyp49u6/peA4O\njjz11HOsXr2K6upqtFotcrkCjUaD1Wpl3brvO9VPr1692bv3V2prawDYuHF9S1vv3sns2fMr1dXN\nW14bNnzfruXqemBvb49Wq70hY18Ondreeu211/jPf/6Do6MjEyZMYNOmTXh6era0JyQkkJx8cW/2\nPxJ2CikTBgXj6aoi1Mf5mowhcXTEd8Fj16RvAQGB7oHVasNktKBQCkGy3YFp06bz7LNPMn36FDw8\nPK+L30lkZBRDhw7nm2+WsmDBYwwePIQ77vgLzs7O9O8/kIyMtE71MWPGLO69927s7e1bOTKHh0dw\n333zefjh+wHw8/Pn6aefv2bz6YhJkybzwQfv8vXXX3ZrR2aRrRMFnhYtWsTEiROJj4+/6Dnnzp0j\nNDS0S4W7EqqqtFitl1+zqklvpqS68aoUHXd3RyoqGi7arjubRdXG9Xjdcy9SF/UVj3M1MlwvuoMc\n3UGG7iJHd5ChK+Sw2WxXvdX8WxlMRgtSmbjLS740NRrZ9O0JqisaUSilOKvtsHdUYLVYsVis2Gxw\n09hoVE7yS3d2DXF3d6SqSotG09binJ6egY9P9yp43K9fT376aQ8qleqGyXA5eWwEbgzFxXn06BHT\nblunHkHuu+8+lEplq2N1dXXo9foWi093UHiuFKPJwntrT5Bf1sDr8wfgYNf1kVQ2s5myZUux6poQ\nK+0ufcH/MCajhYO/5ODl50RIpDtisVCD7HJp1BqQSiXX3Mpgs9k4cagIsViEu5cDGk8HZDJJl49j\ntdo4vDePYwfycXFV4R+sxi9IjUwmoaFej7begEIpJTrBu1OfF7PZSsruHFJTClEopXj5OuHl50xY\ntAdOLspLXt8RjVoDG1eeQFuvp/egQHSNJupqdNRWNyGRiJFKxdTX6fn+m6NMuad3q/fIZLJQkF1D\nQIgaaQf30WS0UF7SgF5nQq8zYTSY8QlwwdPH6apk7+64umqYN282Cxb87YZs45xPTujqqhFqI/5B\n6dQv4gMPPMC//vUvnJ0vWEFKS0t5/vnnWb169TUT7npgs9n46ofTZBbUMu+2mGui8ADU7NiGsagQ\nnwcXIFZe3Y9qV2K12jh2IB83TwcCQzXXdWyTyYJEIm6zSB07WEDakSLSjhTh4JRNbC9fYhK8r/s2\ngclowWazIe/CZJQAVeVaTqWWUpxfy8jxUWg8us6vy2azcfpEKXu2n8XFVcXtdyUh+Y1PWk1VExtW\nHEfjbk9UvDfB4ZoOF9fzWK02RKK2xfxOpZaw76dzLa9FIggIcWXk+CgUys59l7QNBgpzavD2d8ZZ\n3faBoFFrYMfG0xTn1xIcrsFotJB2pIjUlMI25+ZkVjJyfDR2qvbHbjBqKTpVzq/r8qipaiIy1hOR\nWERpUT1556pJO1LE7TOTcHS++He0pqoJALWmrbVB22Bg48pUGhsMjJ0Sh0+AS7t9lJc08P3XR9n/\n8zmGjYkEwGKxsm1dBvnZ1Tg4Keg7NJjwGA9EIhEWi5Wq8kaK8mrIz66htLCuXYt2aJQ7/YYF4+TS\n8YNVU6ORnMxKXF3t8Q1uX8buyLXKQtxZEhN7smzZtzdUBoGro1O/5jk5OURGRrY6FhkZSXZ29jUR\n6nqy50QJBzLKmDg4mH4xnU9bfjmYqiqp2rge+8QkHJJ6Xvb15SUNlBTUERbjjr1D15pTU3bncOxA\nAQDhMR4MvCkUO9Xlmdz1OhOnT5TSo6dPp5/y885WsXPzaTx8HBk7Ja5lMW3SGklNKSAk0o3IWE9S\nDxVx4OdsTh0vYfy0+DaLUa2+DieFI+IOqkGbzVYO7MomOFyDb2DH24p6nYncrCqyz1RQkNvsOBgS\n6U5Mgjc+Ac4dPt2ZzVb0OhNNWiNV5Voqy7RUVTQilUoQS0QoFFJqa5ooL25ALBEhkYjZ9UMmk2Ym\nXdQ6YdCbOJNWRmScZxsloqnRSGFuDWqNClc3eywWK7u3ZZGVXo6ruz2V5VpOHC4iqa8/0Ky47Npy\nBovZSl21jh0bTyFXSOnZ35/Evv6t5lZfq+fHDSfR1hoxmSxYzFa8fJ24dUpci/LZUKdnz86z6J1r\nESdWES9LRNngzIlDRXy/7Dhjp8R2uPjW1+o5djCf0ydKsVqaF3B3L0fCot1ROcjRNzVbMTJSSzAZ\nLAy/NZKo+ObvqMlkobSwDpsNHJ0UODgpycoo59ftWaz56gijJsXg4d1s9bDarJyqzmRf8SEKT2rx\nzItC5SBn3NQ4/IMvJJyrKteyfvlxtqw5ycQZia2U7AajFnuZisrSRjauTMVittJrYCBJ/fyRSMTY\nbDZys6rYu/Mcep2JcVPj8fa7+Fa5h7cj/YeFsu/nc4RFe+Ab6MLurVnkZ1eT2Nefwtwadm46TWpK\nIXKFlPKSeswmKwAad3vi+/jiG6jG3lGBUilFIhVz4nARqQcLyMmqJK6XL70GBLaag8ViJSu9nMz0\nMorza7HZIKKH5x9K6REQuFo6pfRoNBry8vIIDLywv5uXl4eLyx/7y1Je08TyHZlEBbgwrn/QNRun\natMGADzuuPOyrmtsMHDwlxzOnCwD4OAv2UQleJPU17/V4m+xWKmuaKS8pAGz0YpEJsZZrcRZrerQ\nVJ+ZXsaxAwVEJ3jj4CjnyL58CnKq6TkgEN8AF1zd7S+5VWAyWdiyOo2y4gZkcgk9knw6PN9qtZHy\naw7H9hdg7yinILuG1JRCEv+7MB/em4fVYqPfsBCc1XYEhbtRnF/LD2tPsn75ccZNjUetUWE0mPlx\nxzEKTmoRaQxMGtcPL6/2P49H9+WRdriIk0eK6DcshIRkvzbKi8ViJTWlkCN78zCbrTg4KYhN8sFq\ntZGZXs7ZjHKcXJTNWwi+Tnh4OdJQr6ckv47iglpqqppaFqXzyOSSlnvYWG+gWq9FrpQyYEQoEbGe\nFORUs3PTadKPFhPX27flOr1ZT1ZtNvlFFRT+YsHaJOH4yRz+emdflLJmhVSvM7F++XHqqnUAiMUi\nZHIJRoOZ5CFBJPULYNu6dA7/mktopBvu7o6kHS6itKieEeOiiOjhQdqZHPbvz+LArhzK6iu55aZE\nxGIxueXFbP32FGaDDecgMVFugYCIE4cK2bI6jXFT45HKxPy4OQ2TxURV+BlsVhNr6k/jqlQz5Kbh\nFOw2svbrY4yaGIOjs5LGRgP78o7gr/BDVyGipqqRkoI6RGIRqkAzqcoDDFYNx1QI+39u/TDl5unA\nyGlRuLpfqHcnlsD62rXkNxSikChQSBWoFc6E3xJF2V4r6745TtxATyo9cjhYdoRaQx0+5ZF45cXQ\nqK5A1LsR/+DWmW1d3OwYNC6Yn9ed48d16QyaEMjRylQOlx6jQFuMt9UPj7QE7JRyvHxdOfRrLrlZ\nlcT19iPtcBEVpQ0onEQMmRTSRuFpNDWRW19AjGtEy2dv6KgI0lOL2fVDJiGRbpxOK6XXwECSBwdh\ns9k4mVbIob3ZYAR1qBRXb3scPKU0iGsp0BdyWtdEkCyAaFU4nkoPkgcHEZPo3bJtdyatjD6DA4lO\n8Obc6QoO78mjrkaHs6sdSf0DCItyJzLGi+rqP0+6EQGBS9EpR+bFixezZcsWHnvsMfz9/cnPz+fd\nd99lzJgx3H///ddDzk5zOY7MFquVrQfzGRDrjdrx6i0oF3PStOp16HNzUUVFd6ofm81Gakohh/bk\nYrXaSOjjR1i0ByePFnMmrRSbzYZCKUUkFiEWi9E1GVuelEUi+O07mtTfn35DQ9qMUV7SwPrlx/Hw\ncmT8HfFIJGKqKxrZtTWTsqJ6AKQyMT4BLowcF4WynW0/q9XG1u/TyTtbhdJOhlqjYuKdie3eC22D\ngeL8WjKOlVBSWEd0gheDbgpjx6bT5J2t4vZZScjkEr797BA9knwYfEt4q7Eqy7RsWtWcVKzXgACO\nHshHpzWh01Qir3VGbJESHufGwGERqOzlLTKcTCvk+6XHMbhXoxDJocyBgAhnho+OpsHSQLW+hqrS\nRgoOGKmt0hEc4UbP/gG4ezm0LE5mk4VzZyo5e6qcsqJ6DHpzi1wSiQi1px0SFzPeajecHe1R2slx\ndVfhrLZDJBLh7u5Iel42W3J2ojM3cVfMHahkdthsNv7zXRqlRfVMu7c3Kkc5B0uOsCH7ByTFzvjk\nxGGVmdB5VeCY74/Wo5TY4W70dk9iz/o8KkoauOm2aKxWG/mFFZRX1eLfw4GwEG9c7VwxaC18+/kh\nPH2cmDAtgcVv7sYn0InE0e5sz9/FkbJU5GIZntk9cCr3oSmwGJcIaPjVCZlRiTbpLIXSHJ7u8wi+\nDs0L5/YNGXj7u6AJlJP2aznVoVncO3YCbkpXTlRmsDP/F3Lq85nkPZHy3VLqanRtPjcyhQRXjQov\nP2c8YmS8e6q5fo+9VMXC/v+HuVGE2WxFaSdrtlSIbG0sedvzdrH+3BaSvXoiRozeYqCksZSypgok\nJhkheX1QVKtpcC7HoVcTQdpo8lJ0hES64TCwjhUn1/Nw4lyiXJs/Z3WGBt45+jHlukpcKvzwy0mg\nzrWYKo883L0ciHSIpHC7CIvVQmGPQ4T5BqCu8aH2qByz3gZ2Zoq9T1GtKUAulXFbyGiG+g1AhIiD\npUdYd/Y/aE2NjAwYwqTQsS2fixNHC1m//DgAUfFeDB4VSoG2mD3FBzhSdhyT1Ux7KCVK7KRKagy1\nALgonOnlmcBQ3wFo7FypKG1g30/nKM6vQyIVYTHbsHeVEtTbAZFHE+W6SsqaKgjS+DIuYPQfxpFZ\nQKAzdOTI3Cmlx2q1smTJEtasWUNpaSleXl5MmTKF2bNnI+5mifU6q/TojWaU8o4NXVarjT3bzwIQ\nGuWGt79Lh5aP3y70VpOR6s2bcB0ztpUPj9Fgpq5Gh9FgxmiwIJaI8AtUI5E238emRiM/bT5NQU4N\nQWEaBowMbeXnoK3Xk5FaikFnwmq1YbXaUNrJ8PB2xN3LgZBQd1LPZLM78zCFpxpwqPBi6PhQYnr4\nterj+2XHEIlETL6rZ4uSAM0KV0OdntKiesqK6kk/VtyihOjNBhqM2pbzjv1cQvbJagbdHIbRYCZl\ndy6T5yYgVlkI9vGmocZI3rkq9u4417L4KZRSBo4MJTLOi6PlJzhbnkfNDgeQ2JA5gblSyvR5fXBw\nbLstUlPVxKZvT9DYYMDqqKcwIJVHRtxFcU0FW3ccw6XMD7lSQuzNrrj7ONAoquPnbwoQ6+ToBmTR\naGuAHDWehZGIaP0+mhQ61EkWhvSMx2gxklOXT3Z9LiAizi2aOLcYnOSO2Gw26mp0VJRqsXeQ02Rf\nw8cnv0Rv0SNCRJBTANGaCDRKNY5yR+ykCo5VH2dX7gEkIglWmxVfBy8eSpyLvUxFfa2OVZ8fRu2n\nIDv4INWFRnyrI5FWOeLp78joibHYqWRs3XGM3CMNlPmeQdnkhFONF7Ke1TgFijldnUm5rrLVfESI\nUEjkOJf64Z4ThVmmR2SVkBX3C2a5AblYxlC/gdwUMBSZWMb6DYeozjJjlhmQWuXcNDkCbz9nXjz4\nJq5KNU/0ehCJWEJmehk7N50GQOdcwx0z++PtcCF9hcVq4dO0r0mvOs30kCk4VHhxoiadkw0nifYM\nJU2Xir+rFw/3nIvNZuPNIx9Qo6/jrphpfJj6BcP9BzE5fDwAZquZr9JXUqAt5tGk+1Army15Nfpa\nFh18k0h1KPfHz24179LGMo5XnORsTQ6eFaFUp0qQSpstYKFR7s0+VO4qFmz+BwqJnKf7PILRauKd\no4spb6pgbMgt2Gw2Kk5YqEtv/n0QiZqtdiCi30RvUg1HyarNpqKpCrFJhl2jM2h0JPskEecWzY78\n3aRXnSbA0Q+5RMbZ2hyCnQLxULlxsPQIY4JGMi5kFO7ujuSXVLDux71UVdVTGnyKOlMdNmzIJXKS\nvXoy2Kcfrko1TWYdTaZmXyKNnSsqabNCXaWr5nR1FierTnOy6hQ2m414txgCnfzJqsmmNKcBh0ov\n6tSl1LuWcP5jLxPL8FS5MzCoF8O8hghKj8CfiqtWev5IdEbpOZhRxsqdWTx5RxI+bvYXPW//z+c4\nfrAQiVSMxWzFTiUjMs6L5CFBrZxDz3Ne6bE0NlL8wbvosjLxnv9QS/V0m83G6iVHqKpobU6WK6SE\nRrnj5evEwV9yMBjM9BjkhiZcSpg6GKm4c460ZU0V7CrZzZ78QwBEOYej36dBqXOk7+1e9A7tQUlB\nHVvXpWM2W5g0Iwk3z9Y/dkfLT1DeVMnooBEA/Loti/RjxYyf1YMPz31MnbFZqXMvDsOzMJIK77OU\n+2dib3Im6NhASv1OUemTjVgkxlflg/pAHAqZjKTegfgEuKDxcEAsFpFacZJP075GLpZh3+CGT3oS\nIkSU+WbSFFhEtCYCD5U7znJHnOSOuNlp8FS5o2s0s/XYPn42bWVG9F8Y4NOcC6KgoYjP963CNSMK\nqVFJcdBJJGYZ3gUxRI1wZlifhJZ7dOj0KepLTaikdqikKiRyKHA6w/GaE5h/82TtqXLHZDVTra9B\nhIgQ5yBGBgwh3i0GkUhETl0eHxz/HAe5A9MiJ5Fdl8fJylPkN7R2sJWKpQz26cfNgcMpaCjks5PL\n8FS5syBxHpX6Kv6z8xDiM25YpCYkZhn2DnJie/mS2Ne/Rcm22Wz8tPk0menlAEhiash1PUmTuYlw\ndSg9XKOIUIdisBio0FVR0VSJ3mLAZoOaXSrM1VI0fc24hkqxkyqJd+uBo/zCe2+z2fhlayaZJ8sY\nPTmWgJBmX5cjZcdZkr6CSWFjuSlgKIdKj7F59z5cS4MYNyWBUG8/fo/RYuLj1CWcrcshyT2OI+Wp\nDPUbwJTwCWQ0pvNRyteMDhqJQiJnw7kfmBN7Jz094llxeg37Sw7zbPJjeNi5sSR9OccrTiITy9Ao\n1TzWaz4OMnu+OPkNaZUZPN/3CdzsOi4CWVWu5ectZ3B1s2fYrZHNUWbujmxL38fnJ5fxl/DbOFGZ\nwdnabO6Pn00PzQXfRV2TkfLiBsqK66mr0RHfx69VdJTRYqSksYwms45wl5CW76nNZuNoeSqrMzdi\ntVmZEDaG/t7Nn9OVp9eyr+QQ44JvQePsxNr0H9CaGgl1DsLNToOr0gV3Ozfi3XtgJ728gIcafS2/\nFh1gT/EBGk1NeNl7Eq0OJ0wdgkpqh1gkRiwS4yx3Qq10RiwS/yFD1kNDw3jkkcdJT0/jp592AFBY\nWIBa7Yq9ffPv+ZtvvoOnZ+d9NNesWYXFYmXq1Ds6PO989FZWVma7IesbNqzjgw/ewcvLu+XYK6+8\ngUwm48UX/8EHHyzutEy/p66ujs2bNzBjxqxLn9xF/FHD87tE6TEajeTk5FBTU8NvL+nfv3/XSNlF\nXErpKaps5KWlh/H3dODJO5Iumm05K6OcHRtP0SPJh/7DQ8jPriYro5yczEp8A10YNalHm2gid3dH\ne9B55wAAIABJREFUysvrKX7vbZpOZeB1z1wcfxNWWVmmZfWXR0hI9iMw1BW5QkpTo5GzGeVkZ1Zi\nNllx0ajoM8qHj3M+QW/Ro5DIiXKNIMQ5EKPFiM6sR2824OvoTQ/XKNxVGuoMDWzJ3c6+4hRkYikD\nfJIZ7jcYjZ2azJI8tq/IwigxYPGvRnXWD6O8iYKIY8wfOIMwl+AW+WoNdfzzwBsYLUbmx88m1i0a\nXZORFZ+kYHXRcSJoJ1PCJ9BUKCZnlx6XYDFu/azozTr0FgP1vziBRUzMbSqaxFoyDlYgPeVJbuRB\nbuqZzM2BwwAobSznjcPv42nvwWM95yMTSzm6P5/M9FJCxyjIqDvFmeqsFgXrPFKxFG97T4q0JSS5\nxzG7x/RWvjl6s4GimjIO/1hCdaG+JZJozF9iOxVe2mhq4kRFOg5ye4KdAnGQ22Oz2SjSlpBamU5K\nyREq9dX4OnjTz7s3/8nehqPcgUd+Y4U4L0e9sYEGoxatSUtSUBTWxguflVNVmXyS9hVysZxGcxP2\nEnt65A1Fo1QTm+RLQKimXYuixWxl5+bTuLja0WdwEAA22m79/B5tvYGGGj1eAU6XvA8mo+W/Vo1m\nbDYbn6Z9zanqM/TySORA6WHCXIK5p8edOCscL9qP3qznveOfkVdfQB/PJGbFTG1ZZN/evYT9JYeQ\niiTEusUwN24m0Ows/M8Db+Dv6Iu9TMWx8hP8Jfw2/By8+SD1C/wcfLglcBifpn3NuOBbGBN8U4dz\nuRjnv6fvHFvM2docAGZFT6Wvd9cW2DNYjNhsVpS/UV6sNitfZ6ziUFlzLacodTgTQscQ4NRWebxS\nTBYTeouhlVJ7Mf6ISk97eXrmz5/LjBkzGTRoSLvX/b7O1dXQkSKwYcM6Dh06yEsvvdolY/2W85XY\nr2cE259R6enUp+Dw4cM8+uijGI1GtFotDg4ONDY24uXlxc6dO7tU2GuJzmDmo3VpKGRi5k+IRSoR\nY7FY2bXlDMUFdUQneBOT6E2T1siuLWfw8nNi4E2hSCRiQqPcCY1y53RaKb/8kMm6b44xdkpcK4di\nbYOBtPW/kluqwq7PdML6tM5SnZVRjlgsIqlfQKuQ2sBQDUOMFsqK63H1UvF+2mJEIhF3xUzjXF0u\n6ZWnSa04CYBCIkcmlrGvJIXVbMDdTkOdsQGz1cwgn77c2XsipoYLC1uEdyCK2+3Z8t1JyHJA6mEk\neJCK6koJy0+v5tk+jyGTNMuy8dxWrFYLbkpXvstcT4S6OZIrrJeajH0W+oYNJkYWx7q9x/DwcWTC\n5ESk0gsLblpDEXu2nyVCGk1YhAfvrv0Je1858jBf1p/bgtbUyOigEXya9jVSsZS5sTOR/ffpuGf/\nAJL6NUcQ9fFttspYrBbqjQ3UGespb6qkUFtMUUMJYS4h3BF1e5sFXClVEOoeQPB0fw7syib/XDVD\nRkV0Op+GvUxFf5/WWURFIhF+jj74OfowOnAEh8uOszV3J2uzNuFup2mj8JyXQylV4KFqLgKoUTlS\n0XhBgYvWRDA//h7Wnt3ECI8hDPMbgHLopZ/qJVIxt0xs/UX+/TZdezg4KQgOdetUUsDfKjzQPP+p\nkRN56eC/OVB6mJH+Q5gQOgaJuOMoPaVUyUMJc0irPEVvz8RWitlfIyaS31BIrb6Ov0ZMbDnuKHdg\nfMgovstsTrM/OWwcw/0HAXBPj+l8lraMT9O+xs1Ow00BQy85l44QiUT8Jfw23jn6CWOCR3a5wgPN\n39XfIxaJmRn9V3wdvIn1C8Nb0nXKznlkElnLd7qr+dfXh9scS47x5Kbe/hhMFv69sm1hzsEJPgxO\n8KGhycj7a060aR/Ry49+Pbo+cnb8+NGMGTOWw4dTCA+PYM6c+1i48DmamhoxGAwMHTqM+fMfBmDx\n4g+xWCw8+OACNmxYx88/70SlUpGTk42TkxOvvPJmS8HPK+G3Cst5RWL+/IfYtesn6uvrWbDgMYYO\nHQ5AWloqH330ATpd83bmvHkPMGDAQN5881Xq6uqYOXMaKpWKTz5Zwvjxo3n//Y8JCgpumfP51+PH\nj+a22yZw4MB+qqqqmDnzbiZPngJATk427777b2prazGbzUyfPrOlOOvOndv55JOPUCgUDBs24orn\n3F3plNLzyiuvcO+993L33XfTp08fUlJS+OCDD1oVc+vu2Gw2lm49TWl1E09MS0LtqMBisbJj4ymy\nz1Ti4e3IoV9zObIvD7m8OanbqIk92mxjRcV54eikYOv3Gaz+8giOzkpMJgtmo4VGbXPxOrFzJNYK\nMRHZ1S25b2w2G2dPleMXpG43h4hMLsEvSM3qzA3kNxQxL24WCe6xJHv1xBZho8msQylRtCw25U2V\nZFSd4VT1GYKcArk1eCQeKndclI5UNLRe3AKD3Rg5LprGBgMJyc1bJkEaXz5I/ZwfcndyW+hocury\nOVh6hFsChxPjGsk7xxazJWcHE0LHcFy1F5EiAPEpN35IO4lCKWPM7bGtFB5ozhGyd8dZsk6VU1+j\np7HBwIixkfgEJmKfqWJH/i8cKj1Gg0nLw4n3tlEWfq+cSMQS1EoX1EoXgpwCSKZz4f5isYgBI0KZ\nMDWxS7MQS8QS+nr3oo9XEhlVZwhw8sNJfnFrR0dEuobxbPIfoxSJi8KZBxLuwWA2Eq2J6PR1Kpmq\nXWVCLpHxt54PYLAY2liLBvn0Jbsul2DnQIb5XYiuSnCPZUbUX/gucz1TIyZ2yaLu7+jL64MXXlKB\n62okYgk3Bw7rNlmy/8zodDqWLFkGgF6v56233sPOzg6TycTDD99PSsrBdpMcZmSk8803q/Dw8ODF\nFxeydu0q5s6d36kxz1dZh+ayEK+88ka75zk6OvHll99w9OgRFi36O0OHDqeuro433niVt9/+AI1G\nQ3l5OXPmzOLbb9fwxBNPc999cy4rT5DRaOKLL76mqKiQO++cytix45FKJSxc+DwvvfQKAQGBNDZq\nueuuGcTFxWNnZ8drr/2Lzz9fSkBAAF999UWnx/qj0CmlJzc3l1mzWu8jzps3j5EjRzJnzpxrIlhX\n06g3Y7XamDw0lOhANRaLle0bTpGTWcmAEaEkJPtRU9VE+tFiCnKqGTGuOZdHe/gGqrl9ZiIHf8nB\narUhlUmQyST4+LvgpAIXJwmrV5/l+IGCFqWntKgebb0Br0QZ689uoVJfTaWuCpPVTJQ6jFhNNI3m\nJnYV7mW43yAS3GNbxhOJRNjLWptzPVRueKjcGObfOuz2YkT08Gz1OloTQV+vXmzP30WSRzxrsjbi\nLHdkVOBwlFIl/bx7s7NgN3KJjMz6s4zpG0PBbj1SqZiJdya2e29U9nL8g13JSi/nbHo53n7O+Aa6\nNFsLIibiILPnh9wdTA4bR4Q6rFNyd0fEIjGxbp2LxPuzEOIc1KX9nbeG/R6JWMLsHtPbvaa/Tx+S\nvXp2qZJyvRWePzrPzup90TaFTNJhu6NK3mH7teC89QKaA3Lee+8t0tKarU1VVZVkZZ1pV+lJTEzC\nw8MDgNjYOI4fb2vBuhh9+/bv1PbWzTePaum/tLQUs9lEaupxiouLePTRB1vOE4lEFBUVXZGR4fwY\nvr5+qFT2VFSUo9frycvL5bnnnmo5z2w2k5ubg9lsJiamBwEBAQBMnHg7ixd/eNnjdmc6pfQ4Ojqi\n1WpxcnLC3d2ds2fP4uLiQlNT07WWr8twsJPxwKQ4bDYbJYV1HN6TR2FuDQNHhhLfp9nErNaoGHRz\n5xZjtZs9oydfUEz0ubn4JgVR9d8opYQ+Ovb9lE1ZcT3OHgp2paRiE1vZ2LgGUQFolGrc7JoVor3F\nB9lVuBeAAEc/Jobd2pVTvyiTw8eTUXWG949/SqOpiVnRU1v8DyaFjiWtMoP/5GzHx96L0b37cdRc\ngJefM+5eF7duhMd4sHNzc3TP8LGRLdYbkUjEuJBbGOLX/4qtIwICgpIicDn8VlFYvnwpOp2OL7/8\nBrlczksv/QODwdjudXL5hYc6sViCxWLpctnOj3E+AtpisQI2IiOj+PDDT9ucX1CQ3+aYRCJp5WNr\nNLaeT+t5iFvmodFo2rUY/fzzH8dd5UrplNJz880388svvzB+/HgmT57MrFmzkEqljBo16lrL1yVU\n1OowmixU59eRfqyYmsom5AoJQ0aFXzKZ3qWwWa3UbN1C5frvMY0ZjeOkvwIQneDN4b35pOw7xx6P\nTfhl9wM3PXMSphPnFtPqx9toMZJZc45zdbkM9u3X6Witq8VepmJKxASWpC8n0MmfPl5JLW0Ocnv+\nEn4by0+vYUrEbUglUpKHBHfQWzPBEW5IZWJ8A1zazX4sKDwCAgI3goYGLW5ubsjlcsrKStmzZzd/\n/Wv7VsUbRXx8Aq+++jLHjh0hKal5a/jkyTRiY+Owt3dAr9dhsViQSJrXDz8/fzIy0gkODuHAgX3U\n1dVecozg4GDEYjHbtm3llltGA80+Pp6eXsTFxfPqqy9TWFiAn58/Gzeuv3aTvUF0anV97rnnWv6f\nM2cOCQkJNDY2Mnjw4GsmWFeyaV8u+SfL8bQ2p38fNiaCsGiPNk6bl4uloYGSLz6j6eQJHHonE3Dn\nHdQ0NmvScoWU2F4+HN2Xh8LkhtSsYNTAJEI83Nv0I5fIiXWLviFbJj094jFbzYS6BLeJAkr26kmC\ne2y7DpkXQyaXMGF6IgFBrugNpq4WV0BAQOCKmDZtOs8++yTTp0/Bw8OTXr36XPqi64yLi5rXX3+L\nDz98l4aGBsxmM76+vvz73+/h6urK8OEjmT59Ci4uLnzyyRLuv/9BXnxxIatWraB372Tc3duuL79H\nKpXx73+/y9tvv8nSpUuwWq1oNBpefvl13NzcefLJZ/jb3xagVCr/lI7MlwxZt1gsjBo1ii1btrQy\nlV0uOTk5PP3009TW1uLi4sJrr71GUFBQq3Pef/99VqxY0bKX2rNnTxYuXHhZ4/w+ZL2hycjfP9xH\npEVEZKwnI8ZFXfEcfouxvJzCN1/FUl+P+9TprFDnEuoRwCifC2G01XX1rFx8GDFi5HIpdz08oI3z\nb1fTXZwju4Mc3UGG7iJHd5Chu8jRHWToLnL8WULWryd/1DDu/yU6Clm/5AoskUiQSCQYDIarEmLh\nwoVMnz6dH3/8kenTp/PCCy+0e97EiRPZsGEDGzZsuGyFpz1+OVaEvwUUdlIG3hR61f2dx6rXIbF3\nwP/p57H0SyK9+jSbTm8nr76g5ZxjdcepcS8Em4jgcLdrrvAICAgI/JlxddUwb95sUlIO3pDxjx8/\nyuzZd+Lqqul0KgyB7kWntrdmzZrFo48+yn333YeXl1erN9vf3/+S11dVVZGRkcGXX34JwLhx43jx\nxReprq6+qtwHl8JitXI8pRANIoaPiWxTpfpKsNlsiEQilAGBBLzwT0QiESmF+wCwkyn59sw6/q/3\nQ9hsNn4u2INbhAeOBiUxSd6X6FlAQEBAoCOuZ2K+9khM7HlZIeMC3Y9OKT0vvvgiAHv37m11XCQS\ncerUqUteX1JSgqenZ4vzlUQiwcPDg5KSkjZKz3/+8x/27NmDu7s7Dz/8MElJSe112SlOnalEbbCg\n8XMmOMLtivv5LdWbNmBpbMR96h2I/ut1f7LqNG5KV+5ImMD7B79kX3EKdlI7agy1TImbQMKgHl0y\ntoCAgICAgMCV0yml5/Tp09daDgCmTZvG/fffj0wmY+/evTzwwANs2bIFtbptFNDFOL83bbPZyE09\ngUolZ+acZJycLq+Oze+xWSyU/PAjVRvX4zFiGO4ezSn9DWYjmbXnGBkykEGBfdiZvYeNOVtxU7ni\n5eDOiKjk616U1d29e0RIdQc5uoMM0D3k6A4yQPeQozvIAN1Djvb8eQQE/qxcl9hob29vysrKWkLt\nLBYL5eXleHu33vL5ref5wIED8fb2Jisri+Tk5N93eVHOOzJnniyjILeGYWMiMBhMVFRcWSSRzWaj\n8UQqlWu/w1hcjCq6B05TplNZ2Vxt/GTlKUwWE2GqMEQiEZOCx/PKoXfIqy3krxETqapqvMQIXUt3\ncI7sLnJ0Bxm6ixzdQYbuIkd3kKG7yNGRI7OAwJ+RTik906dPv6jT1vLlyy95vUajITo6ms2bNzNh\nwgQ2b95MdHR0m62tsrIyPD2bMwefOnWKoqIigoMvnRvm9xgNZvb+dA6TTIzU9epKZdgMBsq++gKx\nSkX+PWNIVVSxQCxq8QA/WXUauUROmDoEAB8HL24OGMaBksP0876+2UcFBAQEBAQELk6nlJ4pU6a0\nel1RUcHatWsZP358pwf6xz/+wdNPP81HH32Ek5MTr732GgBz585lwYIFxMXF8dZbb5Geno5YLEYm\nk/H66693Ku/A7zmyLw99k4ksLExSXJkxS5+fh8LPH7FSid/jTyL38mZj2hKya3L5Me8nxoWMwmaz\ncbLyFFHq8JbCmQDjQ0YxNvhmIXusgICAgIBAN6JTziaTJk1q9Tdv3jw+/fRT9u3b1+mBQkNDWb16\nNT/++COrV68mJKTZMvLZZ58RFxcHwGuvvcbmzZvZuHEja9euZejQy6+kXFer48ShIqQaOyxyCb5u\n9pd1vc1qpXrrFvJf+ie1P+0AQOHnj00iJq8+H4lIwra8XZQ2llHcWEqNoZZYTevcPyKRSFB4BAQE\nBLqYfv16MmPGX0lJOcj27T8yZ86sNud89tliXnyx43Qnmzdv5Jln/g+AU6cyeOGF59o9r7i4mFGj\nLp2gLzPzDDt2bGt1bObMaej1+kteezFMJhMzZ05j+PCB7Nmzu13ZBg7sw8yZ05gx46/cffcMTpxI\nveLxLsbEiWM5d+5sl/d7nvnz57Y7v9/z6acfs337jwAcOXKYgwf3X9F4V+xh6+npyZkzZ6708mvG\n0X35SGViSkQ2gr2dEIs7l0vBZrOhPXGcvEULqVzzHQ5JPXEaeCHjdEljGQaLkQmhY1BI5Hx7Zh0n\nK5sj13q4dU3CQwEBAQGBjvnss69ITu7L0KHDKSgoIDc3p6XNZrOxZctmxo+f0On+oqNjWLTo5auS\nKTPzDDt3tg6nX7bsW5TKKw+gkclkLFv2LVFR7SfZA3BwcGTZsm9Zvvw7xo+fyMsv//OKx+vuzJs3\nv6WA6tGjhzl48MAV9dOpvZ81a9a0eq3X69m2bRuJiYlXNOi1pu/wEN758Qy3RnR+a6xi5XJqf9qB\nzN0Dr3n349inbys/ppy6PADi3GJQShSsOLOWgoYi/B18cFE4d/kcBAQEBLoT+4sPs68o5Zr0PcA3\nmf4+l+cDKZfLueWW0WzevJGHHnoEgCNHDiGVSklM7InZbObxxxdQV1eHwWAgJqYHTz/9PDJZ63xt\nR44c5v333+arr5r9U9esWcW33y5HpXJg4MBBLeddrL+mpkY++2wxjY1aZs6cRmJiTx5//MlW2aMz\nMtJ5663X0el02NnZ8be/PUlMTA+Ki4uZPftOJk68nf3796LX63n22RdITLz8VC09e/bi/fffbnn9\n448/sGrVSszm5iCehx9+lD59mivKT5w4lltvHUdKygEqKyuZMWMmU6ZMA5oTML7xRnOV+KSknsCF\nCgeXmseECZM4cGAfBoOBf/7zJb7/fi3p6WkoFEreeOMtNJqOU8csWrQQhUJOfn4eZWVlxMXF88IL\nixCJRCxatJDo6GiSknqxbt1abDYrhw4d5OabRzFr1uxO36dOKT0bNmxo9VqlUpGUlMTdd9/d6YGu\nF8NvjaSyVkdiuBvR7RS8/D3nkw06JvdF7uOD86AhiKRtb0tOfT4OMnvc7TS42blyoPQI2XW59LgB\n9bIEBAQEBGD8+Ak8/vgC5s9/CIlEwubNGxk37jagOR/cokX/wtnZBZvNxqJFL7Bp0wZuv/0vF+0v\nKyuTr776gqVLV6LRaHj99Vda2jrqb+7c+9m791deeeWNNn2aTCaeeeb/eP75hfTp05eUlIM888z/\nsWZN87paV1dLXFw88+c/xNatW/jww/f47LMvL/te/PLLzy2WEIB+/fpzyy2jEYlE5OXl8tBD97Np\n09aWdr1ez+efL6W4uJgZM6YwduxtSKVS/v73Z/jHP16mV6/e7NixjTVrvuv0PBISEnnggYf55pul\nPPTQ/Xz00Wc8++zfef31V1i9ehX33//gJedx7tw53n//Y8RiMbNm3UFKykH69u3X0h4WFs6kSZPR\n6XQsWPDYZd+nTik9y5Ytu+yObySuTkoeuj2uw3NsVitVG9dhrq3D867Z2IWFYxcWftHzc+vyCXIK\nQCQSIULE9KjJfJm+gj6e3dPaJSAgINCV9PfpfdnWmGtNZGQUrq4aDhzYR2JiErt3/8KDDzZbfaxW\nK8uXL2P//r1YrVbq6+svud109OgRBgwYjEajAWDixNtbtq2upD+AvLxcZDJpi5UlObkvMpmUvLxc\nVCp7VCoVgwYNASA2No733nu7o+5aodU2MHPmNGprazAYjCxZ8nVLW2FhIZ988iwVFeVIpVKqq6uo\nqqpssbacV5B8fHxwdHSivLwMk8mEQqGkV6/m9/mmm27h1Vdf6vQ8Bv7XJSQyMgoPD08iIiIBiIqK\nJiWlc9tRQ4cOa6lpFhkZRVFRYafvR2folE/P+vXr2yQoPH36NOvXd8+y8zqDucN2m8VCyacfU715\nE9hsYLV2eH6TSUdpUznBzgEtx7ztPXk2+TG87D27RGYBAQEBgctn3LgJbN68ke3bt5GYmNQS8btt\n2w+kph5j8eIvWL78OyZPnoLBYLzicbq6v/PIZBcKeYvFzXnsOst5n57167cwatQYXnjhOc7XEH/h\nhWeZPHkKK1euYenSFf+toXlB3t8WEBeLxRcdt7M1xn4/j872/3taXyfBYul4Pb9cOqX0vPvuu20S\nCXp5efHuu+92qTBdgc1m46nF+1n1U9ZF28tXfIP28CHcJv8Vz7vvQSTpONLqfBHRIKeADs8TEBAQ\nELi+jBo1hoMHD7B69beMG3fBgbmhQYuLixp7e3u02ga2bdvaQS/N9OzZi/3791BdXQ3Axo0XHuw7\n6q/5mLbdPgMDgzCZzBw5cgiAw4dTMJvNBAYGXcl020UikfDQQ49QVVXJ7t27/itvAz4+vgBs2rQB\no/HSClpgYBAGg4Hjx48C8NNPO2hoaLhu8+gs9vYOF73fl6JT21tarRYHh9YZOx0dHamvr7+iQa8l\nVXV6tDoTXq6qdttrtm2l7pefUY++FdcxtwJgtVmx2WwXDTPPqc9DhIhAp0sXVxUQEBAQuH44OzvT\nv/8Ajhw5xJAhQ1qO33rrWHbv3sXUqbejVqtJSEjCYDB02Fd4eAR33XUP8+bNxt7engEDLjgyd9Rf\nnz7JrFixjDvvnEpSUi8ef/zJlutkMhmvvPJGKwfgf/3rjTYO1VeLQqHgvvse4IsvPmXIkGE89tjj\nPPnk33B0dKJ///44O7tcsg+5XM6LL/6rlSOzl5fXdZ1HZxg2bDhPPbWZmTOnXbYjs8h23hbWAdOm\nTWPWrFnceuutLce2bt3KkiVL+O67765M6mvEtn3ZvL82jUVzkvFzb5tavTH9JA2HUvCcdXdLwdDV\nmRs4XHac++LvJsQ5sM01H6Z+Qa2+juf6/q3DsbtLWvkbLUN3kaM7yNBd5OgOMnQXObqDDN1Fjo7K\nUKSnZ+Dj0/b38Eby24io/xXmz5/LjBkzW/x+BC5NcXEePXq0H+rfKUvPE088wbx58/jhhx/w9/cn\nPz+f/fv38+mnn3apoF1BXpkWO4UEn98lJbQaDIgVCux7xGLfI7bluNFi5EDJYfQWA+8d+4S7Y+4g\n0eOCE7TNZiO3Lp9E944dowUEBAQEri2urhrmzZvNggV/Izm5740W55piMpm4556Z1NfXI5crbrQ4\nfxo6pfT07t2bzZs3s3nzZkpKSoiPj+e5555r4+fTHcgtbSDE2wnxb5yvzHW15L+8CNex43EZOrzV\n+Scq0tFbDMyJvZOf8nfz+clvuD18HCP8m73Qy3WVNJl1rZyYBQQEBASuP1u2bL/0SX8SzicnFOha\nOqX0GI1G3N3dmTdvXssxk8mE0Whs5WndHRia4I2ICwqPzWym+OMPsWi1SILaKi4HS4+iVriQ6B5L\nrCaapRkrWZu1icKGYqZGTmpJSig4MQsICAgICPyx6VT01uzZs0lPT291LD09nTlz5lwToa6GXpEe\nJIZfyPpY/u0K9GezODN9EM9lf86p6syWtjpDPaeqM+nr1ROxSIxcImNO7J3cGnwzKaVHef3Qexwp\nS0UpUeJl73EjpiMgICAgICDQRXRK6cnMzCQhIaHVsfj4+Da5e7ob9fv2UrfrJwrH9+UH40ls2Fhx\nei16c7PH/aGyY9iwkezVs+UasUjM2OCbeSjxXhpNTWRUnyHIyR+x6IrLlAkICAgICAh0Azq1kjs6\nOlJZWdnqWGVlJXZ2dtdEqK7C3FBPZb8oNjjlE+YSzEMJ91Kjr2VjdnN+hZTSowQ5BeDZjhUnyjWc\nZ5IfpY9nEoP9+l9v0QUE/r+9O4+qqt77OP5mJlBAEBDUnEpD1FAUSm95RQSuIqAulUvyeNX05jzk\n9TqUKGhlrUhLupaVPvA4lIYTqWmRtzRnuhaJw4OKKPOogB6Bs58/eDxJToiwz4nzfa3FWrDPZv8+\n+7cPm+/ae5/fTwghRAOrU9ETEBDAq6++yrlz57hx4wZnz55l3rx5BAUFNXa+R3Yi9ye2/+9uvrn8\nb37t5kBi5xs4PeHEpO5j6eL4FC+26cv3V37k+yuHuVqWje8dV3l+z97Kjr95/hUv5273XUcIIYQ6\nnnuuFy+9NIpjx46yf//XTJjwX3ets3btGmJioh64naSknSxY8A8A0tJOs3jxonuul5WVRWCg30Nz\nnTt3lm++2VdrWWRkODdv3nzo795PZWUlkZHhDBjQj4MHv79ntn79+hAZGa772rt3NwCzZ0/nypXM\nercNsHnzBt0gjWqZPHniPfe1IdXpQebZs2fz1ltvMXLkSDQaDdbW1owYMYJZs2Y1arj6SCtSL+Ct\nAAAUSElEQVQ8z9HsFKqVmiGv7S2bM/XZ8dha1IzrENIxiF8KTvP5uW2YmZjRy/XZB21OCCGEAVm7\ndj02NjbcunWLd955i0uXLtK+fQegZoiR3buTiIqKrvP2PDy6Eh29/LEynTt3lkOHfsDfP0C37HE/\neXX701uTJ0+87zq3p6H4vffe++Cx2gbYvHkjffr44ujo+NjbMiR1KnqsrKyIiopi8eLFFBcXk5eX\nx44dOwgICODgwYONnfGRRHYdxYjmz3M+5nVsRg6jzfNBWJn99gkza3MrIrqMYPWpT+jm9AzNLGwf\nsDUhhBAAaT/nkPaf7EbZtoeXGx49Wj3S71haWhIQEERS0k6mTauZZPTkyeOYm5vj5dWLqqoqXn11\nBqWlpWg0Grp29WT+/NfuGkH45MkTfPDBe6xfvwGArVs/Z/PmDdjYNKNfv99GZL7f9ioqylm7dg3l\n5WVERobj5dWLV1+dV2sgxdOnf601kvGcOfPo2tWTrKwsxo0bQ1jYcA4fPsTNmzdZuHAxXl49H6s/\nw8KG8O67q+jU6SkmT55I165d+eWXXygoyGfgwEFMnToDgIKCfN59921yc3PQaDQMGhTI3/42gXXr\nPqGgIJ+FC+dhaWlJdPQbJCT8Nx4eHowcGQ5AdHSU7ufo6CisrCy5fDmD3NxcunfvweLF0ZiYmFBe\nXsbKlbGkp59Ho9Hg7d2HmTPnYGZmxsWLF4iJWcKNGxV06vQUt249eMTshlDnp3OLioqIj49nwoQJ\nDBs2jNTUVBYtuvclQX0rPZCMNWa08/lzrYLnNg+nzkzoNoZhTwXrIZ0QQoiGMHRoKHv3fqWbzDIp\naSfBwSFAzXxU0dFvsH79BjZu3IJWq2XXrh0P3N758+dYv/5TPvpoHfHxGyktLdW9dr/t2ds7MHHi\nK/Tp40tCwuZaU1BAzW2qBQv+wd//PoUNG75g0qQpLFjwDyorKwEoLS2he/cexMdvYvz4icTFvV/n\n/b89y/rtr9LSknuul5OTw5o1nxAfv5GdO7dz+fJlAJYuXcyoUeF89lkC69dv4PDhQxw9eoRx416m\nZUtn3njjbRISNtOhQ8eHZklPTyc29gM2bdrKmTNpHDt2FICVK2Pp1asXn32WQELCZoqLi3THYcmS\n13STooaHR5CWdrrO+15fD7zSU1lZSXJyMtu2bePgwYM8+eSTDBkyhKtXr7Jy5UqcnJwaPeCj0t64\nwbUfD9Hcxxfz5nb3Xa+XSw8VUwkhxB+bR49Wj3w1prF16fIMjo5OHDnyI15ePfn++38zdWrNVR+t\nVsuGDQkcPnwIrVbLtWvXsLa2fuD2UlJO0rfvC7r/bWFhw/n22/313h5ARsYlLCzM6dOnZgRpHx9f\nLCzMyci4hI2NLTY2NropJrp16877779X5/2/3+2t3xs4cBCmpqY0a9ac9u3bc/VqJs7OzqSknKSk\npFi3XkVFBZcuXcTX97k6Z7itf/8/Y2VVM3J0ly7PcPXqFQAOHvw3p0+nsnHj/wBw8+ZNXFxcKC8v\n48KFdP7ylyEAdOvWg06dnnrkdh/VA4uefv36YWJiwvDhw5k+fTqenp4AbNq0qdGD1df1lJMoGg0O\nfv76jiKEEKKRBQeHkpS0k/z8fLy8euLs7AzAvn17OHXqJ9as+RRbW1vWr/9Ud4WjPhp6e7dZWPx2\nN8LU1Ex31aoh3TmI8O02tFotJiawbl0C5uYPnzTUzMwMrfa3qTp/fyvq7jaqgJrnrN5+O5bWrdvU\nWr+8vH6zpD+uB97e6tKlC9evX+fUqVP88ssvtS71GaqK06lYd+yI9f8/2CaEEKLpCgz8C0ePHmHL\nls0EB4fqll+/XoaDQwtsbW0pK7vOvn17H7qtXr28OXz4oO5TSzt3bq/T9mqW3fufeLt27amsrOLk\nyeMAnDhxjKqqKtq1a1+f3W0wtra2eHn1JD5+vW5Zbm4OhYUFutfv3Ke2bduSllYzSHFBQT4nT56o\nUzsvvNCf+Ph1umKupKSYrKyr2No2o1Onp/j66z0A/PprKunp/9sQu/ZAD7zSk5CQwNWrV9m+fTuf\nffYZy5Yt409/+hMVFRVUVVU1erj6aDVhErdU/pidEEII/bC3t+f55/ty8uRxXnzxt5nIBw8ewvff\nH2D06OG0aNGCZ5/tiUbz4Adln366M2PHjmfSpHHY2trSt+9vDzI/aHt9+viwcWMCY8aMpmdP71rP\n9VhYWPDmm+/UepD5jTfeueuBan1YunQ5K1e+y0svjQLAxsaGRYuicHJqyahRf2XZsiVYW1sTHf0G\noaHDWLBgHuHhI3jyyXZ4etZtKJdZs+ayevUqIiPDMTExwcLCglmz5uLu3pqoqBhiYpaQkLCeTp2e\nwsPj3jOjNyQTRVGUh69W48SJE+zYsYM9e/ZgZmbGiBEjmDdv3sN/UUWFhWW1LsGpydm5Ofn51/XS\ntiFlMJQchpDBUHIYQgZDyWEIGQwlh7NzcwoLy3ByanbXa7/+ehp393Z6SHV/d34iylhMnjyRl16K\n1D33Ix4uKysDT897F1CPNLdC7969iYmJ4dChQ7z++uucO3fu4b8khBBCNABHRycmTRqn+2RQU3Z7\ncMKsrKtYWlrpO06TUadxen7PysqK4OBggoPlI99CCCHUsXv3fn1HUM3twQlFw5JZNIUQQtzTIzz9\nIIRBeNh7VooeIYQQd7GwsFBlhFwhGlJ1dRXm5mb3fV2KHiGEEHdxdXWhpKQAjeamXPERfwiKoqW0\ntJgWLVrcd516PdMjhBCiabO3twcgNzdPN2WCEIbMxKRmfKGWLVvedx0peoQQQtyTvb29rvgRoimQ\n21tCCCGEMApS9AghhBDCKEjRI4QQQgijIEWPEEIIIYyCFD1CCCGEMApS9AghhBDCKKhW9Fy8eJHR\no0cTGBjI6NGjuXTp0l3rVFdXs3TpUvz9/Rk0aBBbtmxRK54QQgghmjjVip6oqCgiIiL4+uuviYiI\nYPHixXets2vXLi5fvsy+ffv4/PPP+eCDD7hy5YpaEYUQQgjRhKlS9BQWFnL69GndrOzBwcGcPn2a\noqKiWuvt3r2bkSNHYmpqiqOjI/7+/uzdu1eNiEIIIYRo4lQperKzs3F1dcXMrGYSMDMzM1xcXMjO\nzr5rPXd3d93Pbm5u5OTkqBFRCCGEEE1ck5uGwsmpmV7bd3Zurtf2DSUDGEYOQ8gAhpHDEDKAYeQw\nhAxgGDn0fc4UQk2qFD1ubm7k5uZSXV2NmZkZ1dXV5OXl4ebmdtd6WVlZ9OjRA7j7yk9dFBaWodXq\nZ0ZgZ+fm5Odf10vbhpTBUHIYQgZDyWEIGQwlhyFkMJQczs7NKSwsk8JHGA1Vbm85OTnh4eFBUlIS\nAElJSXh4eODo6FhrvaCgILZs2YJWq6WoqIhvvvmGwMBANSIKIYQQoolT7fbWkiVLmD9/Ph9++CF2\ndnasWLECgIkTJzJjxgy6d+9OaGgop06dIiAgAICpU6fStm3bR2rH1NSkwbP/kdo3lAxgGDkMIQMY\nRg5DyACGkcMQMoBh5DCEDEKoxURRFP3cCxJCCCGEUJGMyCyEEEIIoyBFjxBCCCGMghQ9QgghhDAK\nUvQIIYQQwihI0SOEEEIIoyBFjxBCCCGMghQ9QgghhDAKUvQIIYQQwihI0SOEEEIIoyBFTz2sWLEC\nPz8/unTpwrlz53TLL168yOjRowkMDGT06NFcunSp0TIUFxczceJEAgMDGTp0KNOmTaOoqAiA//zn\nP4SEhBAYGMj48eMpLCxstBwAU6ZMISQkhLCwMCIiIkhLSwPU7Y/bVq9eXeu4qN0Xfn5+BAUFERoa\nSmhoKD/88IPqOTQaDVFRUQQEBDB06FBef/11QN3jceXKFV0fhIaG4ufnh4+Pj+o5vvvuO8LCwggN\nDSUkJIR9+/apnuHAgQMMGzaMoUOHMmbMGDIzM1XJUJ/zlD7+ZoVQlSIe2fHjx5WsrCxlwIABytmz\nZ3XLIyMjle3btyuKoijbt29XIiMjGy1DcXGxcuTIEd3Pb731lrJgwQKlurpa8ff3V44fP64oiqLE\nxcUp8+fPb7QciqIo165d032/f/9+JSwsTFEUdftDURQlNTVVmTBhgu646KMvfv+eUBRF9RwxMTHK\n8uXLFa1WqyiKouTn5yuKov7xuNOyZcuUpUuXqppDq9UqvXv31h2PtLQ0xcvLS6murlYtQ0lJieLj\n46NcuHBB19b48eMVRWn8fqjPeUqf7xEh1CBFz2O482RSUFCgeHt7K1VVVYqiKEpVVZXi7e2tFBYW\nqpJl7969ytixY5VTp04pQ4YM0S0vLCxUvLy8VMmgKIqybds2ZdiwYar3h0ajUUaNGqVkZmbqjos+\n+uJeRY+aOcrKyhRvb2+lrKys1nJ9vj81Go3i6+urpKamqppDq9UqPj4+yokTJxRFUZRjx44pAQEB\nqmY4deqUMnjwYN3PxcXFSufOnVXNUNfzlL7PYUKoQbVZ1pu67OxsXF1dMTMzA8DMzAwXFxeys7Nx\ndHRs1La1Wi2bNm3Cz8+P7Oxs3N3dda85Ojqi1WopKSnBwcGh0TIsWrSIQ4cOoSgKn3zyier9sWrV\nKkJCQmjTpo1umb76Yu7cuSiKgre3N3PmzFE1R2ZmJg4ODqxevZqjR49ia2vLzJkzsba21tv7Mzk5\nGVdXVzw9PUlNTVUth4mJCStXrmTKlCnY2NhQXl7Oxx9/rOp7s0OHDhQUFPDzzz/To0cPdu3aBejv\nfPGgdhVF0dt7RAi1yDM9TUBMTAw2NjaMGTNGbxmWL1/OgQMHmD17Nm+//baqbf/000+kpqYSERGh\narv3smHDBnbu3MmXX36JoihER0er2n51dTWZmZl07dqVxMRE5s6dy/Tp06moqFA1x52+/PJLRowY\noXq7VVVVfPTRR3z44Yd89913/Otf/2LWrFmq9kXz5s157733ePPNNxk+fDiFhYXY2dnp9XgIYcyk\n6Gkgbm5u5ObmUl1dDdT888nLy8PNza1R212xYgUZGRmsXLkSU1NT3NzcyMrK0r1eVFSEqalpo17Z\nuFNYWBhHjx6lVatWqvXH8ePHSU9PZ+DAgfj5+ZGTk8OECRPIyMhQvS9u75+lpSURERGkpKSoekzc\n3NwwNzcnODgYgGeffZYWLVpgbW2tl/dnbm4ux48fZ+jQobp8auVIS0sjLy8Pb29vALy9vXniiSew\nsrJStS/69u3Lpk2bSExMZMyYMdy8eZPWrVvr5Xg8qP/1dQ4TQk1S9DQQJycnPDw8SEpKAiApKQkP\nD49GvSwcGxtLamoqcXFxWFpaAtCtWzdu3rzJiRMnANi8eTNBQUGNlqG8vJzs7Gzdz8nJydjb26va\nH5MmTeLgwYMkJyeTnJxMq1at+PTTT3n55ZdV7YuKigquX78OgKIo7N69Gw8PD1WPiaOjI76+vhw6\ndAio+TROYWEh7du3V/39CbBt2zb69+9PixYtAHX/Tlq1akVOTg4XLlwAID09ncLCQtq1a6dqX+Tn\n5wM1t6FjY2MJDw+ndevWejkeD+p/fZzDhFCbiaIoir5D/NEsW7aMffv2UVBQQIsWLXBwcOCrr74i\nPT2d+fPnc+3aNezs7FixYgUdO3ZslAznz58nODiY9u3bY21tDUCbNm2Ii4sjJSWFqKgoNBoNrVu3\n5p133qFly5aNkqOgoIApU6Zw48YNTE1Nsbe355///Ceenp6q9sed/Pz8WLNmDZ07d1a1LzIzM5k+\nfTrV1dVotVo6derEa6+9houLi+o5Fi5cSElJCebm5syaNYv+/fvr5XgEBgayaNEiXnzxRd0yNXPs\n3LmTtWvXYmJiAsCMGTPw9/dXNcOiRYtISUmhsrKSfv36sXDhQqysrBo9Q33OU/r6mxVCLVL0CCGE\nEMIoyO0tIYQQQhgFKXqEEEIIYRSk6BFCCCGEUZCiRwghhBBGQYoeIYQQQhgFKXqEMDBdunQhIyND\n3zGEEKLJkbm3hHgIPz8/CgoKdHMSAQwbNozFixfrMZUQQohHJUWPEHWwZs0a+vbtq+8YQgghHoPc\n3hKinhITEwkPDyc6Ohpvb2+CgoI4fPiw7vXc3FxeeeUVfHx8GDRoEF988YXuterqatasWYO/vz89\ne/Zk+PDhtabz+PHHHwkICKB3794sXboUGUNUCCEen1zpEeIx/PzzzwQFBXHkyBH279/PtGnT+Pbb\nb3FwcGDOnDk8/fTT/PDDD1y4cIFx48bRtm1bnn/+edatW8dXX33Fxx9/TIcOHTh79qxuOhGAAwcO\nsHXrVsrKyhg+fDgDBgyoNZWDEEKIRydXeoSog6lTp9K7d2/d1+2rNo6OjowdOxYLCwsGDx5Mhw4d\nOHDgANnZ2aSkpDB37lysrKzw8PBg5MiR7NixA4AtW7Ywc+ZMOnbsiImJCc8884xuUk6AiRMnYmdn\nh7u7O76+vpw5c0Yv+y2EEE2JXOkRog7i4uLueqYnMTERV1dX3WSWAO7u7uTl5ZGXl4e9vT3NmjWr\n9VpqaioAOTk5PPnkk/dtz9nZWff9E088QXl5eUPtihBCGC250iPEY8jNza31vE12djYuLi64uLhQ\nWlpKWVlZrddcXV0BaNWqFZcvX1Y9rxBCGDMpeoR4DEVFRcTHx1NZWcmePXtIT0+nf//+uLm50bNn\nT2JjY9FoNJw5c4atW7cSEhICwMiRI1m1ahWXLl1CURTOnDlDcXGxnvdGCCGaNrm9JUQdvPLKK7XG\n6enbty8DBw6kR48eZGRk8Nxzz9GyZUvef/993bM5sbGxREVF8cILL2BnZ8f06dN1t8jGjRvHrVu3\nGD9+PMXFxXTs2JG4uDi97JsQQhgLE0U+CytEvSQmJrJlyxY2bdqk7yhCCCHqQG5vCSGEEMIoSNEj\nhBBCCKMgt7eEEEIIYRTkSo8QQgghjIIUPUIIIYQwClL0CCGEEMIoSNEjhBBCCKMgRY8QQgghjIIU\nPUIIIYQwCv8Hf6CdTIy1MSYAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 2 Axes>"
]
},
"metadata": {
"tags": []
}
},
{
"output_type": "stream",
"text": [
"Random init test accuracy: 66.62%\n",
"Finetuned test accuracy: 69.90%\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "2zRIsnuiHcpG",
"colab_type": "code",
"colab": {}
},
"source": [
""
],
"execution_count": 0,
"outputs": []
}
]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment