Skip to content

Instantly share code, notes, and snippets.

@ceceshao1

ceceshao1/Comet-REST-API.ipynb

Created December 27, 2018 19:56
Show Gist options
  • Save ceceshao1/8a8b48e84944fe164124e0d8da118d4e to your computer and use it in GitHub Desktop.
Save ceceshao1/8a8b48e84944fe164124e0d8da118d4e to your computer and use it in GitHub Desktop.
Download ZIP
Comet-REST-API Example
Display the source blob
Display the rendered blob
Raw
Comet-REST-API.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img src=\"https://comet.ml/images/logo_comet_light.png\" width=\"200px\"/>\n",
"\n",
"# Comet.ml REST API\n",
"\n",
"*This page is available as an executable [Jupyter Notebook](https://nbviewer.jupyter.org/github/comet-ml/comet-examples/blob/master/notebooks/Comet-REST-API.ipynb). To launch in MyBinder, follow the link and click the icon in the upper right-hand corner*.\n",
"\n",
"Comet.ml has an extensive interface to all of your data using a [REST API](https://en.wikipedia.org/wiki/Representational_state_transfer) through [Comet.ml endpoints](https://www.comet.ml/docs/rest-api/endpoints/). Now, you can access this information easily through the Comet.ml Python SDK. Requires version comet_ml version 1.0.40 or greater.\n",
"\n",
"## Setup\n",
"\n",
"To run the following experiments, you'll need to set your COMET_API_KEY and COMET_REST_API_KEY. The easiest way to to this is to set the values in a cell like this:\n",
"\n",
"```python\n",
"%%writefile .env\n",
"COMET_API_KEY=\"...\"\n",
"COMET_REST_API_KEY=\"..\"\n",
"```\n",
"where you replace the ...'s with your keys.\n",
"\n",
"You can get your COMET_API_KEY under your quickstart link (replace YOUR_USERNAME with your Comet.ml username):\n",
"\n",
"https://www.comet.ml/YOUR_USERNAME/quickstart\n",
"\n",
"You can get your COMET_REST_API_KEY under your settings (replace YOUR_USERNAME with your Comet.ml username):\n",
"\n",
"https://www.comet.ml/YOUR_USERNAME/settings/account\n",
"\n",
"## Quick Overview\n",
"\n",
"To access the REST API through the Comet.ml SDK, you will need to make an API() instance. First, we import the API class and other libraries we will need:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"from comet_ml import API\n",
"\n",
"import matplotlib.pyplot as plt\n",
"%matplotlib inline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and create the API instance:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"comet_api = API()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using the `comet_api` instance, you can get the name of your workspaces:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['dsblank']"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"comet_api.get()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you reference your workspace by name using comet_api.get(WORKSPACE_NAME), you'll see your projects:"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['pytorch', 'mnist-001', 'visualizations', 'testing', 'pypy', 'tensorflow', 'chainer', 'mnist-014', 'optimizer', 'general', 'jupyter-experiments', 'keras', 'fastai']"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"comet_api.get(\"dsblank\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Or, get the projects from another user's workspace:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['fasttext', 'comet-notebooks', 'parameter-space-exploration', 'home-credit']"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"comet_api.get(\"cometpublic\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using the same method, you can refer to a project by name:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['5dc346a883964bd2b8864c40940fd864', '351afa6e498f452ca743c19d3e131767', '72b89ab961cd42db91535026687f86fb', 'd5d3cf989efd4b5f8841bd64fee5f817', '12ab8382de254a9bb678bf3d6131e255', '51cf6e588a3346cdb560fd0c09d49610']"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"comet_api.get(\"dsblank\", \"keras\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Or, using the slash delimiter:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['5dc346a883964bd2b8864c40940fd864', '351afa6e498f452ca743c19d3e131767', '72b89ab961cd42db91535026687f86fb', 'd5d3cf989efd4b5f8841bd64fee5f817', '12ab8382de254a9bb678bf3d6131e255', '51cf6e588a3346cdb560fd0c09d49610']"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"comet_api.get(\"dsblank/keras\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And one more level, get the details of an experiment:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<APIExperiment 'dsblank/keras/51cf6e588a3346cdb560fd0c09d49610'>"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"comet_api.get(\"dsblank\", \"keras\", '51cf6e588a3346cdb560fd0c09d49610')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Or, again using the slash shorthand:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<APIExperiment 'dsblank/keras/51cf6e588a3346cdb560fd0c09d49610'>"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"comet_api.get(\"dsblank/keras/51cf6e588a3346cdb560fd0c09d49610\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's get the experiment and save it to a variable named `exp`:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"exp = comet_api.get(\"cometpublic/comet-notebooks/d21f94a1c71841d2961da1e6ddb5ab20\")"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<APIExperiment 'cometpublic/comet-notebooks/d21f94a1c71841d2961da1e6ddb5ab20'>"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"exp"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There are a number of items you get from the APIExperiment instance:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Help on APIExperiment in module comet_ml.api object:\n",
"\n",
"class APIExperiment(builtins.object)\n",
" | Methods defined here:\n",
" | \n",
" | __init__(self, api, workspace, project, experiment_key)\n",
" | REST API Experiment interface.\n",
" | \n",
" | __repr__(self)\n",
" | Return repr(self).\n",
" | \n",
" | display(self, clear=False, wait=True, new=0, autoraise=True)\n",
" | Show the comet.ml experiment page in an IFrame in a\n",
" | Jupyter notebook or Jupyter lab, OR open a browser\n",
" | window or tab.\n",
" | \n",
" | For Jupyter environments:\n",
" | \n",
" | Args:\n",
" | clear: to clear the output area, use clear=True\n",
" | wait: to wait for the next displayed item, use\n",
" | wait=True (cuts down on flashing)\n",
" | \n",
" | For non-Jupyter environments:\n",
" | \n",
" | Args:\n",
" | new: open a new browser window if new=1, otherwise re-use\n",
" | existing window/tab\n",
" | autoraise: make the browser tab/window active\n",
" | \n",
" | get_asset(self, asset_id)\n",
" | Get an asset from this experiment. Not cached.\n",
" | \n",
" | ----------------------------------------------------------------------\n",
" | Data descriptors defined here:\n",
" | \n",
" | __dict__\n",
" | dictionary for instance variables (if defined)\n",
" | \n",
" | __weakref__\n",
" | list of weak references to the object (if defined)\n",
" | \n",
" | asset_list\n",
" | Get the asssociated asset-list for this experiment. Not cached.\n",
" | \n",
" | code\n",
" | Get the associated source code for this experiment. Not cached.\n",
" | \n",
" | data\n",
" | The experiment data in JSON-like format.\n",
" | \n",
" | existing_experiment\n",
" | Get an ExistingExperiment() object for this\n",
" | experiment.\n",
" | \n",
" | git_metadata\n",
" | Get the asssociated git-metadata for this experiment. Not cached.\n",
" | \n",
" | git_patch\n",
" | Get the asssociated git-patch for this experiment. Not cached.\n",
" | \n",
" | graph\n",
" | Get the associated graph/model description for this\n",
" | experiment. Not cached.\n",
" | \n",
" | html\n",
" | Get the HTML associated with this experiment. Not cached.\n",
" | \n",
" | images\n",
" | Get the associated image data for this experiment. Not cached.\n",
" | \n",
" | The image data comes as a dictionary with the following\n",
" | keys:\n",
" | \n",
" | apiKey\n",
" | runId\n",
" | experimentKey\n",
" | projectId\n",
" | figCounter\n",
" | figName\n",
" | step\n",
" | runContext\n",
" | fileName\n",
" | imagePath\n",
" | \n",
" | installed_packages\n",
" | Get the associated installed packages for this experiment. Not cached.\n",
" | \n",
" | metrics\n",
" | Get the asssociated metrics for this experiment. Not cached.\n",
" | \n",
" | metrics_raw\n",
" | Get the asssociated raw metrics for this experiment. Not cached.\n",
" | \n",
" | os_packages\n",
" | Get the associated installed packages for this experiment. Not cached.\n",
" | \n",
" | other\n",
" | Get the asssociated other items (things logged with `log_other`)\n",
" | for this experiment. Cached.\n",
" | \n",
" | parameters\n",
" | Get the asssociated parameters for this experiment. Not cached.\n",
" | \n",
" | stdout\n",
" | Get the associated standard output for this experiment. Not cached.\n",
"\n"
]
}
],
"source": [
"help(exp)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For example, we can explore the `other` property, which shows items saved with Experiment.log_other(NAME, VALUE):"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'name': 'Name',\n",
" 'valueMax': 'example 001',\n",
" 'valueMin': 'example 001',\n",
" 'valueCurrent': 'example 001',\n",
" 'timestampMax': 1544540174200,\n",
" 'timestampMin': 1544540174200,\n",
" 'timestampCurrent': 1544540174200}]"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"exp.other"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this example, we see that the experiment has the `Name` \"example 001\". We can use `Name` to also look up experiments:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<APIExperiment 'dsblank/keras/example 001'>"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"exp = comet_api.get(\"dsblank/keras/example 001\")\n",
"exp"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Perhaps one of the most useful abilities for the REST API is to access your experiment's data in order to create a variation of a plot. To access the raw metric data, use the `.metrics_raw` property of the experiment:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"2040"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(exp.metrics_raw)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Thus, there were 2040 metrics logged during the training of this experiment. We can get the first using indexing with an integer:"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'metricName': 'acc',\n",
" 'metricValue': '0.09166666865348816',\n",
" 'timestamp': 1542738532966,\n",
" 'step': 1,\n",
" 'runContext': None,\n",
" 'offset': 21}"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"exp.metrics_raw[0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"That shows that the \"acc\" (accuracy) metric had a value of about 0.09 at step 1 of the experiment.\n",
"\n",
"We can also use a string as an index to query all of the dictionaries in `metrics_raw` to only give those values at each step, like so:"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [],
"source": [
"acc_metrics = exp.metrics_raw[\"acc\"]"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"510"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(acc_metrics)"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(1, 0.09166666865348816)"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"acc_metrics[0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Therefore, `exp.metrics_raw[\"acc\"]` gives us (step, value) for all \"acc\" items. We can then easily use Python's built in zip and matplotlib to plot these values:"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [],
"source": [
"steps, acc = zip(*acc_metrics)"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xl8VNX9//HXmSWZ7CF7WEIAwyogEBFRFBUR0WpbrUrVWmtr6/L9Wrv4dWlta/19tbZft2q1VqutbV2qtVIVURFXQBaRfYvsECBsScg+M+f3x713cmdLAiSEO/k8Hw8ezNy5mTl3MnnPmc85547SWiOEECKxuLq7AUIIITqfhLsQQiQgCXchhEhAEu5CCJGAJNyFECIBSbgLIUQCknAXQogEJOEuhBAJSMJdCCESkKe7HjgvL0+XlpZ218MLIYQjLVmyZK/WOr+9/bot3EtLS1m8eHF3PbwQQjiSUmpLR/aTsowQQiQgCXchhEhAEu5CCJGAJNyFECIBSbgLIUQCajfclVJ/VkrtUUqtjHO7Uko9qpSqUEotV0qN7fxmCiGEOBwd6bk/B0xr4/bzgTLz3/XAE0ffLCGEEEej3XDXWn8E7G9jl4uBv2rDAiBbKVXcWQ0U4mgFg5qXFm2lyR/o7qYctTWVNSzYuK9D+67cUc2byyt55L0NrN9dG3V7MKh5edE2GluOzfOyZMt+Vu6ojnt7SyDIS4u2EghqAubvrNkfDN3+zqpdbNtfH/Yzu6obmblsZ4fb8MG6PWzeW9ehfeua/Ly0aCvBYOtXkVZWN/DIextYU1nT4cfsLp2xiKkPsM12fbu5rTJyR6XU9Ri9e0pKSjrhoYVo3+vLdvA/r65g76FmbjrrhO5uzlE5/5GPAdh8/wXt7nvh7z8JXd68r46HLj8p7PbFWw5w26vLSUv2cMGoru+PXfLEfKMtcdr+l3mbuffNNWgNmSle/ufVFWw/0MCPpw5Ba831zy+J+vnvP7+YZdurGdE7k0H56W0+vtaabz+7qM022D354Zf8/v0KUpI8XDS6NwAvLtzGI3M2sG53DX+4clz7B92NjukKVa31U8BTAOXl5T3mm7kfnbOB80YUMaQoo7ubctSen7+Z0rw0JpW1u/o5ZN2uWt5cvpNbzx2MUqrrGofRG71v1houP7mEyuoGtuyrp7qhBYAa8//DtW1/PX/+dBN3TR+Gxx3/w+7HG6rYsq+eqyb0j3n72ysraWgJ8LUxfdt8vAffXc/kIfmMLenF7ppGfv3Gak47IY8Z41s7RH9bsIXiLB/nDCvklSXb2bCnlsIMH7npSSilQmFk2XuoKepx1pm9+VU7q1m69QBTRxTxzCcb+fbEAZw6KDdu+5r8Af73zTXcdNYJFGT6QtvfXF5JbWMLjS0Bvqyqo6wwnUvH9eU3s9byo6lDQvs98t4GJg3OY2xJL579dBMj+2RRXppDVa3Rxn11zWSmeAH4/fsVnD20gBcWbg39/LyKvSzafIAbzxrEsu3GJ4EfvbyM4kwft0wpY1hxZsx21zb5Q5dfWrSVy08uIRjUPDB7HecOL+T1L3bgUoomf5Cahhbqm439P1xXxUWje/POql08MmcDADsPNobua/6X+1i3q4Z0n5cUrzv0RrlyRzWvLNnOvrpmxpVk0xwIsnDTAU4ZkMP3zhgY9/ntLJ0R7juAfrbrfc1tAmhsCfDgu+t5+uONLP/led3dnKMSDGp+/voqoGM9H8t/lu3ksbkVzDilhOKslK5qHgAb9x7iTx9v4j/LKtlVY/wB3nTWIAB8XvcR3eczn2ziuXmbuay8X9zgALj6mYUAccP9B3/7HKDNcF+/u5ZH52zgyQ+/ZP2957Ng4z7eWF7JmysqmT6ytXf9s38b8xs23Tedn/xzWdT9RIb7vkPNUftsMMP9Dx98CcC/v9jJ3kNNZPq8bYb7F1sP8pf5W1hdWcM/fzARMF4bN/3j86h9axv9/GX+lrDn/qH31vP3z7aw8K4p/Oo/qwHj9RTURn+v2R+kJdBajvnaH+aF3ed/v7iUvYeamTyktYOxbNtBlgFlhelxf0eVtkD+n1dXcPnJJSzZeoAnP/ySJz/8Mu7xfr71AEDokwMY5RnLjD8tCNv/glHG38YlT8yjySwr/ccsHSW5XXy0oYqrT+1/xK/HjuqMqZAzgW+Zs2YmANVa66iSTE9V32zUM5tstUOn2nGwoc3bX1u6nWufXRi1fV+dESzrdx+K+7MVew5xxVPzqW2M7l0HgppLn5jH3HV7APhi20EueuwT/rl4G7e8uDRsX+sxrGAHQj3CQPDwPywGg5q3Vhgv58rqBm742xJeWbKdy/84n8/M2vfcdXu48ukFYT/z2tLt3PT36LAD+PHLy3h58baYt81asQswAu7+WWupbTR6j1oT9hiWRZsPxLyf7z8fft6m3TWNXPLEPC59Yh7PL9jCba8si6rDW737/ebv645/Leev8zeHHdeMpxbw9qpdYY/99spdnPz/3ovZjndX7wbgQH34m4sr4hPcL2euYq/5BvTq59u55cUvYt6f0U5jv4/WVwEw1PaJ+PfvV/Dsp5sAqG5o4fxHPua7f1lEdUML5z38Udj9jPrlbL7x5Py4j2PZtLeOq57+LGzbntomPlxfxRkPzI3a/5YXl/LY+xti/s3feNYgmv1Bvth2sN3HPVrt9tyVUi8Ak4E8pdR24BeAF0Br/STwFjAdqADqgWu7qrFO1GAOVrldXVuOOBY27IkelLO79SWjB9nsD5LkcVHb2EJ6sod9Zmhs2F3LmYNjl3M+33qABRv3s3ZXLSeX5qC1RillfswPsnjLAW55YSmf3TmFO/+1gtWVNfz0leUA3HPxiWT6PCilWBbjj8b6Q6puaOFQk59UrxtXG78P67EPNflZtaOaPeabw7pdh5i1chezVhrhdvlTC9h8/wVca9ZxLdUNLaHn4nfNAXxeV9gby6ufb2dPbSOXlfcLPUcASinW76nF53WR4nXzl3mbud728X3ljuhBvIfeXR/zGGav2h12fV9dc+hNdvEWI5R93th9u73mfi8sNN6AvnVqKf5AkMrqRuZv3Md824BuXZOfW15cGhVkfbJTqGv2h557K5DdLkUgqKlpbAkbqHxu3uZQSG8/0HYnwjJr5S5cCk4ZkMPaXa2vzV/9ZzXXnjaAjVWHWFNZw5rKGn47e23o9ty0JPbVNVPT6I91t5x+Qh63nlvGy4u2k53m5Y8fbuSTir1h+2gNv529lq0RA7wAr38RPcB74+RBeN0urj1tAH/44Es2VtUxYWD8T0edod1w11rPaOd2DdzUaS1KMA1m3c5p0V56+5t857QB3P2V4aFtFXtae95N/gDnPvgR/XNTef66U8J6MIN/NovvnDaAP3+6iZ9MHRzqCVo9xVPvm8OI3lk8fU05Ow42cNr973PKgBwAdh5sYO7aPVz73CL++5wyc7yiEICaRj/Df/E2OqID/vmWA/zwpS/41qn9+eNHG6OOxerN//uLHTy/YAsXje7NozPGhB3rN8b1ZVhxJve8YZQJrjylhL9/ZtR5kz0umvxBfvP22qj7Dsb4NDDm1++GLn+wbg83/P1z7v3qiRFtquWFhVu5418rmDG+H++v3cNPpg6h8mADY0t6cfNZJ/DNpz9j1srwD8F3Xzg81EYgLGjbk+HzoDUcMmvPjS2xP03ur2uKmll0yRPzQvVtuxG/mB26XJzlo7La+MT04GWjeXf1bp7+xOhFv7/W+NT1u2+MoqE5yJ2vrQg7DiAsoC2TyvL4eENrsPq8LtxKUdccYHVlDYPy0ygrjB7L+tuCLaHSlXG9tWb/rxsncuZvPwhd/+l5Q/jt7HWh6/d9fST9clIZ1z+H9btr+eOH4a+pvPQk9h5qZuWOmtBroz23TRsaurz8F1O7vCQDskK1y1llmciPoR21v66Z6oYWdtc0Rk1Z27a/vt1Sw67q8J+rbmgJ9aQBth+op9kfZMX2arbtrw+7zz+bH28te2paf253dRNb99eH/vAiezDWz67ffSisLBMIaiqrG3lvzW6CQc2H64yP1p9tMmbbVlY38oD5h/bY+8bg1cJNrTNxI4Md4LG5FVQ3tPD43ArA+GONxSpx2KcS+s3a7j+XbA+ru1rBDjDefOOJZW9d9ECl3e/eMY7l7tfD1wDurmniGTP4Xlq0jd01Tfz0leV8vvUgxVkpjCnphVLGc5bpa+2DTRiYS5I5qHvLOWXcfeFwBuWnhd335eX9iOXCUcWM7d+rzfYCbNvfEPZG/taKypjBbjemJJtXb5gYul6clUJ5afRjZaV4GVxozGqxl3x6pXq5+8LhPHftyXz1pNbxgrz05LCff+LKcfz9exNC1++/ZBQzxpfw5FXjOKGgdbaMPdgt40tz+Md3T6F/bhoP22YOXX1qf5699uTQ9aKs1kHiwYUZPHX1OJ65ppwHLh3Fs9eezBv/NSl0+/PXnRL302g8xyLYQcK9yzVY4X6EZZkfPL+EO/+1glP+d07UgM6kB+by4Lvr4v5sIKiZcN8cbv5Ha126/N53OeV/5wBG8J/+m7l876+L+cpjnzDpgblMemAuizfHXtZghTSE19/tg1+R7G8mFXsOhb0JPDxnA3e+tiJs/13VjaGBPut960B927NclphlhqA2PnJfdUr0gOZAWwDuqW0K9V6rbG906b7YH2S/Mro3WebsjUhrKtsuVX1ZVRdqm2XKMOOTiBWgke/PvbN9pCS5KclJBYxpgdanl0EFaaHwGd0vi++cPoBvRhzvjeYAcqSLRvehJMcY0C7O8uFxKYaYvd6TzSA+sY8xGHnBo63TKG+MM3Zgd+1pA+idncLFZjAXZiUzOEaPOivFG+ppBzV43cbfxTUTS/nO6QOYPKSAW88dHNo/Ny0p7OfLS3txUr9swKi1n1yag9ulmHZiEWcPLYjZtt7m8zWiTyYTT8gD4Ktj+gAwuDCdTJ+Xs4YUMLJPFgDeiBlRU0cUcc6wQi4r78dZQwooyvKFBm3L+/fiknFtz34a3Terzdu7Srd9WYfTLdy0n4H5aeSlJ/NpxV52Hmzg0nF9CWpjEOm8EYUopahvsXrurT9rlSdivfg/3lDF6H7ZZPq8BIOa5TsOMjDP6JF8tL6K91bv5vSyPDaYpYZZK3YxrDiTSSfk8/nWA5xle4FbCz7eW7Ob5+dv5ozB+bQEWpNktjkw9qE5MGV5yTbYt/dQE+t21XKoyc++umbSkz0cavKHDca9uCj24KD18zWNfooyfeyqaeR3to+/s83atd3KHdX4D2Pg8/tnDAwrxZQVppOZ0vqynv3DM3ApeGTOBjZW1ZHkcdHsD/LsJ5tITfbgtv1eNlaFL275xri+3HjWCZTmpjJ1eCHN/iDPfLop7GP6vRGlBbuJg3L573PK6JeTyvb99WT4vKQne8jLSGL43UY544bJg3jig/CZGlZ4lxVksGVfPenJHn4/YyxVh5pI9rgpzvKxdX996A0nLSm8J1hom54IMKJ3Jn+4ciz9c9P4eIPxu750XF++Ma4f+RnJ1Da14HG52Lyvjo/WV4XV9rNTvRysbwn9b0nxunn/J2cCxrRAK3B/e+lobj9/KMkeN/1zW99QrVp7erKXrBQvhZnJVNU2ce7wQt5asYtMX+ubZ44t0PfXhQ/EWuMTC+88hwxf+BvubecN4eoJ/ZkUMchZlOVjZ3UjxVnhz8uiu6aQYnvuXvr+BOqaOrag6+XvT6ChJYDLpUhPbr2Pj356Fl/7w6fsq2vmvq+P5JyhBXE7DV1Neu5H6LI/zg+NtF/59Gf89JXlrKms5bl5m/nB35aEVs01xCjLTH3oI6Y+9FHUfe6va+bqZxaGetrbDtTT2BIMW5X33b8u5v5Za0PhunFvHTf/Yymj73mHa59bRLXtD9AewD9/fVXYR9XGlkDclY5WqQTgsifnc+XTn/H955fw0foqRvQ2eiz2lYY/j/ER2LJ1n9H2KcONN503V7TWkLcfiB6MWnoYswjcLsXlJ7eWIAbkpTFxUB5KKQYXpjPNXFtQVpgR6qHOMPf/v3fX8+s3VvPL/0SH86kDcynK9HHqoFwG5KWhlCI7NYmCTB+j+2aH7bthT/wZQCP7ZjFhYC59slM4ZWAuw3tnUpKbSmqSh6+e1JsRvTO5dFxf0pLcXHf6gNDPDTd7hVb5IsPnIcnjok92a68bCIV7ZMXP53VTmpsauv7tiaWhoLWmU549tICS3FRSktwUZPjISUtibEkvTjd7tpbvnGa0y+NSfM3s7YLxKac4K4XirBTG9e8VmjCQ5HGFpru6XYqCjGSmjyzivq+NBKDIfOM5bVAe540oCh2D1/YuawU4wHknFgEwZViBeazGfgWZvrBgBvC4XfTLSeXbE0vDtqeZ91cUMQ03PyM57LFSkzzkZ4SXgeLJ8HkpyPCFfs5SkpvKPRcb4ytnDy2gINMXdvuxJD33I2Atid60tw5tKwJXN7SEgtiaHWDV3GMt3gkGNS6XYu2uGv78ySbGmfVQKzitgUD74guALfvqQm8akQ7UN/O3z7ZQ2+gP61FAeO26srox5sIWCC+/bIxYqt0vJ5U1lTX8c8n2mD87ZVghk4fkh95IrLafXJrDxEF5YR/x62Icg1Xvnzq8kHdW7466HWDdvdO48NFPCGhNv5zWEHv/x2eGnud3bj0z7HfzX+eUce3pAwgENH+Zb3xL2ZDCjNBCHsuksjz++p3xQOzfWUYbvbDsVC9f3D2V381ex2NzK0j2xK+tPnxF64Duyl+dh1KKO6cPo8kfCIWB9ckusodabIZ8ZpxSEcC7PzqTQFBH1XdP7JPV5hqF8tIcHrhkFLe9asxEsga6tYaHLj+JIUUZ3D9rLR2tMi68a0ro8mW2N+IHLz8JrTW3mTOe7IvD7M/7eSOK2HTf9MNa/PbLi0bwwyllnHSPMbBtlVmSPV3Tl02LCO8LRhUzfeThtbkrSLgfAXuw2kfKDzX58QeN61ZHpCGiLOO31ad3HGygX04qj71fwRvLK3nFDExrn1jnAwFwu1ysN6cl9s9NRevWAc0dBxtCI/9FER/P7W2trG4IC/GOyk1LijuFDCAlyR2z95Oe7GFSWT6TyvKYPrKYO/61IsZPGzJ9Hh6dMYZrn11EaV5qaEoeGItAkj1uLhzVmySPK6w+GvnHFHk9PdkTFvhXjO/Ha0t3sNw2WHjLOWVt/lHag/bsoQWhWSA+ryv0fKeab6rWTKn2WI/ndqmwXl6Z2XO39y4BzhpSwIbdteSmGc/zucOLePbTzTT5g/x4qlGv9rpdHOm4XXF26+tmTEkvzhqSz/fPNOr4A/OMTwC7a9oeSO4IpRQ/mDyIlTtrOG9EUdhtM8aXMCAvNbTf4bL/nm6bNoR9dc1tLsw6GqnJ0U90dwc79LBwf+DttZTmpoX1II5Eg232SZ2tV13b2IKV3asra7j1pS9CpxywPrZac6bBmPqVl57MnDVGQFil5ppGPy8u3BoaWIz03hqjR/utU/uHPgIu3ryfS5+cH3ZCI/tCnsiaaeXBxrB6ZqbP02ZoW3LTk9q8PcntigojMAIqyePi+etOAeCFhVvDQtVucGEGPq+bF66fwPtrd4eFu8vM8lumlLXb1ljsf3SluWnMvPl0HnlvAw+9t54bJg+ivDT+zBjrOCwPXXYSo+95B4Cc1CR6mz3q7JS2n6OOGpSfjktFD/SOH5ATNoMnJy2Jt394Rqc8JrSWfXpn+UjyuHj22vGh2844zJkh7RmUn86sWyZFbb/v6yOP6n7t60qGFmXy+k2nHdX9tSWy5368OD5b1UWsZdZHG+71th6ZfQDmUJOfgNlzf3mx0Qu3pqVZLzX7suUP1u2hJRCkoSXAbdOG8MDbrYONt/9rBcOKM0lyu2iOMxvFPr/Xql2ui5gr/Ng3x7B060Fy05N44O11FGQks6e2id21jRysb+GCUcVorZkwMJe7zVMLtKVPdirPXXsyH6yrItnripoDDMZH+e9NGkBasoeH3zOmM0aWFn48dQgvfLY1tNox3nGNK8nhqyf1Zuv+ej7fejDm3Ownrxp7RCuArYG7+hbj9xk5MBmLfVqiL8nFM9eUs+9QM0GtQ+H+9bF92LCn9qhPUubzurnj/GEdmr7YmQbkpfPd0wdw9anRs458XjcPXDoq7uyhnigtRs/9eNCjwj2WIT+bxX+dfQI3nx3dE7zlxaUcavTzzLdP5q7XVvBpxV4++OlZoTo6QF2zvefuJzKHd5phvrO6kaE/nxUKpxMK0pm9ajcH61vIS0/iqyf1CQt3ME7vOmFgDgs2xp6a2N9Wb7b+2CIXgpx+Qh4XjuodmhFTkpPKvrrm0GNNGJDD1aeW0uQPdCjcBxemU1aYweQhxgBXZLhrNB63i7suGM7btgU4kbXqMwfnc+bgfEpvfzPqMYYUts5Xzkr18vAVY1i7q4ZpD38cs03TTjyyMxpan0KsnnZHzntj70UnuV2cY05rtPN53fziKyOOqE2RjsUJpiK5XYqfXTg87u2XxZlH31NZpbQS29/j8aBHhnswqKluMKZ3NfmD/O6d9VHhvqemMWwZsX1Ri70sY+/F23vulh22pdT2XufEQbn8df4W3lpZyVWn9I+avnbO0AJG98vmwlHFnP1/H0Ydw7nDC8NmNmSGwr0GpVoX+2SnGsFlzbzIS08m0+cJzR23bk/2uPnTt8oZWpQRNZXMrjQvfMHM+z8+M7x9tlmM+RmtxxSrVAPw2o0T8bpdVB1qoqwgnYWb9jM1ov4KxkyYzmbVrK87fQC56Ulhs0HiSbEVso+HuqqI77UbJ5LURYOodm6X4ulvlTOym+azx9Mjw/0PH1Twu3fWs+COc+LuE3kmOjv7gKq9LFPb2BIW/BC9ctNivctrDeefWBR17pn7LxnV5rSsH04pC1sY5fO68XldNLYEKcxMjhrwKsr0kZeeRJ9eKaTuaA13+5zic4cbvdDe5rzgWCIXeAzMT2doUUboE0PQNmBpn1ccb67vmJLwkkPfXrF7P9bMk4w4bxKHY3S/bJZtOxiaSpfkcXW4NyqB7hyRr62uNGV49Ce47tYjw/3fZo98U8Q0v9U7ayjMTCYnLYnd5mBkksdFhe2EWRt214ZNA7QPqB5q9IdWPlriLcixh9jgiPO8v3PrGe3Ot+0do4SQleKlsaWJQfnp/Pum08LeMJRSvHrDRLJTk3hzuVEumTG+HxNjzCCYfesZfOPJ+azdVcvVE/rz/AJj6uC828+O2RaPbY6y/WgLbMcQ+aZwJObdfnanTGf723XjQ1NVhUhUPXoR0zbbIppDTX6mP/oxN/3jc3NKoyYtyU2zP8iUB1sXHH3vr4tD52aG1rnaGebKzUMdmHECUJDZGnzWEusxJcYCmVgrVyNlp0YPaFlvCGUF6RRnpYQWWVj656aRleINveGcObggzlxuLz8wp759+7TS0HZrwDCSvddrP/dLW19scSR6Z6eQm96xRSZtyfB5u6TMI8TxpEf23K04s1ZPAswxpxcu2LifN8yebb+c1KgBys37wsss1rlRCrN8rNpZE1WWsRvVNys0/c9+zgwrYF/5wcSwskYsM8aX8KuLRsQM5WFFmazcUUNOWtsBaM3Fz2tjWuNXx/Rh2olFHTrJ0dUT+pPidfPTV5bT8RMHCCG6Uo/suVvT5uz18BW2OdfWApv+ue2PflurVc8aks+hRj+BQHS8TRtRRJ/sFK62fUNPTlp0sLpdKmb54s7pracLzUzxxB0kuumsExiYl8ZXRrc9e8RqY6w22FnBPqpvFrdNi32mRTDenCYPKSDJ4+I7tp4+wDWn9mf6yOgBUie7vLzfMfnOUSGORo/suVuLe7bYwr0+Ro+7X5zBvVjunD6Muy4wpo9d+sQ8Fm85wDlDC5izdg+nnZDLk1ePY96Xreeljjd7JJbrzxjEzoONPDdvc5sDiqV5abz/k8nt3p81EJvbTg/fMvPm09vdJz8jmfX3nh+1/VcXnxhjb2f7zaWjursJQrSrx4S7/UsVrN52lW0F5/4YA2wl7fTcLxnbl+kji1i4aX9YmSQlyY3HpfjNpaO4943VoZM12VeyKaW45ZyymOe8jsUqpRzOm0I8f7vuFF5buiPs7IlCiMTSY/66Y81asZ8nPNZKyfZ67v932WiAqIUsaUkeCjN95KUnh50gKnIlm/281e2x3pA640T/I/tmHXdzcoUQnavHhHusbyxqa/ATws+jkuHzhL7J547zh4adjTDSDZMHRZ2HGlpPPXokrHDvjCmFQojE12OSoiVi5ai9vBGr1DF9ZFFoWXFqkjs0NRCMb4OxSi2xjO6XHfalGZajOa+z9SUbx2LFnRDC+RI6Kdbvrg2dPjdyFot9kVDkSZDOHV7Ig5edRKq5gjHFa9TQLUca0h05MVU81gwfCXchREckbFJs21/P1Ic+4r5ZxjfWR9bc822LYXqlhYf7OUML8HndoZNdXTKub9hqz/ZOexuPtajnSKYGnjnYOI+M/UuAhRAinoStuVc3GIOln1YY0w/9EWWZfNsK0cjzb1tTBTN8Xpb+/FwyU7w8P38zYHxZcuTKz8Ox7BdTQ58IDsdVE/ozfWRxp6zQFEIkvoQNd2tmojUQ6Y8oy9jPexK5lN/+fae9zIU+brPXnXmUX3Z7pOfBVkpJsAshOixhyzLW6XWb/EECQc39b68Nuz2/zXCPvj+3GfixZt0IIcTxJmF77o3mNMcmf4AP1u0JnQnRYi+tWGWZMSXZpCd7Qqe+tbMGVOOd5VEIIY4nPSDcg8wxv8TYrpett2713Htnp/D4N8fGvD9rQFV67kIIJ0jYskyDLdyraqO/qd0+ndFa9RlsI7itc5ZLz10I4QSJG+7medab/cHQXHc7+6kArC+AaOt0u60998P/ImYhhDjWEjLcq2qbQlMhgdBpAyy9s3xhA6pWzz3Ge0BIqOYe45S+QghxvEm4mrvWmpP/33th2xZvORB2/YbJg8IGVK2FQSe3cYZGt8t4H5SauxDCCRIu3CvjfLGz5f6vj+SK8SVh24YVZ/LBTyaHvrQ6Fut8XYF2vilJCCGOBwlXllm/u/Vr8TwuxcCI78osK4y9fL80Ly20MjUW6bkLIZwk4cJ9w+5Docs+rzs0y8XicR3IO1VvAAARnUlEQVTZIUvNXQjhJB1KOqXUNKXUOqVUhVLq9hi3lyil5iqlliqlliulpnd+UzvG/lV2Pq87KsztJwD79cUjmBpjwVIsMs9dCOEk7dbclVJu4HHgXGA7sEgpNVNrvdq228+Al7XWTyilhgNvAaVd0N421TS28EmFPdxdeCNOkWvvyV99ailXn1raoftuXaEqUyGFEMe/jvTcxwMVWuuNWutm4EXg4oh9NJBpXs4CdnZeEztuV3UjLQEdOrlXiteN1xVZlolfV2+L9NyFEE7SkXDvA2yzXd9ubrP7JXCVUmo7Rq/9vzqldYepyTxZWJ559sTOrbkbPycrVIUQTtBZA6ozgOe01n2B6cDzSqmo+1ZKXa+UWqyUWlxVVdVJD92qyW+sSrW+TKMoyxf1naPuI+y5W+8J0nMXQjhBR8J9B9DPdr2vuc3uOuBlAK31fMAH5EXekdb6Ka11uda6PD8//8ha3Abrq+gyfOaJwGKEe2RPvqM8MhVSCOEgHQn3RUCZUmqAUioJuAKYGbHPVuAcAKXUMIxw7/yueTusL+Y4WN8MQHF2SlSN/UjLMtZZJMeUZB9FC4UQ4thod7aM1tqvlLoZmA24gT9rrVcppe4BFmutZwI/Bv6klLoVY3D121of+6WcVlnmzMEFfL71IGeU5bNiR3XYPkdalinI9PH2DycxIGJRlBBCHI86dPoBrfVbGAOl9m132y6vBk7r3KYdnprGFp6btxmAr4wu5ruTBpCW7ImaLXM0hhZltr+TEEIcBxJmheovX1/Fgo37AUj2uklLNt63PGbNvU92CileN+nJCXc6HSGEiJIwSbe3rjl0Odm2cMlrDqBeOq4vt547+Ji3SwghukPC9NztA6fJYatSje3eI5wlI4QQTpSg4e6Ovt2dMIcqhBDtSpjEs89nt/fSlXnxSE87IIQQTpQw4W6f4qiU7bL5v4S7EKInSZhwby+8j3R+uxBCOFHChHu88FaS6UKIHihhwr29U77IGWGEED1JwoS7deqBSArpugshep6ECXfrpGGRrLLMsT/TjRBCdJ+ECfemOOEuhBA9UQKFe+yyjKUbTlIphBDdJoHC3ei5//nb5WHbrYq7RLsQoidJnHBvCTJlWAFnDy0M265kLqQQogdKnHD3B2KeU8YiVRkhRE+SMOHeHAiS5Ik+nJF9sgA4oSD9WDdJCCG6TcKcz72pJRhxql/D18f2YXS/bAl3IUSPkjA99yZ/7HBXSkmwCyF6nAQK9wDJ3vg1dyGE6EkSItyb/AEaW4KkJSVMlUkIIY5KQoT7rupGAIqzfd3cEiGEOD4kRLjvPGiEe++slG5uiRBCHB8SItx31TQA0nMXQghLQoS71XMvzpJwF0IISJBw37S3juxUL6kyoCqEEEAChLs/EGTOmt1MKsvv7qYIIcRxw/HhvmpnDQfqW5g6vLD9nYUQoodwfLjvPGgMpg7MT+vmlgghxPHD+eFuzXGXaZBCCBHi+HDfVd1AssdFr1RvdzdFCCGOG44P953VjfTOTpEv5RBCCBvHh/uu6kaKMmV+uxBC2Dk+3KsbWshJS+ruZgghxHHF8eFe1+QnNUlO9SuEEHYdCnel1DSl1DqlVIVS6vY4+1ymlFqtlFqllPpH5zYzvromP2nJsjJVCCHs2k1FpZQbeBw4F9gOLFJKzdRar7btUwbcAZymtT6glCroqgbbaa2pbw6Qliw9dyGEsOtIz308UKG13qi1bgZeBC6O2Od7wONa6wMAWus9ndvM2Jr8QfxBLeeUEUKICB0J9z7ANtv17eY2u8HAYKXUp0qpBUqpabHuSCl1vVJqsVJqcVVV1ZG12Ka+OQBAmtTchRAiTGcNqHqAMmAyMAP4k1IqO3InrfVTWutyrXV5fv7Rn+irrskPIDV3IYSI0JFw3wH0s13va26z2w7M1Fq3aK03Aesxwr5L1TVLuAshRCwdCfdFQJlSaoBSKgm4ApgZsc+/MXrtKKXyMMo0GzuxnTHVNRllGZkKKYQQ4doNd621H7gZmA2sAV7WWq9SSt2jlLrI3G02sE8ptRqYC/xUa72vqxptqZeeuxBCxNShVNRavwW8FbHtbttlDfzI/HfMhGruMltGCCHCOHqFqlWWkXnuQggRztHhbpVlUqTmLoQQYRwd7s0BDUCyW8JdCCHsHB3uLYEgAF6PnMtdCCHsHB3ufivc3Y4+DCGE6HSOTkWrLONxSc9dCCHsHB3uLYEgXreSr9gTQogIjg53fyCIx+XoQxBCiC7h6GRsCWi8bum1CyFEJIeHe5Akj6MPQQghuoSjk7FFyjJCCBGTo5OxJaBljrsQQsTg8HAP4pWeuxBCRHF0MhpTIR19CEII0SUcnYx+KcsIIURMjg73ZhlQFUKImBydjC2BIElSlhFCiCiOTkZ/QOORRUxCCBHF0eEuA6pCCBGbo5PROP2Aow9BCCG6hKOT0TorpBBCiHAJEO6OPgQhhOgSjk7GFhlQFUKImBwe7jIVUgghYnF0MvqDMqAqhBCxODoZW/xBKcsIIUQMjg73ZinLCCFETI5ORn9QBlSFECIWx4Z7MKgJSM1dCCFicmwytgSDABLuQggRg2OTsSWgAWSFqhBCxODccPdLz10IIeJxbDJaZRmPhLsQQkRxbDJaZZkkKcsIIUSUDoW7UmqaUmqdUqpCKXV7G/tdopTSSqnyzmtibP6AlGWEECKedpNRKeUGHgfOB4YDM5RSw2PslwHcAnzW2Y2MpSUgZRkhhIinI8k4HqjQWm/UWjcDLwIXx9jv18BvgMZObF9cUpYRQoj4OhLufYBttuvbzW0hSqmxQD+t9Zud2LY2hXruLum5CyFEpKNORqWUC3gQ+HEH9r1eKbVYKbW4qqrqqB7XCnevR8JdCCEidSQZdwD9bNf7mtssGcCJwAdKqc3ABGBmrEFVrfVTWutyrXV5fn7+kbcaWcQkhBBt6Ui4LwLKlFIDlFJJwBXATOtGrXW11jpPa12qtS4FFgAXaa0Xd0mLTS0yW0YIIeJqNxm11n7gZmA2sAZ4WWu9Sil1j1Lqoq5uYDz+UM9dwl0IISJ5OrKT1vot4K2IbXfH2Xfy0Terfc2hAVUpywghRCTHdnutskySDKgKIUQUxyajlGWEECI+xyajlGWEECI+x4a71XOXsowQQkRzbDK2SM9dCCHicny4ywpVIYSI5thkbD1xmGMPQQghuoxjk1HKMkIIEZ9jw90fCKIUuCXchRAiimPDvTmg8bpcKCXhLoQQkRwb7i2BoJwRUggh4nBsuPsDQZkpI4QQcTg2HZsDWr6FSQgh4nBsOvoDQfn+VCGEiMOx4d4SCOKROe5CCBGTI9OxoTlAXXNABlSFECKODn1Zx/Fm2N1vAzC0KKObWyKEEMcnR/bcLXIudyGEiM3R6ShlGSGEiM3R4S4DqkIIEZuj01HOCCmEELE5Oh2lLCOEELE5OtylLCOEELE5Oh2lLCOEELE5Oh09UpYRQoiYHB3uMs9dCCFic3Q6SrgLIURsjktHrXXossyWEUKI2BwX7sHWbJeeuxBCxOG4dAzY0l0GVIUQIjZHh7tMhRRCiNgcl44BW83dpaTnLoQQsTgv3AO6/Z2EEKKHc16423ruEvNCCBGb48LdHwy2XtES70IIEUuHwl0pNU0ptU4pVaGUuj3G7T9SSq1WSi1XSs1RSvXv/KYa7NkuhBAitnbDXSnlBh4HzgeGAzOUUsMjdlsKlGutRwGvAA90dkMtfkl3IYRoV0d67uOBCq31Rq11M/AicLF9B631XK11vXl1AdC3c5vZKqwq01UPIoQQDteRcO8DbLNd325ui+c6YFasG5RS1yulFiulFldVVXW8lTYBqbMLIUS7OnVAVSl1FVAO/DbW7Vrrp7TW5Vrr8vz8/CN6jICt6y45L4QQsXk6sM8OoJ/tel9zWxil1BTgLuBMrXVT5zQvWiCsLCPpLoQQsXSk574IKFNKDVBKJQFXADPtOyilxgB/BC7SWu/p/Ga2sg+oygpVIYSIrd1w11r7gZuB2cAa4GWt9Sql1D1KqYvM3X4LpAP/VEp9oZSaGefujpqV7YWZyVx/xsCuehghhHC0jpRl0Fq/BbwVse1u2+UpndyuuKye+/1fH0WGz3usHlYIIRzFcStUg+YoqtslJRkhhIjHceHuD0i4CyFEexwX7tY8dxlMFUKI+BwX7taAqnwLkxBCxOe4cLcGVKXnLoQQ8Tku3K0BVY/U3IUQIi7HhbsMqAohRPscF+4yFVIIIdrnuHD3ByXchRCiPY4L94CEuxBCtMu54S6zZYQQIi7nhrv03IUQIi4JdyGESEDOC3eZLSOEEO1yXLgHpecuhBDtcly4+2VAVQgh2uW4cA/V3OXEYUIIEZdzw1167kIIEZfjwn1AXhrTRxbJKX+FEKINHfoO1ePJ1BFFTB1R1N3NEEKI45rjeu5CCCHaJ+EuhBAJSMJdCCESkIS7EEIkIAl3IYRIQBLuQgiRgCTchRAiAUm4CyFEAlLaPIXuMX9gpaqALUf443nA3k5sjhPIMfcMcsw9w9Ecc3+tdX57O3VbuB8NpdRirXV5d7fjWJJj7hnkmHuGY3HMUpYRQogEJOEuhBAJyKnh/lR3N6AbyDH3DHLMPUOXH7Mja+5CCCHa5tSeuxBCiDY4LtyVUtOUUuuUUhVKqdu7uz1HQyn1Z6XUHqXUStu2HKXUu0qpDeb/vcztSin1qHncy5VSY20/c425/wal1DXdcSwdoZTqp5Saq5RarZRapZS6xdyeyMfsU0otVEotM4/5V+b2AUqpz8xje0kplWRuTzavV5i3l9ru6w5z+zql1Hndc0Qdp5RyK6WWKqXeMK8n9DErpTYrpVYopb5QSi02t3Xfa1tr7Zh/gBv4EhgIJAHLgOHd3a6jOJ4zgLHAStu2B4Dbzcu3A78xL08HZgEKmAB8Zm7PATaa//cyL/fq7mOLc7zFwFjzcgawHhie4MesgHTzshf4zDyWl4ErzO1PAjeYl28EnjQvXwG8ZF4ebr7ek4EB5t+Bu7uPr51j/xHwD+AN83pCHzOwGciL2NZtr+1uf0IO88k7FZhtu34HcEd3t+soj6k0ItzXAcXm5WJgnXn5j8CMyP2AGcAfbdvD9jue/wGvA+f2lGMGUoHPgVMwFrB4zO2h1zUwGzjVvOwx91ORr3X7fsfjP6AvMAc4G3jDPIZEP+ZY4d5tr22nlWX6ANts17eb2xJJoda60ry8Cyg0L8c7dkc+J+ZH7zEYPdmEPmazPPEFsAd4F6MHelBr7Td3sbc/dGzm7dVALg47ZuBh4DYgaF7PJfGPWQPvKKWWKKWuN7d122vbcd+h2pNorbVSKuGmMyml0oFXgR9qrWuUav2y80Q8Zq11ADhJKZUNvAYM7eYmdSml1IXAHq31EqXU5O5uzzF0utZ6h1KqAHhXKbXWfuOxfm07ree+A+hnu97X3JZIdiuligHM//eY2+Mdu6OeE6WUFyPY/661/pe5OaGP2aK1PgjMxShJZCulrM6Vvf2hYzNvzwL24axjPg24SCm1GXgRozTzCIl9zGitd5j/78F4Ex9PN762nRbui4Ayc9Q9CWPwZWY3t6mzzQSsEfJrMOrS1vZvmaPsE4Bq8+PebGCqUqqXORI/1dx23FFGF/0ZYI3W+kHbTYl8zPlmjx2lVArGGMMajJC/1Nwt8pit5+JS4H1tFF9nAleYM0sGAGXAwmNzFIdHa32H1rqv1roU42/0fa31lSTwMSul0pRSGdZljNfkSrrztd3dgxBHMGgxHWOWxZfAXd3dnqM8lheASqAFo7Z2HUatcQ6wAXgPyDH3VcDj5nGvAMpt9/MdoML8d213H1cbx3s6Rl1yOfCF+W96gh/zKGCpecwrgbvN7QMxgqoC+CeQbG73mdcrzNsH2u7rLvO5WAec393H1sHjn0zrbJmEPWbz2JaZ/1ZZ2dSdr21ZoSqEEAnIaWUZIYQQHSDhLoQQCUjCXQghEpCEuxBCJCAJdyGESEAS7kIIkYAk3IUQIgFJuAshRAL6/25B+iC0Wu/sAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"plt.plot(steps, acc);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"That's it for a quick overview. Now let's look in detail at each component, and introduce the low-level REST API as well."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Workspaces\n",
"\n",
"By default, comet_api.get() reports only your workspace names:"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['dsblank']"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"comet_api.get()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can also interate over those names:"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"dsblank\n"
]
}
],
"source": [
"for workspace in comet_api.get():\n",
" print(workspace)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As we saw above, you can also access other public workspaces as well:"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['fasttext', 'comet-notebooks', 'parameter-space-exploration', 'home-credit']"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"comet_api.get(\"cometpublic\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Projects\n",
"\n",
"Under get(WORKSPACE_NAME), you'll find the projects:"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['fasttext', 'comet-notebooks', 'parameter-space-exploration', 'home-credit']"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"comet_api.get(\"cometpublic\")"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [],
"source": [
"project = comet_api.get(\"cometpublic\", \"comet-notebooks\")\n",
"## OR:\n",
"#project = comet_api.get(\"cometpublic/comet-notebooks\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you just print out, or iterate over a project, you get access to the experiment ids:"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['7092a5e4c362453fb0b3f06785a1d30c', 'example 001']"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"project"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'7092a5e4c362453fb0b3f06785a1d30c'"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"project[0].experiment_key"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'d21f94a1c71841d2961da1e6ddb5ab20'"
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"project[1].experiment_key"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"However, you can also access all of the project data via the `.data` property:"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'project_id': '63776506878949eb9ac225a2a24b87f6',\n",
" 'user_name': 'cometpublic',\n",
" 'project_name': 'comet-notebooks',\n",
" 'project_desc': '',\n",
" 'team_id': 'cometpublic-default',\n",
" 'is_owner': False,\n",
" 'is_public': True,\n",
" 'is_shared': False,\n",
" 'num_of_experiments': 2,\n",
" 'last_updated': 1544540181888,\n",
" 'team_name': 'cometpublic',\n",
" 'experiments': {'7092a5e4c362453fb0b3f06785a1d30c': {'code_sha': '0193bacf',\n",
" 'file_name': 'Jupyter interactive',\n",
" 'file_path': 'Jupyter interactive',\n",
" 'duration_millis': 123000,\n",
" 'start_server_timestamp': 1542824533698,\n",
" 'end_server_timestamp': 1542824657041,\n",
" 'has_images': True,\n",
" 'experiment_key': '7092a5e4c362453fb0b3f06785a1d30c',\n",
" 'is_key': True},\n",
" 'd21f94a1c71841d2961da1e6ddb5ab20': {'code_sha': '66e8551f',\n",
" 'file_name': 'Jupyter interactive',\n",
" 'file_path': 'Jupyter interactive',\n",
" 'duration_millis': 1716489000,\n",
" 'start_server_timestamp': 1542823692825,\n",
" 'end_server_timestamp': 1544540181881,\n",
" 'has_images': False,\n",
" 'experiment_key': 'd21f94a1c71841d2961da1e6ddb5ab20',\n",
" 'is_key': True,\n",
" '_other': [{'name': 'Name',\n",
" 'valueMax': 'example 001',\n",
" 'valueMin': 'example 001',\n",
" 'valueCurrent': 'example 001',\n",
" 'timestampMax': 1544540174200,\n",
" 'timestampMin': 1544540174200,\n",
" 'timestampCurrent': 1544540174200}],\n",
" 'other': ['Name'],\n",
" '_name': 'example 001'},\n",
" 'example 001': {'code_sha': '66e8551f',\n",
" 'file_name': 'Jupyter interactive',\n",
" 'file_path': 'Jupyter interactive',\n",
" 'duration_millis': 1716489000,\n",
" 'start_server_timestamp': 1542823692825,\n",
" 'end_server_timestamp': 1544540181881,\n",
" 'has_images': False,\n",
" 'experiment_key': 'd21f94a1c71841d2961da1e6ddb5ab20',\n",
" 'is_key': False,\n",
" '_other': [{'name': 'Name',\n",
" 'valueMax': 'example 001',\n",
" 'valueMin': 'example 001',\n",
" 'valueCurrent': 'example 001',\n",
" 'timestampMax': 1544540174200,\n",
" 'timestampMin': 1544540174200,\n",
" 'timestampCurrent': 1544540174200}],\n",
" 'other': ['Name']}}}"
]
},
"execution_count": 30,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"project.data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And that's everything there is to a project object:"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Help on APIExperiments in module comet_ml.api object:\n",
"\n",
"class APIExperiments(builtins.object)\n",
" | Methods defined here:\n",
" | \n",
" | __getitem__(self, item)\n",
" | \n",
" | __init__(self, api, workspace, project)\n",
" | Initialize self. See help(type(self)) for accurate signature.\n",
" | \n",
" | __iter__(self)\n",
" | \n",
" | __len__(self)\n",
" | \n",
" | __repr__(self)\n",
" | Return repr(self).\n",
" | \n",
" | ----------------------------------------------------------------------\n",
" | Data descriptors defined here:\n",
" | \n",
" | __dict__\n",
" | dictionary for instance variables (if defined)\n",
" | \n",
" | __weakref__\n",
" | list of weak references to the object (if defined)\n",
" | \n",
" | data\n",
" | The project data in JSON format.\n",
"\n"
]
}
],
"source": [
"help(project)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Experiments\n",
"\n",
"Continuing with the dictionary-like access, you can see and iterate over the experiment ids:"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['7092a5e4c362453fb0b3f06785a1d30c', 'example 001']"
]
},
"execution_count": 32,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"comet_api.get(\"cometpublic\", \"comet-notebooks\")"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [],
"source": [
"exp = comet_api.get(\"cometpublic\", \"comet-notebooks\", 'd21f94a1c71841d2961da1e6ddb5ab20')\n",
"## OR\n",
"# exp = comet_api.get(\"cometpublic/comet-notebooks/d21f94a1c71841d2961da1e6ddb5ab20\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Regular Expression Experiment Name Matching\n",
"\n",
"You can also use regular expressions as the name for the experiment:"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[<APIExperiment 'cometpublic/comet-notebooks/example 001'>]"
]
},
"execution_count": 34,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"comet_api.get(\"cometpublic\", \"comet-notebooks\", \"example.*\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Experiment Properties"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"APIExperiments also have a `.data` property:"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'code_sha': '66e8551f',\n",
" 'file_name': 'Jupyter interactive',\n",
" 'file_path': 'Jupyter interactive',\n",
" 'duration_millis': 1716489000,\n",
" 'start_server_timestamp': 1542823692825,\n",
" 'end_server_timestamp': 1544540181881,\n",
" 'has_images': False,\n",
" 'experiment_key': 'd21f94a1c71841d2961da1e6ddb5ab20',\n",
" 'is_key': True,\n",
" '_other': [{'name': 'Name',\n",
" 'valueMax': 'example 001',\n",
" 'valueMin': 'example 001',\n",
" 'valueCurrent': 'example 001',\n",
" 'timestampMax': 1544540174200,\n",
" 'timestampMin': 1544540174200,\n",
" 'timestampCurrent': 1544540174200}],\n",
" 'other': ['Name'],\n",
" '_name': 'example 001',\n",
" 'metrics': ['train_accuracy',\n",
" 'train_curr_epoch',\n",
" 'train_loss',\n",
" 'train_val_loss'],\n",
" 'parameters': ['f']}"
]
},
"execution_count": 35,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"exp.data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this brief dictionary representation, you will see that `other`, `metrics` and `parameters` give a list of names. However, as we saw above, you can get more information through properties of those same names:\n",
"\n",
"names through exp.data[\"properties\"] and more detail at exp.properties:"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'name': 'f',\n",
" 'valueMax': '/run/user/1000/jupyter/kernel-b1c4403c-c673-4e7f-90dc-853616d231e2.json',\n",
" 'valueMin': '/run/user/1000/jupyter/kernel-b1c4403c-c673-4e7f-90dc-853616d231e2.json',\n",
" 'valueCurrent': '/run/user/1000/jupyter/kernel-b1c4403c-c673-4e7f-90dc-853616d231e2.json',\n",
" 'timestampMax': 1542823692798,\n",
" 'timestampMin': 1542823692798,\n",
" 'timestampCurrent': 1542823692798}"
]
},
"execution_count": 36,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"exp.parameters[0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"names through exp.data[\"other\"] and more detail at exp.other:"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"('Name', 'example 001')"
]
},
"execution_count": 37,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"exp.other[0][\"name\"], exp.other[0][\"valueCurrent\"]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"names through exp.data[\"metrics\"] and more detail at exp.metrics:"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'name': 'train_accuracy',\n",
" 'valueMax': '0.0',\n",
" 'valueMin': '0.0',\n",
" 'valueCurrent': '0.0',\n",
" 'timestampMax': 1542823699380,\n",
" 'timestampMin': 1542823699380,\n",
" 'timestampCurrent': 1542823704609,\n",
" 'runContextMax': 'train',\n",
" 'runContextMin': 'train',\n",
" 'runContextCurrent': 'train',\n",
" 'stepMax': 1,\n",
" 'stepMin': 1,\n",
" 'stepCurrent': 3},\n",
" {'name': 'train_curr_epoch',\n",
" 'valueMax': '1.0',\n",
" 'valueMin': '0.0',\n",
" 'valueCurrent': '1.0',\n",
" 'timestampMax': 1542823693885,\n",
" 'timestampMin': 1542823693885,\n",
" 'timestampCurrent': 1542823699380,\n",
" 'runContextMax': 'train',\n",
" 'runContextMin': 'train',\n",
" 'runContextCurrent': 'train'},\n",
" {'name': 'train_loss',\n",
" 'valueMax': '2.3872790336608887',\n",
" 'valueMin': '2.3872790336608887',\n",
" 'valueCurrent': '2.3872790336608887',\n",
" 'timestampMax': 1542823695820,\n",
" 'timestampMin': 1542823695820,\n",
" 'timestampCurrent': 1542823695820,\n",
" 'runContextMax': 'train',\n",
" 'runContextMin': 'train',\n",
" 'runContextCurrent': 'train',\n",
" 'stepMax': 0,\n",
" 'stepMin': 0,\n",
" 'stepCurrent': 0},\n",
" {'name': 'train_val_loss',\n",
" 'valueMax': '2.3066751956939697',\n",
" 'valueMin': '2.2724671363830566',\n",
" 'valueCurrent': '2.2724671363830566',\n",
" 'timestampMax': 1542823699379,\n",
" 'timestampMin': 1542823699379,\n",
" 'timestampCurrent': 1542823704609,\n",
" 'runContextMax': 'train',\n",
" 'runContextMin': 'train',\n",
" 'runContextCurrent': 'train',\n",
" 'stepMax': 1,\n",
" 'stepMin': 3,\n",
" 'stepCurrent': 3}]"
]
},
"execution_count": 38,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"exp.metrics"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can see all of the methods and propeties on an experiment instance:"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Help on APIExperiment in module comet_ml.api object:\n",
"\n",
"class APIExperiment(builtins.object)\n",
" | Methods defined here:\n",
" | \n",
" | __init__(self, api, workspace, project, experiment_key)\n",
" | REST API Experiment interface.\n",
" | \n",
" | __repr__(self)\n",
" | Return repr(self).\n",
" | \n",
" | display(self, clear=False, wait=True, new=0, autoraise=True)\n",
" | Show the comet.ml experiment page in an IFrame in a\n",
" | Jupyter notebook or Jupyter lab, OR open a browser\n",
" | window or tab.\n",
" | \n",
" | For Jupyter environments:\n",
" | \n",
" | Args:\n",
" | clear: to clear the output area, use clear=True\n",
" | wait: to wait for the next displayed item, use\n",
" | wait=True (cuts down on flashing)\n",
" | \n",
" | For non-Jupyter environments:\n",
" | \n",
" | Args:\n",
" | new: open a new browser window if new=1, otherwise re-use\n",
" | existing window/tab\n",
" | autoraise: make the browser tab/window active\n",
" | \n",
" | get_asset(self, asset_id)\n",
" | Get an asset from this experiment. Not cached.\n",
" | \n",
" | ----------------------------------------------------------------------\n",
" | Data descriptors defined here:\n",
" | \n",
" | __dict__\n",
" | dictionary for instance variables (if defined)\n",
" | \n",
" | __weakref__\n",
" | list of weak references to the object (if defined)\n",
" | \n",
" | asset_list\n",
" | Get the asssociated asset-list for this experiment. Not cached.\n",
" | \n",
" | code\n",
" | Get the associated source code for this experiment. Not cached.\n",
" | \n",
" | data\n",
" | The experiment data in JSON-like format.\n",
" | \n",
" | existing_experiment\n",
" | Get an ExistingExperiment() object for this\n",
" | experiment.\n",
" | \n",
" | git_metadata\n",
" | Get the asssociated git-metadata for this experiment. Not cached.\n",
" | \n",
" | git_patch\n",
" | Get the asssociated git-patch for this experiment. Not cached.\n",
" | \n",
" | graph\n",
" | Get the associated graph/model description for this\n",
" | experiment. Not cached.\n",
" | \n",
" | html\n",
" | Get the HTML associated with this experiment. Not cached.\n",
" | \n",
" | images\n",
" | Get the associated image data for this experiment. Not cached.\n",
" | \n",
" | The image data comes as a dictionary with the following\n",
" | keys:\n",
" | \n",
" | apiKey\n",
" | runId\n",
" | experimentKey\n",
" | projectId\n",
" | figCounter\n",
" | figName\n",
" | step\n",
" | runContext\n",
" | fileName\n",
" | imagePath\n",
" | \n",
" | installed_packages\n",
" | Get the associated installed packages for this experiment. Not cached.\n",
" | \n",
" | metrics\n",
" | Get the asssociated metrics for this experiment. Not cached.\n",
" | \n",
" | metrics_raw\n",
" | Get the asssociated raw metrics for this experiment. Not cached.\n",
" | \n",
" | os_packages\n",
" | Get the associated installed packages for this experiment. Not cached.\n",
" | \n",
" | other\n",
" | Get the asssociated other items (things logged with `log_other`)\n",
" | for this experiment. Cached.\n",
" | \n",
" | parameters\n",
" | Get the asssociated parameters for this experiment. Not cached.\n",
" | \n",
" | stdout\n",
" | Get the associated standard output for this experiment. Not cached.\n",
"\n"
]
}
],
"source": [
"help(exp)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For example, just like when creating and logging data, you can also use the `.display()` method to show the Comet.ml page for that experiment right in the notebook:"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" <iframe\n",
" width=\"100%\"\n",
" height=\"800px\"\n",
" src=\"https://www.comet.ml/cometpublic/comet-notebooks/d21f94a1c71841d2961da1e6ddb5ab20\"\n",
" frameborder=\"0\"\n",
" allowfullscreen\n",
" ></iframe>\n",
" "
],
"text/plain": [
"<IPython.lib.display.IFrame at 0x7f50076c7e10>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"exp.display()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can get an existing experiment:"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"COMET INFO: old comet version (1.0.39) detected. current: 1.0.40 please update your comet lib with command: `pip install --no-cache-dir --upgrade comet_ml`\n",
"COMET INFO: Experiment is live on comet.ml https://www.comet.ml/dsblank/chainer/596d91ae1dbc420c9b13a3ced858de3c\n",
"\n"
]
}
],
"source": [
"e = comet_api.get(\"dsblank\", \"chainer\", \"596d91ae1dbc420c9b13a3ced858de3c\")\n",
"ee = e.existing_experiment"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can make changes to the saved data using the existing experiment:\n",
"\n",
"https://www.comet.ml/docs/python-sdk/ExistingExperiment/"
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"COMET INFO: Uploading stats to Comet before program termination (may take several seconds)\n",
"COMET INFO: Experiment is live on comet.ml https://www.comet.ml/dsblank/chainer/596d91ae1dbc420c9b13a3ced858de3c\n",
"\n"
]
}
],
"source": [
"ee.end()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Examples\n",
"\n",
"Comet.ml is working on a query API which will allow highly efficient queries of your data. However, you can also write your own query of sorts.\n",
"\n",
"Here is some code that prints out the names of experiments that have associated HTML (this can take a long time if you have many experiments):"
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"processing workspace dsblank ...\n",
" processing project pytorch ...\n",
" processing project mnist-001 ...\n",
" processing project visualizations ...\n",
" processing project testing ...\n",
" processing project pypy ...\n",
" processing project tensorflow ...\n",
" processing project chainer ...\n",
"found html!\n",
"found html!\n",
"found html!\n",
"found html!\n",
"found html!\n",
"found html!\n",
"found html!\n",
"found html!\n",
"found html!\n",
"found html!\n",
"found html!\n",
"found html!\n",
"found html!\n",
"found html!\n",
" processing project mnist-014 ...\n",
" processing project optimizer ...\n",
" processing project general ...\n",
" processing project jupyter-experiments ...\n",
" processing project keras ...\n",
" processing project fastai ...\n",
"CPU times: user 2.78 s, sys: 86.7 ms, total: 2.87 s\n",
"Wall time: 17.5 s\n"
]
}
],
"source": [
"%%time\n",
"for workspace in comet_api.get():\n",
" print(\"processing workspace\", workspace, \"...\")\n",
" for project in comet_api.get(workspace):\n",
" print(\" processing project\", project, \"...\")\n",
" for exp_id in comet_api.get(workspace, project):\n",
" exp = comet_api.get(workspace, project, exp_id)\n",
" if exp.html != None:\n",
" print(\"found html!\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here is a function that will find the first experiment that has associated images:"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {},
"outputs": [],
"source": [
"def find_image():\n",
" for workspace in comet_api.get():\n",
" for project in comet_api.get(workspace):\n",
" for exp_id in comet_api.get(workspace, project):\n",
" exp = comet_api.get(workspace, project, exp_id)\n",
" if exp.images != []:\n",
" return exp"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<APIExperiment 'dsblank/pytorch/3b56611892b7447aa8c4486a6eeb27d0'>"
]
},
"execution_count": 45,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"find_image()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, we get the experiment API and explore the `.images` property:"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'apiKey': 'x0yluTXX4ysG4z1k0FaTAz3qj',\n",
" 'runId': 'EgBEDhAJMA6OxWXZ1dHv4AkSz',\n",
" 'experimentKey': '3b56611892b7447aa8c4486a6eeb27d0',\n",
" 'projectId': '1f1999d917104dbe8b6a17b2c28c171c',\n",
" 'figCounter': 0,\n",
" 'figName': 'Exponential Slowdown',\n",
" 'step': None,\n",
" 'runContext': None,\n",
" 'fileName': '4ykmnOHJFZkrRD36KDrTXvHJk.svg',\n",
" 'fileSize': 0,\n",
" 'imagePath': 'https://s3.amazonaws.com/comet.ml/4ykmnOHJFZkrRD36KDrTXvHJk.svg',\n",
" 'createdAt': 1542116223922}]"
]
},
"execution_count": 46,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"comet_api.get('dsblank/pytorch/3b56611892b7447aa8c4486a6eeb27d0').images"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can get a URL for the image, and display it in the notebook:"
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'https://s3.amazonaws.com/comet.ml/4ykmnOHJFZkrRD36KDrTXvHJk.svg'"
]
},
"execution_count": 47,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"url = comet_api.get('dsblank/pytorch/3b56611892b7447aa8c4486a6eeb27d0').images[0][\"imagePath\"]\n",
"url"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {},
"outputs": [],
"source": [
"from IPython.display import Image"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<img src=\"https://s3.amazonaws.com/comet.ml/4ykmnOHJFZkrRD36KDrTXvHJk.svg\"/>"
],
"text/plain": [
"<IPython.core.display.Image object>"
]
},
"execution_count": 49,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"Image(url=url)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, let's write a short program that will find the run with the best accuracy given a workspace/project string:"
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {},
"outputs": [],
"source": [
"def find_best_run(project):\n",
" runs = []\n",
" for exp_id in comet_api.get(project):\n",
" exp = comet_api.get(project, experiment=exp_id)\n",
" accs = [x[\"valueMax\"] for x in exp.metrics if x[\"name\"] == \"acc\"]\n",
" if len(accs) > 0:\n",
" runs.append([float(accs[0]), exp])\n",
" if runs:\n",
" return sorted(runs, key=lambda v: v[0], reverse=True)[0]"
]
},
{
"cell_type": "code",
"execution_count": 51,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[1.0, <APIExperiment 'cometpublic/fasttext/44ea2d68794d4c5e9e18b86c30562bf8'>]"
]
},
"execution_count": 51,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"find_best_run(\"cometpublic/fasttext\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Can we get all of the `hidden_size` parameter values for the experiments in dsblank/pytorch?"
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(<APIExperiment 'dsblank/pytorch/5f540fc7894146f4bbfd21ebe8549fcf'>,\n",
" 'hidden_size',\n",
" 128)"
]
},
"execution_count": 52,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"experiments = [[(exp, \"hidden_size\", int(param[\"valueCurrent\"])) for param in exp.parameters \n",
" if param[\"name\"] == \"hidden_size\"][0] \n",
" for exp in comet_api.get(\"dsblank/pytorch/.*\")]\n",
"experiments[0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Assets\n",
"\n",
"To get an asset, you need to get the asset_id. You can see all of the assets related to a project using the `APIExperiment.asset_list`:"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {},
"outputs": [],
"source": [
"for exp in comet_api.get('dsblank/visualizations/.*'):\n",
" if exp.asset_list != []:\n",
" print(exp)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"From there, you can use the `APIExperiment.get_asset(asset_id)` method to get the asset."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Low-level API\n",
"\n",
"If you would prefer a more direct, lower-level API, you can also use the following functions from the API() instance. See the full documentation in [REST API](../../rest-api/getting-started/).\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Utility functions\n",
"\n",
"* get_url(version=None)\n",
"* get_url_server(version=None)\n",
"* get_url_end_point(end_point, version=None)\n",
"* get_request(end_point, params)\n",
"* get_version(self)\n",
"\n",
"### Access functions\n",
"* get_workspaces(workspace_name=None)\n",
"* get_projects(workspace)\n",
"* get_experiment_keys(project_id)\n",
"* get_experiment_data(project_id)\n",
"* get_experiment(experiment_key)\n",
"* get_experiment_html(experiment_key)\n",
"* get_experiment_code(experiment_key)\n",
"* get_experiment_stdout(experiment_key)\n",
"* get_experiment_installed_packages(experiment_key)\n",
"* get_experiment_graph(experiment_key)\n",
"* get_experiment_images(experiment_key)\n",
"* get_experiment_parameters(experiment_key, param=None)\n",
"* get_experiment_metrics(experiment_key, metric=None)\n",
"* get_experiment_other(experiment_key, other=None, value=None)\n",
"* get_experiment_metrics_raw(experiment_key, metric=None)\n",
"* get_experiment_asset_list(experiment_key)\n",
"* get_experiment_asset(experiment_key, asset_id)\n",
"* get_experiment_git_patch(experiment_key)\n",
"* get_experiment_git_metadata(experiment_key)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We hope that this gives you some ideas of how you can use the Comet REST API!"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.5"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment