7.2 Splice MLflow Support.ipynb

{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Splice + MLflow: What you need to know\n## Just like the last notebook, maybe an image here of the Splice Logo with a <3 and the MLflow Logo\n<blockquote><p class='quotation'><span style='font-size:15px'>Mlflow allows you to track experiments and share results with teammates easily.<br>At Splice Machine, MLflow is embedded directly into your database (MLManager). This means that all of the configuration is taken care of for you, and <b>everything</b> you track in mlflow is persisted to the database.<br><br>\n    MLflow requires the NSDS (or ExtNSDS) as a parameter to connect to the database. If are unfamliar with our NSDS, check out the <a href=\"./7.1 Splice and Spark.ipynb\">previous notebook</a> on using Splice Machine and Spark.<footer>Splice Machine</footer>\n</blockquote>\n\n#### Let's start our Spark Session"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Setup\nfrom pyspark.sql import SparkSession\nfrom splicemachine.spark import PySpliceContext\n\nspark = SparkSession.builder.getOrCreate()\nsplice = PySpliceContext(spark)\n","execution_count":1,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Importing MLflow Support\n<blockquote><p class='quotation'><span style='font-size:15px'>Using MLflow on Splice is as easy as a single import. After imporing, you immediately have access to the <code>mlflow</code> module. <br>You will have access to all of the functions in the standard MLflow API as well as some extra ones that are custom to Splice Machine.<br> You can check out our full <a href='https://pysplice.readthedocs.io/en/dbaas-4100/splicemachine.mlflow_support.html'>documentation</a> for everything available and our <a href=\"https://www.github.com/splicemachine/pysplice\">GitHub</a> repo to raise issues and ask questions. <br>After importing, you can register your Splice Context for access to even more functions.<br><br><footer>Splice Machine</footer>\n</blockquote>"},{"metadata":{"trusted":true},"cell_type":"code","source":"# MLFlow Setup\nfrom splicemachine.mlflow_support import *\nmlflow.register_splice_context(splice)","execution_count":3,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Step 0: The MLflow UI\n<blockquote> You can access the MLflow UI in 2 ways:\n    <ul>\n        <li>From the url at <a href=/mlflow>/mlflow</a></li>\n        <li>From the Notebook as an IFrame using the <code>get_mlflow_ui</code> function. You can also pass in an optional experiment ID and/or run ID to open the IFrame directly to your experiment/run.</li>\n    </ul>\n</blockquote>"},{"metadata":{"trusted":true},"cell_type":"code","source":"from splicemachine.notebook import get_mlflow_ui\nget_mlflow_ui()","execution_count":4,"outputs":[{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.HTML object>","text/html":"<font size=\"+1\"><a target=\"_blank\" href=/mlflow/#/experiments/0>MLFlow UI</a></font>"},"metadata":{}},{"output_type":"execute_result","execution_count":4,"data":{"text/plain":"<IPython.lib.display.IFrame at 0x7f6d86fa0c50>","text/html":"\n        <iframe\n            width=\"100%\"\n            height=\"500px\"\n            src=\"/mlflow/#/experiments/0\"\n            frameborder=\"0\"\n            allowfullscreen\n        ></iframe>\n        "},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"## MLflow concepts\n<blockquote>MLflow Tracking is organized around the concept <code>experiments</code> and <code>runs</code>:<br> \n    <ul>\n        <li>Experiments can be thought of as the problem you are trying to track or solve (ie Performance Testing TPC-C</li>\n        <li>Runs are single executions of some piece of code (ie a single full execution of TPC-C with some database configuration). Experiments have multiple runs (1-to-many).</li>\n    </ul>\n</blockquote>"},{"metadata":{},"cell_type":"markdown","source":"### Setting an Experiment\n<blockquote>To start an Experiment, you can call <code>mlflow.set_experiment('EXP_NAME')</code> and pass in an experiment name.<br> \n    If the Experiment exists, it will be set to the <code>active</code> experiment. Otherwise, mlflow will create the Experiment for you and set it to active.\n\n</blockquote>"},{"metadata":{"trusted":true},"cell_type":"code","source":"help(mlflow.set_experiment)","execution_count":5,"outputs":[{"output_type":"stream","text":"Help on function set_experiment in module mlflow.tracking.fluent:\n\nset_experiment(experiment_name)\n    Set given experiment as active experiment. If experiment does not exist, create an experiment\n    with provided name.\n    \n    :param experiment_name: Name of experiment to be activated.\n\n","name":"stdout"}]},{"metadata":{"trusted":true},"cell_type":"code","source":"mlflow.set_experiment('mlflow_api_demo')","execution_count":6,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### View Your [Experiment](/mlflow)"},{"metadata":{"trusted":true},"cell_type":"code","source":"exp_id = mlflow.client.get_experiment_by_name('mlflow_api_demo').experiment_id\nget_mlflow_ui(exp_id)","execution_count":7,"outputs":[{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.HTML object>","text/html":"<font size=\"+1\"><a target=\"_blank\" href=/mlflow/#/experiments/26>MLFlow UI</a></font>"},"metadata":{}},{"output_type":"execute_result","execution_count":7,"data":{"text/plain":"<IPython.lib.display.IFrame at 0x7f6dee735b10>","text/html":"\n        <iframe\n            width=\"100%\"\n            height=\"500px\"\n            src=\"/mlflow/#/experiments/26\"\n            frameborder=\"0\"\n            allowfullscreen\n        ></iframe>\n        "},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"### Starting a run\n<blockquote>Once you have an Experiment, you can start your run by calling <code>mlflow.start_run(run_name='RUN_NAME')</code> and pass in a run name. You can also pass in the optional <code>tags</code> parameter as a dictionary and store key value pairs associated to the run.<br> \nWhen you start a run, MLFlow (MLManager) automatically logs some information for you:\n    <ul>\n        <li>Start Date</li>\n        <li>Current User</li>\n        <li>Run ID</li>\n        <li>DB Transaction ID</li>\n        <li>Source (where the run came from)</li>\n    </ul>\n</blockquote>"},{"metadata":{"trusted":true},"cell_type":"code","source":"help(mlflow.start_run)","execution_count":8,"outputs":[{"output_type":"stream","text":"Help on function _start_run in module splicemachine.mlflow_support.mlflow_support:\n\n_start_run(run_id=None, tags=None, experiment_id=None, run_name=None, nested=False)\n    Start a new run\n    :param tags: a dictionary containing metadata about the current run.\n        For example:\n            {\n                'team': 'pd',\n                'purpose': 'r&d'\n            }\n    :param run_name: an optional name for the run to show up in the MLFlow UI\n    :param run_id: if you want to reincarnate an existing run, pass in the run id\n    :param experiment_id: if you would like to create an experiment/use one for this run\n    :param nested: Controls whether run is nested in parent run. True creates a nest run\n\n","name":"stdout"}]},{"metadata":{"trusted":true},"cell_type":"code","source":"mlflow.start_run(run_name='First_pass_default_settings', tags={'team': 'pd', 'purpose':'performance testing'})","execution_count":16,"outputs":[{"output_type":"execute_result","execution_count":16,"data":{"text/plain":"<ActiveRun: >"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"### Tracking Concepts\n<blockquote>There are 4 main conepts when tracking a run:<br>\n    <ul>\n        <li><b>Tags</b>: Any key value pair that likely won't be used for comparison between runs (non-measurable items). Only tags can be overwritten</li>\n        <li><b>Parameters</b>: Configuration options that were made before starting the run that may have a measurable effect on the outcome</li>\n        <li><b>Metrics</b>: The measured outcomes between runs that can be compared. Metrics have an optional <code>step</code> parmeter if you want to track metrics over time for a specific run</li>\n        <li><b>Artifacts</b>: Objects (files, images, notebooks, etc) to be associated with a run</li>\n    </ul>\n</blockquote>"},{"metadata":{"trusted":true},"cell_type":"code","source":"help(mlflow.set_tag)\nprint('---------------------------------------------------------------------------------')\nhelp(mlflow.lp)\nprint('---------------------------------------------------------------------------------')\nhelp(mlflow.lm)\nprint('---------------------------------------------------------------------------------')\nhelp(mlflow.log_artifact)","execution_count":17,"outputs":[{"output_type":"stream","text":"Help on function set_tag in module mlflow.tracking.fluent:\n\nset_tag(key, value)\n    Set a tag under the current run. If no run is active, this method will create a\n    new active run.\n    \n    :param key: Tag name (string)\n    :param value: Tag value (string, but will be string-ified if not)\n\n---------------------------------------------------------------------------------\nHelp on function _lp in module splicemachine.mlflow_support.mlflow_support:\n\n_lp(key, value)\n    Add a shortcut for logging parameters in MLFlow.\n    Accessible from mlflow.lp\n    :param key: key for the parameter\n    :param value: value for the parameter\n\n---------------------------------------------------------------------------------\nHelp on function _lm in module splicemachine.mlflow_support.mlflow_support:\n\n_lm(key, value, step=None)\n    Add a shortcut for logging metrics in MLFlow.\n    Accessible from mlflow.lm\n    :param key: key for the parameter\n    :param value: value for the parameter\n\n---------------------------------------------------------------------------------\nHelp on function _log_artifact in module splicemachine.mlflow_support.mlflow_support:\n\n_log_artifact(file_name, name, run_uuid=None)\n    Log an artifact for the active run\n    :param file_name: (str) the name of the file name to log\n    :param name: (str) the name of the run relative name to store the model under\n    :param run_uuid: the run uuid of a previous run, if none, defaults to current run\n    NOTE: We do not currently support logging directories. If you would like to log a directory, please zip it first\n          and log the zip file\n\n","name":"stdout"}]},{"metadata":{"trusted":true},"cell_type":"code","source":"mlflow.set_tag('teammates', 'carol, daniel')","execution_count":18,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mlflow.lp('spark executors', '5')","execution_count":19,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mlflow.lm('execution time sec', 25)","execution_count":20,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Setting metrics over \"steps\"\nfor i in range(10):\n    mlflow.lm('Build time', i*3, step=i)","execution_count":21,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"get_mlflow_ui(mlflow.current_exp_id(), mlflow.current_run_id())","execution_count":22,"outputs":[{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.HTML object>","text/html":"<font size=\"+1\"><a target=\"_blank\" href=/mlflow/#/experiments/26/runs/8b0159a70a5c>MLFlow UI</a></font>"},"metadata":{}},{"output_type":"execute_result","execution_count":22,"data":{"text/plain":"<IPython.lib.display.IFrame at 0x7f6d8638c2d0>","text/html":"\n        <iframe\n            width=\"100%\"\n            height=\"500px\"\n            src=\"/mlflow/#/experiments/26/runs/8b0159a70a5c\"\n            frameborder=\"0\"\n            allowfullscreen\n        ></iframe>\n        "},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"### End Run\n<blockquote>When you finish a run, you call <code>mlflow.end_run()</code>.<br> You know a run is ended in the MLFlow UI because there is a green check mark next to it</blockquote>"},{"metadata":{"trusted":true},"cell_type":"code","source":"mlflow.end_run()","execution_count":23,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Artifacts\n<blockquote>Artifacts can be any file type. The artifact is serialized as a BLOB and stored in the database. When storing artifacts in the database, files with file extensions such as <code>.txt</code>, <code>.pdf</code>, <code>.yaml</code>, <code>.pdf</code>, <code>.jpeg</code> etc. will be available for preview in the mlflow ui <br>We can use some neat Jupyter tricks like <code>writefile</code> to make artifacts even more useful.\n</blockquote>\n\n#### Write a yaml file"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%writefile my_env.yaml\n\nname: datatest  \nchannels:\n- defaults\n- conda-forge\n- ericmjl\ndependencies:\n- python=3.6\n- colorama=0.3.9\n- jupyter=1.0.0\n- ipykernel=4.6.1\n- jupyterlab=0.25.2\n- pytest=3.1.3\n- pytest-cov=2.5.1\n- tinydb=3.3.1\n- pyyaml=3.12\n- pandas-summary=0.0.41\n- environment_kernels=1.1\n- missingno=0.3.7\n","execution_count":24,"outputs":[{"output_type":"stream","text":"Overwriting my_env.yaml\n","name":"stdout"}]},{"metadata":{},"cell_type":"markdown","source":"#### Write a code snippet"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%writefile harm_mean.py\ndef harm_mean(nums, rnd=4):\n    \"\"\"\n    Calculates the harmonic mean of n numbers rounded to rnd decimal places\n    :param nums: List of numbers\n    :param rnd: Number of decimal places to round the result\n    \"\"\"\n    return round(len(nums)/sum([1/i for i in nums]),rnd)","execution_count":25,"outputs":[{"output_type":"stream","text":"Overwriting harm_mean.py\n","name":"stdout"}]},{"metadata":{},"cell_type":"markdown","source":"## Put it together\n#### Start a run, log our artifacts, view the results"},{"metadata":{"trusted":true},"cell_type":"code","source":"!jupyter nbconvert --to html '7.2 Splice MLflow Support.ipynb'\nwith mlflow.start_run(run_name='environment_requirements'):\n    run_id = mlflow.current_run_id()\n    exp_id = mlflow.current_exp_id()\n    mlflow.log_artifact('my_env.yaml', name='my_env.yaml')\n    mlflow.log_artifact('harm_mean.py', name='harm_mean.py')\n    mlflow.log_artifact('7.2 Splice MLflow Support.ipynb', name='training_notebook.ipynb')\n    mlflow.log_artifact('7.2 Splice MLflow Support.html', name='training_notebook.html')","execution_count":26,"outputs":[{"output_type":"stream","text":"[NbConvertApp] Converting notebook 7.2 Splice MLflow Support.ipynb to html\n[NbConvertApp] Writing 358977 bytes to 7.2 Splice MLflow Support.html\nSaving artifact of size: 0.349 KB to Splice Machine DB\nSaving artifact of size: 0.328 KB to Splice Machine DB\nSaving artifact of size: 53.573 KB to Splice Machine DB\nSaving artifact of size: 359.035 KB to Splice Machine DB\n","name":"stdout"}]},{"metadata":{},"cell_type":"markdown","source":"#### Click on one of your artifacts to render the results!"},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"get_mlflow_ui(exp_id, run_id)","execution_count":27,"outputs":[{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.HTML object>","text/html":"<font size=\"+1\"><a target=\"_blank\" href=/mlflow/#/experiments/26/runs/26342e253ff0>MLFlow UI</a></font>"},"metadata":{}},{"output_type":"execute_result","execution_count":27,"data":{"text/plain":"<IPython.lib.display.IFrame at 0x7f6d86390290>","text/html":"\n        <iframe\n            width=\"100%\"\n            height=\"500px\"\n            src=\"/mlflow/#/experiments/26/runs/26342e253ff0\"\n            frameborder=\"0\"\n            allowfullscreen\n        ></iframe>\n        "},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"#### Another Artifact Example"},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nfrom random import random\nwith mlflow.start_run(run_name='my_plot'):\n    plt.rcParams.update({\n        \"pgf.texsystem\": \"pdflatex\",\n        \"pgf.preamble\": [\n             r\"\\usepackage[utf8x]{inputenc}\",\n             r\"\\usepackage[T1]{fontenc}\",\n             r\"\\usepackage{cmbright}\",\n             ]\n    })\n\n    plt.figure(figsize=(4.5, 2.5))\n    plt.plot([random()*19 for _ in range(10)])\n    plt.text(0.5, 3., \"serif\", family=\"serif\")\n    plt.text(0.5, 2., \"monospace\", family=\"monospace\")\n    plt.text(2.5, 2., \"sans-serif\", family=\"sans-serif\")\n    plt.xlabel(r\"µ is not $\\mu$\")\n    plt.tight_layout(.5)\n\n    plt.savefig(\"pgf_texsystem.png\")\n    mlflow.log_artifact('pgf_texsystem.png', 'results.png')\n    display(get_mlflow_ui(mlflow.current_exp_id(), mlflow.current_run_id()))","execution_count":28,"outputs":[{"output_type":"stream","text":"/opt/conda/lib/python3.7/site-packages/matplotlib/backend_bases.py:57: DeprecationWarning: PILLOW_VERSION is deprecated and will be removed in a future release. Use __version__ instead.\n  from PIL import PILLOW_VERSION\n","name":"stderr"},{"output_type":"stream","text":"Saving artifact of size: 11.711 KB to Splice Machine DB\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.HTML object>","text/html":"<font size=\"+1\"><a target=\"_blank\" href=/mlflow/#/experiments/26/runs/7c10c68d11b3>MLFlow UI</a></font>"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"<IPython.lib.display.IFrame at 0x7f6d85bab550>","text/html":"\n        <iframe\n            width=\"100%\"\n            height=\"500px\"\n            src=\"/mlflow/#/experiments/26/runs/7c10c68d11b3\"\n            frameborder=\"0\"\n            allowfullscreen\n        ></iframe>\n        "},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"<Figure size 324x180 with 1 Axes>","image/png":"iVBORw0KGgoAAAANSUhEUgAAAUgAAAC4CAYAAACB8XWeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deVxVdf7H8deXXUAUBVSEQEUEZFVwzVyyrFxTq8mxbVLbzBqnUqfSlqnRalrMsmmfxMwSTVNTM3cTFxBwQ0UEBWTT2ES2e7+/P0B+LVxluTvf5+PBA7ice87n4uXtOee7CSkliqIoyp/ZmLoARVEUc6UCUlEURQcVkIqiKDqogFQURdFBBaSiKIoOKiAVRVF0sDPmwTw8PKS/v78xD6koinJNCQkJhVJKz4Z+ZtSA9Pf359ChQ8Y8pKIoyjUJITJ1/UxdYiuKouigAlJRFEUHFZCKoig6GPUepKKY2rtbTyEQzLo5ACGEqctRzJwKSKXVKCit5P1taWi0kpyiK7w+MQxbGxWSim7qEltpNdYmZaPRSib39WHlofM8vTKJao3W1GUpZkydQSqtRlxiNhE+7XjrrggCvFxZ+GMqV6o0LJkShZO9ranLU8yQOoNUWoXjOSWcuFDCpL4+ADw6tAevju/N1hN5PPy/g5RX1Zi4QsUcqYBUWoXViVnY2wrGhnvXP3bfQH/+c1cE+85c5L7PDlB8pdqEFSrmSAWkYvVqNFq+T8phRJAX7i4Ov/vZpL4+fDClDylZRUz5JJ5Ll6tMVKVijlRAKlZv1+kCCssqmdTHp8Gf3x7WhY/vjyYtv4x7/ruPvJIKI1eomCsVkIrVi0vMxt3ZnmG9vHRuM7yXF18+1I+coivc9dE+zl8qN2KFirlSAalYteLyan46nsf4yK442F377T6wR0dip/WnqLyKu/+7jzMFZUaqUjFXKiAVq7bhyAWqarRM7NO1UdtH3eDONzMGUlWj5Z7/7uPEhRIDV6iYMxWQilWLS8yip5crYV3bNfo5Id5ufPvoQOxtbfjLx/EknS8yYIWKOVMBqVits4WXScj8lUl9fZo87rqHpyvfPjKQdm3s+esn8cSnXzRQlYo5UwGpWK01iVnYCJgQ2bjL6z/y7eDMd48OxLt9Gx74/ADbT+bruULF3F03IIUQnwsh8oUQR3/z2EtCiGwhRFLdxx2GLVNRmkarlcQlZjM4wIPO7ZyavZ9Obk6sfGQgAV6uzPjqED8euaDHKhVz15gzyC+B2xp4/B0pZWTdx0b9lqUoLbP/7CWyi64wuW/DfR+booOLA19PH0C4T3ue+DqRuIQsPVSoWILrBqSUchdwyQi1KIrerE7MwtXRjltDOutlf+3a2PPV3/oxsEdH/vFdMsvidS5joliRltyDnCmESKm7BHfXtZEQYoYQ4pAQ4lBBQUELDqcojVNeVcPGIxe4I6wzbRz0N0uPi6Mdnz0Qw8hgL178/ij/3XlGb/tWzFNzA3Ip0AOIBC4A/9G1oZTyYylltJQy2tOzwZUVFUWvNh/L5XKVhok6hha2hJO9LUun9mVMeBf+/WMqb285iZRS78dRzEOz5oOUUuZd/VoI8QmwXm8VKUoLrU7Mxse9Df38Oxhk//a2Nrz3lyhcHOxYvC2NskoNL44JVks4WKFmBaQQoouU8mpz3p3A0WttryjGcqH4CnvSCnlyRE9sDLicgq2N4N8Tw2jjYMvne89SXlXDa3eqJRyszXUDUgixAhgGeAghsoAFwDAhRCQggQzgEQPWqCiN9v3hHKSEiVHN6/vYFDY2ggVjQ3B1tGPJ9jTKqzT85+4I7G1V92Jrcd2AlFLe28DDnxmgFkVpESklcYlZRPu54+/hYpRjCiF4ZlQvXBztWLQplSvVGt6/Vy3hYC3Uf3WK1UjJKiYtv6x+WQVjemxYD14Z35ufjucx7X+H1BIOVkIFpGI1Vidm4WBnwx1hXUxy/PsH+vPWXRH8cqaQ+z87QEmFWsLB0qmAVKxCVY2Wdck53BrSiXZt7E1Wx+S+Prx/bx+SzqslHKyBCkjFKmxLzefX8mqTXF7/0ejwLnxyfzSn82qXcMhXSzhYLBWQilVYnZiFZ1tHhgR4mLoUAIYHefHFQzFkF13hrv/uI+tXtYSDJVIBqVi8S5er2H4ynwmR3tiZURebQT08iJ3Wn18vV3HXR/s4d1GFpKUxn3eTYhKx8ZkM+vfPJFvwrNk/JOdQrZEGGVrYUn3qlnAoq6zhH98lodWqYYmWRAVkK1VVo+Wfa47wwvdHuVBSwYJ1xyz2jzcuMYuQLm4Ed3EzdSkNCvF2Y/6YEA5m/MpX+zJMXY7SBCogW6HCskr++mk8X+8/x+PDerBoYjhJ54tYm5xt6tKa7HReKSlZxWbROHMtk/v6MKyXJ4s2nVSX2hZEBWQrczS7mHHv7+FIdjGL743iuduCmNzXhwifdiz8MZXLlZbVwTkuMRtbG8G4CG9Tl3JNQgherxurPScuxWLP1lsbFZCtyLrkHCZ/9AsAqx4dVB8qNjaC+WN7k1dSyUcWNMehRitZcziLYYGeeLZ1NHU51+Xdvg3/vCOYfekXWXHwnKnLURpBBWQroNFKFm1KZdaKw4R1bce6J28k9A/LoPb1c2d8pDf/3ZXO+UuWcQm4N62QvJJKs7+8/q17+/kyOKAj/96YSnbRFVOXo1yHCkgrV1JRzbT/HWTpjjNM6X8Dy6cNwMO14bOtObcFYSNg4Y+pRq6yeVYnZuHmZMeIIC9Tl9JoQggWTgxHKyVz41LUZLtmTgWkFTtTUMaED/ay+3Qh/5oQyut3huFgp/uf3Lt9Gx4bGsCGIxfMfh3o0opqNh3LZWyEt8XNnOPbwZk5twWx+3Qh36kFwMyaCkgrtT01nwlL9lJcXs3yaf2ZOsCvUc+bcVN3vNs58coPx9GYcUPCj0dyqajWmmXfx8a4b4Af/bp14NX1x8ktVkMRzZUKSCsjpWTpjjP87X8H8e3gzNqZg+nfvWOjn9/GwZZ5dwRz/EIJ3x46b8BKWyYuMYtuHi70uaG9qUtpFhsbwRuTwqnWaHl+zRF1qW2mVEBakStVGp76JolFm1IZHdaFuMcG4ePu3OT9jAnvQoy/O29tPmmWU3adv1TO/rOXmNSnq0WvA+Pv4cIzt/bi59R81iblmLocpQEqIK1E7aQIv/BDSg7P3daL9++NavaSp0II5o/pzaXyKt7/+bSeK225NYdrO7RPMMKyCob20OBu9LmhPS/9cIz8UnWpbW5UQFqBgxmXGL9kD5mF5Xz2QDSPDwto8ZlVmE877urrw5e/ZJBeUKanSltOSsnqxCwGdu/YrLNjc2NrI3hjcgTlVRoWrD1m6nKUP1ABaeG+3n+OKZ/E4+Zkz5onBjMiqJPe9v3MqF442tny2oYTettnSyVk/krGxXKL6vt4PQFerjw9sic/Hs1lQ8qF6z9BMRoVkBaqqkbLC98f4Z9rjjCohwdrnhhMgJerXo/h1daJmSMC+Dk1n52nCvS67+aKS8ymjb0tt4V2NnUpejVjSHfCfdoxf+1RLpZVmrocpY4KSAtUWFbJ1M/2Ext/jkeGdufzB2MMtszAQ4P98evozKvrj1Ot0RrkGI1VUa1hfUoOt4d2xtWxWUu6my07WxvemBxOSUU1L/9w3NTlKHVUQFqYo9nFjF+yl+TzRbz3l0jm3R5s0MXqHe1sef6OYNLyy1gen2mw4zTGT8fzKK2osdi+j9cT1NmNmcN7si45hy3Hck1djoIKSIvyQ91kE1opWfXoIMZHGqcV95aQTtwY4ME7W0/zqwkXoVqdmEWXdk4M7NH4fp2W5vHhPQju4sbz3x+lqFwt+GVqKiAtgEYreWNTKk+uOEyodzvWzbyRMJ9213+inggheHFMCKUV1byz9ZTRjvtb+aUV7DpdyJ1RXQ16xmxq9rY2vDk5nEuXq3h1vfk0jrVWKiDNXElFNdO/OsSHO85wbz9fvp4+wCRTe/Xq3JapA/yIjc/kZG6p0Y+/LikHjdY8l1XQt9Cu7XhsaA/iErPYfjLf1OW0aiogzVh63WQTu04V8GojJpswtL+PDKStkz2vrD9m9KFxqxKyiPBtr/eWenP15M0BBHZyZV7cEbMczdRaqIA0U9tP5jP+g70UlVcTO60/9w3wM/mwOncXB/4+sid70y7y0/E8ox33WE4xqbmlTO5j+SNnGsvRzpY3JkeQX1rBvzeqS21TUQFpZqSUfLTzDH/78iA+7s6smzmYAU2YbMLQ/jrAjwAvV17beILKGo1Rjrk6MRt7W8GYcPNeVkHfIn3bM31Id1YcOM+e04WmLsfsPfjFAT7fc1av+1QBaUYqqjU8vTKJhT+mckdoF+IeG2h2w+nsbW14cUwImRfL+WJvhsGPV63RsjYpm5uDOuHu4mDw45mbv98SSHcPF+bEpVBmYesFGVNKVhE7Thag7/Y7FZBmorSimnv+u491yTk8O6oXS6ZE4exgnp2hhwZ6cnOQF0u2pRl8goVdpwooLKuyqqGFTeFkb8sbk8PJKb7CG5ssY6Z3U1i2LxNnB1sm6vl9ogLSDFRrtDy+PJGjOSUs/Wtfnhje8skmDO350cFU1mh4a/NJgx5ndWI2HVwcGBroadDjmLNo/w48OMifr/Zlmv1M76ZQVF7FuuQcJkR1xc1JvyPKrhuQQojPhRD5Qoijv3msgxDiJyHE6brP7nqtqhWRUvL8miPsPl3I63eGWswY4+6erjw4yJ/vErI4klVskGMUl1fz0/E8xkV4m7T13hw8O6oXN3RwZk5cCleqjHPv11KsSsiiskbL1P6NmzW/KRrzrvsSuO0Pj80FfpZS9gR+rvteaYb3t6Xx7aEsZo0I4J6YG0xdTpM8eXNPOjg78PIPhun280NKDlUaLZNaQd/H63F2sGPRpHAyL5bz1hbDnrVbEq1WEhufSbSfOyHebnrf/3UDUkq5C7j0h4fHA/+r+/p/wAQ919UqrErI4u2fTjExqit/vyXQ1OU0mZuTPc+M6sWhzF9Zb4BpulYnZhHYyZXQrvp/41uigT06MnXADXy+9ywJmX/8k2yd9qQVknGxnPsG6v/sEZp/D7KTlPICQN1nnetuCiFmCCEOCSEOFRSYx5RZ5mBvWiFz41IY1KMjCyeFm/09R13ujvYlpIsb/954Qq+XfukFZSSeK2JSHx+L/d0Ywtzbg/Fu14ZnV6VQUa0utZfFZ9LRxcFgt6YMfmNHSvmxlDJaShnt6dl6b7T/VmpuCY8uS6CHpysf3dfXou+v2doIFowNIae4go93pettv2sOZ2MjrGNZBX1ydbRj4aQw0gsu8+5W81sOw5iyi67w84k8/tLPF0c7wyz929y/zDwhRBeAus9qwGgj5RZX8NAXB3F2tOWLh2L03upmCv27d2R0WBeW7kwjp+hKi/en1UpWJ2ZzY09POrk56aFC6zKkpyf3RPvy8a4zJJ8vMnU5JvP1/trp9+7tZ7h7980NyHXAA3VfPwCs1U851q20opqHvjxIyZVqPn8wBu/2bUxdkt7MvT0IrYRFeuirF3/2ItlFV5jUioYWNtXzY4LxauvEs6uSjTaiyZxU1mhYefA8I4I6GXQwRWO6+awA9gG9hBBZQoiHgYXALUKI08Atdd8r13C1r+OpvFI+nNqX3t7Gm67MGHw7OPPITd1Zm5TT4gaE1YnZuDracWuIZXR5MgU3J3tenxjKqbwyPtiWZupyjG7T0VwKy6oM1jhzVWNase+VUnaRUtpLKX2klJ9JKS9KKW+WUvas+6ya1K5BSskLa47W93W01k7Pjw7tQSc3R17+4ThabfO6/ZRX1fDjkQuMDuvS7GVrW4sRQZ2YGNWVD3ec4ViOYfqimqvY+Ez8OjozJMDDoMex3NYBC7JkWxorD523yL6OTeHiaMfc24NIySomLjGrWfvYdDSXy1UaJqrL60aZPzYEdxcHnv0uxeRrBhnLiQslHMz4lan9/bAx8OTJKiANLC4hi/9YcF/Hphof0ZVI3/a8sflksyZXWJ2YjW+HNsT4dzBAddanvbMD/5oQyvELJXy044ypyzGKZfGZONrZcFe04QcQqIA0oL1phcyxgr6OTWFT1+2noLSSD7Y37d5YTtEV9p4pZGKUj8HPDKzJqN6dGRPehcXbTptktndjKqmo5vvD2YyL8Ka9s+Fnd1IBaSAnc0utpq9jU0Xd4M7EqK58tvss5y6WN/p53ydlIyXq8roZXh7XGzcne55blUyNFV9qr0nMprxKY/DGmataz1+tEeUWV/DgFwesqq9jUz13WxB2toLXNjZujWcpJXEJWcT4u+PX0cXA1Vmfjq6OvDy+N8lZxXyq50ljzYWUkmXxmUT4tCPcp71RjqkCUs/KKmustq9jU3Ru58Tjw3qw+Vgev6Rdfzbs5KxizhRcVhNTtMDosC7c1rszb/90irT8MlOXo3fx6ZdIyy9j6gDjnD2CCki9sva+jk01bUh3fNzb8Mr649e97FudmIWjnQ13hHcxUnXWRwjBKxN64+xgy3OrktE0s6uVuYqNz6RdG3vGRhhv6Q0VkHpyta/jrlMFVt3XsSmc7G355x3BpOaWsuLgeZ3bVdZoWJecw629O7fK2xH65NXWiQVjQ0g8V8SXv2SYuhy9ySupYPOxXO6O9sHJ3nj9Y1VA6klr6evYVLeHdqZ/tw68veUkxeUNL1+6PTWfovJqNbRQTyZEduXmIC/e3JxKRuFlU5ejFysOnKNGK416eQ0qIPWitfV1bAohBPPHhlB0pZp3fz7V4DZxidl4tnXkRgOPimgthBC8dmcY9rY2PBeX0uxRTeaiWqNlxYFzDA30NHoDngrIFmqNfR2bqrd3O/4ScwPL9mWSlv/7fnoXyyrZnprPnVFdsbNVb0d96dzOiRdGB3Pg7CVWNXNUk7nYejyPvJJK7jPy2SOogGyR1tzXsan+cWsgbexteXX9id89vi45hxqtVH0fDeCuvr5E+7mz8MdUnbc3LMGy+Ey6tm/D8CCd83IbjPqLbqa8kgoeauV9HZvCw9WRp0b2ZOepAran/v/0oasTs+nt7UZQZ7Wsgr7Z2AheGR9KUXmVxa5jk5Zfyi9nLjKl/w3YmmB0lQrIZiirrOGhLw5S3Mr7OjbV/QP96e7hwqvrj1NVo+VUXilHsotV30cDCvF24/6B/sTuzzTY6pOGFBt/DgdbG+6J8TXJ8VVANtHVvo4nVV/HJnOws+GFMcGkF17mq30ZxCVmYWcjGBdpvH5trdHsWwPp6OLIi2uPWlSDzeXKGuISsrgjrDMero4mqUEFZBNIKXnxe9XXsSWG9/LipkBP3vv5NHEJ2Qzr5WmyN39r4eZkz/Ojg0g6X8S3h3T3RzU3a5NyKK2sMdq464aogGyCD7an8c1B1dexJYQQvDg6mPIqDYVllery2kgmRHaln38HFm1K5dfLVaYu57qklHy1L4PgLm70ucHdZHWogGykNYezeGuL6uuoDz07tWXajd3o7ObEiGDjt0y2RleHIZZU1PCmBTTYJJ77ldTcUu4b4GfSrnMqIBvhl7RCnlul+jrq09zbg9j13HCDLdep/FlQZzceHOTPigPnzH41xGX7MmnraMd4E9+fVgF5HSdzS3kkNoHuHqqvoz4JIdTv0gSeHtkTD9faBhtzncyisKySjUdymdTXBxdHO5PWot6h11Df19FB9XVUrENbJ3teGB1MSlYxK68xgYgpfXvoPFUardHHXTdEBaQOqq+jYq3GRXjTv1sH3ticyiUza7DRaCXL488xqEdHArxcTV2OCsiGqL6OijUTQvDqhFBKK2p4c3Oqqcv5ne2p+WQXXTHJuOuGqID8A61WMm/1EdXXUbFqgZ3a8rfB/nxz8DyHz/1q6nLqLYvPpJObIyNDOpm6FEAF5O9otJLn4lJYlZDF7FsCVV9Hxao9NTIQr7bm02CTefEyO08VcG+/G7A3k5mdzKMKM6DRSp5bVRuOT4/syaybe5q6JEUxKFdHO14YHcLR7BK+PnDO1OWwfP85bG0E9/YznxMTFZDUhuOzq5KJS8zi7yMDeXqk6giutA5jwrswqEdH3tyUysWySpPVUVGt4dtD5xnVuxOd3JxMVscftfqA1Gglz36XzOrEbP5xSyBPjVRnjkrrIYTglfG9Ka/SsGiT6RpsfkjOoai8mvsG+Jushoa06oDUaCXPfJfM6sPZPHNrIE+qy2qlFQrwasvDQ7rx7aEsEjJN02ATG59JgJcrA7p3MMnxdWm1AanRSv7xbRJrDmfz7KhezByhwlFpvWaN6ElnNyde/N74DTbJ54tIzio2+bjrhrTKgKzRaJn9bRLfJ+Xw7KhePDE8wNQlKYpJuTja8eKYEI5fKGH5/kyjHjs2PhNnB1vuNMNlN1pdQNaGYzJrk3KYc1uQCkdFqXNHWGduDPDgzc0nKSg1ToNNUXkV65JzmBDV1SyH8rYoIIUQGUKII0KIJCHEIX0VZSg1Gi1//zaZdck5zL09iMeG9TB1SYpiNoQQvDy+NxXVGhb+aJwGm1UJWVTWaM1m5Mwf6eMMcriUMlJKGa2HfRlMjUbL0yuT+CE5h3m3B/HoUBWOivJHPTxdmT6kO3GJWRzMuGTQY2m1ktj4TGL83QnuYp6LtrWKS+wajZanViaxPuUC/7wjiEdUOCqKTjNHBODdrrbBpkajNdhxdqcVknGx3Cxm7dGlpQEpgS1CiAQhxIyGNhBCzBBCHBJCHCooKGjh4ZquWqPlqW+S2JBygRdGBzPjJhWOinItzg52zB8bQmpuKcviDddgs2xfJh6uDtwW2tlgx2iplgbkYCllH+B24AkhxE1/3EBK+bGUMlpKGe3padyJH2rD8TAbjtSG47Qh3Y16fEWxVKN6d+amQE/e3nKK/NIKve8/69dytqXmcU+Mr1nPKt+igJRS5tR9zgfWAP30UZQ+VGu0zFpxmI1HclU4KkoTCSF4eVxvKmu0LNyo/wabFXVjv81p3HVDmh2QQggXIUTbq18DtwJH9VVYS1RrtDz59WF+PJrLi2NCVDgqSjN083Bhxk3dWX04m/3pF/W238oaDSsPnmdEUCd83J31tl9DaMkZZCdgjxAiGTgAbJBSbtJPWc1XVaNl5teJbDqWy4KxITx8YzdTl6QoFuuJ4QF0bd+G+WuPUa2nBptNR3MpLKvifhOud91YzQ5IKWW6lDKi7qO3lPI1fRbWHFfDcfOxPF4aG8JDg1U4KkpLtHGwZf7YEE7mlfK/XzL0ss/Y+Ez8OzpzY4CHXvZnSFbTzaeqRssTXyey5XgeL4/rzYMqHBVFL24N6cSwXp68u/U0eSUta7A5caGEgxm/MnWAHzY25jXuuiFWEZBVNbVryPx0PI9XxvfmgUH+pi5JUazG1QabKo2W1zeeaNG+lsVn4mhnw+S+PnqqzrAsPiArazQ8vjyBrSfyeHV8b+4f6G/qkhTF6vh1dOHRoT1Ym5TDvjPNa7Apqajm+8PZjIvwpr2zg54rNAyLDsjKGg2PxSay9UQ+/5oQyn0qHBXFYB4f1gMf9zbMX3u0WQ02axKzKa/ScJ8FNM5cZbEBeTUct6Xm89qdoQYZrlRSUsLQoUPrv9+2bRvTpk3jmWeeYdmyZXo/nqKYMyd7W14a25vT+WV8uTejSc+VUrIsPpMI3/aE+7Q3TIEGYGfqApqjolrDY7EJbD9ZwOt3hjGlv2E6m7q5ubFjx47671esWMGUKVMYMWIE1dXVBjmmopizkSGduDnIi3e3nmJshDed2zVu/Zj49Euk5Zfx1l0RBq5QvywuICuqNTwam8CORobjyZMnef311wkJCeHo0aO8+OKLxMbGUlNTg62tLW3btuW5557jiy++YN68eTz22GNkZWWRnJzMzJkzmTVrFkVFRezatYuDBw9SXV1Nbm4uU6ZMMdIrVhTzsmBsb0a+s5N/bTjOkil9GvWcZfEZtHe2Z0x4FwNXp2dSSqN99O3bV7bElaoaef9n+6XfnPVyxf7MRj3nnXfekTNmzJCVlZUyPT1dfvnll/KWW26p//nQoUPl4cOH67/esGGDlFLKgwcPSiml9PPzq9/2gQcekNu3b2/Ra1AUa/DOTyel35z1cs/pgutum1t8RXaft0G+tuG4ESprOuCQ1JFZFnMPsqJaw4xlCew6XcCiSWH8pZFjOKdPn46XlxdDhgxhwYIF5OfnU15ezsKFC1m4cCG+vr78dpah4OBgAKKjzXp6S0UxqUeH9uCGDs7MX3uUqpprN9isOHAOjVbyVwPdCjMkiwjIimoN0786xO7TBSyaGM49MY3/Re/fv5+5c+eyf/9+OnXqhJQSLy8v5s6dy9y5c3nooYfo1atX/fbmtmiQopgjJ3tbXhoXwpmCy3y+96zO7ao1WlYcOMfQQE/8OroYsUL9MPt7kFfDcU9aIYsmhXN3tG+Tnn/p0iVmz55N9+7dKSgo4PHHH6eqqop58+ZhZ2dHRUUFCxcu5KeffiIzM5MlS5YwZ84cPD09Wb58OcXFxXz00UeEhoaSkpLCsmXLcHNzo0+fxt17URRrNSKoE7eEdGLxz6cZF+GNd/s2f9pm6/E88koqeW2C5XTt+S1RewluHNHR0fLQocYvXXOlqjYc954p5I1J4dzVxHBUzNMdd9zB119/Tfv27Vm8eDFLly6lT58+LF++3NSlKU10/lI5I9/eycjgTnzw1z+fNEz5JJ7Mi+Xsem44tmY6tFAIkSB1LBlj1pfYn+xOZ++ZQt6cHKHC0QpIKdFqtWzcuJH27Wv7wn344Yds3LhRhaOF8u3gzMzhAWw4coHdp3+/YkBafim/nLnIXwfcYLbheD1mHZCPDu1B7MP9WzRu86WXXmLo0KEEBATwxBNPEBgYSEFBAYsXLyY0NJTQ0FC++OKL+u1dXV2ZN28evXv3Zvr06fWP69r+vffeIzg4mPDwcObMmQNARkYGAQEB3H333YSEhLBo0aL67ceNG0dERAR9+/ZlyZIl9Y/n5ub+7menTp0C4IcffqBfv35ERkYye/bsa77Wy5cvM3r0aCIiIggNDWXlypW88sorxMTEEBoayowZM7h6xTBs2DDmzJlDv379CAwMZPfu3QAcO3as/njh4eGcPn36T8fZuXMnkZGRREZGEhUVRfyQqrYAAAsBSURBVGlpKQBvvvkmMTExhIeHs2DBgvrfRXBwMI8//jh9+vTh/Pnz+Pv7U1hYyKOPPkp6ejrjxo3jnXfeacS/pmKOpt/UHf+OzixYe4zKGk3947Hx53CwtWnybTGzoqt52xAfLe3m0xwLFiyQb731lpw1a5Z866235JNPPinfe+892aNHD1laWiovXbokfX19ZV5enpRSSkDu3btXVlVVSV9fX5mdnS3Pnj2rc/t27drJkpISKaWsf+zs2bNSCCGPHz8uy8vLZUBAgMzIyJBSSnn+/HkppZRVVVWyZ8+e8sKFC1JKKSdPniw//PBDKaWUxcXFMi8vT+bl5cmYmBh5+fLl+m1+/vlnna911apVctq0afXfFxUVyYsXL9Z/P3XqVLlu3TopZW2XptmzZ0sppdywYYO8+eabpZRSzpw5U8bGxkoppaysrJTl5eV/Os6YMWPknj17pJRSlpaWyurqarl582Y5ffp0qdVqpUajkaNHj5Y7d+6s/13s27ev/vl+fn6yoKDgT18rlmt7ap70m7NeLtl2WkopZVlFtQydv0k+tSLRxJVdH9bQzaclOnToQPv27enQoQPu7u64u7szZMgQXF1dcXd3p1+/fqSkpADg4ODAoEGDsLe3p1u3buTm5nL48GGd28fExPC3v/2NL7/8EkdHx/pj+vn5ERwcTJs2bbjxxhtJTEwE4LPPPiMqKoqYmBhycnLIyckBYMeOHTz88MNA7QgeLy8v4uPjSU9PZ9CgQURGRnL48GHS09N1vs6wsDC2bt3KnDlz2L17N+3atWP79u3079+fsLAwtm3bxrFjx+q3nzhxIgB9+/YlIyMDgIEDB/L666+zaNEiMjMzadPmzzfeBw8ezOzZs1m8eDFFRUXY2dmxZcsWtmzZQlRUFH369CE1NbX+7NPPz48BAwY0699OsQzDenkxqncn3t92muyiK6xNyqG0ssaixl03xOxbsfVBCPG7j2utrmhvb/+752m12vrL0oZs2rSJPXv28M033/D++++TkJBQ/9w/2rFjB5s3b2bv3r04OzsTHR2NVnvtPmSjRo1q9P25wMBAEhIS2LhxI/PmzePWW2/lgw8+4NChQ/j6+vLSSy9RUfH/8/ldDXRbW1tqamoAmDJlCv3792fDhg2MGjWKTz/9lBMnTvDJJ58AsHHjRubOncvo0aPZuHEjAwYMYOvWrUgpmTdvHo888sjvasrIyMDFxfK6dyhN9+KYEHa+vZNXfjhG5sVygru40ecGd1OX1SKt4gzyjzp06MCePXu4fPkyRUVFHDx4kPDwcJ3b9+nTp8HtpZScO3eOoUOH8tprr9WfhUFtMJw8eZKKigr27t1LVFQUJSUleHh44OzszLFjx0hOTq7ffvjw4fX3Ni9fvkxBQQEDBgxg9+7dZGVlAZCZmUlubq7OOnNycnB2dmbq1Kk888wz9WetHh4elJWVsWrVquv+btLT0+nevTuzZs1i3LhxpKSk8MQTT5CUlERSUhLe3t6cOXOGsLAw5syZQ3R0NKmpqYwaNYrPP/+csrIyALKzs8nPz7/u8RTr4ePuzJMjerL5WB6puaXcP9DP4vsVt4ozyIY8+eST9O/fH4AFCxbg5eWlc1t/f/8Gt9dqtdx3332UlJSg0Wj4z3/+U/+cbt26MXfuXE6ePMm0adPw9/fH29ubjz/+mIiICIKCgn7Xl3Lx4sXMmDGDpUuXYm9vz/LlywkMDGTp0qWMHTuWmpoaXFxcrnk2eeTIEZ599llsbGywt7dn6dKlfP/994SFheHv709MTMx1fy8rV64kNjYWe3t7OnfuzPz58/+0zbvvvsv27duxtbUlJCSE22+/HUdHR06cOMHAgQOB2sau2NhYbG3Nd0lPRf+mDelGXEIWBaWVjI/0NnU5LWbW/SAtVUZGBmPGjOHoUbNY5FFRjCrz4mWKyquJ8LWMac2u1Q+y1Z5BKopiGH4dXfDraOoq9KNV3oM0NH9/f3X2qChWQAWkoiiKDiogFUVRdFABqSiKooNRW7GFEAVAZhOf5gEUGqAcc9WaXm9req3Qul6vJb1WPymlZ0M/MGpANocQ4pCuJnhr1Jpeb2t6rdC6Xq+1vFZ1ia0oiqKDCkhFURQdLCEgPzZ1AUbWml5va3qt0Lper1W8VrO/B6koimIqlnAGqSiKYhJmHZBCiNuEECeFEGlCiLmmrsdQhBC+QojtQogTQohjQoinTF2ToQkhbIUQh4UQ601di6EJIdoLIVYJIVLr/o0HmromQxJC/L3ufXxUCLFCCOFk6pqay2wDUghhC3wA3A6EAPcKIUJMW5XB1AD/kFIGAwOAJ6z4tV71FHDC1EUYyXvAJillEBCBFb9uIURXYBYQLaUMBWyBv5i2quYz24AE+gFpUsp0KWUV8A0w3sQ1GYSU8oKUMrHu61Jq/4C6mrYqwxFC+ACjgU9NXYuhCSHcgJuAzwCklFVSyiLTVmVwdkAbIYQd4AzkmLieZjPngOwKnP/N91lYcWhcJYTwB6KA/aatxKDeBZ4Drr3ehHXoDhQAX9TdUvhUCGG1a1BIKbOBt4BzwAWgWEq5xbRVNZ85B2RDc7VbdZO7EMIViAOellKWmLoeQxBCjAHypZQJpq7FSOyAPsBSKWUUcBmw5vvp7tRe6XUDvAEXIcRU01bVfOYckFnAbxfU9cGCT9WvRwhhT204LpdSrjZ1PQY0GBgnhMig9rbJCCFErGlLMqgsIEtKefWKYBW1gWmtRgJnpZQFUspqYDUwyMQ1NZs5B+RBoKcQopsQwoHaG73rTFyTQYjalY0+A05IKd82dT2GJKWcJ6X0kVL6U/tvuk1KabFnGNcjpcwFzgshetU9dDNw3IQlGdo5YIAQwrnufX0zFtwoZbZLLkgpa4QQM4HN1LaEfS6lPHadp1mqwcB9wBEhRFLdY/+UUm40YU2K/jwJLK/7jz4deMjE9RiMlHK/EGIVkEht74zDWPCoGjWSRlEURQdzvsRWFEUxKRWQiqIoOqiAVBRF0UEFpKIoig4qIBVFUXRQAakoiqKDCkhFURQdVEAqJieE+MWA+24vhHjcUPtXrJvqKK5YtbrZkdbXzU2oKE2iziAVgxFC+Ashjv7m+2eEEC81sF1Z3WcXIcQGIURy3WzU9zSwvxNCiE/qZqzeIoRo85ufz6573lEhxNN1Dy8EegghkoQQbzZw7B1Xx0kLITr+tl5FMdux2EqrdBuQI6UcDSCEaNfANj2Be6WU04UQ3wKTgFghRF9qxzj3p3aqvP1CiJ3UTi0WKqWM1HHMAOB03dfhwBG9vRrF4qkzSMWcHAFGCiEWCSGGSCmLG9jmrJTy6oQeCYB/3dc3AmuklJellGXUTrM15FoHE0L4AdlSyqsT94YDKS19EYr1UAGpGNpvJz62v9aGUspTQF9qg/LfQoj5DWxW+ZuvNfz/VVBDEyxfTyS/D8S+qIBUfkMFpGJofkIITyGEDbVrs9jq2lAI4Q2USyljqZ22vykTy+4CJtTNQ+gC3AnsBkqBtjqeEwE41R27J7UzYatLbKWeCkjF0C4CX1F7OXwUuF8I0UPHtmHAgbo5MZ8H/tXYg9QtevYlcIDa9Xw+lVIellJeBPbWNdz8sZEmErARQiQD86md2PWBRr8yxeqpbj6KwZh7FxshRBoQVbeSpKL8iTqDVFolIURbQKvCUbkWdQapKIqigzqDVBRF0UEFpKIoig4qIBVFUXRQAakoiqKDCkhFURQdVEAqiqLooAJSURRFBxWQiqIoOvwflYE3lvJ5lrIAAAAASUVORK5CYII=\n"},"metadata":{"needs_background":"light"}}]},{"metadata":{},"cell_type":"markdown","source":"### Context Managers in Runs\n<blockquote>There are 2 Context Managers in MLManager/MLflow. <code>start_run</code> and <code>timer</code>.<br>\nContext managers enable some autologging and cleanup functions for you. To use a Context Manager, prepend the command with the <code>with</code> call append a <code>:</code> after the call, and indent all lines after it.<br>\nAnother great feature is if the run fails for some reason MLflow will track that for you</blockquote>"},{"metadata":{"trusted":true},"cell_type":"code","source":"with mlflow.start_run(run_name='run with context manager'):\n    mlflow.lp('foo','bar')\n    mlflow.lm('score', 92)","execution_count":29,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with mlflow.start_run(run_name='a run that failed'):\n    raise Exception","execution_count":30,"outputs":[{"output_type":"error","ename":"Exception","evalue":"","traceback":["\u001b[0;31m---------------------------------------------------------------------------\u001b[0m","\u001b[0;31mException\u001b[0m                                 Traceback (most recent call last)","\u001b[0;32m<ipython-input-30-583755d4e51d>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mmlflow\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstart_run\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'a run that failed'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m     \u001b[0;32mraise\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m","\u001b[0;31mException\u001b[0m: "]}]},{"metadata":{"trusted":true},"cell_type":"code","source":"from time import sleep\n# Multiple context managers\nwith mlflow.start_run(run_name='using the timer too'):\n    with mlflow.timer('run time'):\n        sleep(2)\n    print('done!')","execution_count":31,"outputs":[{"output_type":"stream","text":"Starting Code Block run time... Done.\nCode Block run time:\nRan in 2.001 secs\nRan in 0.033 mins\ndone!\n","name":"stdout"}]},{"metadata":{},"cell_type":"markdown","source":"#### Timers are default stored as parameters, but can also be stored as metrics"},{"metadata":{"trusted":true},"cell_type":"code","source":"from time import sleep\n# Multiple context managers\nwith mlflow.start_run(run_name='using the timer as a metric'):\n    with mlflow.timer('run time', param=False):\n        sleep(2)\n    print('done!')","execution_count":32,"outputs":[{"output_type":"stream","text":"Starting Code Block run time... Done.\nCode Block run time:\nRan in 2.002 secs\nRan in 0.033 mins\ndone!\n","name":"stdout"}]},{"metadata":{},"cell_type":"markdown","source":"### Nested Runs\n<blockquote>MLFlow supports the concept of <code>nested</code> runs. A nested run is a run that occurs underneath a parent run. In machine learning, this could be used for hyperparmeter tuning (like choosing K in a k-means clustering algorithm). But it can be used for anything you find useful.<br> To use it, simply pass <code>nested=True</code> to the <code>start_run</code> function</blockquote>"},{"metadata":{"trusted":true,"scrolled":false},"cell_type":"code","source":"from random import randint, sample\nfrom time import sleep\nfrom tqdm.notebook import tqdm\nexec_time = [1,3,5,2]\nnum_execs = []\nwith mlflow.start_run(run_name='parent run'):\n    for i in tqdm(range(4)):\n        with mlflow.start_run(run_name=f'child {i+1}', nested=True):\n            with mlflow.timer('run time', param=False):\n                sleep(exec_time[i])\n            mlflow.set_tag('child', 'yes')\n            mlflow.lp('num_executors', i+1)\n            num_execs.append(i+1)\n    # Plot results\n    plt.figure(figsize=(4.5, 2.5))\n    plt.plot(num_execs, exec_time)\n\n    plt.ylabel('exec time')\n    plt.xlabel('num executors')\n    plt.tight_layout(.5)\n    plt.savefig(\"spark_results.png\")\n    mlflow.log_artifact('spark_results.png','spark_results.png')\n    display(get_mlflow_ui(mlflow.current_exp_id()))","execution_count":33,"outputs":[{"output_type":"display_data","data":{"text/plain":"HBox(children=(FloatProgress(value=0.0, max=4.0), HTML(value='')))","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"b6847259d96749388c62fea2fd98f271"}},"metadata":{}},{"output_type":"stream","text":"Starting Code Block run time... Done.\nCode Block run time:\nRan in 1.001 secs\nRan in 0.017 mins\nStarting Code Block run time... Done.\nCode Block run time:\nRan in 3.002 secs\nRan in 0.05 mins\nStarting Code Block run time... Done.\nCode Block run time:\nRan in 5.004 secs\nRan in 0.083 mins\nStarting Code Block run time... Done.\nCode Block run time:\nRan in 2.002 secs\nRan in 0.033 mins\n\nSaving artifact of size: 8.37 KB to Splice Machine DB\n","name":"stdout"},{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.HTML object>","text/html":"<font size=\"+1\"><a target=\"_blank\" href=/mlflow/#/experiments/26>MLFlow UI</a></font>"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"<IPython.lib.display.IFrame at 0x7f6d856c2c10>","text/html":"\n        <iframe\n            width=\"100%\"\n            height=\"500px\"\n            src=\"/mlflow/#/experiments/26\"\n            frameborder=\"0\"\n            allowfullscreen\n        ></iframe>\n        "},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"<Figure size 324x180 with 1 Axes>","image/png":"iVBORw0KGgoAAAANSUhEUgAAAUgAAAC4CAYAAACB8XWeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAcTElEQVR4nO3deXxU5dXA8d9JSAj7vkOIYAVRIGBAAcGlVsG6IlLwVRYXxFpAQXFpX7VWbdXWsrnLJhYUVN4KgkVRxF0S9k0WC4qsCoY1ZDvvH3OjQ8yQSTJ35s7M+X4+88mdmZu5hwc93Ln3nOcRVcUYY8wvJUQ6AGOM8SpLkMYYE4AlSGOMCcASpDHGBGAJ0hhjArAEaYwxAVSKdAD+6tevr2lpaZEOwxgTR7Kysr5X1QYlveepBJmWlkZmZmakwzDGxBER2R7oPfuKbYwxAbh6Biki24BDQAGQr6oZbh7PGGNCKRxfsS9Q1e/DcBxjjAkp+4ptTBhlbtvPPxZ9xfH8gkiHYoLg9hmkAotERIHnVfWF4juIyDBgGEBqaqrL4RgTGQWFyqT3tzB+8SYKFbbsPcyk6zqTmCCRDs2chNtnkD1UtTPQB7hdRHoV30FVX1DVDFXNaNCgxDvtxkS1XdnHGPji5/zzvU1cmd6Muy9pw8K1u/nff6/FZtPyNlfPIFV1p/Nzr4jMBboCS908pjFe8p91u7nnjdXk5RfyVP+O9O3cHICjufk8/cFW6lVLZszFbSIcpQnEtQQpItWABFU95GxfDDzs1vGM8ZKcvAIefXsDMz7fTvtmtZgwsBOn1K/20/t3XdyG/Udymfj+FupWS2Zoj1MiGK0JxM0zyEbAXBEpOs5MVX3HxeMZ4wmb9hxixMwVfLXnEMN6teKui9uQXOnEq1kiwiNXtefAkTz+PG89daomc1WnZhGK2ATiWoJU1a+Bjm59vjFeo6rM+vJbHp6/juqVKzH9xq6cd1rg6+qJCcK4AekMnbqMu+asolbVJC5o0zCMEZvSWJmPMSGQfTSP3/9rOffPXUOXtLosGNXzpMmxSEpSIi8MOou2TWpw2ytZZG0/EIZoTbAsQRpTQcu27afP+KW8u34P91/alulDu9KwRkrQv18jJYlpQ7vSuGYKN05bxqY9h1yM1pSFJUhjyqmgUBn/3mZ+9/xnJFVK4I3bujOsV2sSylHbWL96ZWbcdDaVKyUwaPKX7Dhw1IWITVlZgjSmHPxrG6/o2JT5I86lY4vaFfrMFnWrMuOmszmam8+gyV/yw+HjIYrWlJclSGPKaNG63fQZ/xFrv8vmqf4dGTegEzVSkkLy2W0a12DKkC7szD7G0GnLOHw8PySfa8rHEqQxQcrJK+CBf69l2Iwsmtepwtsje/5U+B1KGWl1eeZ/OrNu50GGvZxpfdsRZAnSmCBs3nOIq57+hJc/284tPU/hzdt6nFD4HWoXtm3Ek/068OnWH7jztZUUFFpLYiR4akZxY7ymeG3jtKFdOD9MtYp9Ozdn/5FcHnl7A7WrruXRq87EabwwYWIJ0pgAso/mcd/c1SxYs5uev6rPP/p3LFP5Tijc3LMV+4/k8swS69uOBEuQxpQgc9t+Rr26kj0Hc7ivT1tu6dmqXOU7oXD3Jda3HSmWII3xU1CoPP3BFsa9t4kWdavyxm3dK1y+U1G+vu0zOXA0lz/PW0/daslcmW592+FgCdIYx67sY9zx6kq++O9+rkpvyl+uOjNk5TsVVSkxgfEDOjFk6peMmb2KWlWSwnYtNJ7ZXWxj+Lm2cY0LtY2hkpKUyIuDMmjTuAa3vbLc+rbDwBKkiWvhqm0MlaK+7UY1K1vfdhhYgjRxy7+28eZz3a9tDJUGNaxvO1wsQZq446tt/IbLJ33MvkPHmTq0C3+6rN0vJrX1shZ1q/LyTV2tb9tl0fNfhDEhkH00j9tnLue+N33zNi68o2fUTlLbtnFN69t2mSVIEzeytu/n0gkfsWjdHu7rU/Z5G73Iv2/71hnWtx1qliBNzCsoVCYu3kz/5z8nMUF4/bbu3Hpe+eZt9KKivu1PtljfdqhZHaSJaf61jVemN+URD9U2hpL1bbvDEqSJWYvW7WbsG6vJzS/kH9d2pG/nZjGdNG7u2YofjuTy7JKt1K+WzGjr264wS5Am5uTkFfDXBRuY/tl2zmxWkwkDOtGqQfVIhxUWYy9pw4EjuUx4fwt1rG+7wlxPkCKSCGQC36nqZW4fz8S3zXsOMWLWCjbuPsTN557C3b3bULlSYqTDChvr2w6tcNykGQVsCMNxTBwLVNsYT8mxSFHf9jmt6jJm9iqWfLU30iFFLVcTpIg0B34LvOTmcUx8i6XaxlAp3re9/Bvr2y4Pt88gxwFjgUKXj2PilH9t470xUtsYKsX7tjdb33aZuZYgReQyYK+qZpWy3zARyRSRzH379rkVjokxBYXKpPdPrG0cHkO1jaFS1LednJjADda3XWZunkH2AK4QkW3Aq8CFIvJK8Z1U9QVVzVDVjAYNGrgYjokVu7Nz+J+XPufvizbx2/ZNeHvkuaRHeFJbL7O+7fJzLUGq6n2q2lxV04ABwPuqer1bxzPx4d31e+g9fimrd2Tz92s7Mn5AekwWfoda28Y1mTykC9/9aH3bZWGthiYq5OQV8OC/13LLy5k0q12F+SPOpd9ZzWO68DvUuqTV5dnrrW+7LMKSIFV1idVAmvLastc3b+P0onkbf989bgq/Q+3Cto144hrr2w6WddIYz1JVXlv2LQ/NW0e15EpMHdKFC9rGd/lOKFxzVnMOHPX1bdepupZHrG87IEuQxpOyj+Vx/5treHvNLs49tT5P9e9Iw5pWvhMq/n3b9apXZvRvTot0SJ5kCdJ4Ttb2/Yyc5VuT+t4+bRkWwTWpY9nYS9qw/3AuExZvpm7VJIZY3/YvlJogRaQqMAZIVdVbRORXQBtVne96dCauFBQqzy7Zwj/f20yz2lV4/bbuVr7jIhHh0at9fdsPzVtPHevb/oVgbtJMBY4D3ZznO4BHXIvIxCWrbYyMSokJTBjYibNPsb7tkgSTIFur6hNAHoCqHgPs+44JGattjKyUpEReHJzBaY2sb7u4YBJkrohUARRARFrjO6M0pkKK1zbOs9rGiKmZksT0G7vS0Pq2TxBMgnwQeAdoISL/Ahbjm4DCmHLzr228yaltbG21jRHVoEZlZtx4NklO3/Z3Px6LdEgRV2qCVNV3gb7AEGAWkKGqS9wNy8QqX23jN1w+8RPfvI1DuvC/cTpvoxel1qvKyzd25UhuPjdM/iLu+7aD7aRpBiQCyUAvEenrXkgmVmUfy+MPs1Zwzxtr6NyyNgtH9bTCbw86vYlvve3vDljfdjBlPlOADsA6fp7XUYE3XYzLxBj/2sZ7erfl1l5W2+hlXZz1tofNyGL4jCwmD8mIy7P8YArFz1HVdq5HYmKSf21j09opzBnejU6pdSIdlgnCr0/39W2PmbOK0a+tYsLATiTG2T9qwSTIz0Sknaqudz0aE1N2Z+dw52sr+ezrH7iiY1MeufpMalr5TlTx79uuXTUp7vq2g0mQ0/Elyd34ynsEUFXt4GpkJqq9t34Pd7++iuP5hTzZr4OV70Sxm3u24vvDuTz3Yfz1bQeTIKcANwBrsLVlTCly8gr428KNTPt0G2c0rcmEgZ2sfCcG3NO7DfuPHGfC4s3Uq5bM4O5pkQ4pLIJJkN+o6luuR2Ki3pa9h/jDTN+a1Df2OIV7+sTXmtSxTER47Or2/Hg0j4fmraN21aS46NsOJkFuFJGZwDz8OmhU1e5iG8BX2zg781seems9VZITmTIkgwvbNop0WCbEivq2B0/5kjGzV1G7ajLnnRbb60gFUwdZBV9ivBi43HnY7OAG+GVt4zujelpyjGH+fdvDZ2SxIsb7tkXVO1OuZ2RkaGZmZqTDMEHK2n6AUa+uYFd2DmMuPo3hvWzZ1Xix79Bx+j33KdnH8phzazd+1ahGpEMqNxHJUtWMkt4LeAYpImOdnxNFZELxh1vBGu8rKFSe/mAL/Z//DBGYM7wbvz//VEuOccS/b3vQlNjt2z7ZV+wNzs9MIKuEh4lDu7NzuP6lL3jyP19xafsmvD2yJ52t8DsuFfVtHz7u69vefyQ30iGFXMAEqarznM2jqjrd/wEcDU94xksWb9hDn/FLWfntjzzZrwMTBqRb4XecO71JTSYPdvq2p34Zc33bwdykuS/I10yMyskr4KG31nHT9Eya1KrC/JHncm1GCyv8NgB0PaUuT1/XmbU7DzJ8RlZMrbcdsMxHRPoAlwLNil1zrAmU+s+EiKQAS4HKznFeV9UHKxauCbctew8zYtYKNuw6aLWNJqCL2jXi8Ws6cNecVYyevYoJA2Kjb/tkdZA78V1/vIITrzkeAu4M4rOPAxeq6mERSQI+FpGFqvp5uaM1YWO1jaas+p3VnANHcnl0wQbqVE3iL1dGf992wASpqquAVSIyU1XzyvrB6qsfOuw8TXIe3qkpMgFlH8vjj3PXMH/1LnqcWo9/9k+3NalNUG7p5Vtv+7kPt1KvWmXujPK+7VI7acqTHIuISCK+s89TgadV9YsS9hkGDANITU0t76FMiPjXNo7t3cZqG02ZFfVtj1+8mbpR3rcdTKthualqAZAuIrWBuSJypqquLbbPC8AL4CsUdzMeE1hBofLch1t56t1NNKnlm7fRyndMeRT1bR+Igb7tYJdcqBBV/RFYAvQOx/FM2fjXNvY5szELRllto6mYSokJTBzYiS5pvvW2P9y0L9IhlUupCVJE3nXOAIue1xGR/wTxew2Kfs9ZNvYiYGNFgjWh51/b+ES/Dkwc2MlqG01IpCQl8lKU920HcwZZ3zkDBEBVDwDBrLTUBPhARFYDy4B3VXV++cI0oVZSbWN/q200Iea/3vbQacvYsje61tsOJkEWishPd09EpCVB3I1W1dWq2klVO6jqmar6cEUCNaGzZe9h+j7zKdM+3cbQHmnMvd3WpDbuieb1toNJkH/EV8M4Q0Rm4Cv+tk6aKKSqzF72LZdP/JjdB3OYPDiDBy8/wwq/jeuitW+71ASpqu8AnYHXgNnAWapa6jVI4y0Hc/IYMWsFY99YTadU35rUvz7dCr9N+BTv2z4SBX3bwdykEXx3nzs7E1hUFZGurkdmQmb5Nwe4dPxHLFy7m7G92zDjprNpZIXfJgJO6Nt+xft928F8xX4G6AYMdJ4fAp52LSITMkXzNl773GfAz/M2xkKPrIleRX3bH23+ntGzV1FQ6N3y52AKxc9W1c4isgJ8d7FFJNnluEwF7TnoW5P6060/cFmHJjzWt72V7xjP6HdWc/YfOc5jCzZSt2oyD195hicrKIJJkHlOy6CCr74RW/7V0xZv2MNdc1aRk1fIE/06cK2tSW08aFiv1vxwJJfnP/yautWSPdm3HUyCnADMBRqKyKNAP+BPrkZlyuV4fgF/XeBbk7pdk5pMvM7WpDbedm/vthw4ksv4xZupVz2ZQd3SIh3SCYKZrOJfIpIF/BoQ4CpV3VDKr5kw27L3MCNnrWD9roMM7ZHGvX3aWvmO8Tz/vu0H31pH7arJXNGxaaTD+kmpCVJELlLV9/BrExSRwc7SCybCVJU5mTt48K11pCQlMHlwhpXvmKhS1Lc9aMqXjJm9ktpVkujlkfW2g7mL/YCIPCsi1USkkYjMw7c2tokw/9rG9Ba1eeeOXpYcTVQq6ts+tWENhr/inb7tYBLkecBWYCXwMTBTVfu5GpUplX9t492XtOGVm6220UQ3X992FxrU8E7fdjAJsg5wNr4keRxoKXZLNGJKqm28/QKrbTSxoWGNlBP6tndGuG87mAT5ObBQVXsDXYCmwCeuRmVKtOdgDjdMtnkbTWxLrVeV6UO90bcdTIK8SFWnAKjqMVUdCdzrblimuMUb9tB73FJWfPMjT1xj8zaa2Nauqa9ve8eBYwydtixifdvBJMhvReR6EXkAwJn6LMfdsEyR4/k/z9vYuFYV5o04l/5dbN5GE/u6nlKXSdd1Zu132Qx/JYvc/PD3p1gvtodt3XeYq5/2m7fx9905taEVfpv48Zt2jfhb3/ZO3/bKsPdtWy+2B6kqc7J28OC/rbbRmGszWnDgaC6PLdhInTD3bVsvtscczMnjj3PXMm/VTrq1qse4AelWvmPi3rBerfnhcC7PL/2aetWTueOi8PRtWy+2hyz/5gAjZ/nWpL77kjYMP6+1le8Y47i3T1v2H8ll3HubqVctmRvC0LdtvdgeUFioPGtrUhtzUiLCX/v6+rYfcPq2L3e5bzuYM0hUdSO2ZKsr9hzMYfTslXyy5Qd+26EJj13dnlpVrHzHmJJUSkxg0nW+vu3Rs1dSy+W+7WDuYhuXvL9xD33Gf8Ty7T/y+DXtmTSwkyVHY0oRzr5t1xKkiLQQkQ9EZIOIrBORUW4dK9oczy/gz/PWceO0TBrVTGHeiB78rkuq1TYaE6Sivu361Stzo4t9226eQeYDY1T1dOAc4HYRaefi8aJCUW3j1E+2MaR7UW1jjUiHZUzUaVgjhRk3dSUxwb2+bdcSpKruUtXlzvYhYAPQzK3jeZ2qMjvzWy6b8DG7so/x0qAMHrriDFKSbFJbY8qrZb1qvvW2c9zp2w7LNUgRSQM6AV+E43heczAnj5GvrmTs6755GxeO6sVF7azw25hQaNe0Ji8Nzvipb/tobuj6toO6i10RIlIdeAO4Q1UPlvD+MGAYQGpqqtvhhJ3VNhrjvrNb1WPSdZ1Z8tXekC41Iqru9TaKSBIwH/iPqj5V2v4ZGRmamZnpWjzh5F/b2LhmChMGduKsllbbaIzXiEiWqmaU9J5rZ5DOpLqTgQ3BJMdYYrWNxsQGN79i9wBuANaIyErntftVdYGLx4y49zfu4a45qzmWW8Dj17Snf4ZNTWZMtHItQarqx/haE+PC8fwC/rZwI1M/2cbpTWoycWC6le8YE+Vcv0kTD7buO8yImb41qYd0961JbeU7xkQ/S5AVUHzexpcGZVj5jjExxBJkOfnP23hOq7qM+10nGteyeRuNiSWWIMthxTcHGPnqCnb+mMNdF5/GbefbsqvGxCJLkGVQWKg8t3QrTy3aRKOaKcy+9RzOalk30mEZY1xiCTJIew/mcGdRbWP7JjzW12objYl1liCD8MHGvYyZs4qjuflW22hMHLEEeRLH8wt4fOFXTPnkv7RtXINJ151jtY3GxBFLkAFYbaMxxhJkMVbbaIwpYgnSz8GcPP40dy1vWW2jMQZLkD+x2kZjTHFxnyCtttEYE0hcJ0irbTTGnEzcJkj/2sa/9W3P77pYbaMx5kRxlyCtttEYE6y4SpD+tY2Du7XkvktPt9pGY0xAcZEg/WsbKycl8OKgDH5jtY3GmFLEfIK02kZjTHnFdIK02kZjTEXEZIIsLFSeX/o1/1j0ldU2GmPKLeYS5N6DOYyevYqPt3xvtY3GmAqJqQRptY3GmFByLUGKyBTgMmCvqp7p1nHAV9v4xDtfMfljq200xoSOm2eQ04BJwMsuHoOv9x1mxKwVrNtptY3GmNByLUGq6lIRSXPx83k9awcPvrWO5EpW22iMCb2IX4MUkWHAMIDU1NSgf2/Vjmzufn211TYaY1wT8QSpqi8ALwBkZGRosL+X3qI2U4d2odevGlhtozHGFRFPkBVxQZuGkQ7BGBPDEiIdgDHGeJVrCVJEZgGfAW1EZIeI3OTWsYwxxg1u3sUe6NZnG2NMONhXbGOMCUBUg75x7DoR2QdsL+Ov1Qe+dyEcN1is7ommeKMpVoiueMsTa0tVbVDSG55KkOUhIpmqmhHpOIJhsbonmuKNplghuuINdaz2FdsYYwKwBGmMMQHEQoJ8IdIBlIHF6p5oijeaYoXoijeksUb9NUhjjHFLLJxBGmOMK6IiQYrIFBHZKyJrA7wvIjJBRLaIyGoR6RzuGP1iKS3W80UkW0RWOo8Hwh2jXywtROQDEdkgIutEZFQJ+3hpbIOJ1xPjKyIpIvKliKxyYv1zCft4YmyDjNUT41ospkQRWSEi80t4LzRjq6qefwC9gM7A2gDvXwosBAQ4B/jCw7GeD8yP9Jg6sTQBOjvbNYBNQDsPj20w8XpifJ3xqu5sJwFfAOd4cWyDjNUT41osptHAzJLiCtXYRsUZpKouBfafZJcrgZfV53Ogtog0CU90JwoiVs9Q1V2qutzZPgRsAJoV281LYxtMvJ7gjNdh52mS8yh+wd8TYxtkrJ4iIs2B3wIvBdglJGMbFQkyCM2Ab/2e78Cj/+M4ujlfZxaKyBmRDgbAmf29E76zB3+eHNuTxAseGV/nK+BKYC/wrqp6dmyDiBU8Mq6OccBYoDDA+yEZ21hJkCXNmOvVfwGX42tt6ghMBP4vwvEgItWBN4A7VPVg8bdL+JWIjm0p8XpmfFW1QFXTgeZAVxEpvnidZ8Y2iFg9M64iUrQYYNbJdivhtTKPbawkyB1AC7/nzYGdEYrlpFT1YNHXGVVdACSJSP1IxSMiSfiSzb9U9c0SdvHU2JYWr9fG14njR2AJ0LvYW54aWwgcq8fGtQdwhYhsA14FLhSRV4rtE5KxjZUE+RYwyLlzdQ6Qraq7Ih1USUSksYhvsW4R6Yrv7+CHCMUiwGRgg6o+FWA3z4xtMPF6ZXxFpIGI1Ha2qwAXARuL7eaJsQ0mVq+MK4Cq3qeqzVU1DRgAvK+q1xfbLSRjGxVLLohv8t3zgfoisgN4EN+FZFT1OWABvrtWW4CjwNDIRBpUrP2A20QkHzgGDFDntlsE9ABuANY4158A7gdSwXtjS3DxemV8mwDTRSQRXzKZrarzRWS4X6xeGdtgYvXKuAbkxthaJ40xxgQQK1+xjTEm5CxBGmNMAJYgjTEmAEuQxhgTgCVIY4wJwBKkMcWIyFUi0i7ScZjIswRpzC9dBZQpQYpIVNQUm7KxBGnKTETSxDcn44vO/IGLnA4MRGSJiGQ42/WddjBEZIiI/J+IzBOR/4rIH0RktDOf3+ciUreE4zQQkTdEZJnz6OG8PkGc+QhF5BIRWSoiCSfZv7qITBWRNeKbG/Aa5/XDfsfqJyLTRKQ7cAXwpPjmPWwtIulOjKtFZK6I1PH7sz4mIh8Co0TkWhFZK74JHZa69zdgwiaU87PZIz4eQBqQD6Q7z2cD1zvbS4AMZ7s+sM3ZHoKvq6EG0ADIBoY77/0T38QTxY8zEzjX2U7F12IIUBVYB1wAfAW0LmX/x4Fxfp9bx/l52O+1fsA0Z3sa0M/vvdXAec72w0Wf5fxZn/Hbbw3QzNmuHem/J3tU/GFfC0x5/VdVi9r9svAlzdJ8oL55HA+JSDYwz3l9DdChhP0vAto5LcAANUWkhqoeEpFbgKXAnaq69WT7O68PKHpRVQ8E8wcEEJFa+JLdh85L04E5fru85rf9CTBNRGYDJU38YaKMJUhTXsf9tguAKs52Pj9fukk5ye8U+j0vpOT/FhOAbqp6rIT32uObLKFpafs7kyyU1FPr/1rxWIN15KcPUx0uImfjm8h1pYikq2pEJnQwoWHXIE2obQPOcrb7VfCzFgF/KHoiIunOz5bAGHwT5vZxklLA/Ut4vY6zuUdETheRBOBqv+MewncpAFXNBg6ISE/nvRuADymBiLRW1S9U9QHge06cbstEIUuQJtT+jm/Wl0/xXYOsiJFAhnNzZD0w3G/Ks7tUdSdwE/CSiKSUtL/zOY8AdYpuoOC7dglwLzAfeB/wnwrrVeBu5wZSa2Awvps2q4F0fNchS/KkcyNoLb6v/6sq+Oc3EWaz+RhjTAB2BmmMMQFYgjTGmAAsQRpjTACWII0xJgBLkMYYE4AlSGOMCcASpDHGBGAJ0hhjAvh/pN1nIiLnfhgAAAAASUVORK5CYII=\n"},"metadata":{"needs_background":"light"}}]},{"metadata":{},"cell_type":"markdown","source":"## Storing ML Models\n<blockquote><p class='quotation'><span style='font-size:15px'>Just like everything else we've tracked so far, tracking ML Models is easy with Splice Machine's MLManager. The <code>log_model</code> and <code>load_model</code> functions are all you need. \n    <footer>Splice Machine</footer>   \n</blockquote>\n\n#### Let's try it out"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn import svm\nfrom sklearn import datasets\nfrom sklearn.metrics import accuracy_score\n\n# Start a run\nwith mlflow.start_run(run_name='my first model'):\n    # Load some sklearn data\n    digits = datasets.load_digits()\n\n    # Build a simple model\n    clf = svm.SVC(gamma=0.001, C=100.)\n    # Log parameters to mlflow\n    mlflow.lp('gamma', 0.001)\n    mlflow.lp('C', 100.0)\n\n    # Train the model\n    with mlflow.timer('train_time'):\n        clf.fit(digits.data[:-1], digits.target[:-1])\n\n    # Predict with some data\n    preds = clf.predict(digits.data[:-1])\n\n    # Measure accuracy\n    acc = accuracy_score(digits.target[:-1], preds)\n    print('Accuracy:',acc)\n    # Log metric to mlflow\n    mlflow.lm('accuracy', acc)\n    \n    # Save model\n    mlflow.log_model(clf, 'clf_model')\n    rid = mlflow.current_run_id()","execution_count":45,"outputs":[{"output_type":"stream","text":"Starting Code Block train_time... Done.\nCode Block train_time:\nRan in 0.156 secs\nRan in 0.003 mins\nAccuracy: 1.0\nSaving artifact of size: 473.58 KB to Splice Machine DB\n","name":"stdout"}]},{"metadata":{},"cell_type":"markdown","source":"#### Load our model back and make new predictions"},{"metadata":{"trusted":true,"scrolled":true},"cell_type":"code","source":"loaded_model = mlflow.load_model(run_id=rid, name='clf_model')\ndisplay(loaded_model)\n# Make a new prediction\nnew_data = [ \n    0.,  0., 12., 10.,  0.,  0.,  0.,  0.,  0.,  0., 14., 16., 16.,\n    14.,  0.,  0.,  0.,  0., 13., 16., 15., 10.,  1.,  0.,  0.,  0.,\n    11., 16., 16.,  7.,  0.,  0.,  0.,  0.,  0.,  4.,  7., 16.,  7.,\n    0.,  0.,  0.,  0.,  0.,  4., 16.,  9.,  0.,  0.,  0.,  5.,  4.,\n    12., 16.,  4.,  0.,  0.,  0.,  9., 16., 16., 10.,  0.,  0.\n]\nprint('Prediction on new data:', loaded_model.predict([new_data])[0])","execution_count":56,"outputs":[{"output_type":"display_data","data":{"text/plain":"SVC(C=100.0, cache_size=200, class_weight=None, coef0=0.0,\n    decision_function_shape='ovr', degree=3, gamma=0.001, kernel='rbf',\n    max_iter=-1, probability=False, random_state=None, shrinking=True,\n    tol=0.001, verbose=False)"},"metadata":{}},{"output_type":"stream","text":"Prediction on new data: 5\n","name":"stdout"}]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# Fantastic!\n<blockquote> \nNow you have all of the tools necessary to start tracking your experiments and sharing results! Again, feel free to check out our <a href=\"https://pysplice.readthedocs.io/en/dbaas-4100/splicemachine.mlflow_support.html\">documentation</a>!<br><br>\n    Next Up: <a href='./7.3 Data Exploration.ipynb'>Using MLManager to explore and analyze your data</a>\n<footer>Splice Machine</footer>\n</blockquote>"}],"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.7.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"toc":{"nav_menu":{},"number_sections":false,"sideBar":true,"skip_h1_title":false,"base_numbering":1,"title_cell":"Table of Contents","title_sidebar":"Contents","toc_cell":false,"toc_position":{"height":"calc(100% - 180px)","width":"242px","left":"10px","top":"150px"},"toc_section_display":true,"toc_window_display":true}},"nbformat":4,"nbformat_minor":4}