Skip to content

Instantly share code, notes, and snippets.

@guschmue

guschmue/SubmissionExample.ipynb

Created September 9, 2019 17:59
Show Gist options
  • Save guschmue/13a7dcfea4038acf0d1c41845401eadd to your computer and use it in GitHub Desktop.
Save guschmue/13a7dcfea4038acf0d1c41845401eadd to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
SubmissionExample.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## End to End Mlperf Submission example\n",
"\n",
"This is following the [General MLPerf Submission Rules](https://docs.google.com/document/d/1Kt79dyaM9w1ckHJoLS6dBxfCqf-LABMrH2JRZm1p_bk/edit#heading=h.k6uj9xqxzwoe).\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Get the mlperf source code"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!git clone https://github.com/mlperf/inference.git --depth 1"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Build loadgen"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"/home/gs/submission/inference\n"
]
}
],
"source": [
"%cd inference"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# build loadgen\n",
"!pip install pybind11\n",
"!cd loadgen; CFLAGS=\"-std=c++14 -O3\" python setup.py develop"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"%cd v0.5/classification_and_detection\n",
"!python setup.py develop"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Download data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%bash\n",
"\n",
"## download imagenet and coco.\n",
"## we keep a copy under /data so just symlink it.\n",
"mkdir data\n",
"ln -s /data/imagenet2012 data/\n",
"ln -s /data/coco data/"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Download models"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%bash\n",
"\n",
"mkdir models\n",
"\n",
"# resnet50\n",
"wget -q https://zenodo.org/record/2535873/files/resnet50_v1.pb -O models/resnet50_v1.pb \n",
"wget -q https://zenodo.org/record/2592612/files/resnet50_v1.onnx -O models/resnet50_v1.onnx\n",
"\n",
"# mobilenet\n",
"wget -q https://zenodo.org/record/2269307/files/mobilenet_v1_1.0_224.tgz -O models/mobilenet_v1_1.0_224.tgz\n",
"cd models; tar zxvf mobilenet_v1_1.0_224.tgz\n",
"wget -q https://zenodo.org/record/3157894/files/mobilenet_v1_1.0_224.onnx -O models/mobilenet_v1_1.0_224.onnx\n",
"\n",
"# ssd-mobilenet\n",
"wget -q http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_coco_2018_01_28.tar.gz -O models/ssd_mobilenet_v1_coco_2018_01_28.tar.gz\n",
"cd models; tar zxvf ssd_mobilenet_v1_coco_2018_01_28.tar.gz; mv ssd_mobilenet_v1_coco_2018_01_28/frozen_inference_graph.pb ssd_mobilenet_v1_coco_2018_01_28.pb\n",
"wget -q https://zenodo.org/record/3163026/files/ssd_mobilenet_v1_coco_2018_01_28.onnx -O models/ssd_mobilenet_v1_coco_2018_01_28.onnx \n",
"\n",
"# ssd-resnet34\n",
"wget -q https://zenodo.org/record/3345892/files/tf_ssd_resnet34_22.1.zip -O models/tf_ssd_resnet34_22.1.zip\n",
"cd models; unzip tf_ssd_resnet34_22.1.zip; mv tf_ssd_resnet34_22.1/resnet34_tf.22.1.pb .\n",
"wget -q https://zenodo.org/record/3228411/files/resnet34-ssd1200.onnx -O models/resnet34-ssd1200.onnx"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Run benchmarks using the reference implementation\n",
"\n",
"Lets pretent we have 2 platforms (tf-gpu and onnxruntime-cpu) and want to submit the following:\n",
"```\n",
"onnxruntime-cpu: mobilenet\n",
"tf-gpu: resnet50\n",
"tf gpu: ssd-mobilenet\n",
"tf gpu: ssd-resnet34\n",
"```\n",
"\n",
"The following script will run those combinations.\n",
"To make this a little faster we do accuracy runs only with 500 images (which would not be a valid submission) and since I run this on a small server some of the latencies don't meet the mlperf rules (which would not be a valid submission)."
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {},
"outputs": [],
"source": [
"import logging\n",
"import os\n",
"logger = logging.getLogger()\n",
"logger.setLevel(logging.CRITICAL)\n",
"\n",
"os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' \n",
"os.environ['CUDA_VISIBLE_DEVICES'] = \"0\"\n",
"\n",
"# final results go here\n",
"SUBMISSION_DIR = \"/tmp/Mlperf\"\n",
"os.environ['SUBMISSION_DIR'] = SUBMISSION_DIR"
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {
"scrolled": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"====== mobilenet/SingleStream =====\n",
"TestScenario.SingleStream qps=21.98, mean=0.0452, time=22.744, acc=72.600%, queries=500, tiles=50.0:0.0463,80.0:0.0465,90.0:0.0466,95.0:0.0468,99.0:0.0578,99.9:0.1085\n",
"accuracy=72.600%, good=363, total=500\n",
"TestScenario.SingleStream qps=22.83, mean=0.0437, time=21.905, queries=500, tiles=50.0:0.0462,80.0:0.0464,90.0:0.0465,95.0:0.0468,99.0:0.0497,99.9:0.0805\n",
"====== mobilenet/MultiStream =====\n",
"TestScenario.MultiStream qps=1.00, mean=0.0920, time=250.125, acc=72.600%, queries=250, tiles=50.0:0.0924,80.0:0.0985,90.0:0.1052,95.0:0.1103,99.0:0.1146,99.9:0.1260\n",
"accuracy=72.600%, good=363, total=500\n",
"TestScenario.MultiStream qps=2.00, mean=0.0848, time=500.131, queries=1000, tiles=50.0:0.0858,80.0:0.0955,90.0:0.0996,95.0:0.1049,99.0:0.1140,99.9:0.1512\n",
"====== mobilenet/Server =====\n",
"TestScenario.Server qps=0.99, mean=0.0417, time=504.482, acc=72.600%, queries=500, tiles=50.0:0.0417,80.0:0.0499,90.0:0.0564,95.0:0.0667,99.0:0.0847,99.9:0.1275\n",
"accuracy=72.600%, good=363, total=500\n",
"TestScenario.Server qps=21.01, mean=0.0237, time=0.048, queries=1, tiles=50.0:0.0237,80.0:0.0237,90.0:0.0237,95.0:0.0237,99.0:0.0237,99.9:0.0237\n",
"====== mobilenet/Offline =====\n",
"TestScenario.Offline qps=2.28, mean=3.3837, time=7.024, acc=72.600%, queries=16, tiles=50.0:3.7338,80.0:4.5426,90.0:5.1020,95.0:5.1272,99.0:5.1410,99.9:5.1441\n",
"accuracy=72.600%, good=363, total=500\n",
"TestScenario.Offline qps=72.16, mean=3.8105, time=6.930, queries=500, tiles=50.0:3.6139,80.0:5.9200,90.0:6.8850,95.0:6.9063,99.0:6.9063,99.9:6.9063\n",
"====== resnet50/SingleStream =====\n",
"TestScenario.SingleStream qps=101.70, mean=0.0095, time=4.916, acc=76.000%, queries=500, tiles=50.0:0.0096,80.0:0.0103,90.0:0.0104,95.0:0.0105,99.0:0.0106,99.9:0.0115\n",
"accuracy=76.000%, good=380, total=500\n",
"TestScenario.SingleStream qps=105.73, mean=0.0093, time=4.729, queries=500, tiles=50.0:0.0089,80.0:0.0103,90.0:0.0107,95.0:0.0109,99.0:0.0110,99.9:0.0114\n",
"====== resnet50/MultiStream =====\n",
"TestScenario.MultiStream qps=1.00, mean=0.0135, time=250.043, acc=76.000%, queries=250, tiles=50.0:0.0128,80.0:0.0129,90.0:0.0130,95.0:0.0130,99.0:0.0132,99.9:0.1519\n",
"accuracy=76.000%, good=380, total=500\n",
"TestScenario.MultiStream qps=2.00, mean=0.0138, time=500.037, queries=1000, tiles=50.0:0.0134,80.0:0.0135,90.0:0.0135,95.0:0.0136,99.0:0.0142,99.9:0.2063\n",
"====== resnet50/Server =====\n",
"TestScenario.Server qps=0.99, mean=0.0108, time=504.436, acc=76.000%, queries=500, tiles=50.0:0.0109,80.0:0.0110,90.0:0.0111,95.0:0.0114,99.0:0.0140,99.9:0.0193\n",
"accuracy=76.000%, good=380, total=500\n",
"TestScenario.Server qps=24.50, mean=0.0095, time=0.041, queries=1, tiles=50.0:0.0095,80.0:0.0095,90.0:0.0095,95.0:0.0095,99.0:0.0095,99.9:0.0095\n",
"====== resnet50/Offline =====\n",
"TestScenario.Offline qps=7.66, mean=0.9524, time=2.088, acc=76.000%, queries=16, tiles=50.0:0.9529,80.0:1.1776,90.0:1.2437,95.0:1.3075,99.0:1.3305,99.9:1.3357\n",
"accuracy=76.000%, good=380, total=500\n",
"TestScenario.Offline qps=240.24, mean=1.1732, time=2.081, queries=500, tiles=50.0:1.0933,80.0:1.6170,90.0:1.8518,95.0:1.8518,99.0:2.0509,99.9:2.0509\n",
"====== ssd-mobilenet/SingleStream =====\n",
"loading annotations into memory...\n",
"Done (t=0.46s)\n",
"creating index...\n",
"index created!\n",
"Loading and preparing results...\n",
"Converting ndarray to lists...\n",
"(2106, 7)\n",
"0/2106\n",
"DONE (t=0.01s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=1.02s).\n",
"Accumulating evaluation results...\n",
"DONE (t=0.61s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.256\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.380\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.280\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.019\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.180\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.557\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.231\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.283\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.284\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.022\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.189\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.597\n",
"TestScenario.SingleStream qps=216.16, mean=0.0276, time=2.318, acc=92.165%, mAP=25.649%, queries=501, tiles=50.0:0.0273,80.0:0.0283,90.0:0.0289,95.0:0.0301,99.0:0.0334,99.9:0.0373\n",
"mAP=25.649%\n",
"TestScenario.SingleStream qps=35.88, mean=0.0278, time=13.965, queries=501, tiles=50.0:0.0275,80.0:0.0284,90.0:0.0292,95.0:0.0302,99.0:0.0341,99.9:0.0403\n",
"====== ssd-mobilenet/MultiStream =====\n",
"loading annotations into memory...\n",
"Done (t=0.49s)\n",
"creating index...\n",
"index created!\n",
"Loading and preparing results...\n",
"Converting ndarray to lists...\n",
"(2107, 7)\n",
"0/2107\n",
"DONE (t=0.01s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=1.04s).\n",
"Accumulating evaluation results...\n",
"DONE (t=0.48s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.256\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.380\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.280\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.019\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.180\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.557\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.231\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.283\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.284\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.022\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.189\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.597\n",
"TestScenario.MultiStream qps=75.48, mean=0.0892, time=3.325, acc=92.169%, mAP=25.648%, queries=251, tiles=50.0:0.0896,80.0:0.0968,90.0:0.0992,95.0:0.1015,99.0:0.1788,99.9:0.2639\n",
"mAP=25.649%\n",
"TestScenario.MultiStream qps=2.00, mean=0.0878, time=501.111, queries=1002, tiles=50.0:0.0885,80.0:0.0957,90.0:0.0980,95.0:0.0994,99.0:0.1075,99.9:0.3103\n",
"====== ssd-mobilenet/Server =====\n",
"loading annotations into memory...\n",
"Done (t=0.50s)\n",
"creating index...\n",
"index created!\n",
"Loading and preparing results...\n",
"Converting ndarray to lists...\n",
"(2106, 7)\n",
"0/2106\n",
"DONE (t=0.01s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=1.05s).\n",
"Accumulating evaluation results...\n",
"DONE (t=0.48s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.256\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.380\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.280\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.019\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.180\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.557\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.231\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.283\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.284\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.022\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.189\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.597\n",
"TestScenario.Server qps=217.79, mean=0.0618, time=2.300, acc=92.165%, mAP=25.649%, queries=501, tiles=50.0:0.0611,80.0:0.0659,90.0:0.0690,95.0:0.0845,99.0:0.1041,99.9:0.1311\n",
"mAP=25.649%\n",
"TestScenario.Server qps=16.50, mean=0.0317, time=0.061, queries=1, tiles=50.0:0.0317,80.0:0.0317,90.0:0.0317,95.0:0.0317,99.0:0.0317,99.9:0.0317\n",
"====== ssd-mobilenet/Offline =====\n",
"loading annotations into memory...\n",
"Done (t=0.46s)\n",
"creating index...\n",
"index created!\n",
"Loading and preparing results...\n",
"Converting ndarray to lists...\n",
"(2106, 7)\n",
"0/2106\n",
"DONE (t=0.01s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=1.00s).\n",
"Accumulating evaluation results...\n",
"DONE (t=0.57s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.256\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.380\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.280\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.019\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.180\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.557\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.231\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.283\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.284\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.022\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.189\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.597\n",
"TestScenario.Offline qps=7.57, mean=4.6831, time=2.245, acc=92.165%, mAP=25.649%, queries=17, tiles=50.0:5.5412,80.0:6.3013,90.0:6.6706,95.0:6.8977,99.0:7.3473,99.9:7.4485\n",
"mAP=25.649%\n",
"TestScenario.Offline qps=51.87, mean=5.5460, time=9.658, queries=501, tiles=50.0:5.3666,80.0:8.3505,90.0:9.2078,95.0:9.2078,99.0:9.6345,99.9:9.6345\n",
"====== ssd-resnet34/SingleStream =====\n",
"loading annotations into memory...\n",
"Done (t=0.44s)\n",
"creating index...\n",
"index created!\n",
"Loading and preparing results...\n",
"Converting ndarray to lists...\n",
"(89779, 7)\n",
"0/89779\n",
"DONE (t=0.40s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=5.32s).\n",
"Accumulating evaluation results...\n",
"DONE (t=1.51s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.253\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.449\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.258\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.178\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.349\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.298\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.236\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.373\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.391\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.221\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.486\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.436\n",
"TestScenario.SingleStream qps=43.03, mean=0.1077, time=11.643, acc=33.184%, mAP=25.346%, queries=501, tiles=50.0:0.1061,80.0:0.1118,90.0:0.1184,95.0:0.1229,99.0:0.1302,99.9:0.1489\n",
"mAP=0.430%\n",
"TestScenario.SingleStream qps=9.26, mean=0.1079, time=54.120, queries=501, tiles=50.0:0.1059,80.0:0.1141,90.0:0.1198,95.0:0.1231,99.0:0.1319,99.9:0.1508\n",
"====== ssd-resnet34/MultiStream =====\n",
"loading annotations into memory...\n",
"Done (t=0.44s)\n",
"creating index...\n",
"index created!\n",
"Loading and preparing results...\n",
"Converting ndarray to lists...\n",
"(89936, 7)\n",
"0/89936\n",
"DONE (t=0.40s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=6.00s).\n",
"Accumulating evaluation results...\n",
"DONE (t=1.52s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.253\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.449\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.258\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.178\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.349\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.298\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.236\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.373\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.391\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.221\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.486\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.436\n",
"TestScenario.MultiStream qps=18.66, mean=0.2203, time=13.449, acc=33.271%, mAP=25.340%, queries=251, tiles=50.0:0.2154,80.0:0.2271,90.0:0.2323,95.0:0.2387,99.0:0.2789,99.9:0.6477\n",
"mAP=0.430%\n",
"TestScenario.MultiStream qps=2.00, mean=0.2281, time=501.643, queries=1002, tiles=50.0:0.2252,80.0:0.2334,90.0:0.2426,95.0:0.2490,99.0:0.2974,99.9:0.7694\n",
"====== ssd-resnet34/Server =====\n",
"loading annotations into memory...\n",
"Done (t=0.45s)\n",
"creating index...\n",
"index created!\n",
"Loading and preparing results...\n",
"Converting ndarray to lists...\n",
"(89779, 7)\n",
"0/89779\n",
"DONE (t=0.40s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=5.28s).\n",
"Accumulating evaluation results...\n",
"DONE (t=1.44s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.253\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.449\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.258\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.178\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.349\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.298\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.236\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.373\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.391\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.221\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.486\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.436\n",
"TestScenario.Server qps=43.52, mean=0.1348, time=11.511, acc=33.184%, mAP=25.346%, queries=501, tiles=50.0:0.1264,80.0:0.1401,90.0:0.1869,95.0:0.2038,99.0:0.2186,99.9:0.2462\n",
"mAP=0.430%\n",
"TestScenario.Server qps=7.08, mean=0.1139, time=0.141, queries=1, tiles=50.0:0.1139,80.0:0.1139,90.0:0.1139,95.0:0.1139,99.0:0.1139,99.9:0.1139\n",
"====== ssd-resnet34/Offline =====\n",
"loading annotations into memory...\n",
"Done (t=0.44s)\n",
"creating index...\n",
"index created!\n",
"Loading and preparing results...\n",
"Converting ndarray to lists...\n",
"(89778, 7)\n",
"0/89778\n",
"DONE (t=0.40s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=5.44s).\n",
"Accumulating evaluation results...\n",
"DONE (t=1.46s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.253\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.449\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.258\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.178\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.349\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.298\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.236\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.373\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.391\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.221\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.486\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.436\n",
"TestScenario.Offline qps=1.42, mean=24.5135, time=11.936, acc=33.184%, mAP=25.346%, queries=17, tiles=50.0:27.1318,80.0:32.9352,90.0:36.7293,95.0:38.2327,99.0:38.6533,99.9:38.7480\n",
"mAP=0.430%\n",
"TestScenario.Offline qps=9.29, mean=31.5222, time=53.929, queries=501, tiles=50.0:28.9873,80.0:46.5904,90.0:53.2797,95.0:53.5744,99.0:53.5744,99.9:53.5744\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
]
}
],
"source": [
"%%bash\n",
"\n",
"# where to find stuff\n",
"export DATA_ROOT=`pwd`/data\n",
"export MODEL_DIR=`pwd`/models\n",
"\n",
"# options for official runs\n",
"gopt=\"--max-batchsize 32 --samples-per-query 2 --threads 2\"\n",
"gopt=\"$gopt --count 500\"\n",
"\n",
"rm -rf output\n",
"result=output/results.csv\n",
"\n",
"\n",
"function one_run {\n",
" # args: mode framework device model ...\n",
" scenario=$1; shift\n",
" model=$3\n",
" output_dir=output/$1-$2/$model\n",
" echo \"====== $model/$scenario =====\"\n",
"\n",
" # accuracy run\n",
" ./run_local.sh $@ --scenario $scenario --accuracy --output $output_dir/$scenario.Accuracy\n",
" if [ $model == \"resnet50\" -o $model == \"mobilenet\" ] ; then\n",
" python tools/accuracy-imagenet.py --mlperf-accuracy-file $output_dir/$scenario.Accuracy/mlperf_log_accuracy.json \\\n",
" --imagenet-val-file $DATA_ROOT/imagenet2012/val_map.txt > $output_dir/$scenario.Accuracy/Accuracy.txt\n",
" else\n",
" python tools/accuracy-coco.py --mlperf-accuracy-file $output_dir/$scenario.Accuracy/mlperf_log_accuracy.json \\\n",
" --coco-dir $DATA_ROOT/coco | grep mAP > $output_dir/$scenario.Accuracy/Accuracy.txt\n",
" fi\n",
" cat $output_dir/$scenario.Accuracy/Accuracy.txt\n",
"\n",
" # performance run\n",
" ./run_local.sh $@ --scenario $scenario --output $output_dir/$scenario\n",
"\n",
" # summary to csv\n",
" python tools/lglog2csv.py --input $output_dir/$scenario/mlperf_log_summary.txt --runtime \"$1-$2\" --machine $HOSTNAME --model $3 --name $1-$2-py >> $result\n",
"}\n",
"\n",
"function one_model {\n",
" # args: framework device model ...\n",
" one_run SingleStream $@\n",
" one_run MultiStream $@\n",
" one_run Server $@\n",
" one_run Offline $@\n",
"}\n",
"\n",
"\n",
"mkdir output\n",
"echo \"build,date,machine,runtime,model,mode,qps,mean,latency_90,latency_99\" > $result\n",
"\n",
"# run image classifier benchmarks \n",
"export DATA_DIR=$DATA_ROOT/imagenet2012\n",
"one_model onnxruntime cpu mobilenet $gopt\n",
"one_model tf gpu resnet50 $gopt\n",
"\n",
"# run ssd benchmarks\n",
"export DATA_DIR=$DATA_ROOT/coco\n",
"one_model tf gpu ssd-mobilenet $gopt\n",
"one_model tf gpu ssd-resnet34 $gopt\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Create submission directory\n",
"\n",
"The mlperf submission needs to contain a json file that describes the systems used.\n"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [],
"source": [
"%%bash\n",
"\n",
"mkdir -p ${SUBMISSION_DIR}/results ${SUBMISSION_DIR}/benchmarks ${SUBMISSION_DIR}/systems\n",
"\n",
"# copy results\n",
"cp -r output/tf-gpu ${SUBMISSION_DIR}/results\n",
"cp -r output/onnxruntime-cpu ${SUBMISSION_DIR}/results\n",
"\n",
"# copy/idenify benchmark source code\n",
"echo \"git clone https://github.com/mlperf/inference.git\" > ${SUBMISSION_DIR}/benchmarks/VERSION.txt\n",
"git rev-parse HEAD >> ${SUBMISSION_DIR}/benchmarks/VERSION.txt\n",
"\n",
"cat > ${SUBMISSION_DIR}/systems/tf-gpu.json <<EOF\n",
"{\n",
" \"system_name\": \"tf-gpu\",\n",
" \"submission\": {\n",
" \"submitter\": \"Mlperf\",\n",
" \"division\": \"closed\",\n",
" \"status\": \"cloud\"\n",
" },\n",
" \"sw\": {\n",
" \"framework\": \"tf-14.1\",\n",
" \"cublas_version\": \"10.2.0.163\",\n",
" \"cuda_version\": \"10.1.163\",\n",
" \"cudnn_version\": \"7.6.0.64\",\n",
" \"container_base\": \"Ubuntu-16.04\",\n",
" \"cuda_driver_version\": \"418.67\",\n",
" \"os\": \"Ubuntu 16.04.2 LTS\" \n",
" },\n",
" \"notes\": \"\",\n",
" \"hw\": {\n",
" \"sys_storage_type\": \"SATA SSD\",\n",
" \"cpu\": \"2x Intel(R) Xeon(R) CPU E5-2698 v4 @2.20GHz\",\n",
" \"host_processors_per_node\": \"1\",\n",
" \"host_processor_model_name\": \"Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GH\",\n",
" \"host_processor_core_count\": \"4\",\n",
" \"host_processor_vcpu_count\": \"8\",\n",
" \"host_processor_frequency\": \"3.50GH\",\n",
" \"host_processor_caches\": \"10240KB\",\n",
" \"host_processor_interconnect\": \"-\",\n",
" \"host_memory_capacity\": \"32GB\",\n",
" \"host_storage_type\": \"SSD\",\n",
" \"host_storage_capacity\": \"200GB\",\n",
" \"host_networking\": \"-\",\n",
" \"host_networking_topology\": \"-\",\n",
" \"host_memory_configuration\": \"-\",\n",
" \"accelerator\": \"Tesla V100-SXM2-16GB\",\n",
" \"accelerator_model_name\": \"GTX-1080\",\n",
" \"accelerator_host_interconnect\": \"-\",\n",
" \"accelerator_core_count\": \"2560\",\n",
" \"accelerator_frequency\": \"1733MHz\",\n",
" \"accelerator_on-chip_memories\": \"1\",\n",
" \"accelerator_memory_configuration\": \"HBM\",\n",
" \"accelerator_memory_capacity\": \"8 GB\",\n",
" \"num_accelerators\": \"1\",\n",
" \"num_nodes\": \"1\",\n",
" \"accelerators_per_node\": \"1\",\n",
" \"hw_notes\": \"\"\n",
" }\n",
"}\n",
"EOF\n",
"\n",
"cat > ${SUBMISSION_DIR}/systems/onnxruntime-cpu.json <<EOF\n",
"{\n",
" \"system_name\": \"onnxruntime-cpu\",\n",
" \"submission\": {\n",
" \"submitter\": \"Mlperf\",\n",
" \"division\": \"closed\",\n",
" \"status\": \"cloud\"\n",
" },\n",
" \"sw\": {\n",
" \"framework\": \"tf-14.1\",\n",
" \"os\": \"Ubuntu 16.04.2 LTS\" \n",
" },\n",
" \"notes\": \"\",\n",
" \"hw\": {\n",
" \"sys_storage_type\": \"SATA SSD\",\n",
" \"cpu\": \"2x Intel(R) Xeon(R) CPU E5-2698 v4 @2.20GHz\",\n",
" \"host_processors_per_node\": \"1\",\n",
" \"host_processor_model_name\": \"Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GH\",\n",
" \"host_processor_core_count\": \"4\",\n",
" \"host_processor_vcpu_count\": \"8\",\n",
" \"host_processor_frequency\": \"3.50GH\",\n",
" \"host_processor_caches\": \"10240KB\",\n",
" \"host_processor_interconnect\": \"-\",\n",
" \"host_memory_capacity\": \"32GB\",\n",
" \"host_storage_type\": \"SSD\",\n",
" \"host_storage_capacity\": \"200GB\",\n",
" \"host_networking\": \"-\",\n",
" \"host_networking_topology\": \"-\",\n",
" \"host_memory_configuration\": \"-\",\n",
" \"accelerator\": \"-\",\n",
" \"num_accelerators\": \"0\",\n",
" \"num_nodes\": \"1\",\n",
" \"accelerators_per_node\": \"1\",\n",
" \"hw_notes\": \"\"\n",
" }\n",
"}\n",
"EOF"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### What's in the submission directory now ?\n"
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"/tmp/Mlperf\r\n",
"/tmp/Mlperf/systems\r\n",
"/tmp/Mlperf/benchmarks\r\n",
"/tmp/Mlperf/results\r\n",
"/tmp/Mlperf/results/tf-gpu\r\n",
"/tmp/Mlperf/results/tf-gpu/ssd-mobilenet\r\n",
"/tmp/Mlperf/results/tf-gpu/ssd-mobilenet/Server.Accuracy\r\n",
"/tmp/Mlperf/results/tf-gpu/ssd-mobilenet/Offline.Accuracy\r\n",
"/tmp/Mlperf/results/tf-gpu/ssd-mobilenet/SingleStream\r\n",
"/tmp/Mlperf/results/tf-gpu/ssd-mobilenet/SingleStream.Accuracy\r\n",
"/tmp/Mlperf/results/tf-gpu/ssd-mobilenet/MultiStream\r\n",
"/tmp/Mlperf/results/tf-gpu/ssd-mobilenet/Offline\r\n",
"/tmp/Mlperf/results/tf-gpu/ssd-mobilenet/MultiStream.Accuracy\r\n",
"/tmp/Mlperf/results/tf-gpu/ssd-mobilenet/Server\r\n",
"/tmp/Mlperf/results/tf-gpu/resnet50\r\n",
"/tmp/Mlperf/results/tf-gpu/resnet50/Server.Accuracy\r\n",
"/tmp/Mlperf/results/tf-gpu/resnet50/Offline.Accuracy\r\n",
"/tmp/Mlperf/results/tf-gpu/resnet50/SingleStream\r\n",
"/tmp/Mlperf/results/tf-gpu/resnet50/SingleStream.Accuracy\r\n",
"/tmp/Mlperf/results/tf-gpu/resnet50/MultiStream\r\n",
"/tmp/Mlperf/results/tf-gpu/resnet50/Offline\r\n",
"/tmp/Mlperf/results/tf-gpu/resnet50/MultiStream.Accuracy\r\n",
"/tmp/Mlperf/results/tf-gpu/resnet50/Server\r\n",
"/tmp/Mlperf/results/tf-gpu/ssd-resnet34\r\n",
"/tmp/Mlperf/results/tf-gpu/ssd-resnet34/Server.Accuracy\r\n",
"/tmp/Mlperf/results/tf-gpu/ssd-resnet34/Offline.Accuracy\r\n",
"/tmp/Mlperf/results/tf-gpu/ssd-resnet34/SingleStream\r\n",
"/tmp/Mlperf/results/tf-gpu/ssd-resnet34/SingleStream.Accuracy\r\n",
"/tmp/Mlperf/results/tf-gpu/ssd-resnet34/MultiStream\r\n",
"/tmp/Mlperf/results/tf-gpu/ssd-resnet34/Offline\r\n",
"/tmp/Mlperf/results/tf-gpu/ssd-resnet34/MultiStream.Accuracy\r\n",
"/tmp/Mlperf/results/tf-gpu/ssd-resnet34/Server\r\n",
"/tmp/Mlperf/results/onnxruntime-cpu\r\n",
"/tmp/Mlperf/results/onnxruntime-cpu/mobilenet\r\n",
"/tmp/Mlperf/results/onnxruntime-cpu/mobilenet/Server.Accuracy\r\n",
"/tmp/Mlperf/results/onnxruntime-cpu/mobilenet/Offline.Accuracy\r\n",
"/tmp/Mlperf/results/onnxruntime-cpu/mobilenet/SingleStream\r\n",
"/tmp/Mlperf/results/onnxruntime-cpu/mobilenet/SingleStream.Accuracy\r\n",
"/tmp/Mlperf/results/onnxruntime-cpu/mobilenet/MultiStream\r\n",
"/tmp/Mlperf/results/onnxruntime-cpu/mobilenet/Offline\r\n",
"/tmp/Mlperf/results/onnxruntime-cpu/mobilenet/MultiStream.Accuracy\r\n",
"/tmp/Mlperf/results/onnxruntime-cpu/mobilenet/Server\r\n"
]
}
],
"source": [
"!find {SUBMISSION_DIR} -type d"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And if we look at the files for a specific model/platform/scenario"
]
},
{
"cell_type": "code",
"execution_count": 63,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\u001b[31mFor example: tf-gpu resnet50 SingleStream\u001b[0m\n",
"\n",
"mlperf_log_accuracy.json mlperf_log_summary.txt results.json\n",
"mlperf_log_detail.txt\t mlperf_log_trace.json\n",
"\n",
"Accuracy in results/tf-gpu/resnet50/SingleStream.Accuracy/Accuracy.txt\n",
"accuracy=76.000%, good=380, total=500\n",
"\n",
"Summary in results/tf-gpu/resnet50/SingleStream/mlperf_log_summary.txt\n",
"================================================\n",
"MLPerf Results Summary\n",
"================================================\n",
"SUT name : PySUT\n",
"Scenario : Single Stream\n",
"Mode : Performance\n",
"90th percentile latency (ns) : 10690255\n",
"Result is : INVALID\n",
" Min duration satisfied : NO\n",
" Min queries satisfied : Yes\n",
"Recommendations:\n",
" * Decrease the expected latency so the loadgen pre-generates more queries.\n",
"\n",
"================================================\n",
"Additional Stats\n",
"================================================\n",
"QPS w/ loadgen overhead : 106.82\n",
"QPS w/o loadgen overhead : 107.18\n",
"\n",
"Min latency (ns) : 8248658\n",
"Max latency (ns) : 11679843\n",
"Mean latency (ns) : 9330370\n",
"50.00 percentile latency (ns) : 8944492\n",
"90.00 percentile latency (ns) : 10690255\n",
"95.00 percentile latency (ns) : 10913400\n",
"97.00 percentile latency (ns) : 10957330\n",
"99.00 percentile latency (ns) : 11057972\n",
"99.90 percentile latency (ns) : 11679843\n",
"\n",
"================================================\n",
"Test Parameters Used\n",
"================================================\n",
"samples_per_query : 1\n",
"target_qps : 100\n",
"target_latency (ns): 0\n",
"max_async_queries : 1\n",
"min_duration (ms): 60000\n",
"max_duration (ms): 0\n",
"min_query_count : 500\n",
"max_query_count : 500\n",
"qsl_rng_seed : 0\n",
"sample_index_rng_seed : 0\n",
"schedule_rng_seed : 0\n",
"accuracy_log_rng_seed : 0\n",
"accuracy_log_probability : 0\n",
"\n",
"1 ERROR encountered. See detailed log.\n",
"\n",
"The system info that goes with tf-gpu\n",
"{\n",
" \"system_name\": \"tf-gpu\",\n",
" \"submission\": {\n",
" \"submitter\": \"Mlperf\",\n",
" \"division\": \"closed\",\n",
" \"status\": \"cloud\"\n",
" },\n",
" \"sw\": {\n",
" \"framework\": \"tf-14.1\",\n",
" \"cublas_version\": \"10.2.0.163\",\n",
" \"cuda_version\": \"10.1.163\",\n",
" \"cudnn_version\": \"7.6.0.64\",\n",
" \"container_base\": \"Ubuntu-16.04\",\n",
" \"cuda_driver_version\": \"418.67\",\n",
" \"os\": \"Ubuntu 16.04.2 LTS\" \n",
" },\n",
" \"notes\": \"\",\n",
" \"hw\": {\n",
" \"sys_storage_type\": \"SATA SSD\",\n",
" \"cpu\": \"2x Intel(R) Xeon(R) CPU E5-2698 v4 @2.20GHz\",\n",
" \"host_processors_per_node\": \"1\",\n",
" \"host_processor_model_name\": \"Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GH\",\n",
" \"host_processor_core_count\": \"4\",\n",
" \"host_processor_vcpu_count\": \"8\",\n",
" \"host_processor_frequency\": \"3.50GH\",\n",
" \"host_processor_caches\": \"10240KB\",\n",
" \"host_processor_interconnect\": \"-\",\n",
" \"host_memory_capacity\": \"32GB\",\n",
" \"host_storage_type\": \"SSD\",\n",
" \"host_storage_capacity\": \"200GB\",\n",
" \"host_networking\": \"-\",\n",
" \"host_networking_topology\": \"-\",\n",
" \"host_memory_configuration\": \"-\",\n",
" \"accelerator\": \"Tesla V100-SXM2-16GB\",\n",
" \"accelerator_model_name\": \"GTX-1080\",\n",
" \"accelerator_host_interconnect\": \"-\",\n",
" \"accelerator_core_count\": \"2560\",\n",
" \"accelerator_frequency\": \"1733MHz\",\n",
" \"accelerator_on-chip_memories\": \"1\",\n",
" \"accelerator_memory_configuration\": \"HBM\",\n",
" \"accelerator_memory_capacity\": \"8 GB\",\n",
" \"num_accelerators\": \"1\",\n",
" \"num_nodes\": \"1\",\n",
" \"accelerators_per_node\": \"1\",\n",
" \"hw_notes\": \"\"\n",
" }\n",
"}\n"
]
}
],
"source": [
"print(\"\\n\\x1b[31mFor example: tf-gpu resnet50 SingleStream\\x1b[0m\\n\")\n",
"\n",
"!ls {SUBMISSION_DIR}/results/tf-gpu/resnet50/SingleStream\n",
"\n",
"print(\"\\nAccuracy in results/tf-gpu/resnet50/SingleStream.Accuracy/Accuracy.txt\")\n",
"!cat {SUBMISSION_DIR}/results/tf-gpu/resnet50/SingleStream.Accuracy/Accuracy.txt\n",
"\n",
"print(\"\\nSummary in results/tf-gpu/resnet50/SingleStream/mlperf_log_summary.txt\")\n",
"!cat {SUBMISSION_DIR}/results/tf-gpu/resnet50/SingleStream/mlperf_log_summary.txt\n",
"\n",
"print(\"\\nThe system info that goes with tf-gpu\")\n",
"!cat {SUBMISSION_DIR}/systems/tf-gpu.json\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment