Skip to content

Instantly share code, notes, and snippets.

@sayakpaul

sayakpaul/trt-image.ipynb

Created July 27, 2022 13:24
Show Gist options
  • Save sayakpaul/d8b9876fad1421b7cb8301d0dcaf4946 to your computer and use it in GitHub Desktop.
Save sayakpaul/d8b9876fad1421b7cb8301d0dcaf4946 to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
trt-image.ipynb
{
"cells": [
{
"cell_type": "markdown",
"id": "04188740",
"metadata": {},
"source": [
"## Initial setup"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "5a5b62bd-68a3-429d-b4c9-c70588ebb7a5",
"metadata": {},
"outputs": [],
"source": [
"# Storage bucket\n",
"GCS_BUCKET = \"gs://hf-tf-vision\"\n",
"REGION = \"us-central1\""
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "c9309446-1c97-4b2e-9b1c-ca0f4ead7c34",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Creating gs://hf-tf-vision/...\n"
]
}
],
"source": [
"!gsutil mb -l $REGION $GCS_BUCKET"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "157f1ffe",
"metadata": {},
"outputs": [],
"source": [
"# # Install Vertex AI SDK\n",
"# !pip install --upgrade google-cloud-aiplatform -q\n",
"\n",
"# Install TensorFlow Serving API\n",
"# !pip install -q tensorflow-serving-api"
]
},
{
"cell_type": "markdown",
"id": "aba67801",
"metadata": {},
"source": [
"## Initial imports"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "354874fb-98e8-4fd7-891e-4c9ce1e6f75f",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"2022-07-27 12:17:49.547428: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.\n"
]
}
],
"source": [
"from transformers import ViTFeatureExtractor, TFViTForImageClassification\n",
"import tensorflow as tf\n",
"import tempfile\n",
"import requests\n",
"import base64\n",
"import json\n",
"import os"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "57bd7259-5a06-40d3-9ef6-37e7380ca42e",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2.9.0-rc2\n",
"4.20.1\n"
]
}
],
"source": [
"import transformers\n",
"\n",
"print(tf.__version__)\n",
"print(transformers.__version__)"
]
},
{
"cell_type": "markdown",
"id": "dad72899",
"metadata": {},
"source": [
"## Save the model locally\n",
"\n",
"We will work with a [Vision Transformer B-16 model provided by 🤗 Transformers](https://huggingface.co/docs/transformers/main/en/model_doc/vit). We will first initialize it, load the model weights, and then save it locally as a [SavedModel](https://www.tensorflow.org/guide/saved_model) resource. "
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "a8f1ba75-a1f7-4ada-91bc-c3d07ec146f0",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"All model checkpoint layers were used when initializing TFViTForImageClassification.\n",
"\n",
"All the layers of TFViTForImageClassification were initialized from the model checkpoint at google/vit-base-patch16-224.\n",
"If your task is similar to the task the model of the checkpoint was trained on, you can already use TFViTForImageClassification for predictions without further training.\n",
"WARNING:absl:Found untraced functions such as embeddings_layer_call_fn, embeddings_layer_call_and_return_conditional_losses, encoder_layer_call_fn, encoder_layer_call_and_return_conditional_losses, layernorm_layer_call_fn while saving (showing 5 of 421). These functions will not be directly callable after loading.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:Assets written to: vit/saved_model/1/assets\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO:tensorflow:Assets written to: vit/saved_model/1/assets\n"
]
}
],
"source": [
"# the saved_model parameter is a flag to create a saved model version of the model\n",
"LOCAL_MODEL_DIR = \"vit\"\n",
"model = TFViTForImageClassification.from_pretrained(\"google/vit-base-patch16-224\")\n",
"model.save_pretrained(LOCAL_MODEL_DIR, saved_model=True)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "215a0d0f-45e6-438c-b02f-152427da958b",
"metadata": {},
"outputs": [],
"source": [
"# # Inspect the input and output signatures of the model\n",
"# !saved_model_cli show --dir {LOCAL_MODEL_DIR}/saved_model/1 --all"
]
},
{
"cell_type": "markdown",
"id": "e7ae5a29",
"metadata": {
"tags": []
},
"source": [
"## Preprocessing utilities"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "c4862e37-9701-4782-9da1-e4fac4dd9d32",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"ViTFeatureExtractor {\n",
" \"do_normalize\": true,\n",
" \"do_resize\": true,\n",
" \"feature_extractor_type\": \"ViTFeatureExtractor\",\n",
" \"image_mean\": [\n",
" 0.5,\n",
" 0.5,\n",
" 0.5\n",
" ],\n",
" \"image_std\": [\n",
" 0.5,\n",
" 0.5,\n",
" 0.5\n",
" ],\n",
" \"resample\": 2,\n",
" \"size\": 224\n",
"}"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"feature_extractor = ViTFeatureExtractor()\n",
"feature_extractor"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "0bfc1d8f-94f2-4b4e-b117-9918fa25a233",
"metadata": {},
"outputs": [],
"source": [
"CONCRETE_INPUT = \"pixel_values\"\n",
"SIZE = feature_extractor.size\n",
"INPUT_SHAPE = (SIZE, SIZE, 3)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "06843d8c-db3e-4650-8358-f17b265818f9",
"metadata": {},
"outputs": [],
"source": [
"def normalize_img(img, mean=feature_extractor.image_mean, std=feature_extractor.image_std):\n",
" # Scale to the value range of [0, 1] first and then normalize.\n",
" img = img / 255\n",
" mean = tf.constant(mean)\n",
" std = tf.constant(std)\n",
" return (img - mean) / std\n",
"\n",
"def preprocess(image):\n",
" resized = tf.image.resize(image, (SIZE, SIZE))\n",
" normalized = normalize_img(resized)\n",
" normalized = tf.transpose(normalized, (2, 0, 1)) # Since HF models are channel-first.\n",
" return normalized"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "359fca61-995f-451b-a89c-d2fbf493e6cf",
"metadata": {},
"outputs": [],
"source": [
"# # To deploy the model on Vertex AI we must have the model in a storage bucket.\n",
"# tf.saved_model.save(\n",
"# model,\n",
"# os.path.join(GCS_BUCKET, LOCAL_MODEL_DIR),\n",
"# signatures={\"serving_default\": model_exporter(model)},\n",
"# )"
]
},
{
"cell_type": "markdown",
"id": "2d65d3af-d456-43f9-a9da-84e529c179d2",
"metadata": {},
"source": [
"## Export prediction requests for warmup"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "4c3ed3d1-1135-4b5d-9a57-70e96c2ada6b",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'image': TensorSpec(shape=(None, None, 3), dtype=tf.uint8, name=None),\n",
" 'label': TensorSpec(shape=(), dtype=tf.int64, name=None)}"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import tensorflow_datasets as tfds\n",
"\n",
"validation_ds = tfds.load(\"imagenette\", split=\"validation\")\n",
"validation_ds.element_spec"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "15963b1b-0c3a-4a54-b30a-43bbeca0dba8",
"metadata": {},
"outputs": [],
"source": [
"!mkdir -p $LOCAL_MODEL_DIR/requests"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "8418f8bb-6f67-401d-bc49-f39a8f7927a6",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Serving function input: pixel_values\n"
]
}
],
"source": [
"loaded = tf.saved_model.load(os.path.join(LOCAL_MODEL_DIR, \"saved_model\", \"1\"))\n",
"serving_input = list(\n",
" loaded.signatures[\"serving_default\"].structured_input_signature[1].keys()\n",
")[0]\n",
"print(\"Serving function input:\", serving_input)\n",
"\n",
"SERVE_INP = serving_input"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "43487618-3cf7-4bf2-ba7b-a7b7fe712b6a",
"metadata": {},
"outputs": [],
"source": [
"def encode(images):\n",
" rows = []\n",
" for row in images.numpy().tolist():\n",
" rows.append(row)\n",
" return {SERVE_INP: rows}\n",
"\n",
"\n",
"def export_requests_jsonl(file_name, rows=2, batch_size=32):\n",
" with tf.io.gfile.GFile(file_name, mode=\"w\") as f:\n",
" for batch in validation_ds.map(lambda x: preprocess(x[\"image\"])).batch(batch_size).take(rows):\n",
" d = encode(batch)\n",
" f.write(json.dumps(d))\n",
" f.write(\"\\n\")"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "b40a3bf9-b90a-4601-a785-e032f26ffd16",
"metadata": {},
"outputs": [],
"source": [
"export_requests_jsonl(os.path.join(LOCAL_MODEL_DIR, \"requests\", \"requests_1_1.jsonl\"), rows=1, batch_size=1)\n",
"export_requests_jsonl(os.path.join(LOCAL_MODEL_DIR, \"requests\", \"requests_1_32.jsonl\"), rows=1, batch_size=32)\n",
"export_requests_jsonl(os.path.join(LOCAL_MODEL_DIR, \"requests\", \"requests_10_32.jsonl\"), rows=10, batch_size=32)"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "9e9ba818-0029-4edc-bf5d-f14dded3166d",
"metadata": {},
"outputs": [],
"source": [
"!mkdir -p $LOCAL_MODEL_DIR/assets.extra"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "e4a9ba6e-86dc-4257-bd64-252a594fc99d",
"metadata": {},
"outputs": [],
"source": [
"from tensorflow_serving.apis import predict_pb2, prediction_log_pb2\n",
"\n",
"\n",
"def build_grpc_request(\n",
" row_dict, model_name=\"vit\", signature_name=\"serving_default\"\n",
"):\n",
" \"\"\"Generate gRPC inference request with payload.\"\"\"\n",
"\n",
" request = predict_pb2.PredictRequest()\n",
" request.model_spec.name = model_name\n",
" request.model_spec.signature_name = signature_name\n",
" for key, value in row_dict.items():\n",
" proto = tf.make_tensor_proto(value)\n",
" request.inputs[key].CopyFrom(proto)\n",
" return request\n",
"\n",
"\n",
"def export_warmup_file(\n",
" request_files, export_path, model_name=\"vit\", signature_name=\"serving_default\"\n",
"):\n",
" with tf.io.TFRecordWriter(export_path) as writer:\n",
" for request_file_path in request_files:\n",
" with open(request_file_path) as f:\n",
" row_dict = json.loads(f.readline())\n",
" request = build_grpc_request(row_dict, model_name, signature_name)\n",
" log = prediction_log_pb2.PredictionLog(\n",
" predict_log=prediction_log_pb2.PredictLog(request=request)\n",
" )\n",
" writer.write(log.SerializeToString())\n",
"\n",
"\n",
"# https://www.tensorflow.org/tfx/serving/saved_model_warmup\n",
"export_warmup_file(\n",
" [\n",
" os.path.join(LOCAL_MODEL_DIR, \"requests\", \"requests_1_1.jsonl\"),\n",
" os.path.join(LOCAL_MODEL_DIR, \"requests\", \"requests_1_32.jsonl\"),\n",
" ],\n",
" os.path.join(LOCAL_MODEL_DIR, \"assets.extra\", \"tf_serving_warmup_requests\"),\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "85870e22-c9ea-4e06-b8a2-7b2c385c470b",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Copying file://vit/assets.extra/tf_serving_warmup_requests [Content-Type=application/octet-stream]...\n",
"/ [1/1 files][ 19.0 MiB/ 19.0 MiB] 100% Done \n",
"Operation completed over 1 objects/19.0 MiB. \n",
"Copying file://vit/saved_model/1/keras_metadata.pb [Content-Type=application/octet-stream]...\n",
"Copying file://vit/saved_model/1/saved_model.pb [Content-Type=application/octet-stream]...\n",
"Copying file://vit/saved_model/1/variables/variables.data-00000-of-00001 [Content-Type=application/octet-stream]...\n",
"Copying file://vit/saved_model/1/variables/variables.index [Content-Type=application/octet-stream]...\n",
"==> NOTE: You are uploading one or more large file(s), which would run \n",
"significantly faster if you enable parallel composite uploads. This\n",
"feature can be enabled by editing the\n",
"\"parallel_composite_upload_threshold\" value in your .boto\n",
"configuration file. However, note that if you do this large files will\n",
"be uploaded as `composite objects\n",
"<https://cloud.google.com/storage/docs/composite-objects>`_,which\n",
"means that any user who downloads such objects will need to have a\n",
"compiled crcmod installed (see \"gsutil help crcmod\"). This is because\n",
"without a compiled crcmod, computing checksums on composite objects is\n",
"so slow that gsutil disables downloads of composite objects.\n",
"\n",
"- [4/4 files][336.8 MiB/336.8 MiB] 100% Done \n",
"Operation completed over 4 objects/336.8 MiB. \n"
]
}
],
"source": [
"!gsutil -m cp -r $LOCAL_MODEL_DIR/assets.extra $GCS_BUCKET/\n",
"!gsutil -m cp -r $LOCAL_MODEL_DIR/saved_model/1/* $GCS_BUCKET/"
]
},
{
"cell_type": "markdown",
"id": "d5a46991",
"metadata": {},
"source": [
"## Deployment on Vertex AI\n",
"\n",
"[This resource](https://cloud.google.com/vertex-ai/docs/general/general-concepts) shows some relevant concepts on Vertex AI. "
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "5edb508a-80f4-4652-947b-f0d95b9a8ce3",
"metadata": {},
"outputs": [],
"source": [
"from google.cloud.aiplatform import gapic as aip"
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "f8edaee9-1068-4ffb-bfc1-9e63a39c8bc5",
"metadata": {},
"outputs": [],
"source": [
"# Deployment hardware\n",
"DEPLOY_COMPUTE = \"n1-standard-16\"\n",
"DEPLOY_GPU = aip.AcceleratorType.NVIDIA_TESLA_T4\n",
"PROJECT_ID = \"fast-ai-exploration\""
]
},
{
"cell_type": "code",
"execution_count": 22,
"id": "4e8d6fcd-2e63-4e39-9c2e-2047308be3f0",
"metadata": {},
"outputs": [],
"source": [
"# Initialize clients.\n",
"API_ENDPOINT = f\"{REGION}-aiplatform.googleapis.com\"\n",
"PARENT = f\"projects/{PROJECT_ID}/locations/{REGION}\"\n",
"\n",
"client_options = {\"api_endpoint\": API_ENDPOINT}\n",
"model_service_client = aip.ModelServiceClient(client_options=client_options)\n",
"endpoint_service_client = aip.EndpointServiceClient(client_options=client_options)\n",
"prediction_service_client = aip.PredictionServiceClient(client_options=client_options)"
]
},
{
"cell_type": "markdown",
"id": "fb4ad9cf-8963-4b66-9e56-5005ac302ca8",
"metadata": {},
"source": [
"## Create Models"
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "d5fb346f-bfce-43e7-9c0c-5463709d8c6f",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'projects/29880397572/locations/us-central1/models/6743520317429252096'"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tf28_gpu_model_dict = {\n",
" \"display_name\": \"ViT Base TF2.8 GPU model\",\n",
" \"artifact_uri\": GCS_BUCKET,\n",
" \"container_spec\": {\n",
" \"image_uri\": \"us-docker.pkg.dev/vertex-ai/prediction/tf2-gpu.2-8:latest\",\n",
" },\n",
"}\n",
"tf28_gpu_model = (\n",
" model_service_client.upload_model(parent=PARENT, model=tf28_gpu_model_dict)\n",
" .result(timeout=180)\n",
" .model\n",
")\n",
"tf28_gpu_model"
]
},
{
"cell_type": "code",
"execution_count": 24,
"id": "fc9e1ded-0e0b-46f4-9817-789a90d7f1dd",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'projects/29880397572/locations/us-central1/models/5242695741608034304'"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tf_opt_gpu_model_dict = {\n",
" \"display_name\": \"ViT Base optimized TensorFlow runtime GPU model\",\n",
" \"artifact_uri\": GCS_BUCKET,\n",
" \"container_spec\": {\n",
" \"image_uri\": \"us-docker.pkg.dev/vertex-ai-restricted/prediction/tf_opt-gpu.nightly:latest\",\n",
" \"args\": [\n",
" \"--allow_precompilation=true\",\n",
" \"--allow_precision_affecting_optimizations=false\",\n",
" ],\n",
" },\n",
"}\n",
"\n",
"tf_opt_gpu_model = (\n",
" model_service_client.upload_model(parent=PARENT, model=tf_opt_gpu_model_dict)\n",
" .result(timeout=180)\n",
" .model\n",
")\n",
"tf_opt_gpu_model"
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "557a681a-4af8-4896-bada-a134d207c1eb",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'projects/29880397572/locations/us-central1/models/3847705757030023168'"
]
},
"execution_count": 25,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tf_opt_lossy_gpu_model_dict = {\n",
" \"display_name\": \"ViT Base optimized TensorFlow runtime GPU model with lossy optimizations\",\n",
" \"artifact_uri\": GCS_BUCKET,\n",
" \"container_spec\": {\n",
" \"image_uri\": \"us-docker.pkg.dev/vertex-ai-restricted/prediction/tf_opt-gpu.nightly:latest\",\n",
" \"args\": [\n",
" \"--allow_precompilation=true\",\n",
" \"--allow_precision_affecting_optimizations=true\",\n",
" ],\n",
" },\n",
"}\n",
"\n",
"tf_opt_lossy_gpu_model = (\n",
" model_service_client.upload_model(parent=PARENT, model=tf_opt_lossy_gpu_model_dict)\n",
" .result(timeout=180)\n",
" .model\n",
")\n",
"tf_opt_lossy_gpu_model"
]
},
{
"cell_type": "markdown",
"id": "ea120606-d72d-41a4-936a-bb8547e3d503",
"metadata": {},
"source": [
"## Create Endpoints"
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "5a0dc3ef-6609-4374-b934-75c382f7fc6d",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'projects/29880397572/locations/us-central1/endpoints/2991014675178586112'"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tf28_gpu_endpoint_dict = {\n",
" \"display_name\": \"ViT Base TF2.8 GPU endpoint\",\n",
"}\n",
"tf28_gpu_endpoint = (\n",
" endpoint_service_client.create_endpoint(\n",
" parent=PARENT, endpoint=tf28_gpu_endpoint_dict\n",
" )\n",
" .result(timeout=300)\n",
" .name\n",
")\n",
"tf28_gpu_endpoint"
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "a88e56e2-14ff-412f-a4f2-35c7c8431f18",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'projects/29880397572/locations/us-central1/endpoints/4884778318487879680'"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tf_opt_gpu_endpoint_dict = {\n",
" \"display_name\": \"ViT Base optimized TensorFlow runtime GPU endpoint\",\n",
"}\n",
"tf_opt_gpu_endpoint = (\n",
" endpoint_service_client.create_endpoint(\n",
" parent=PARENT, endpoint=tf_opt_gpu_endpoint_dict\n",
" )\n",
" .result(timeout=300)\n",
" .name\n",
")\n",
"tf_opt_gpu_endpoint"
]
},
{
"cell_type": "code",
"execution_count": 28,
"id": "c7a8c050-43d3-494f-8da3-a88c106ed50e",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'projects/29880397572/locations/us-central1/endpoints/7156844330496294912'"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tf_opt_lossy_gpu_endpoint_dict = {\n",
" \"display_name\": \"ViT Base optimized TensorFlow runtime GPU with lossy optimizations endpoint\",\n",
"}\n",
"tf_opt_lossy_gpu_endpoint = (\n",
" endpoint_service_client.create_endpoint(\n",
" parent=PARENT, endpoint=tf_opt_lossy_gpu_endpoint_dict\n",
" )\n",
" .result(timeout=300)\n",
" .name\n",
")\n",
"tf_opt_lossy_gpu_endpoint"
]
},
{
"cell_type": "markdown",
"id": "14098bc5-beb6-4f49-a7ab-535f3c98907a",
"metadata": {},
"source": [
"## Deploy to Endpoints"
]
},
{
"cell_type": "code",
"execution_count": 30,
"id": "6a5a98af-720e-4854-a97e-76cd02ed5f7a",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"deployed_model {\n",
" id: \"7290522954201300992\"\n",
"}"
]
},
"execution_count": 30,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tf28_gpu_deployed_model_dict = {\n",
" \"model\": tf28_gpu_model,\n",
" \"display_name\": \"ViT Base TF2.8 GPU deployed model\",\n",
" \"dedicated_resources\": {\n",
" \"min_replica_count\": 1,\n",
" \"max_replica_count\": 1,\n",
" \"machine_spec\": {\n",
" \"machine_type\": DEPLOY_COMPUTE,\n",
" \"accelerator_type\": DEPLOY_GPU,\n",
" \"accelerator_count\": 1,\n",
" },\n",
" },\n",
"}\n",
"\n",
"tf28_gpu_deployed_model = endpoint_service_client.deploy_model(\n",
" endpoint=tf28_gpu_endpoint,\n",
" deployed_model=tf28_gpu_deployed_model_dict,\n",
" traffic_split={\"0\": 100},\n",
").result()\n",
"tf28_gpu_deployed_model"
]
},
{
"cell_type": "code",
"execution_count": 35,
"id": "5295d3d9-1925-4a33-80db-77f2950641b5",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"1"
]
},
"execution_count": 35,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# sample_batch = for batch in validation_ds.map(lambda x: preprocess(x[\"image\"])).batch(batch_size).take(rows)\n",
"prediction_samples = [\n",
" sample.numpy().tolist() for sample in validation_ds.map(lambda x: preprocess(x[\"image\"])).take(1)\n",
"]\n",
"len(prediction_samples)"
]
},
{
"cell_type": "code",
"execution_count": 36,
"id": "69d70050-58a5-419b-89db-ec04f75f9008",
"metadata": {},
"outputs": [
{
"ename": "FailedPrecondition",
"evalue": "400 The request size (1659946 bytes) exceeds 1.500MB limit.",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31m_InactiveRpcError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m/opt/conda/lib/python3.7/site-packages/google/api_core/grpc_helpers.py\u001b[0m in \u001b[0;36merror_remapped_callable\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 56\u001b[0m \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 57\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mcallable_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 58\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mgrpc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mRpcError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mexc\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/opt/conda/lib/python3.7/site-packages/grpc/_channel.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, request, timeout, metadata, credentials, wait_for_ready, compression)\u001b[0m\n\u001b[1;32m 945\u001b[0m wait_for_ready, compression)\n\u001b[0;32m--> 946\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0m_end_unary_response_blocking\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstate\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcall\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 947\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/opt/conda/lib/python3.7/site-packages/grpc/_channel.py\u001b[0m in \u001b[0;36m_end_unary_response_blocking\u001b[0;34m(state, call, with_call, deadline)\u001b[0m\n\u001b[1;32m 848\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 849\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0m_InactiveRpcError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstate\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 850\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31m_InactiveRpcError\u001b[0m: <_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.FAILED_PRECONDITION\n\tdetails = \"The request size (1659946 bytes) exceeds 1.500MB limit.\"\n\tdebug_error_string = \"{\"created\":\"@1658926563.663880441\",\"description\":\"Error received from peer ipv4:74.125.69.95:443\",\"file\":\"/home/conda/feedstock_root/build_artifacts/grpc-split_1656146941531/work/src/core/lib/surface/call.cc\",\"file_line\":966,\"grpc_message\":\"The request size (1659946 bytes) exceeds 1.500MB limit.\",\"grpc_status\":9}\"\n>",
"\nThe above exception was the direct cause of the following exception:\n",
"\u001b[0;31mFailedPrecondition\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m/tmp/ipykernel_3307/550173685.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m 1\u001b[0m prediction_service_client.predict(\n\u001b[1;32m 2\u001b[0m \u001b[0mendpoint\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtf28_gpu_endpoint\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0minstances\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mprediction_samples\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 4\u001b[0m )\n",
"\u001b[0;32m/opt/conda/lib/python3.7/site-packages/google/cloud/aiplatform_v1/services/prediction_service/client.py\u001b[0m in \u001b[0;36mpredict\u001b[0;34m(self, request, endpoint, instances, parameters, retry, timeout, metadata)\u001b[0m\n\u001b[1;32m 583\u001b[0m \u001b[0mretry\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mretry\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 584\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtimeout\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 585\u001b[0;31m \u001b[0mmetadata\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmetadata\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 586\u001b[0m )\n\u001b[1;32m 587\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/opt/conda/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, timeout, retry, *args, **kwargs)\u001b[0m\n\u001b[1;32m 152\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"metadata\"\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmetadata\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 153\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 154\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mwrapped_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 155\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 156\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;32m/opt/conda/lib/python3.7/site-packages/google/api_core/grpc_helpers.py\u001b[0m in \u001b[0;36merror_remapped_callable\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 57\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mcallable_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 58\u001b[0m \u001b[0;32mexcept\u001b[0m \u001b[0mgrpc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mRpcError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mexc\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 59\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mexceptions\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfrom_grpc_error\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mexc\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mexc\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 60\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 61\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0merror_remapped_callable\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mFailedPrecondition\u001b[0m: 400 The request size (1659946 bytes) exceeds 1.500MB limit."
]
}
],
"source": [
"prediction_service_client.predict(\n",
" endpoint=tf28_gpu_endpoint,\n",
" instances=prediction_samples,\n",
")\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "656715fa-e386-4ab7-8a2f-1e21084f2c0a",
"metadata": {},
"outputs": [],
"source": [
"# tf_opt_gpu_deployed_model_dict = {\n",
"# \"model\": tf_opt_gpu_model,\n",
"# \"display_name\": \"ViT Base optimized TensorFlow runtime GPU model\",\n",
"# \"dedicated_resources\": {\n",
"# \"min_replica_count\": 1,\n",
"# \"max_replica_count\": 1,\n",
"# \"machine_spec\": {\n",
"# \"machine_type\": DEPLOY_COMPUTE,\n",
"# \"accelerator_type\": DEPLOY_GPU,\n",
"# \"accelerator_count\": 1,\n",
"# },\n",
"# },\n",
"# }\n",
"\n",
"# tf_opt_gpu_deployed_model = endpoint_service_client.deploy_model(\n",
"# endpoint=tf_opt_gpu_endpoint,\n",
"# deployed_model=tf_opt_gpu_deployed_model_dict,\n",
"# traffic_split={\"0\": 100},\n",
"# ).result()\n",
"# tf_opt_gpu_deployed_model"
]
},
{
"cell_type": "code",
"execution_count": 37,
"id": "64e200db-c9e7-4ae4-b965-b29dbd1a554a",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"deployed_model {\n",
" id: \"4163617437922623488\"\n",
"}"
]
},
"execution_count": 37,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# tf_opt_lossy_gpu_deployed_model_dict = {\n",
"# \"model\": tf_opt_lossy_gpu_model,\n",
"# \"display_name\": \"ViT Base optimized TensorFlow runtime GPU model with lossy optimizations\",\n",
"# \"dedicated_resources\": {\n",
"# \"min_replica_count\": 1,\n",
"# \"max_replica_count\": 1,\n",
"# \"machine_spec\": {\n",
"# \"machine_type\": DEPLOY_COMPUTE,\n",
"# \"accelerator_type\": DEPLOY_GPU,\n",
"# \"accelerator_count\": 1,\n",
"# },\n",
"# },\n",
"# }\n",
"\n",
"# tf_opt_lossy_gpu_deployed_model = endpoint_service_client.deploy_model(\n",
"# endpoint=tf_opt_lossy_gpu_endpoint,\n",
"# deployed_model=tf_opt_lossy_gpu_deployed_model_dict,\n",
"# traffic_split={\"0\": 100},\n",
"# ).result()\n",
"# tf_opt_lossy_gpu_deployed_model"
]
},
{
"cell_type": "code",
"execution_count": 44,
"id": "2de61a97-d898-4279-9474-16733157a1e4",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" % Total % Received % Xferd Average Speed Time Time Time Current\n",
" Dload Upload Total Spent Left Speed\n",
"100 4110 100 4110 0 0 97k 0 --:--:-- --:--:-- --:--:-- 100k\n"
]
}
],
"source": [
"!curl https://raw.githubusercontent.com/GoogleCloudPlatform/vertex-ai-samples/main/notebooks/community/vertex_endpoints/optimized_tensorflow_runtime/benchmark.py -o benchmark.py"
]
},
{
"cell_type": "markdown",
"id": "a6f739e7-df44-4fd6-96f3-9e3902b98954",
"metadata": {},
"source": [
"## Benchmarking"
]
},
{
"cell_type": "code",
"execution_count": 45,
"id": "ff7a96e0-5511-4541-8765-e9986ef10c74",
"metadata": {},
"outputs": [],
"source": [
"from benchmark import benchmark"
]
},
{
"cell_type": "code",
"execution_count": 47,
"id": "bcd17295-de82-4331-a56a-9b74b88593a8",
"metadata": {},
"outputs": [],
"source": [
"import subprocess\n",
"\n",
"def get_headers():\n",
" gcloud_access_token = (\n",
" subprocess.check_output(\"gcloud auth print-access-token\".split(\" \"))\n",
" .decode()\n",
" .rstrip(\"\\n\")\n",
" )\n",
" return {\"authorization\": \"Bearer \" + gcloud_access_token}"
]
},
{
"cell_type": "code",
"execution_count": 40,
"id": "1afee670-c4f3-431f-b81e-c705e2c54ac6",
"metadata": {},
"outputs": [],
"source": [
"def build_rest_request(row_dict, model_name):\n",
" payload = json.dumps({\"instances\": row_dict})\n",
" return payload"
]
},
{
"cell_type": "code",
"execution_count": 48,
"id": "08c38b25-3b2e-4992-88ac-13d6e7476c99",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:benchmarking at 1 QPS, sending 10 requests\n",
"INFO:tensorflow:Running benchmark at 1 qps\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"INFO:tensorflow:received 10 responses.\n",
"INFO:tensorflow:num_qps:1 requests/second: 1.00 #success:0 #error:10 latencies: [avg:0.45ms p50:0.44ms p90:0.63ms p99:0.64ms]\n",
"INFO:tensorflow:benchmarking at 2 QPS, sending 10 requests\n",
"INFO:tensorflow:Running benchmark at 2 qps\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"INFO:tensorflow:num_qps:2 requests/second: 2.00 #success:0 #error:10 latencies: [avg:0.33ms p50:0.27ms p90:0.61ms p99:0.63ms]\n",
"INFO:tensorflow:benchmarking at 3 QPS, sending 15 requests\n",
"INFO:tensorflow:Running benchmark at 3 qps\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"INFO:tensorflow:num_qps:3 requests/second: 3.00 #success:0 #error:15 latencies: [avg:0.29ms p50:0.26ms p90:0.31ms p99:0.63ms]\n",
"INFO:tensorflow:benchmarking at 4 QPS, sending 20 requests\n",
"INFO:tensorflow:Running benchmark at 4 qps\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"INFO:tensorflow:num_qps:4 requests/second: 3.99 #success:0 #error:20 latencies: [avg:0.28ms p50:0.26ms p90:0.29ms p99:0.61ms]\n",
"INFO:tensorflow:benchmarking at 5 QPS, sending 25 requests\n",
"INFO:tensorflow:Running benchmark at 5 qps\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"INFO:tensorflow:num_qps:5 requests/second: 4.99 #success:0 #error:25 latencies: [avg:0.27ms p50:0.25ms p90:0.29ms p99:0.89ms]\n"
]
},
{
"data": {
"text/plain": [
"defaultdict(list,\n",
" {'reqested_qps': [1, 2, 3, 4, 5],\n",
" 'actual_qps': [0.9988986238060823,\n",
" 1.9975657562986389,\n",
" 2.9960220225706826,\n",
" 3.994268250049734,\n",
" 4.99237652464746],\n",
" 'avg_latency': [0.45332908630371094,\n",
" 0.33142566680908203,\n",
" 0.28967857360839844,\n",
" 0.28089284896850586,\n",
" 0.27065277099609375],\n",
" 'p50': [0.44035911560058594,\n",
" 0.26619434356689453,\n",
" 0.2598762512207031,\n",
" 0.26214122772216797,\n",
" 0.25081634521484375],\n",
" 'p90': [0.6313323974609375,\n",
" 0.6090879440307617,\n",
" 0.3138542175292969,\n",
" 0.287318229675293,\n",
" 0.2880096435546875],\n",
" 'p99': [0.6429195404052734,\n",
" 0.6278204917907715,\n",
" 0.6320953369140623,\n",
" 0.6146121025085444,\n",
" 0.8917045593261703],\n",
" 'success': [0, 0, 0, 0, 0],\n",
" 'error': [10, 10, 15, 20, 25]})"
]
},
"execution_count": 48,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"headers = get_headers()\n",
"\n",
"\n",
"def send_rest_request(request):\n",
" res = r.post(\n",
" f\"https://{REGION}-aiplatform.googleapis.com/v1/{tf28_gpu_endpoint}:predict\",\n",
" data=request,\n",
" headers=headers,\n",
" verify=False,\n",
" )\n",
" assert res.status_code == 200\n",
" return res\n",
"\n",
"\n",
"tf28_gpu_results = benchmark(\n",
" send_rest_request,\n",
" build_rest_request,\n",
" f\"{LOCAL_MODEL_DIR}/requests/requests_10_32.jsonl\",\n",
" [1, 2, 3, 4, 5],\n",
" 5,\n",
")\n",
"\n",
"tf28_gpu_results"
]
},
{
"cell_type": "code",
"execution_count": 50,
"id": "4dc8ab23-1813-4301-ae92-b999d7979c0d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:benchmarking at 1 QPS, sending 10 requests\n",
"INFO:tensorflow:Running benchmark at 1 qps\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"INFO:tensorflow:received 10 responses.\n",
"INFO:tensorflow:num_qps:1 requests/second: 1.00 #success:0 #error:10 latencies: [avg:0.34ms p50:0.27ms p90:0.60ms p99:0.63ms]\n",
"INFO:tensorflow:benchmarking at 2 QPS, sending 10 requests\n",
"INFO:tensorflow:Running benchmark at 2 qps\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"INFO:tensorflow:num_qps:2 requests/second: 2.00 #success:0 #error:10 latencies: [avg:0.32ms p50:0.26ms p90:0.46ms p99:0.62ms]\n",
"INFO:tensorflow:benchmarking at 3 QPS, sending 15 requests\n",
"INFO:tensorflow:Running benchmark at 3 qps\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"INFO:tensorflow:num_qps:3 requests/second: 3.00 #success:0 #error:15 latencies: [avg:0.32ms p50:0.26ms p90:0.52ms p99:0.62ms]\n",
"INFO:tensorflow:benchmarking at 4 QPS, sending 20 requests\n",
"INFO:tensorflow:Running benchmark at 4 qps\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"INFO:tensorflow:num_qps:4 requests/second: 3.99 #success:0 #error:20 latencies: [avg:0.30ms p50:0.26ms p90:0.39ms p99:0.64ms]\n",
"INFO:tensorflow:benchmarking at 5 QPS, sending 25 requests\n",
"INFO:tensorflow:Running benchmark at 5 qps\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"INFO:tensorflow:num_qps:5 requests/second: 4.99 #success:0 #error:25 latencies: [avg:0.30ms p50:0.25ms p90:0.78ms p99:1.14ms]\n",
"INFO:tensorflow:benchmarking at 6 QPS, sending 30 requests\n",
"INFO:tensorflow:Running benchmark at 6 qps\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"INFO:tensorflow:num_qps:6 requests/second: 5.99 #success:0 #error:30 latencies: [avg:0.18ms p50:0.12ms p90:0.29ms p99:0.67ms]\n",
"INFO:tensorflow:benchmarking at 7 QPS, sending 35 requests\n",
"INFO:tensorflow:Running benchmark at 7 qps\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"INFO:tensorflow:num_qps:7 requests/second: 6.99 #success:0 #error:35 latencies: [avg:0.17ms p50:0.18ms p90:0.29ms p99:0.65ms]\n",
"INFO:tensorflow:benchmarking at 8 QPS, sending 40 requests\n",
"INFO:tensorflow:Running benchmark at 8 qps\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"INFO:tensorflow:num_qps:8 requests/second: 7.99 #success:0 #error:40 latencies: [avg:0.34ms p50:0.27ms p90:0.61ms p99:0.70ms]\n",
"INFO:tensorflow:benchmarking at 9 QPS, sending 45 requests\n",
"INFO:tensorflow:Running benchmark at 9 qps\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"INFO:tensorflow:num_qps:9 requests/second: 8.98 #success:0 #error:45 latencies: [avg:0.36ms p50:0.27ms p90:0.60ms p99:0.64ms]\n",
"INFO:tensorflow:benchmarking at 10 QPS, sending 50 requests\n",
"INFO:tensorflow:Running benchmark at 10 qps\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"INFO:tensorflow:num_qps:10 requests/second: 9.98 #success:0 #error:50 latencies: [avg:0.22ms p50:0.26ms p90:0.30ms p99:0.89ms]\n"
]
},
{
"data": {
"text/plain": [
"defaultdict(list,\n",
" {'reqested_qps': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],\n",
" 'actual_qps': [0.9988864200088217,\n",
" 1.997598293144067,\n",
" 2.9961337389353244,\n",
" 3.994600917672566,\n",
" 4.992526275163532,\n",
" 5.990180757336857,\n",
" 6.988187586700395,\n",
" 7.985983843454074,\n",
" 8.984993107148929,\n",
" 9.980267438456147],\n",
" 'avg_latency': [0.3353595733642578,\n",
" 0.3162384033203125,\n",
" 0.31762123107910156,\n",
" 0.29555559158325195,\n",
" 0.29839515686035156,\n",
" 0.17777283986409506,\n",
" 0.1730987003871373,\n",
" 0.33721923828125,\n",
" 0.35749541388617617,\n",
" 0.2232217788696289],\n",
" 'p50': [0.2655982971191406,\n",
" 0.26428699493408203,\n",
" 0.2636909484863281,\n",
" 0.26285648345947266,\n",
" 0.2543926239013672,\n",
" 0.12433528900146484,\n",
" 0.18072128295898438,\n",
" 0.2690553665161133,\n",
" 0.2715587615966797,\n",
" 0.2554655075073242],\n",
" 'p90': [0.5975961685180664,\n",
" 0.458693504333496,\n",
" 0.5248546600341796,\n",
" 0.39420127868652377,\n",
" 0.7801532745361331,\n",
" 0.2889633178710938,\n",
" 0.2853870391845703,\n",
" 0.6131887435913086,\n",
" 0.5972385406494141,\n",
" 0.3047466278076172],\n",
" 'p99': [0.6313920021057129,\n",
" 0.6207203865051271,\n",
" 0.6157016754150391,\n",
" 0.6397843360900879,\n",
" 1.1397552490234375,\n",
" 0.6668758392333985,\n",
" 0.6463527679443357,\n",
" 0.6954407691955566,\n",
" 0.6371402740478516,\n",
" 0.8882355690002433],\n",
" 'success': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
" 'error': [10, 10, 15, 20, 25, 30, 35, 40, 45, 50]})"
]
},
"execution_count": 50,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"headers = get_headers()\n",
"\n",
"\n",
"def send_rest_request(request):\n",
" res = r.post(\n",
" f\"https://{REGION}-aiplatform.googleapis.com/v1/{tf_opt_gpu_endpoint}:predict\",\n",
" data=request,\n",
" headers=headers,\n",
" verify=False,\n",
" )\n",
" assert res.status_code == 200\n",
" return res\n",
"\n",
"\n",
"tf_opt_gpu_results = benchmark(\n",
" send_rest_request,\n",
" build_rest_request,\n",
" f\"{LOCAL_MODEL_DIR}/requests/requests_10_32.jsonl\",\n",
" [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],\n",
" 5,\n",
")\n",
"\n",
"tf_opt_gpu_results"
]
},
{
"cell_type": "code",
"execution_count": 51,
"id": "4467d381-a19b-4c33-98d1-8f8b0ea000a5",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO:tensorflow:benchmarking at 1 QPS, sending 10 requests\n",
"INFO:tensorflow:Running benchmark at 1 qps\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"INFO:tensorflow:received 10 responses.\n",
"INFO:tensorflow:num_qps:1 requests/second: 1.00 #success:0 #error:10 latencies: [avg:0.30ms p50:0.27ms p90:0.33ms p99:0.58ms]\n",
"INFO:tensorflow:benchmarking at 5 QPS, sending 25 requests\n",
"INFO:tensorflow:Running benchmark at 5 qps\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"INFO:tensorflow:num_qps:5 requests/second: 4.99 #success:0 #error:25 latencies: [avg:0.23ms p50:0.24ms p90:0.52ms p99:1.00ms]\n",
"INFO:tensorflow:benchmarking at 10 QPS, sending 50 requests\n",
"INFO:tensorflow:Running benchmark at 10 qps\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"INFO:tensorflow:num_qps:10 requests/second: 9.98 #success:0 #error:50 latencies: [avg:0.14ms p50:0.04ms p90:0.27ms p99:0.87ms]\n",
"INFO:tensorflow:benchmarking at 15 QPS, sending 75 requests\n",
"INFO:tensorflow:Running benchmark at 15 qps\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"INFO:tensorflow:num_qps:15 requests/second: 14.96 #success:0 #error:75 latencies: [avg:0.14ms p50:0.04ms p90:0.29ms p99:0.86ms]\n",
"INFO:tensorflow:benchmarking at 20 QPS, sending 100 requests\n",
"INFO:tensorflow:Running benchmark at 20 qps\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"INFO:tensorflow:num_qps:20 requests/second: 19.94 #success:0 #error:100 latencies: [avg:0.12ms p50:0.04ms p90:0.28ms p99:0.63ms]\n",
"INFO:tensorflow:benchmarking at 21 QPS, sending 105 requests\n",
"INFO:tensorflow:Running benchmark at 21 qps\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"INFO:tensorflow:num_qps:21 requests/second: 20.94 #success:0 #error:105 latencies: [avg:0.33ms p50:0.04ms p90:0.29ms p99:0.65ms]\n",
"INFO:tensorflow:benchmarking at 22 QPS, sending 110 requests\n",
"INFO:tensorflow:Running benchmark at 22 qps\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"INFO:tensorflow:num_qps:22 requests/second: 21.94 #success:0 #error:110 latencies: [avg:0.17ms p50:0.04ms p90:0.48ms p99:0.67ms]\n",
"INFO:tensorflow:benchmarking at 23 QPS, sending 115 requests\n",
"INFO:tensorflow:Running benchmark at 23 qps\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"INFO:tensorflow:num_qps:23 requests/second: 22.93 #success:0 #error:115 latencies: [avg:0.17ms p50:0.04ms p90:0.54ms p99:0.68ms]\n",
"INFO:tensorflow:benchmarking at 24 QPS, sending 120 requests\n",
"INFO:tensorflow:Running benchmark at 24 qps\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"INFO:tensorflow:num_qps:24 requests/second: 23.93 #success:0 #error:120 latencies: [avg:0.16ms p50:0.04ms p90:0.52ms p99:0.68ms]\n",
"INFO:tensorflow:benchmarking at 25 QPS, sending 125 requests\n",
"INFO:tensorflow:Running benchmark at 25 qps\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"name 'r' is not defined\n",
"INFO:tensorflow:num_qps:25 requests/second: 24.91 #success:0 #error:125 latencies: [avg:0.15ms p50:0.04ms p90:0.29ms p99:0.72ms]\n"
]
},
{
"data": {
"text/plain": [
"defaultdict(list,\n",
" {'reqested_qps': [1, 5, 10, 15, 20, 21, 22, 23, 24, 25],\n",
" 'actual_qps': [0.9988656052532046,\n",
" 4.992522234158652,\n",
" 9.980418477122322,\n",
" 14.964420512409697,\n",
" 19.94308691664755,\n",
" 20.940827726828378,\n",
" 21.937771123476345,\n",
" 22.93165397167941,\n",
" 23.925615780406833,\n",
" 24.912406629199157],\n",
" 'avg_latency': [0.30303001403808594,\n",
" 0.2335071563720703,\n",
" 0.1437520980834961,\n",
" 0.14041900634765625,\n",
" 0.12353658676147461,\n",
" 0.3323078155517578,\n",
" 0.16604553569446912,\n",
" 0.167896436608356,\n",
" 0.16114115715026855,\n",
" 0.1512012481689453],\n",
" 'p50': [0.26869773864746094,\n",
" 0.23937225341796875,\n",
" 0.04029273986816406,\n",
" 0.040531158447265625,\n",
" 0.03993511199951172,\n",
" 0.04076957702636719,\n",
" 0.04470348358154297,\n",
" 0.04172325134277344,\n",
" 0.042438507080078125,\n",
" 0.039577484130859375],\n",
" 'p90': [0.33054351806640614,\n",
" 0.5188941955566412,\n",
" 0.27430057525634766,\n",
" 0.28672218322753923,\n",
" 0.2783775329589844,\n",
" 0.2945423126220703,\n",
" 0.48184394836425837,\n",
" 0.5406379699707037,\n",
" 0.5233764648437501,\n",
" 0.28672218322753906],\n",
" 'p99': [0.5843019485473634,\n",
" 1.00111961364746,\n",
" 0.8677244186401358,\n",
" 0.8563232421875013,\n",
" 0.6256604194641141,\n",
" 0.6508445739746094,\n",
" 0.6688451766967772,\n",
" 0.680384635925293,\n",
" 0.6778383255004884,\n",
" 0.7153034210205081],\n",
" 'success': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0],\n",
" 'error': [10, 25, 50, 75, 100, 105, 110, 115, 120, 125]})"
]
},
"execution_count": 51,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"headers = get_headers()\n",
"\n",
"\n",
"def send_rest_request(request):\n",
" res = r.post(\n",
" f\"https://{REGION}-aiplatform.googleapis.com/v1/{tf_opt_lossy_gpu_endpoint}:predict\",\n",
" data=request,\n",
" headers=headers,\n",
" verify=False,\n",
" )\n",
" assert res.status_code == 200\n",
" return res\n",
"\n",
"\n",
"tf_opt_lossy_gpu_results = benchmark(\n",
" send_rest_request,\n",
" build_rest_request,\n",
" f\"{LOCAL_MODEL_DIR}/requests/requests_10_32.jsonl\",\n",
" [1, 5, 10, 15, 20, 21, 22, 23, 24, 25],\n",
" 5,\n",
")\n",
"\n",
"tf_opt_lossy_gpu_results"
]
},
{
"cell_type": "code",
"execution_count": 55,
"id": "001e74a9-b717-4263-8c82-a3e4e8826e60",
"metadata": {},
"outputs": [],
"source": [
"import matplotlib\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"\n",
"\n",
"def build_graph(x_key, y_key, results_dict, axis):\n",
" matplotlib.rcParams[\"figure.figsize\"] = [10.0, 7.0]\n",
"\n",
" fig, ax = plt.subplots(facecolor=(1, 1, 1))\n",
" ax.set_xlabel(\"QPS\")\n",
" ax.set_ylabel(\"Latency(ms)\")\n",
" for title, results in results_dict.items():\n",
" x = np.array(results[x_key])\n",
" y = np.array(results[y_key])\n",
" ax.plot(x, y, label=title)\n",
" ax.legend()\n",
" ax.axis(axis)\n",
" ax.set_title(f\"BERT base model {y_key} latency, batch size 32\")\n",
" return fig"
]
},
{
"cell_type": "code",
"execution_count": 56,
"id": "8932b151-ed51-4f6e-910a-5f979e710bbc",
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAnEAAAG5CAYAAADh3mJ8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAA9h0lEQVR4nO3deVxVdf7H8fdlCVBQBBFQEFFIARdU3EtcBlt0cDRzNEsszcqpbLOYdEbblBybtHSaNCtLHUxTMWs0sxxzl1JLUEMFE2RcMURFWc7vD3/ekQS8InA59no+Hj6Ge873nPM5X67ju+9ZvhbDMAwBAADAVBzsXQAAAACuHyEOAADAhAhxAAAAJkSIAwAAMCFCHAAAgAkR4gAAAEyIEAeYwKRJk3T//ffbu4wqcz3n16NHD7333ntVXJF56qgsTZo00VdffVXlx6ms7/Kjjz6qV155pRIqAsyLEAeUo0mTJnJzc5O7u7vq1aunvn376vDhw9b1I0aM0C233CJ3d3frnzZt2kiSMjIyZLFYrMubNGmihIQESVJERIR1uaOjo1xdXa2fJ0+ebJdz/S3o0aNHib5u3rx5ifVr165VixYtVKtWLfXs2VOHDh2qlONWV0CyF3sE2n/+85/6y1/+Uqn7fP755xUYGKg6deooKChIr732mnXdTz/9pP79+8vHx0deXl664447tG/fvko9PnC9CHHANXz22WfKy8tTdna2fH199cQTT5RY//zzzysvL8/6Z9euXSXWnz59Wnl5eVqyZIleeeUVrVmzRikpKdb2t99+u2bOnGn9/OKLL1bn6f3mXNnXV/4jfOLECQ0cOFCvvPKKTp06paioKP3xj3+0Y6WobiNHjtTevXuVm5urTZs2aeHChVq6dKmkS3+PY2NjtW/fPh09elQdO3ZU//797VwxfusIcYCNXF1dNWjQIKWmplZo+6ioKEVERGjnzp0V2j4/P19//OMf5eHhoXbt2pUIiwkJCWrWrJk8PDwUHh6uZcuWWdft379f0dHRqlu3rurXr18imOzdu1cxMTHy8vJS8+bN9cknn5R5/B49emjChAnq2rWr3N3d9fvf/14nT57UsGHDVKdOHXXo0EEZGRnW9ps2bVKHDh1Ut25ddejQQZs2bbKuS09PV3R0tDw8PBQTE6MTJ06UONaWLVvUtWtXeXp6qk2bNlq3bp1NfTRp0iQNGjSozH4qz9KlSxUREaF7771Xrq6umjRpknbt2qW9e/dec9sDBw6oV69e8vb2Vv369TVs2DCdPn1akvTAAw/o559/1u9//3u5u7tr6tSp1zzHHj166C9/+Yu6desmDw8P9enTp0QfbdiwwbptYGCgPvzwQ23fvl2+vr4qLCy0tvv0008VGRlp0/lL0vbt2xUeHq569erpwQcfVH5+viQpJydH/fr1k4+Pj+rVq6d+/fopMzNTkjR+/Hh9++23evzxx+Xu7q7HH39ckpSSkmL9bvn6+pYYYb548aKGDx8uDw8PRUREKDk5udR6DMPQ008/rQYNGqhu3bpq3bq1du/eLenSKPiECRMkydq3l/84ODjoww8/lHR93/HmzZurdu3a1s8ODg7av3+/JKljx44aOXKkvLy85OzsrKefflr79u3TyZMnbe5foNIZAMoUFBRkrFmzxjAMwzh79qwxfPhw44EHHrCuj4uLM8aPH1/qtunp6YYko6CgwDAMw9i8ebPh5uZmLF26tES76OhoY86cOeXWMXHiRMPJyclYvHixcfHiReNvf/ub0aRJE+PixYuGYRjGJ598YmRlZRlFRUVGYmKiUatWLePIkSOGYRjGkCFDjFdffdUoKioyzp8/b3z77beGYRhGXl6eERAQYLz//vtGQUGB8d133xne3t7G7t27S60hOjraaNasmbF//37j9OnTRlhYmBEaGmqsWbPGKCgoMB544AFjxIgRhmEYxsmTJw1PT0/jo48+MgoKCoyFCxcanp6exokTJwzDMIzOnTsbTz/9tJGfn2/85z//Mdzd3Y1hw4YZhmEYmZmZhpeXl/H5558bRUVFxpdffml4eXkZx44du2Z/XaufoqOjjfr16xve3t5G165djW+++ca67ZNPPmk8+uijJfYXERFhLFmypMz+uFxHWlqa8eWXXxr5+fnGsWPHjNtvv90YO3aste2V3yNbz7Fp06bGvn37jHPnzhnR0dHGCy+8YBiGYRw6dMhwd3c3Fi5caFy8eNE4ceKEsWPHDsMwDCMsLMz44osvrMf5wx/+YEybNq3U+n8tKCjIiIiIMH7++Wfj5MmTRteuXa3f7RMnThhLliwxzp49a+Tm5hqDBg0y+vfvX2pfGIZh5ObmGn5+fsa0adOM8+fPG7m5ucaWLVsMw7j0O3JxcTE+//xzo7Cw0IiPjzc6depUak2rVq0y2rVrZ+Tk5BjFxcVGamqq9Xtd1t+9f//734a/v7/x888/X/d33DAMY8qUKUbt2rUNSUZwcLBx+PDhUtstW7bM8PPzK79TgSpGiAPKERQUZNSuXduoW7eu4ejoaPj7+xs//PCDdX1cXJzh4uJi1K1b1/pn+PDhhmH8L8TVrVvXcHV1NSQZzz77rFFcXFziGLaGuCv/oSsqKjL8/PyM9evXl9q+TZs2xvLlyw3DMIwHHnjAePjhh6/6xygxMdG47bbbSiwbPXq0MWnSpFL3GR0dbbz66qvWz88884xx5513Wj+vWLHCaNOmjWEYhvHRRx8ZHTp0KLF9586djQ8++MA4dOiQ4ejoaOTl5VnXDR061BriEhISjPvvv7/Etn369DE+/PBDax3lhbjy+mnLli1Gbm6ukZ+fb3z44YeGu7u7sX//fsMwDOOhhx6yBqXLunbtanzwwQdl9kdZdSxbtsyIjIy0fv51iLPlHF955RXrulmzZhl33HGHYRiGMXnyZOMPf/hDqcdNSEgw7rvvPsMwLgVpNzc3a+i5lqCgIOOdd96xfv7888+Npk2bltp2x44dhqenp/Xzr/ti4cKFJc7/ShMnTjR69+5t/ZySkmK4urqW2nbt2rVGaGiosXnzZqOoqKjEutJC3L59+wwfHx/r7/t6v+OXFRcXG99//73x17/+1cjNzb1q/eHDh42GDRsaCxcuLHc/QFXjcipwDcuXL9fp06d14cIFzZw5U9HR0frvf/9rXf/cc8/p9OnT1j/z5s0rsf2JEyeUl5enadOmad26dSooKKhQHYGBgdafHRwcFBAQoCNHjkiSPvroI0VGRsrT01Oenp7avXu39fLb1KlTZRiGOnbsqIiICL3//vuSpEOHDmnr1q3WbTw9PbVgwYIS5/Zrvr6+1p/d3Nyu+pyXlydJOnLkiIKCgkpsGxQUpKysLB05ckT16tUrcdnqyraHDh3S4sWLS9S1YcMGZWdn33A/derUSR4eHnJxcVFcXJy6deumL774QpLk7u6u3NzcEvvKzc2Vh4fHNY957NgxDRkyRI0aNVKdOnV0//33X3WJ+Eq2nKOfn5/151q1aln79vDhw2rWrFmp+73//vut93B+8sknuv322+Xv73/N+i+7su+CgoKs/Xbu3Dk98sgjCgoKUp06ddS9e3edPn1aRUVFpe6nvBpLO7f8/PwSl4Ev69Wrlx5//HH96U9/kq+vr0aPHn3V7+iyX375Rf3799crr7yi22+/XVLFvuOSZLFY1LZtW7m5uWnixIkl1h0/flx9+vTRmDFjNHTo0HL3A1Q1QhxgI0dHRw0cOFCOjo7asGHDdW/77LPPytXVVf/4xz8qdPwrn4otLi5WZmamGjZsqEOHDunhhx/WzJkzdfLkSZ0+fVotW7aUYRiSLv2DOWfOHB05ckTvvvuuxowZo/379yswMFDR0dElAmheXp7eeeedCtV3pct1Xennn39Wo0aN5O/vr5ycHJ09e7bEussCAwP1wAMPlKjr7Nmzio+Pt+nYZfVTaSwWi7WfIiIiStw/d/bsWR04cEARERHXPOaf//xnWSwW/fDDD8rNzdX8+fOt+718nCvdyDkGBgbqwIEDpa5r1KiRunTpomXLlunjjz/WAw88cM39XenKvvv555+t/fbGG29o37592rp1q3Jzc7V+/XpJsp5jaedXVo3X68knn9R3332nlJQU/fTTT/rb3/52VZvi4mLdd9996tmzpx555JESddzId7ywsLDEeeTk5KhPnz6KjY3V+PHjb/zkgBtEiANsZBiGkpKSlJOTo7CwsArtIz4+XlOnTrXeMH49vvvuOy1dulSFhYWaPn26XFxc1LlzZ509e1YWi0U+Pj6SpA8++MB687ckLV682HoTer169WSxWOTo6Kh+/frpp59+0scff6yCggIVFBRo+/bt2rNnT4XO7Up33323fvrpJy1cuFCFhYVatGiRUlNT1a9fPwUFBSkqKkoTJ07UxYsXtWHDBn322WfWbS+PJq1evVpFRUXKz8/XunXrrOdQ0X46ffq0Vq9ebR31WbBggdavX6877rhDkjRgwADt3r1bn376qfLz8/Xyyy+rdevWatGixTWPeebMGbm7u8vT01NZWVlXBQ1fX18dPHiwUs5x2LBh+uqrr/TJJ5+osLBQJ0+eLPGwzPDhwzV16lT9+OOPGjBggHX5unXrrgpbvzZr1ixlZmbq1KlTmjx5svUhmDNnzsjNzU2enp46deqUXnrppXLPr1+/fvrvf/+r6dOn68KFCzpz5oy2bt16zXP7te3bt2vr1q0qKChQ7dq15erqKkdHx6vajR8/XmfPntWMGTNKLL+e73hxcbHeffdd5eTkyDAMbdu2TbNmzVLv3r0lXRqVveOOO9StWzfrq4IAeyPEAddw+cm3OnXqaPz48Zo3b16J0ZmpU6eWeDKufv36Ze6rb9++qlevnubMmXPddfTv31+LFi1SvXr19PHHH2vp0qVydnZWeHi4nn32WXXp0kW+vr768ccf1a1bN+t227dvV6dOneTu7q7Y2FjNmDFDwcHB8vDw0JdffqnExEQ1bNhQfn5+euGFF3ThwoXrru3XvL29tXLlSr3xxhvy9vbW1KlTtXLlSmvfLFy4UFu3bpWXl5deeuklDR8+3LptYGCgkpKSNHnyZPn4+CgwMFB/+9vfVFxcfEP9VFBQoAkTJsjHx0f169fX22+/reXLl1vfFefj46NPP/1U48ePV7169bR161YlJibadMyJEyfq+++/V926ddW3b18NHDiwxPo///nPevXVV+Xp6alp06bd0Dk2btxYX3zxhd544w15eXkpMjKyxAjigAEDdOjQIQ0YMKDEJevDhw+rS5cu5e77vvvuU58+fdS0aVM1bdrU+vTnU089pfPnz6t+/frq3Lmz7rzzzhLbjR07VkuWLFG9evX05JNPysPDQ2vWrNFnn30mPz8/hYaG6ptvvrnmuf1abm6uHn74YdWrV09BQUHy9vbWc889d1W7f/3rX9qyZYvq1atn/Xu4YMGC6/6OL1u2zPqU9/33368nnnjC+kqhZcuWafv27frggw9K/H2/chQZqG4W48oxfwAwsUmTJmn//v2aP3++vUuxq2bNmundd9/V7373O+uyUaNG6d5777WOPAIwPyd7FwAAqDyffvqpLBaLevXqVWL5zTRFGIBLquxy6kMPPaQGDRqoZcuW1mWnTp1STEyMQkNDFRMTo5ycHOu6KVOmKCQkRM2bN9fq1auty7/77ju1atVKISEhevLJJ8XAIQCUrkePHnrsscc0a9YsOThwtwxws6uyy6nr16+Xu7u7hg8fbr3J+vnnn5eXl5fi4+OVkJCgnJwcvf7660pNTdXQoUO1bds2HTlyRL/73e/0008/ydHRUR07dtSMGTPUuXNn3X333XryySd11113VUXJAAAAplFl/6nWvXt3eXl5lViWlJSkuLg4SVJcXJyWL19uXT5kyBC5uLgoODhYISEh2rZtm7Kzs5Wbm6suXbrIYrFo+PDh1m0AAAB+y6r1nrijR49aXzzp7++vY8eOSZKysrLUuXNna7uAgABlZWXJ2dlZAQEBVy0vy+zZszV79mxJl+bLs+XVAAAAAPaWkZFR7kvCS1MjHmwo7YrulS/h/PXysowePVqjR4+WdGmy8bImVQYAAKhJoqKirnubar3z1dfX1zqtTHZ2tho0aCDp0gjblW8Kv/yG9YCAgBIvvyzvzesAAAC/JdUa4mJjY63zSs6bN0/9+/e3Lk9MTNSFCxeUnp6utLQ0dezYUf7+/vLw8NCWLVtkGIY++ugj6zYAAAC/ZVV2OXXo0KFat26dTpw4oYCAAL300kuKj4/X4MGDNXfuXDVu3FiLFy+WdGnOwsGDBys8PFxOTk6aNWuWdWqVd955RyNGjND58+d111138WQqAACAbuIZG7gnDgBgVgUFBcrMzKzQPMuo2VxdXRUQECBnZ+cSyyuSW2rEgw0AAOB/MjMz5eHhoSZNmpT7QB/MxTAMnTx5UpmZmQoODr7h/fFKbwAAapj8/Hx5e3sT4G4yFotF3t7elTbCSogDAKAGIsDdnCrz90qIAwAAMCFCHAAAKOHkyZOKjIxUZGSk/Pz81KhRI+tni8Vi/TkyMlIZGRlas2aN2rdvr1atWql9+/b6+uuvS93vzp071blzZ0VGRioqKkrbtm0rtV1aWpr69eunZs2aqX379urZs6fWr18vSfrwww/l4+OjyMhIhYeHa86cOZKkSZMmadq0aSX206RJk+ueBcFMeLABAACU4O3trZ07d0q6FI7c3d313HPPSZLc3d2t6y7LycnRZ599poYNG2r37t264447Sp0m8/nnn9fEiRN111136YsvvtDzzz+vdevWlWiTn5+vvn37atq0aYqNjZUk7d69W8nJyerevbsk6Y9//KNmzpypY8eOKSIiwtrut4YQBwAAbkjbtm2tP0dERCg/P18XLlyQi4tLiXYWi0W5ubmSpF9++aXUWZgWLFigLl26lAhmLVu2VMuWLa9q26BBAzVr1kyHDh2qrFMxFUIcAAA12EufpSj1SG6l7jO8YR1N/H1EhbY9f/68IiMjJUnBwcFatmxZifWffvqp2rZte1WAk6Tp06frjjvu0HPPPafi4mJt2rTpqjYpKSlq166dTbUcPHhQBw8eVEhIyPWfyE2AEAcAAGzm5uZ21eXUy1JSUvTCCy/oyy+/LHX9O++8ozfffFP33HOPPvnkE40cOVJfffVVuccbMGCA0tLSdOutt2rp0qWSpEWLFmnDhg1ycXHRu+++Ky8vrzKf+ryZn/IlxAEAUINVdMSsumVmZmrAgAH66KOP1KxZs1LbzJs3TzNmzJAk3XvvvRo1atRVbSIiIqwPMUjSsmXLlJycbL0nT/rfPXFX8vb2VnZ2dollZ86ckaenZ0VPqcbj6VQAAHBDTp8+rb59+2rKlCnq1q1bme0aNmyo//znP5Kkr7/+WqGhoVe1ue+++7Rx40atWLHCuuzcuXPXrKF79+5asWKFzpw5I0launSp2rRpY52L/WbESBwAALghM2fO1P79+/XKK6/olVdekSR9+eWXatCggUaNGqVHH31UUVFRmjNnjsaOHavCwkK5urpq9uzZV+3Lzc1NK1eu1DPPPKOnnnpKvr6+8vDw0IQJE8qtoXXr1nr88cd12223yWKxqEGDBnrvvfeq5HxrCothGIa9i6gKFZlIFgCAmmDPnj0KCwuzdxmoIqX9fiuSW7icCgAAYEKEOAAAABMixAEAAJgQIQ4AAMCECHEAAAAmRIgDAAAwIUIcAAAo4eTJk4qMjFRkZKT8/PzUqFEj62eLxWL9OTIyUhkZGTd8vMmTJ5e5Li8vT4899piaNWumtm3bqn379pozZ44kKSMjQ25uboqMjFR4eLgeffRRFRcXa926derXr1+J/YwYMUJLliy54VprEl72CwAASvD29rbOjzpp0iS5u7tbp71yd3cvc+7Uipo8ebJefPHFUteNGjVKTZs2VVpamhwcHHT8+HG9//771vXNmjXTzp07VVhYqF69emn58uXy8vKq1PpqKkbiAABApTEMQ+PGjVPLli3VqlUrLVq0SJK0bt06de/eXQMGDCgxahYfH6/z588rMjJSw4YNK7GvAwcOaNu2bXr11Vfl4HApsvj4+OiFF1646rhOTk7q2rWr9u/fX/UnWUMwEgcAQE3273jpvz9W7j79Wkl3JVRo08uBS5KCg4O1bNmyEuuXLl2qnTt3ateuXTpx4oQ6dOig7t27S5K2bdum1NRUBQUF6c4779TSpUuVkJCgmTNnljq6l5KSojZt2lgDXHnOnTuntWvX6uWXX67QeZkRI3EAAMBmbm5u2rlzp3bu3HlVgJOkDRs2aOjQoXJ0dJSvr6+io6O1fft2SVLHjh3VtGlTOTo6aujQodqwYcN1Hfu1115TZGSkGjZsaF124MABRUZGqlu3burbt6/uuusuWSyWUrcva7lZMRIHAEBNVsERM3spb0r2X4eoa4Wq8PBw7dq1S8XFxXJwcND48eM1fvx4ubu7W9tcvifuSt7e3srJySmx7NSpU6pfv76NZ2EOjMQBAIBK0717dy1atEhFRUU6fvy41q9fr44dO0q6dDk1PT1dxcXFWrRokW677TZJkrOzswoKCq7aV0hIiKKiojRhwgQVFRVJkvLz88sNipIUGhqqI0eOaM+ePZKkQ4cOadeuXdbLwDcLRuIAAEClGTBggDZv3qw2bdrIYrFo6tSp8vPz0969e9WlSxfFx8frxx9/tD7kIEmjR49W69at1a5dOy1YsKDE/t577z2NGzdOISEh8vLykpubm15//fVya3BxcdH8+fP14IMPKj8/X87OznrvvfdUt27dKjtve7AY14qzJhUVFaXk5GR7lwEAwHXbs2ePwsLC7F1GpVq3bp2mTZumlStX2rsUuyvt91uR3MLlVAAAABPicioAAKhyPXr0UI8ePexdxk2FkTgAAAATIsQBAACYECEOAADAhAhxAAAAJkSIAwAAJZw8eVKRkZGKjIyUn5+fGjVqZP1ssVisP0dGRiojI+OGjzd58uQy1+Xl5emxxx5Ts2bN1LZtW7Vv315z5syRJGVkZMjNzU2RkZEKDw/Xo48+quLiYq1bt079+vUrsZ8RI0ZoyZIlV+2/rOVmwNOpAACgBG9vb+tUVpMmTZK7u7uee+45SZK7u3upk9XfiMmTJ+vFF18sdd2oUaPUtGlTpaWlycHBQcePH9f7779vXX952q3CwkL16tVLy5cvl5eXV6XWV1MxEgcAACqNYRgaN26cWrZsqVatWmnRokWSLr3s9/IsDVeOmsXHx+v8+fOKjIzUsGHDSuzrwIED2rZtm1599VU5OFyKLD4+PnrhhReuOq6Tk5O6du2q/fv3V7j2tWvXqm3btmrVqpUeeughXbhwQZIUHx+v8PBwtW7d2hpmFy9erJYtW6pNmzbq3r27JOn2228vEXC7deumH374ocL1XAsjcQAA1GCvb3tde0/trdR9tvBqoRc6Xh2EbHE5cElScHCwli1bVmL90qVLtXPnTu3atUsnTpxQhw4drCFn27ZtSk1NVVBQkO68804tXbpUCQkJmjlzZqmjeykpKWrTpo01wJXn3LlzWrt2rV5++eUKnVd+fr5GjBihtWvX6tZbb9Xw4cP1zjvvaPjw4Vq2bJn27t0ri8Wi06dPS5JefvllrV69Wo0aNbIuGzVqlD788ENNnz5dP/30ky5cuKDWrVtXqB5bMBIHAABs5ubmpp07d2rnzp1XBThJ2rBhg4YOHSpHR0f5+voqOjpa27dvlyR17NhRTZs2laOjo4YOHaoNGzZc17Ffe+01RUZGqmHDhtZlBw4cUGRkpLp166a+ffvqrrvuksViKXX7spZL0r59+xQcHKxbb71VkhQXF6f169erTp06cnV11ahRo7R06VLVqlVL0qVRthEjRmjOnDkqKiqSJN17771auXKlCgoK9P7772vEiBHXdX7Xi5E4AABqsIqOmNlLeVOy/zpElReqJCk8PFy7du1ScXGxHBwcNH78eI0fP17u7u7WNpfvibuSt7e3cnJySiw7deqU6tevf911Ozk5adu2bVq7dq0SExM1c+ZMff311/rnP/+prVu36vPPP1dkZKR27twpb29vxcTEKCkpSZ988kmVz+HOSBwAAKg03bt316JFi1RUVKTjx49r/fr16tixo6RLl1PT09NVXFysRYsW6bbbbpMkOTs7q6Cg4Kp9hYSEKCoqShMmTLCOduXn55cbFCUpNDRUR44c0Z49eyRJhw4d0q5du6yXgUvTokULZWRkWO+p+/jjjxUdHa28vDz98ssvuvvuuzV9+nRrYDxw4IA6deqkl19+WfXr19fhw4clXbqk+uSTT6pDhw5V/oAFI3EAAKDSDBgwQJs3b1abNm1ksVg0depU+fn5ae/everSpYvi4+P1448/Wh9ykKTRo0erdevWateunRYsWFBif++9957GjRunkJAQeXl5yc3NTa+//nq5Nbi4uGj+/Pl68MEHlZ+fL2dnZ7333nuqW7dumdu4urrqgw8+0L333qvCwkJ16NBBjz76qE6dOqX+/ftbw+Obb74pSRo3bpzS0tJkGIZ69+6tNm3aSJLat2+vOnXq6MEHH7yRbrSJxbhWnDWpqKioKh/GBACgKuzZs0dhYWH2LqNSrVu3TtOmTdPKlSvtXUqVOnLkiHr06KG9e/eW+UBGab/fiuQWLqcCAABUgo8++kidOnXSa6+9ZtMTtTeKy6kAAKDK9ejRQz169LB3GVVq+PDhGj58eLUdj5E4AABqoJv0bqffvMr8vRLiAACoYVxdXXXy5EmC3E3GMAydPHlSrq6ulbI/LqcCAFDDBAQEKDMzU8ePH7d3Kahkrq6uCggIqJR9EeIAAKhhnJ2dFRwcbO8yUMNxORUAAMCECHEAAAAmRIgDAAAwIUIcAACACRHiAAAATIgQBwAAYEKEOAAAABMixAEAAJgQIQ4AAMCECHEAAAAmRIgDAAAwIUIcAACACRHiAAAATIgQBwAAYEKEOAAAABMixAEAAJgQIQ4AAMCECHEAAAAmZJcQ9+abbyoiIkItW7bU0KFDlZ+fr1OnTikmJkahoaGKiYlRTk6Otf2UKVMUEhKi5s2ba/Xq1fYoGQAAoEap9hCXlZWlt956S8nJydq9e7eKioqUmJiohIQE9e7dW2lpaerdu7cSEhIkSampqUpMTFRKSopWrVqlMWPGqKioqLrLBgAAqFHsMhJXWFio8+fPq7CwUOfOnVPDhg2VlJSkuLg4SVJcXJyWL18uSUpKStKQIUPk4uKi4OBghYSEaNu2bfYoGwAAoMao9hDXqFEjPffcc2rcuLH8/f1Vt25d9enTR0ePHpW/v78kyd/fX8eOHZN0aeQuMDDQun1AQICysrJK3ffs2bMVFRWlqKgoHT9+vOpPBgAAwE6qPcTl5OQoKSlJ6enpOnLkiM6ePav58+eX2d4wjKuWWSyWUtuOHj1aycnJSk5Olo+PT6XVDAAAUNNUe4j76quvFBwcLB8fHzk7O2vgwIHatGmTfH19lZ2dLUnKzs5WgwYNJF0aeTt8+LB1+8zMTDVs2LC6ywYAAKhRqj3ENW7cWFu2bNG5c+dkGIbWrl2rsLAwxcbGat68eZKkefPmqX///pKk2NhYJSYm6sKFC0pPT1daWpo6duxY3WUDAADUKE7VfcBOnTpp0KBBateunZycnNS2bVuNHj1aeXl5Gjx4sObOnavGjRtr8eLFkqSIiAgNHjxY4eHhcnJy0qxZs+To6FjdZQMAANQoFqO0m85uAlFRUUpOTrZ3GQAAANdUkdzCjA0AAAAmRIgDAAAwIUIcAACACRHiAAAATIgQBwAAYEKEOAAAABMixAEAAJgQIQ4AAMCECHEAAAAmRIgDAAAwIUIcAACACRHiAAAATIgQBwAAYEKEOAAAABMixAEAAJgQIQ4AAMCECHEAAAAmRIgDAAAwIUIcAACACRHiAAAATIgQBwAAYEKEOAAAABMixAEAAJgQIQ4AAMCECHEAAAAmRIgDAAAwIUIcAACACRHiAAAATIgQBwAAYEKEOAAAABMixAEAAJgQIQ4AAMCECHEAAAAmRIgDAAAwIUIcAACACRHiAAAATIgQBwAAYEKEOAAAABMixAEAAJgQIQ4AAMCECHEAAAAmRIgDAAAwIUIcAACACRHiAAAATIgQBwAAYEKEOAAAABMixAEAAJgQIQ4AAMCECHEAAAAmRIgDAAAwIUIcAACACRHiAAAATIgQBwAAYEKEOAAAABMixAEAAJgQIQ4AAMCECHEAAAAmRIgDAAAwIUIcAACACRHiAAAATIgQBwAAYEKEOAAAABMixAEAAJgQIQ4AAMCECHEAAAAmRIgDAAAwIUIcAACACdklxJ0+fVqDBg1SixYtFBYWps2bN+vUqVOKiYlRaGioYmJilJOTY20/ZcoUhYSEqHnz5lq9erU9SgYAAKhR7BLixo4dqzvvvFN79+7Vrl27FBYWpoSEBPXu3VtpaWnq3bu3EhISJEmpqalKTExUSkqKVq1apTFjxqioqMgeZQMAANQY1R7icnNztX79eo0cOVKSdMstt8jT01NJSUmKi4uTJMXFxWn58uWSpKSkJA0ZMkQuLi4KDg5WSEiItm3bVt1lAwAA1CjVHuIOHjwoHx8fPfjgg2rbtq1GjRqls2fP6ujRo/L395ck+fv769ixY5KkrKwsBQYGWrcPCAhQVlZWqfuePXu2oqKiFBUVpePHj1f9yQAAANhJtYe4wsJCff/993rssce0Y8cO1a5d23rptDSGYVy1zGKxlNp29OjRSk5OVnJysnx8fCqtZgAAgJqm2kNcQECAAgIC1KlTJ0nSoEGD9P3338vX11fZ2dmSpOzsbDVo0MDa/vDhw9btMzMz1bBhw+ouGwAAoEap9hDn5+enwMBA7du3T5K0du1ahYeHKzY2VvPmzZMkzZs3T/3795ckxcbGKjExURcuXFB6errS0tLUsWPH6i4bAACgRnGyx0HffvttDRs2TBcvXlTTpk31wQcfqLi4WIMHD9bcuXPVuHFjLV68WJIUERGhwYMHKzw8XE5OTpo1a5YcHR3tUTYAAECNYTFKu+nsJhAVFaXk5GR7lwEAAHBNFcktzNgAAABgQoQ4AAAAEyLEAQAAmBAhDgAAwIRsfjo1JydHR44ckZubm5o0aSIHB/IfAACAvZQb4n755RfNmjVL//rXv3Tx4kX5+PgoPz9fR48eVefOnTVmzBj17NmzumoFAADA/ys3xA0aNEjDhw/Xt99+K09PzxLrvvvuO3388cc6ePCgdTJ7AAAAVI9yQ9yaNWvKXNe+fXu1b9++0gsCAADAtdl0Y9vGjRt19uxZSdL8+fP1zDPP6NChQ1VaGAAAAMpmU4h77LHHVKtWLe3atUtTp05VUFCQhg8fXtW1AQAAoAw2hTgnJydZLBYlJSVp7NixGjt2rM6cOVPVtQEAAKAMNr1ixMPDQ1OmTNH8+fO1fv16FRUVqaCgoKprAwAAQBlsGolbtGiRXFxcNHfuXPn5+SkrK0vjxo2r6toAAABQBothGIatjXNzc1VYWGj97OXlVSVFVYaoqCglJyfbuwwAAIBrqkhusely6rvvvqu//vWvcnNzk8VikSRZLBYdPHjw+qsEAADADbMpxE2bNk0pKSmqX79+VdcDAAAAG9h0T1yzZs1Uq1atqq4FAAAANrJpJG7KlCnq2rWrOnXqJBcXF+vyt956q8oKAwAAQNlsCnGPPPKIevXqpVatWsnBwabBOwAAAFQhm0Kck5OT/v73v1d1LQAAALCRTcNqPXv21OzZs5Wdna1Tp05Z/wAAAMA+bBqJW7hwoaRL98ZdxitGAAAA7MemEJeenl7VdQAAAOA6lHs5dcOGDeVunJubq927d1dqQQAAALi2ckfiPv30Uz3//PO688471b59e/n4+Cg/P1/79+/XN998o0OHDumNN96orloBAADw/8oNcW+++aZycnK0ZMkSLV68WNnZ2XJzc1NYWJgeeeQR3XbbbdVVJwAAAK5gMQzDuFajoqIiOTo6Vkc9laYiE8kCAADYQ0Vyi02vGAkJCdG4ceO0Z8+eChUGAACAymVTiPvhhx906623auTIkercubNmz56t3Nzcqq4NAAAAZbApxHl4eOjhhx/Wpk2bNHXqVL300kvy9/dXXFyc9u/fX9U1AgAA4FdsCnFFRUVasWKFBgwYoLFjx+rZZ5/VwYMH9fvf/1533313VdcIAACAX7HpZb+hoaHq2bOnxo0bp65du1qXDxo0SOvXr6+y4gAAAFA6m0LcDz/8IHd391LXvfXWW5VaEAAAAK7Npsupf/rTn3T69Gnr55ycHD300ENVVRMAAACuweanUz09Pa2f69Wrpx07dlRVTQAAALgGm0JccXGxcnJyrJ9PnTqlwsLCKisKAAAA5bPpnrhnn31WXbt21aBBgyRJixcv1vjx46u0MAAAAJTNphA3fPhwtW/fXt98840Mw9DSpUsVHh5e1bUBAACgDDaFOElq0aKF6tWrZ72M+vPPP6tx48ZVVhgAAADKZlOIe/vtt/XSSy/J19dXjo6OMgxDFotFP/zwQ1XXBwAAgFLYFOJmzJihffv2ydvbu6rrAQAAgA1sejo1MDBQdevWrepaAAAAYCObRuKaNm2qHj16qG/fvnJxcbEuf+aZZ6qsMAAAAJTNphDXuHFjNW7cWBcvXtTFixeruiYAAABcg00hbuLEiZKks2fPqnbt2lVaEAAAAK7NpnviNm/erPDwcIWFhUmSdu3apTFjxlRpYQAAACibTSHuqaee0urVq61Pp7Zp00br16+v0sIAAABQNptCnHTpCdUrOTo6VnoxAAAAsI1N98QFBgZq06ZNslgsunjxot566y3rpVUAAABUP5tG4v75z39q1qxZysrKUkBAgHbu3Kl//OMfVV0bAAAAymDTSNy+ffu0YMGCEss2btyobt26VUlRAAAAKJ9NI3FPPPGETcsAAABQPcodidu8ebM2bdqk48eP6+9//7t1eW5uroqKiqq8OAAAAJSu3BB38eJF5eXlqbCwUGfOnLEur1OnjpYsWVLlxQEAAKB05Ya46OhoRUdHa8SIEQoKCqqumgAAAHANNj3YUKtWLY0bN04pKSnKz8+3Lv/666+rrDAAAACUzaYHG4YNG6YWLVooPT1dEydOVJMmTdShQ4eqrg0AAABlsCnEnTx5UiNHjpSzs7Oio6P1/vvva8uWLVVdGwAAAMpg0+VUZ2dnSZK/v78+//xzNWzYUJmZmVVaGAAAAMpmU4ibMGGCfvnlF73xxht64oknlJubq+nTp1dxaQAAACiLTSGuX79+kqS6devqm2++kSRCHAAAgB3ZdE9caa58+S8AAACqV4VDnGEYlVkHAAAArkOFQ5zFYqnMOgAAAHAdyr0nzsPDo9SwZhiGzp8/X2VFAQAAoHzlhrgr50sFAABAzVHhy6kAAACwH0IcAACACdktxBUVFalt27bWd9CdOnVKMTExCg0NVUxMjHJycqxtp0yZopCQEDVv3lyrV6+2V8kAAAA1ht1C3IwZMxQWFmb9nJCQoN69eystLU29e/dWQkKCJCk1NVWJiYlKSUnRqlWrNGbMGBUVFdmrbAAAgBrBLiEuMzNTn3/+uUaNGmVdlpSUpLi4OElSXFycli9fbl0+ZMgQubi4KDg4WCEhIdq2bZs9ygYAAKgx7BLinnrqKU2dOlUODv87/NGjR+Xv7y9J8vf317FjxyRJWVlZCgwMtLYLCAhQVlZWqfudPXu2oqKiFBUVpePHj1fhGQAAANhXtYe4lStXqkGDBmrfvr1N7UubGaKsFw2PHj1aycnJSk5Olo+Pzw3VCQAAUJOV+564qrBx40atWLFCX3zxhfLz85Wbm6v7779fvr6+ys7Olr+/v7Kzs9WgQQNJl0beDh8+bN0+MzNTDRs2rO6yAQAAapRqH4mbMmWKMjMzlZGRocTERPXq1Uvz589XbGys5s2bJ0maN2+e+vfvL0mKjY1VYmKiLly4oPT0dKWlpaljx47VXTYAAECNUu0jcWWJj4/X4MGDNXfuXDVu3FiLFy+WJEVERGjw4MEKDw+Xk5OTZs2aJUdHRztXCwAAYF8Wo7Sbzm4CUVFRSk5OtncZAAAA11SR3MKMDQAAACZEiAMAADAhQhwAAIAJEeIAAABMiBAHAABgQoQ4AAAAEyLEAQAAmBAhDgAAwIQIcQAAACZEiAMAADAhQhwAAIAJEeIAAABMiBAHAABgQoQ4AAAAEyLEAQAAmBAhDgAAwIQIcQAAACZEiAMAADAhQhwAAIAJEeIAAABMiBAHAABgQoQ4AAAAEyLEAQAAmBAhDgAAwIQIcQAAACZEiAMAADAhQhwAAIAJEeIAAABMiBAHAABgQoQ4AAAAEyLEAQAAmBAhDgAAwIQIcQAAACZEiAMAADAhQhwAAIAJEeIAAABMiBAHAABgQoQ4AAAAEyLEAQAAmBAhDgAAwIQIcQAAACZEiAMAADAhQhwAAIAJEeIAAABMiBAHAABgQoQ4AAAAEyLEAQAAmBAhDgAAwIQIcQAAACZEiAMAADAhQhwAAIAJEeIAAABMiBAHAABgQoQ4AAAAEyLEAQAAmBAhDgAAwIQIcQAAACZEiAMAADAhQhwAAIAJEeIAAABMiBAHAABgQoQ4AAAAEyLEAQAAmBAhDgAAwIQIcQAAACZEiAMAADAhQhwAAIAJVXuIO3z4sHr27KmwsDBFRERoxowZkqRTp04pJiZGoaGhiomJUU5OjnWbKVOmKCQkRM2bN9fq1auru2QAAIAap9pDnJOTk9544w3t2bNHW7Zs0axZs5SamqqEhAT17t1baWlp6t27txISEiRJqampSkxMVEpKilatWqUxY8aoqKioussGAACoUao9xPn7+6tdu3aSJA8PD4WFhSkrK0tJSUmKi4uTJMXFxWn58uWSpKSkJA0ZMkQuLi4KDg5WSEiItm3bVt1lAwAA1Ch2vScuIyNDO3bsUKdOnXT06FH5+/tLuhT0jh07JknKyspSYGCgdZuAgABlZWWVur/Zs2crKipKUVFROn78eNWfAAAAgJ3YLcTl5eXpnnvu0fTp01WnTp0y2xmGcdUyi8VSatvRo0crOTlZycnJ8vHxqbRaAQAAahq7hLiCggLdc889GjZsmAYOHChJ8vX1VXZ2tiQpOztbDRo0kHRp5O3w4cPWbTMzM9WwYcPqLxoAAKAGqfYQZxiGRo4cqbCwMD3zzDPW5bGxsZo3b54kad68eerfv791eWJioi5cuKD09HSlpaWpY8eO1V02AABAjeJU3QfcuHGjPv74Y7Vq1UqRkZGSpMmTJys+Pl6DBw/W3Llz1bhxYy1evFiSFBERocGDBys8PFxOTk6aNWuWHB0dq7tsAACAGsVilHbT2U0gKipKycnJ9i4DAADgmiqSW5ixAQAAwIQIcQAAACZEiAMAADAhQhwAAIAJEeIAAABMiBAHAABgQoQ4AAAAEyLEAQAAmBAhDgAAwIQIcQAAACZEiAMAADAhQhwAAIAJEeIAAABMiBAHAABgQoQ4AAAAEyLEAQAAmBAhDgAAwIQIcQAAACZEiAMAADAhQhwAAIAJEeIAAABMiBAHAABgQoQ4AAAAEyLEAQAAmBAhDgAAwIQIcQAAACZEiAMAADAhQhwAAIAJEeIAAABMiBAHAABgQoQ4AAAAEyLEAQAAmBAhDgAAwIQIcQAAACZEiAMAADAhQhwAAIAJEeIAAABMiBAHAABgQoQ4AAAAEyLEAQAAmBAhDgAAwIQIcQAAACZEiAMAADAhQhwAAIAJEeIAAABMiBAHAABgQoQ4AAAAEyLEAQAAmBAhDgAAwIQIcQAAACZEiAMAADAhQhwAAIAJEeIAAABMiBAHAABgQoQ4AAAAEyLEAQAAmBAhDgAAwIQIcQAAACZEiAMAADAhQhwAAIAJEeIAAABMiBAHAABgQoQ4AAAAEyLEAQAAmBAhDgAAwIQIcQAAACZEiAMAADAh04S4VatWqXnz5goJCVFCQoK9ywEAALArJ3sXYIuioiL96U9/0po1axQQEKAOHTooNjZW4eHh9i7tuuWdP6v/5pywfrZc/l/L5c+W/62zlGxz5YfL7SwlVl7ezvLr5lf9b8kNrl5nKW3HNUQNLs0UXJyc5Oz4///9VqIzr/jZYrniu1hy+a9Z25Wzr0v/Yyl7XyW+s2X8gg2j9OP+avmvNir9S1/KNv87bjn7M4xS6iulvWGU8neojHaVvL/fnMr8PwTnWpKDY+XtD6hipghx27ZtU0hIiJo2bSpJGjJkiJKSkkwZ4lZvnqdJh9+xdxkAfsMsNoY9W+NRZbezlc3HtTHbWpxukSymuUAFmCPEZWVlKTAw0Po5ICBAW7duvard7NmzNXv2bEnS7t27FRUVVW01Xg8Xudi7hHIdP35cPj4+9i7DtOi/G0P/3Rj6r+LouxtD/92YvXv3Xvc2pghxRmmXPkoZQh89erRGjx4tSYqKilJycnKV13Yzou9uDP13Y+i/G0P/VRx9d2PovxtTkYEnU4wbBwQE6PDhw9bPmZmZatiwoR0rAgAAsC9ThLgOHTooLS1N6enpunjxohITExUbG2vvsgAAAOzGFJdTnZycNHPmTN1xxx0qKirSQw89pIiIiHK3uXxZFdePvrsx9N+Nof9uDP1XcfTdjaH/bkxF+s9ilHbDGQAAAGo0U1xOBQAAQEmEOAAAABO66UIc03NV3OHDh9WzZ0+FhYUpIiJCM2bMsHdJplNUVKS2bduqX79+9i7FdE6fPq1BgwapRYsWCgsL0+bNm+1dkqm8+eabioiIUMuWLTV06FDl5+fbu6Qa7aGHHlKDBg3UsmVL67JTp04pJiZGoaGhiomJUU5Ojh0rrNlK679x48apRYsWat26tQYMGKDTp0/br8AarLS+u2zatGmyWCw6ceJEKVte7aYKcZen5/r3v/+t1NRU/etf/1Jqaqq9yzINJycnvfHGG9qzZ4+2bNmiWbNm0X/XacaMGQoLC7N3GaY0duxY3Xnnndq7d6927dpFP16HrKwsvfXWW0pOTtbu3btVVFSkxMREe5dVo40YMUKrVq0qsSwhIUG9e/dWWlqaevfuzUBAOUrrv5iYGO3evVs//PCDbr31Vk2ZMsVO1dVspfWddGkgZc2aNWrcuLHN+7qpQtyV03Pdcsst1um5YBt/f3+1a9dOkuTh4aGwsDBlZWXZuSrzyMzM1Oeff65Ro0bZuxTTyc3N1fr16zVy5EhJ0i233CJPT0/7FmUyhYWFOn/+vAoLC3Xu3DnepXkN3bt3l5eXV4llSUlJiouLkyTFxcVp+fLldqjMHErrvz59+sjJ6dJLLzp37qzMzEx7lFbjldZ3kvT0009r6tSp1zV3+U0V4kqbnosQUjEZGRnasWOHOnXqZO9STOOpp57S1KlT5eBwU/21qhYHDx6Uj4+PHnzwQbVt21ajRo3S2bNn7V2WaTRq1EjPPfecGjduLH9/f9WtW1d9+vSxd1mmc/ToUfn7+0u69B+1x44ds3NF5vX+++/rrrvusncZprFixQo1atRIbdq0ua7tbqp/bWydngvly8vL0z333KPp06erTp069i7HFFauXKkGDRqoffv29i7FlAoLC/X999/rscce044dO1S7dm0uZV2HnJwcJSUlKT09XUeOHNHZs2c1f/58e5eF36jXXntNTk5OGjZsmL1LMYVz587ptdde08svv3zd295UIY7puW5cQUGB7rnnHg0bNkwDBw60dzmmsXHjRq1YsUJNmjTRkCFD9PXXX+v++++3d1mmERAQoICAAOvI76BBg/T999/buSrz+OqrrxQcHCwfHx85Oztr4MCB2rRpk73LMh1fX19lZ2dLkrKzs9WgQQM7V2Q+8+bN08qVK7VgwQIGUWx04MABpaenq02bNmrSpIkyMzPVrl07/fe//73mtjdViGN6rhtjGIZGjhypsLAwPfPMM/Yux1SmTJmizMxMZWRkKDExUb169WIk5Dr4+fkpMDBQ+/btkyStXbtW4eHhdq7KPBo3bqwtW7bo3LlzMgxDa9eu5cGQCoiNjdW8efMkXQoj/fv3t3NF5rJq1Sq9/vrrWrFihWrVqmXvckyjVatWOnbsmDIyMpSRkaGAgAB9//338vPzu+a2N1WIu3J6rrCwMA0ePPia03PhfzZu3KiPP/5YX3/9tSIjIxUZGakvvvjC3mXhN+Ltt9/WsGHD1Lp1a+3cuVMvvviivUsyjU6dOmnQoEFq166dWrVqpeLiYqZAuoahQ4eqS5cu2rdvnwICAjR37lzFx8drzZo1Cg0N1Zo1axQfH2/vMmus0vrv8ccf15kzZxQTE6PIyEg9+uij9i6zRiqt7yqKabcAAABM6KYaiQMAAPitIMQBAACYECEOAADAhAhxAAAAJkSIAwAAMCFCHAD8v8zMTPXv31+hoaFq2rSpHn/8cV24cEHr1q1T3bp11bZtW4WFhemll16SdOlN68OGDVOrVq3UsmVL3XbbbcrLy7PzWQD4rSDEAYAuvex64MCB+sMf/qC0tDSlpaXp/Pnzev755yVJt99+u3bs2KHk5GTNnz9f3333nWbMmCFfX1/9+OOP2r17t+bOnStnZ2c7nwmA3wonexcAADXB119/LVdXVz344IOSJEdHR7355psKCgpSTEyMtV3t2rXVvn17HThwQNnZ2QoKCrKua968ebXXDeC3i5E4AJCUkpKi9u3bl1hWp04dNWnSRPv377cuO3nypLZs2aKIiAg99NBDev3119WlSxdNmDBBaWlp1V02gN8wRuIAQJcup5Y2YfflSW2+/fZbtW3bVg4ODoqPj7dO6Xfw4EF9+eWX+uqrr9ShQwdt3ryZeUsBVAtCHABIioiI0KefflpiWW5uro4eParmzZvr9ttv18qVK6/azt3dXQMHDtTAgQPl4OCgL774ghAHoFpwORUAJPXu3Vvnzp3TRx99JEkqKirSs88+q8cff1xubm6lbrNx40bl5ORIki5evKjU1NQS98gBQFUixAGAJIvFomXLlmnJkiUKDQ2Vt7e3HBwcNH78+DK3OXDggKKjo9WqVSu1bdtWUVFRuueee6qxagC/ZRbj8g0fAACrTZs2aejQoVq6dOlVDzwAQE1AiAMAADAhLqcCAACYECEOAADAhAhxAAAAJkSIAwAAMCFCHAAAgAkR4gAAAEzo/wARkj4eatHhRwAAAABJRU5ErkJggg==\n",
"text/plain": [
"<Figure size 720x504 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"fig = build_graph(\n",
" \"actual_qps\",\n",
" \"p50\",\n",
" {\n",
" \"TF2.8 GPU\": tf28_gpu_results,\n",
" \"TF opt GPU\": tf_opt_gpu_results,\n",
" \"TF opt GPU lossy\": tf_opt_lossy_gpu_results,\n",
" },\n",
" (0, 14, 0, 1000),\n",
")\n",
"fig.savefig(\"vit_p50_latency_32.png\", bbox_inches=\"tight\")"
]
},
{
"cell_type": "code",
"execution_count": 57,
"id": "a9d565f6-4f1c-45a3-a9cc-71c4d41bfac9",
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAnEAAAG5CAYAAADh3mJ8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAA9/0lEQVR4nO3deVxVdf7H8fcVCEkQBRFQEFFIARdUxK3UdDBLR0czR9PE0qxssc1isvnZquTolI1OZVmZZpimYtpoZjnmllIuiUu4Jsi4gSEqyvL9/eF4RxQQUbgcez0fD3twzznfcz7ny/XR2+9ZvjZjjBEAAAAspYqjCwAAAMDVI8QBAABYECEOAADAgghxAAAAFkSIAwAAsCBCHAAAgAUR4gALeOmllzR48GBHl1Furub8OnfurA8++KCcK7JOHddL/fr19c0335T7ca7Xd/nhhx/Wq6++eh0qAqyLEAeUoH79+nJzc5O7u7tq1qypHj166ODBg/b1Q4cO1U033SR3d3f7n+bNm0uS9u/fL5vNZl9ev359xcfHS5IiIiLsy52cnFS1alX753HjxjnkXH8PduzYoS5dusjT01MhISFasGBBofUffPCBQkJC5O7uru7du+vQoUPX5bgVFZAcxRGB9t1339Vf//rX67rP5557ToGBgapevbqCgoL0+uuv29f98ssv6t27t3x8fOTl5aU77rhDu3btuq7HB64WIQ64gi+//FLZ2dlKT0+Xr6+vHn/88ULrn3vuOWVnZ9v/bNmypdD6EydOKDs7W/PmzdOrr76q5cuXKzk52b79bbfdpilTptg/v/DCCxV5er8beXl56t27t3r27KmMjAxNmzZNgwcP1i+//CJJ+ve//60XXnhBiYmJysjIUHBwsAYOHOjgqlGRhg0bpp07dyorK0tr167V7NmzNX/+fEnn/x736tVLu3bt0uHDhxUdHa3evXs7uGL83hHigFKqWrWq+vXrp+3bt5epfVRUlCIiIrR58+Yytc/JydGf//xneXh4qGXLloXCYnx8vBo2bCgPDw+Fh4cXGmHavXu3OnXqJE9PT9WqVUt//vOf7et27typmJgYeXl5qVGjRvr888+LPX7nzp314osvqn379nJ3d9cf//hHHT9+XIMGDVL16tXVunVr7d+/37792rVr1bp1a3l6eqp169Zau3atfd2+ffvUqVMneXh4KCYmRseOHSt0rPXr16t9+/aqUaOGmjdvrpUrV5aqj1566SX169evyH7auXOnDh06pKeeekpOTk7q0qWLOnTooJkzZ0o6H9bvueceRURE6KabbtJf//pXrVq1Snv27Lnicffs2aMuXbrI29tbtWrV0qBBg3TixAlJ0n333adff/1Vf/zjH+Xu7q4JEyZc8Rw7d+6sv/71r+rQoYM8PDzUrVu3Qn20evVqe9vAwEB9/PHH2rhxo3x9fZWXl2ff7osvvlBkZGSp+k6SNm7cqPDwcNWsWVP333+/cnJyJEmZmZnq2bOnfHx8VLNmTfXs2VOpqamSpDFjxuj777/XY489Jnd3dz322GOSpOTkZPt3y9fXt9AI87lz5zRkyBB5eHgoIiJCSUlJRdZjjNFTTz2l2rVry9PTU82aNdO2bdsknR8Ff/HFFyXJ3rcX/lSpUkUff/yxpKv7jjdq1EjVqlWzf65SpYp2794tSYqOjtawYcPk5eUlFxcXPfXUU9q1a5eOHz9e6v4FrjsDoFhBQUFm+fLlxhhjTp06ZYYMGWLuu+8++/rY2FgzZsyYItvu27fPSDK5ubnGGGPWrVtn3NzczPz58wtt16lTJ/P++++XWMfYsWONs7OzmTt3rjl37pz529/+ZurXr2/OnTtnjDHm888/N2lpaSY/P98kJCSYm2++2Rw6dMgYY8yAAQPMa6+9ZvLz882ZM2fM999/b4wxJjs72wQEBJgPP/zQ5Obmmh9//NF4e3ubbdu2FVlDp06dTMOGDc3u3bvNiRMnTFhYmAkNDTXLly83ubm55r777jNDhw41xhhz/PhxU6NGDfPJJ5+Y3NxcM3v2bFOjRg1z7NgxY4wxbdu2NU899ZTJyckx//73v427u7sZNGiQMcaY1NRU4+XlZZYsWWLy8/PN119/bby8vMyRI0eu2F8l9dPWrVtNtWrVTEFBgX37P/zhD+ZPf/qTMcaYp59+2jzyyCP2dampqUaSWbhwYbH9caGOlJQU8/XXX5ucnBxz5MgRc9ttt5lRo0bZt734e1Tac2zQoIHZtWuXOX36tOnUqZN5/vnnjTHGHDhwwLi7u5vZs2ebc+fOmWPHjplNmzYZY4wJCwszX331lf04f/rTn8zEiROLrP9SQUFBJiIiwvz666/m+PHjpn379vbv9rFjx8y8efPMqVOnTFZWlunXr5/p3bt3kX1hjDFZWVnGz8/PTJw40Zw5c8ZkZWWZ9evX239Hrq6uZsmSJSYvL8/ExcWZNm3aFFnT0qVLTcuWLU1mZqYpKCgw27dvt3+vi/u7969//cv4+/ubX3/99aq/48YYM378eFOtWjUjyQQHB5uDBw8Wud2CBQuMn59fyZ0KlDNCHFCCoKAgU61aNePp6WmcnJyMv7+/2bp1q319bGyscXV1NZ6envY/Q4YMMcb8L8R5enqaqlWrGknmmWeeKRQijCl9iLv4f3T5+fnGz8/PrFq1qsjtmzdvbg8f9913n3nwwQcv+59RQkKCufXWWwstGzFihHnppZeK3GenTp3Ma6+9Zv/89NNPm+7du9s/L1q0yDRv3twYY8wnn3xiWrduXah927ZtzUcffWQOHDhgnJycTHZ2tn3dwIED7SEuPj7eDB48uFDbbt26mY8//theR0khrrh+OnfunAkODjZvvPGGOXfunFm2bJlxcXEx3bp1M8YY88033xhvb2+zZcsWc/r0aTNixAhjs9nM7Nmzi+2P4upYsGCBiYyMtH++NMSV5hxfffVV+7qpU6eaO+64wxhjzLhx4+zB81Lx8fHm3nvvNcacD9Jubm720HMlQUFB5p133rF/XrJkiWnQoEGR227atMnUqFHD/vnSvpg9e3ah87/Y2LFjTdeuXe2fk5OTTdWqVYvcdsWKFSY0NNSsW7fO5OfnF1pXVIjbtWuX8fHxsf+9uNrv+AUFBQXmp59+Mv/3f/9nsrKyLlt/8OBBU6dOnWK/G0BF4XIqcAULFy7UiRMndPbsWU2ZMkWdOnXSf/7zH/v6Z599VidOnLD/mTFjRqH2x44dU3Z2tiZOnKiVK1cqNze3THUEBgbaf65SpYoCAgLsN95/8sknioyMVI0aNVSjRg1t27bNfvltwoQJMsYoOjpaERER+vDDDyVJBw4c0A8//GBvU6NGDX366aeFzu1Svr6+9p/d3Nwu+5ydnS1JOnTokIKCggq1DQoKUlpamg4dOqSaNWsWumx18bYHDhzQ3LlzC9W1evVqpaenX1M/ubi4aOHChVqyZIn8/Pw0adIk9e/fXwEBAZKkrl276uWXX9bdd9+toKAg1a9fXx4eHvb1JTly5IgGDBigunXrqnr16ho8ePBll4gvVppz9PPzs/9888032/v24MGDatiwYZH7HTx4sP0ezs8//1y33Xab/P39r1j/BRf3XVBQkP37dfr0aT300EMKCgpS9erV1bFjR504cUL5+flF7qekGos6t5ycnEKXgS/o0qWLHnvsMT366KPy9fXViBEjlJWVVeQ+f/vtN/Xu3VuvvvqqbrvtNkll+45Lks1mU4sWLeTm5qaxY8cWWnf06FF169ZNI0eO5J5JOBwhDiglJycn9e3bV05OTlq9evVVt33mmWdUtWpV/fOf/yzT8S9+KragoECpqamqU6eODhw4oAcffFBTpkzR8ePHdeLECTVp0kTGGEnn/4f5/vvv69ChQ3rvvfc0cuRI7d69W4GBgerUqVOhAJqdna133nmnTPVd7EJdF/v1119Vt25d+fv7KzMzU6dOnSq07oLAwEDdd999heo6deqU4uLiSnXs4vpJkpo1a6Z///vfOn78uJYtW6a9e/cqOjravv2jjz6qlJQUHTlyRHfffbfy8vLUpEmTKx7zL3/5i2w2m7Zu3aqsrCzNmjXL3v/S+VBwsWs5x8DAwGLv06tbt67atWunBQsWaObMmbrvvvuuuL+LXdx3v/76q73fJk2apF27dumHH35QVlaWVq1aJUn2cyzq/EpzL2FpPPHEE/rxxx+VnJysX375RX/7298u26agoED33nuvbr/9dj300EOF6riW73heXl6h88jMzFS3bt3Uq1cvjRkz5tpPDrhGhDiglIwxSkxMVGZmpsLCwsq0j7i4OE2YMMF+w/jV+PHHHzV//nzl5eXprbfekqurq9q2batTp07JZrPJx8dHkvTRRx/Zb/6WpLlz59pvQq9Zs6ZsNpucnJzUs2dP/fLLL5o5c6Zyc3OVm5urjRs3aseOHWU6t4vddddd+uWXXzR79mzl5eVpzpw52r59u3r27KmgoCBFRUVp7NixOnfunFavXq0vv/zS3vbCaNKyZcuUn5+vnJwcrVy50n4OZe0nSdq6datycnJ0+vRpTZw4Uenp6Ro6dKik8w+ObNu2TcYY/frrrxoxYoRGjRqlmjVrXvGYJ0+elLu7u2rUqKG0tLTLgoavr6/27t17Xc5x0KBB+uabb/T5558rLy9Px48fL/SwzJAhQzRhwgT9/PPP6tOnj335ypUrLwtbl5o6dapSU1OVkZGhcePG2R+COXnypNzc3FSjRg1lZGTo5ZdfLvH8evbsqf/85z966623dPbsWZ08eVI//PDDFc/tUhs3btQPP/yg3NxcVatWTVWrVpWTk9Nl240ZM0anTp3S5MmTCy2/mu94QUGB3nvvPWVmZsoYow0bNmjq1Knq2rWrJCkrK0t33HGHOnToYH9VEOBohDjgCi48+Va9enWNGTNGM2bMUEREhH39hAkTCj0ZV6tWrWL31aNHD9WsWVPvv//+VdfRu3dvzZkzRzVr1tTMmTM1f/58ubi4KDw8XM8884zatWsnX19f/fzzz+rQoYO93caNG9WmTRu5u7urV69emjx5soKDg+Xh4aGvv/5aCQkJqlOnjvz8/PT888/r7NmzV13bpby9vbV48WJNmjRJ3t7emjBhghYvXmzvm9mzZ+uHH36Ql5eXXn75ZQ0ZMsTeNjAwUImJiRo3bpx8fHwUGBiov/3tbyooKLimfpKkmTNnyt/fX7Vr19aKFSu0fPlyubq6Sjof4u699165u7srOjpa7dq1K/XLZMeOHauffvpJnp6e6tGjh/r27Vto/V/+8he99tprqlGjhiZOnHhN51ivXj199dVXmjRpkry8vBQZGVnoSeU+ffrowIED6tOnT6FL1gcPHlS7du1K3Pe9996rbt26qUGDBmrQoIH96c8nn3xSZ86cUa1atdS2bVt17969ULtRo0Zp3rx5qlmzpp544gl5eHho+fLl+vLLL+Xn56fQ0FB99913Vzy3S2VlZenBBx9UzZo1FRQUJG9vbz377LOXbffZZ59p/fr1qlmzpv3v4aeffnrV3/EFCxbYn/IePHiwHn/8cfsrhRYsWKCNGzfqo48+KvT3/eJRZKCi2czFY/4AYGEvvfSSdu/erVmzZjm6FIdq2LCh3nvvPf3hD3+wLxs+fLjuuece3XHHHQ6sDMD15OzoAgAA188XX3whm82mLl26FFp+I00RBuC8cruc+sADD6h27dqFbgrOyMhQTEyMQkNDFRMTo8zMTPu68ePHKyQkRI0aNdKyZcvsy3/88Uc1bdpUISEheuKJJ8TAIQAUrXPnznrkkUc0depUVanC3TLAja7cLqeuWrVK7u7uGjJkiP0m6+eee05eXl6Ki4tTfHy8MjMz9cYbb2j79u0aOHCgNmzYoEOHDukPf/iDfvnlFzk5OSk6OlqTJ09W27Ztddddd+mJJ57QnXfeWR4lAwAAWEa5/VOtY8eO8vLyKrQsMTFRsbGxkqTY2FgtXLjQvnzAgAFydXVVcHCwQkJCtGHDBqWnpysrK0vt2rWTzWbTkCFD7G0AAAB+zyr0nrjDhw/bXzzp7++vI0eOSJLS0tLsrwCQpICAAKWlpcnFxaXQizYvLC/OtGnTNG3aNEnn58tr3LhxeZwGAADAdbV///4SXxJelErxYENRV3RtNluxy4szYsQIjRgxQtL5ycaLm1QZAACgMomKirrqNhV656uvr699Wpn09HTVrl1b0vkRtovfFH7hDesBAQGFXn558ZvXAQAAfs8qNMT16tXLPq/kjBkz1Lt3b/vyhIQEnT17Vvv27VNKSoqio6Pl7+8vDw8PrV+/XsYYffLJJ/Y2AAAAv2fldjl14MCBWrlypY4dO6aAgAC9/PLLiouLU//+/TV9+nTVq1dPc+fOlSRFRESof//+Cg8Pl7Ozs6ZOnWqfWuWdd97R0KFDdebMGd155508mQoAAKAbeMYG7okDAFhVbm6uUlNTyzTPMiq3qlWrKiAgwD4d4AVlyS2V4sEGAADwP6mpqfLw8FD9+vVLfKAP1mKM0fHjx5Wamqrg4OBr3h+v9AYAoJLJycmRt7c3Ae4GY7PZ5O3tfd1GWAlxAABUQgS4G9P1/L0S4gAAACyIEAcAAAo5fvy4IiMjFRkZKT8/P9WtW9f+2Waz2X+OjIzU/v37tXz5crVq1UpNmzZVq1at9O233xa53z//+c/2dvXr11dkZGSR26WkpKhnz55q2LChWrVqpdtvv12rVq2SJH388cfy8fFRZGSkwsPD9f7770uSXnrpJU2cOLHQfurXr3/VsyBYCQ82AACAQry9vbV582ZJ58ORu7u7nn32WUmSu7u7fd0FmZmZ+vLLL1WnTh1t27ZNd9xxR5HTZM6ZM8f+8zPPPCNPT8/LtsnJyVGPHj00ceJE9erVS5K0bds2JSUlqWPHjpLOh8EpU6boyJEjioiIsG/3e0OIAwAA16RFixb2nyMiIpSTk6OzZ8/K1dW1yO2NMfr888+LHLH79NNP1a5du0LBrEmTJmrSpMll29auXVsNGzbUgQMHrsNZWA8hDgCASuzlL5O1/VDWdd1neJ3qGvvHiDK1PXPmjP0yaHBwsBYsWFBo/RdffKEWLVoUG+Ak6fvvv5evr69CQ0MvW5ecnKyWLVuWqpa9e/dq7969CgkJKf0J3EAIcQAAoNTc3Nwuu5x6QXJysp5//nl9/fXXJe7js88+08CBA0t1vD59+iglJUW33HKL5s+fL+n8ZdnVq1fL1dVV7733nry8vIp96vNGfsqXEAcAQCVW1hGzipaamqo+ffrok08+UcOGDYvdLi8vT/Pnz9ePP/5Y5PqIiAj7QwyStGDBAiUlJdnvyZP+d0/cxby9vZWenl5o2cmTJ1WjRo0ynI018HQqAAC4JidOnFCPHj00fvx4dejQocRtv/nmGzVu3FgBAQFFrr/33nu1Zs0aLVq0yL7s9OnTV6yhY8eOWrRokU6ePClJmj9/vpo3b26fi/1GRIgDAADXZMqUKdq9e7deffVV+ytEjhw5IkkaPnx4oTlBExISSryU6ubmpsWLF+vdd99VgwYN1K5dO7322mt68cUXS6yhWbNmeuyxx3TrrbcqMjJS7777rj744IPrc4KVlM0YYxxdRHkoy0SyAABUBjt27FBYWJijy0A5Ker3W5bcwkgcAACABRHiAAAALIgQBwAAYEGEOAAAAAsixAEAAFgQIQ4AAMCCCHEAAKCQ48eP29/35ufnp7p169o/22w2+8+RkZHav3//NR9v3Lhxxa7Lzs7WI488ooYNG6pFixZq1aqV3n//fUnS/v375ebmpsjISIWHh+vhhx9WQUGBVq5cqZ49exbaz9ChQzVv3rxrrrUyYdotAABQiLe3t31+1Jdeeknu7u72aa/c3d2LnTu1rMaNG6cXXnihyHXDhw9XgwYNlJKSoipVqujo0aP68MMP7esbNmyozZs3Ky8vT126dNHChQvl5eV1XeurrBiJAwAA140xRqNHj1aTJk3UtGlTzZkzR5K0cuVKdezYUX369Ck0ahYXF6czZ84oMjJSgwYNKrSvPXv2aMOGDXrttddUpcr5yOLj46Pnn3/+suM6Ozurffv22r17d/mfZCXBSBwAAJXZv+Kk//x8fffp11S6M75MTS8ELkkKDg7WggULCq2fP3++Nm/erC1btujYsWNq3bq1OnbsKEnasGGDtm/frqCgIHXv3l3z589XfHy8pkyZUuToXnJyspo3b24PcCU5ffq0VqxYoVdeeaVM52VFjMQBAIBSc3Nz0+bNm7V58+bLApwkrV69WgMHDpSTk5N8fX3VqVMnbdy4UZIUHR2tBg0ayMnJSQMHDtTq1auv6tivv/66IiMjVadOHfuyPXv2KDIyUh06dFCPHj105513ymazFdm+uOVWxUgcAACVWRlHzBylpCnZLw1RVwpV4eHh2rJliwoKClSlShWNGTNGY8aMkbu7u32bC/fEXczb21uZmZmFlmVkZKhWrVqlPAtrYCQOAABcNx07dtScOXOUn5+vo0ePatWqVYqOjpZ0/nLqvn37VFBQoDlz5ujWW2+VJLm4uCg3N/eyfYWEhCgqKkovvvii8vPzJUk5OTklBkVJCg0N1aFDh7Rjxw5J0oEDB7Rlyxb7ZeAbBSNxAADguunTp4/WrVun5s2by2azacKECfLz89POnTvVrl07xcXF6eeff7Y/5CBJI0aMULNmzdSyZUt9+umnhfb3wQcfaPTo0QoJCZGXl5fc3Nz0xhtvlFiDq6urZs2apfvvv185OTlycXHRBx98IE9Pz3I7b0ewmSvFWYuKiopSUlKSo8sAAOCq7dixQ2FhYY4u47pauXKlJk6cqMWLFzu6FIcr6vdbltzC5VQAAAAL4nIqAAAod507d1bnzp0dXcYNhZE4AAAACyLEAQAAWBAhDgAAwIIIcQAAABZEiAMAAIUcP35ckZGRioyMlJ+fn+rWrWv/bLPZ7D9HRkZq//7913y8cePGFbsuOztbjzzyiBo2bKgWLVqoVatWev/99yVJ+/fvl5ubmyIjIxUeHq6HH35YBQUFWrlypXr27FloP0OHDtW8efMu239xy62Ap1MBAEAh3t7e9qmsXnrpJbm7u+vZZ5+VJLm7uxc5Wf21GDdunF544YUi1w0fPlwNGjRQSkqKqlSpoqNHj+rDDz+0r78w7VZeXp66dOmihQsXysvL67rWV1kxEgcAAK4bY4xGjx6tJk2aqGnTppozZ46k8y/7vTBLw8WjZnFxcTpz5owiIyM1aNCgQvvas2ePNmzYoNdee01VqpyPLD4+Pnr++ecvO66zs7Pat2+v3bt3l7n2FStWqEWLFmratKkeeOABnT17VpIUFxen8PBwNWvWzB5m586dqyZNmqh58+bq2LGjJOm2224rFHA7dOigrVu3lrmeK2EkDgCASuyNDW9oZ8bO67rPxl6N9Xz05UGoNC4ELkkKDg7WggULCq2fP3++Nm/erC1btujYsWNq3bq1PeRs2LBB27dvV1BQkLp376758+crPj5eU6ZMKXJ0Lzk5Wc2bN7cHuJKcPn1aK1as0CuvvFKm88rJydHQoUO1YsUK3XLLLRoyZIjeeecdDRkyRAsWLNDOnTtls9l04sQJSdIrr7yiZcuWqW7duvZlw4cP18cff6y33npLv/zyi86ePatmzZqVqZ7SYCQOAACUmpubmzZv3qzNmzdfFuAkafXq1Ro4cKCcnJzk6+urTp06aePGjZKk6OhoNWjQQE5OTho4cKBWr159Vcd+/fXXFRkZqTp16tiX7dmzR5GRkerQoYN69OihO++8Uzabrcj2xS2XpF27dik4OFi33HKLJCk2NlarVq1S9erVVbVqVQ0fPlzz58/XzTffLOn8KNvQoUP1/vvvKz8/X5J0zz33aPHixcrNzdWHH36ooUOHXtX5XS1G4gAAqMTKOmLmKCVNyX5piCopVElSeHi4tmzZooKCAlWpUkVjxozRmDFj5O7ubt/mwj1xF/P29lZmZmahZRkZGapVq9ZV1+3s7KwNGzZoxYoVSkhI0JQpU/Ttt9/q3Xff1Q8//KAlS5YoMjJSmzdvlre3t2JiYpSYmKjPP/+83OdwZyQOAABcNx07dtScOXOUn5+vo0ePatWqVYqOjpZ0/nLqvn37VFBQoDlz5ujWW2+VJLm4uCg3N/eyfYWEhCgqKkovvviifbQrJyenxKAoSaGhoTp06JB27NghSTpw4IC2bNlivwxclMaNG2v//v32e+pmzpypTp06KTs7W7/99pvuuusuvfXWW/bAuGfPHrVp00avvPKKatWqpYMHD0o6f0n1iSeeUOvWrcv9AQtG4gAAwHXTp08frVu3Ts2bN5fNZtOECRPk5+ennTt3ql27doqLi9PPP/9sf8hBkkaMGKFmzZqpZcuW+vTTTwvt74MPPtDo0aMVEhIiLy8vubm56Y033iixBldXV82aNUv333+/cnJy5OLiog8++ECenp7Ftqlatao++ugj3XPPPcrLy1Pr1q318MMPKyMjQ71797aHxzfffFOSNHr0aKWkpMgYo65du6p58+aSpFatWql69eq6//77r6UbS8VmrhRnLSoqKqrchzEBACgPO3bsUFhYmKPLuK5WrlypiRMnavHixY4upVwdOnRInTt31s6dO4t9IKOo329ZcguXUwEAAK6DTz75RG3atNHrr79eqidqrxWXUwEAQLnr3LmzOnfu7OgyytWQIUM0ZMiQCjseI3EAAFRCN+jdTr971/P3SogDAKCSqVq1qo4fP06Qu8EYY3T8+HFVrVr1uuyPy6kAAFQyAQEBSk1N1dGjRx1dCq6zqlWrKiAg4LrsixAHAEAl4+LiouDgYEeXgUqOy6kAAAAWRIgDAACwIEIcAACABRHiAAAALIgQBwAAYEGEOAAAAAsixAEAAFgQIQ4AAMCCCHEAAAAWRIgDAACwIEIcAACABRHiAAAALIgQBwAAYEGEOAAAAAsixAEAAFgQIQ4AAMCCCHEAAAAWRIgDAACwIIeEuDfffFMRERFq0qSJBg4cqJycHGVkZCgmJkahoaGKiYlRZmamffvx48crJCREjRo10rJlyxxRMgAAQKVS4SEuLS1Nb7/9tpKSkrRt2zbl5+crISFB8fHx6tq1q1JSUtS1a1fFx8dLkrZv366EhAQlJydr6dKlGjlypPLz8yu6bAAAgErFISNxeXl5OnPmjPLy8nT69GnVqVNHiYmJio2NlSTFxsZq4cKFkqTExEQNGDBArq6uCg4OVkhIiDZs2OCIsgEAACqNCg9xdevW1bPPPqt69erJ399fnp6e6tatmw4fPix/f39Jkr+/v44cOSLp/MhdYGCgvX1AQIDS0tKK3Pe0adMUFRWlqKgoHT16tPxPBgAAwEEqPMRlZmYqMTFR+/bt06FDh3Tq1CnNmjWr2O2NMZcts9lsRW47YsQIJSUlKSkpST4+PtetZgAAgMqmwkPcN998o+DgYPn4+MjFxUV9+/bV2rVr5evrq/T0dElSenq6ateuLen8yNvBgwft7VNTU1WnTp2KLhsAAKBSqfAQV69ePa1fv16nT5+WMUYrVqxQWFiYevXqpRkzZkiSZsyYod69e0uSevXqpYSEBJ09e1b79u1TSkqKoqOjK7psAACASsW5og/Ypk0b9evXTy1btpSzs7NatGihESNGKDs7W/3799f06dNVr149zZ07V5IUERGh/v37Kzw8XM7Ozpo6daqcnJwqumwAAIBKxWaKuunsBhAVFaWkpCRHlwEAAHBFZcktzNgAAABgQYQ4AAAACyLEAQAAWBAhDgAAwIIIcQAAABZEiAMAALAgQhwAAIAFEeIAAAAsiBAHAABgQYQ4AAAACyLEAQAAWBAhDgAAwIIIcQAAABZEiAMAALAgQhwAAIAFEeIAAAAsiBAHAABgQYQ4AAAACyLEAQAAWBAhDgAAwIIIcQAAABZEiAMAALAgQhwAAIAFEeIAAAAsiBAHAABgQYQ4AAAACyLEAQAAWBAhDgAAwIIIcQAAABZEiAMAALAgQhwAAIAFEeIAAAAsiBAHAABgQYQ4AAAACyLEAQAAWBAhDgAAwIIIcQAAABZEiAMAALAgQhwAAIAFEeIAAAAsiBAHAABgQYQ4AAAACyLEAQAAWBAhDgAAwIIIcQAAABZEiAMAALAgQhwAAIAFEeIAAAAsiBAHAABgQYQ4AAAACyLEAQAAWBAhDgAAwIIIcQAAABZEiAMAALAgQhwAAIAFEeIAAAAsiBAHAABgQYQ4AAAACyLEAQAAWBAhDgAAwIIIcQAAABZEiAMAALAgQhwAAIAFEeIAAAAsiBAHAABgQYQ4AAAACyLEAQAAWJBDQtyJEyfUr18/NW7cWGFhYVq3bp0yMjIUExOj0NBQxcTEKDMz0779+PHjFRISokaNGmnZsmWOKBkAAKBScUiIGzVqlLp3766dO3dqy5YtCgsLU3x8vLp27aqUlBR17dpV8fHxkqTt27crISFBycnJWrp0qUaOHKn8/HxHlA0AAFBpVHiIy8rK0qpVqzRs2DBJ0k033aQaNWooMTFRsbGxkqTY2FgtXLhQkpSYmKgBAwbI1dVVwcHBCgkJ0YYNGyq6bAAAgEqlwkPc3r175ePjo/vvv18tWrTQ8OHDderUKR0+fFj+/v6SJH9/fx05ckSSlJaWpsDAQHv7gIAApaWlFbnvadOmKSoqSlFRUTp69Gj5nwwAAICDVHiIy8vL008//aRHHnlEmzZtUrVq1eyXTotijLlsmc1mK3LbESNGKCkpSUlJSfLx8bluNQMAAFQ2FR7iAgICFBAQoDZt2kiS+vXrp59++km+vr5KT0+XJKWnp6t27dr27Q8ePGhvn5qaqjp16lR02QAAAJVKhYc4Pz8/BQYGateuXZKkFStWKDw8XL169dKMGTMkSTNmzFDv3r0lSb169VJCQoLOnj2rffv2KSUlRdHR0RVdNgAAQKXi7IiD/uMf/9CgQYN07tw5NWjQQB999JEKCgrUv39/TZ8+XfXq1dPcuXMlSREREerfv7/Cw8Pl7OysqVOnysnJyRFlAwAAVBo2U9RNZzeAqKgoJSUlOboMAACAKypLbmHGBgAAAAsixAEAAFgQIQ4AAMCCCHEAAAAWVOqnUzMzM3Xo0CG5ubmpfv36qlKF/AcAAOAoJYa43377TVOnTtVnn32mc+fOycfHRzk5OTp8+LDatm2rkSNH6vbbb6+oWgEAAPBfJYa4fv36aciQIfr+++9Vo0aNQut+/PFHzZw5U3v37rVPZg8AAICKUWKIW758ebHrWrVqpVatWl33ggAAAHBlpbqxbc2aNTp16pQkadasWXr66ad14MCBci0MAAAAxStViHvkkUd08803a8uWLZowYYKCgoI0ZMiQ8q4NAAAAxShViHN2dpbNZlNiYqJGjRqlUaNG6eTJk+VdGwAAAIpRqleMeHh4aPz48Zo1a5ZWrVql/Px85ebmlndtAAAAKEapRuLmzJkjV1dXTZ8+XX5+fkpLS9Po0aPLuzYAAAAUw2aMMaXdOCsrS3l5efbPXl5e5VLU9RAVFaWkpCRHlwEAAHBFZcktpbqc+t577+n//u//5ObmJpvNJkmy2Wzau3fv1VcJAACAa1aqEDdx4kQlJyerVq1a5V0PAAAASqFU98Q1bNhQN998c3nXAgAAgFIq1Ujc+PHj1b59e7Vp00aurq725W+//Xa5FQYAAIDilSrEPfTQQ+rSpYuaNm2qKlVKNXgHAACAclSqEOfs7Ky///3v5V0LAAAASqlUw2q33367pk2bpvT0dGVkZNj/AAAAwDFKNRI3e/ZsSefvjbuAV4wAAAA4TqlC3L59+8q7DgAAAFyFEi+nrl69usTGWVlZ2rZt23UtCAAAAFdW4kjcF198oeeee07du3dXq1at5OPjo5ycHO3evVvfffedDhw4oEmTJlVUrQAAAPivEkPcm2++qczMTM2bN09z585Venq63NzcFBYWpoceeki33nprRdUJAACAi9iMMeZKG+Xn58vJyaki6rluyjKRLAAAgCOUJbeU6hUjISEhGj16tHbs2FGmwgAAAHB9lSrEbd26VbfccouGDRumtm3batq0acrKyirv2gAAAFCMUoU4Dw8PPfjgg1q7dq0mTJigl19+Wf7+/oqNjdXu3bvLu0YAAABcolQhLj8/X4sWLVKfPn00atQoPfPMM9q7d6/++Mc/6q677irvGgEAAHCJUr3sNzQ0VLfffrtGjx6t9u3b25f369dPq1atKrfiAAAAULRShbitW7fK3d29yHVvv/32dS0IAAAAV1aqy6mPPvqoTpw4Yf+cmZmpBx54oLxqAgAAwBWU+unUGjVq2D/XrFlTmzZtKq+aAAAAcAWlCnEFBQXKzMy0f87IyFBeXl65FQUAAICSleqeuGeeeUbt27dXv379JElz587VmDFjyrUwAAAAFK9UIW7IkCFq1aqVvvvuOxljNH/+fIWHh5d3bQAAAChGqUKcJDVu3Fg1a9a0X0b99ddfVa9evXIrDAAAAMUrVYj7xz/+oZdfflm+vr5ycnKSMUY2m01bt24t7/oAAABQhFKFuMmTJ2vXrl3y9vYu73oAAABQCqV6OjUwMFCenp7lXQsAAABKqVQjcQ0aNFDnzp3Vo0cPubq62pc//fTT5VYYAAAAileqEFevXj3Vq1dP586d07lz58q7JgAAAFxBqULc2LFjJUmnTp1StWrVyrUgAAAAXFmp7olbt26dwsPDFRYWJknasmWLRo4cWa6FAQAAoHilCnFPPvmkli1bZn86tXnz5lq1alW5FgYAAIDilSrESeefUL2Yk5PTdS8GAAAApVOqe+ICAwO1du1a2Ww2nTt3Tm+//bb90ioAAAAqXqlG4t59911NnTpVaWlpCggI0ObNm/XPf/6zvGsDAABAMUo1Erdr1y59+umnhZatWbNGHTp0KJeiAAAAULJSjcQ9/vjjpVoGAACAilHiSNy6deu0du1aHT16VH//+9/ty7OyspSfn1/uxQEAAKBoJYa4c+fOKTs7W3l5eTp58qR9efXq1TVv3rxyLw4AAABFKzHEderUSZ06ddLQoUMVFBRUUTUBAADgCkr1YMPNN9+s0aNHKzk5WTk5Ofbl3377bbkVBgAAgOKV6sGGQYMGqXHjxtq3b5/Gjh2r+vXrq3Xr1uVdGwAAAIpRqhB3/PhxDRs2TC4uLurUqZM+/PBDrV+/vrxrAwAAQDFKdTnVxcVFkuTv768lS5aoTp06Sk1NLdfCAAAAULxShbgXX3xRv/32myZNmqTHH39cWVlZeuutt8q5NAAAABSnVCGuZ8+ekiRPT0999913kkSIAwAAcKBS3RNXlItf/gsAAICKVeYQZ4y5nnUAAADgKpQ5xNlstutZBwAAAK5CiffEeXh4FBnWjDE6c+ZMuRUFAACAkpUY4i6eLxUAAACVR5kvpwIAAMBxCHEAAAAW5LAQl5+frxYtWtjfQZeRkaGYmBiFhoYqJiZGmZmZ9m3Hjx+vkJAQNWrUSMuWLXNUyQAAAJWGw0Lc5MmTFRYWZv8cHx+vrl27KiUlRV27dlV8fLwkafv27UpISFBycrKWLl2qkSNHKj8/31FlAwAAVAoOCXGpqalasmSJhg8fbl+WmJio2NhYSVJsbKwWLlxoXz5gwAC5uroqODhYISEh2rBhgyPKBgAAqDQcEuKefPJJTZgwQVWq/O/whw8flr+/vyTJ399fR44ckSSlpaUpMDDQvl1AQIDS0tKK3O+0adMUFRWlqKgoHT16tBzPAAAAwLEqPMQtXrxYtWvXVqtWrUq1fVEzQxT3ouERI0YoKSlJSUlJ8vHxuaY6AQAAKrMS3xNXHtasWaNFixbpq6++Uk5OjrKysjR48GD5+voqPT1d/v7+Sk9PV+3atSWdH3k7ePCgvX1qaqrq1KlT0WUDAABUKhU+Ejd+/HilpqZq//79SkhIUJcuXTRr1iz16tVLM2bMkCTNmDFDvXv3liT16tVLCQkJOnv2rPbt26eUlBRFR0dXdNkAAACVSoWPxBUnLi5O/fv31/Tp01WvXj3NnTtXkhQREaH+/fsrPDxczs7Omjp1qpycnBxcLQAAgGPZTFE3nd0AoqKilJSU5OgyAAAArqgsuYUZGwAAACyIEAcAAGBBhDgAAAALIsQBAABYECEOAADAgghxAAAAFkSIAwAAsCBCHAAAgAUR4gAAACyIEAcAAGBBhDgAAAALIsQBAABYECEOAADAgghxAAAAFkSIAwAAsCBCHAAAgAUR4gAAACyIEAcAAGBBhDgAAAALIsQBAABYECEOAADAgghxAAAAFkSIAwAAsCBCHAAAgAUR4gAAACyIEAcAAGBBhDgAAAALIsQBAABYECEOAADAgghxAAAAFkSIAwAAsCBCHAAAgAUR4gAAACyIEAcAAGBBhDgAAAALIsQBAABYECEOAADAgghxAAAAFkSIAwAAsCBCHAAAgAUR4gAAACyIEAcAAGBBhDgAAAALIsQBAABYECEOAADAgghxAAAAFkSIAwAAsCBCHAAAgAUR4gAAACyIEAcAAGBBhDgAAAALIsQBAABYECEOAADAgghxAAAAFkSIAwAAsCBCHAAAgAUR4gAAACyIEAcAAGBBhDgAAAALIsQBAABYECEOAADAgghxAAAAFkSIAwAAsCBCHAAAgAUR4gAAACyIEAcAAGBBhDgAAAALqvAQd/DgQd1+++0KCwtTRESEJk+eLEnKyMhQTEyMQkNDFRMTo8zMTHub8ePHKyQkRI0aNdKyZcsqumQAAIBKp8JDnLOzsyZNmqQdO3Zo/fr1mjp1qrZv3674+Hh17dpVKSkp6tq1q+Lj4yVJ27dvV0JCgpKTk7V06VKNHDlS+fn5FV02AABApVLhIc7f318tW7aUJHl4eCgsLExpaWlKTExUbGysJCk2NlYLFy6UJCUmJmrAgAFydXVVcHCwQkJCtGHDhoouGwAAoFJx6D1x+/fv16ZNm9SmTRsdPnxY/v7+ks4HvSNHjkiS0tLSFBgYaG8TEBCgtLS0Ivc3bdo0RUVFKSoqSkePHi3/EwAAAHAQh4W47Oxs3X333XrrrbdUvXr1Yrczxly2zGazFbntiBEjlJSUpKSkJPn4+Fy3WgEAACobh4S43Nxc3X333Ro0aJD69u0rSfL19VV6erokKT09XbVr15Z0fuTt4MGD9rapqamqU6dOxRcNAABQiVR4iDPGaNiwYQoLC9PTTz9tX96rVy/NmDFDkjRjxgz17t3bvjwhIUFnz57Vvn37lJKSoujo6IouGwAAoFJxrugDrlmzRjNnzlTTpk0VGRkpSRo3bpzi4uLUv39/TZ8+XfXq1dPcuXMlSREREerfv7/Cw8Pl7OysqVOnysnJqaLLBgAAqFRspqibzm4AUVFRSkpKcnQZAAAAV1SW3MKMDQAAABZEiAMAALAgQhwAAIAFEeIAAAAsiBAHAABgQYQ4AAAACyLEAQAAWBAhDgAAwIIIcQAAABZEiAMAALAgQhwAAIAFEeIAAAAsiBAHAABgQYQ4AAAACyLEAQAAWBAhDgAAwIIIcQAAABZEiAMAALAgQhwAAIAFEeIAAAAsiBAHAABgQYQ4AAAACyLEAQAAWBAhDgAAwIIIcQAAABZEiAMAALAgQhwAAIAFEeIAAAAsiBAHAABgQYQ4AAAACyLEAQAAWBAhDgAAwIIIcQAAABZEiAMAALAgQhwAAIAFEeIAAAAsiBAHAABgQYQ4AAAACyLEAQAAWBAhDgAAwIIIcQAAABZEiAMAALAgQhwAAIAFEeIAAAAsiBAHAABgQYQ4AAAACyLEAQAAWBAhDgAAwIIIcQAAABZEiAMAALAgQhwAAIAFEeIAAAAsiBAHAABgQYQ4AAAACyLEAQAAWBAhDgAAwIIIcQAAABZEiAMAALAgQhwAAIAFEeIAAAAsiBAHAABgQYQ4AAAACyLEAQAAWBAhDgAAwIIIcQAAABZEiAMAALAgQhwAAIAFWSbELV26VI0aNVJISIji4+MdXQ4AAIBDOTu6gNLIz8/Xo48+quXLlysgIECtW7dWr169FB4e7ujSrtqxzHRt2fW9/bORKbTeXNrAXLr+CtuXsP6ytuby1nY2WxHHumT7yz5ebW3/+3x5KSXv67JaLju3S9cWlLS5ZCvqeP/972W/gxJqNeZ/tZkL7QskU2DftEAF55uYgvNncmH/pqCI9kbGFMgY6fxvxPx3O9nbGhnZ7NteqMX8b1//XXdhuWdVF7lXdZFkk2ySbBf+LWf73zJdstx2fqHNZpP93362i7e/5Geb7ZJ9nv9ju3SdzXbJ9hfaXM5mK3p5aVxL2//uoXSbFfF3ylaobQl/54o86qXHvdb2xW5YTNsS2pe4qqTxgYu+o/bvehGrL14vXdS35pLPhfdpu7i9uXx9kcf5775sPrdIrh4l1A5ULpYIcRs2bFBISIgaNGggSRowYIASExMtGeL+vWmBXjr4jqPLwO9ZlqMLACqpnY4uAL9nrnK96jaWCHFpaWkKDAy0fw4ICNAPP/xw2XbTpk3TtGnTJEnbtm1TVFRUhdV4Ncryi6pIR48elY+Pj6PLsCz679rQf9eG/is7+u7a0H/XZufOq/9XhCVCXFGX/Yq6PDJixAiNGDFCkhQVFaWkpKRyr+1GRN9dG/rv2tB/14b+Kzv67trQf9emLANPlniwISAgQAcPHrR/Tk1NVZ06dRxYEQAAgGNZIsS1bt1aKSkp2rdvn86dO6eEhAT16tXL0WUBAAA4jCUupzo7O2vKlCm64447lJ+frwceeEAREREltrlwWRVXj767NvTftaH/rg39V3b03bWh/65NWfrPZkp8zwQAAAAqI0tcTgUAAEBhhDgAAAALuuFCHNNzld3Bgwd1++23KywsTBEREZo8ebKjS7Kc/Px8tWjRQj179nR0KZZz4sQJ9evXT40bN1ZYWJjWrVvn6JIs5c0331RERISaNGmigQMHKicnx9ElVWoPPPCAateurSZNmtiXZWRkKCYmRqGhoYqJiVFmZqYDK6zciuq/0aNHq3HjxmrWrJn69OmjEydOOK7ASqyovrtg4sSJstlsOnbsWKn2dUOFuAvTc/3rX//S9u3b9dlnn2n79u2OLssynJ2dNWnSJO3YsUPr16/X1KlT6b+rNHnyZIWFhTm6DEsaNWqUunfvrp07d2rLli3041VIS0vT22+/raSkJG3btk35+flKSEhwdFmV2tChQ7V06dJCy+Lj49W1a1elpKSoa9euDASUoKj+i4mJ0bZt27R161bdcsstGj9+vIOqq9yK6jvp/EDK8uXLVa9evVLv64YKcRdPz3XTTTfZp+dC6fj7+6tly5aSJA8PD4WFhSktLc3BVVlHamqqlixZouHDhzu6FMvJysrSqlWrNGzYMEnSTTfdpBo1aji2KIvJy8vTmTNnlJeXp9OnT/MuzSvo2LGjvLy8Ci1LTExUbGysJCk2NlYLFy50QGXWUFT/devWTc7O51960bZtW6WmpjqitEqvqL6TpKeeekoTJky4qrmeb6gQV9T0XISQstm/f782bdqkNm3aOLoUy3jyySc1YcIEValyQ/21qhB79+6Vj4+P7r//frVo0ULDhw/XqVOnHF2WZdStW1fPPvus6tWrJ39/f3l6eqpbt26OLstyDh8+LH9/f0nn/1F75MgRB1dkXR9++KHuvPNOR5dhGYsWLVLdunXVvHnzq2p3Q/3fprTTc6Fk2dnZuvvuu/XWW2+pevXqji7HEhYvXqzatWurVatWji7FkvLy8vTTTz/pkUce0aZNm1StWjUuZV2FzMxMJSYmat++fTp06JBOnTqlWbNmObos/E69/vrrcnZ21qBBgxxdiiWcPn1ar7/+ul555ZWrbntDhTim57p2ubm5uvvuuzVo0CD17dvX0eVYxpo1a7Ro0SLVr19fAwYM0LfffqvBgwc7uizLCAgIUEBAgH3kt1+/fvrpp58cXJV1fPPNNwoODpaPj49cXFzUt29frV271tFlWY6vr6/S09MlSenp6apdu7aDK7KeGTNmaPHixfr0008ZRCmlPXv2aN++fWrevLnq16+v1NRUtWzZUv/5z3+u2PaGCnFMz3VtjDEaNmyYwsLC9PTTTzu6HEsZP368UlNTtX//fiUkJKhLly6MhFwFPz8/BQYGateuXZKkFStWKDw83MFVWUe9evW0fv16nT59WsYYrVixggdDyqBXr16aMWOGpPNhpHfv3g6uyFqWLl2qN954Q4sWLdLNN9/s6HIso2nTpjpy5Ij279+v/fv3KyAgQD/99JP8/Pyu2PaGCnEXT88VFham/v37X3F6LvzPmjVrNHPmTH377beKjIxUZGSkvvrqK0eXhd+Jf/zjHxo0aJCaNWumzZs364UXXnB0SZbRpk0b9evXTy1btlTTpk1VUFDAFEhXMHDgQLVr1067du1SQECApk+frri4OC1fvlyhoaFavny54uLiHF1mpVVU/z322GM6efKkYmJiFBkZqYcfftjRZVZKRfVdWTHtFgAAgAXdUCNxAAAAvxeEOAAAAAsixAEAAFgQIQ4AAMCCCHEAAAAWRIgDgP9KTU1V7969FRoaqgYNGuixxx7T2bNntXLlSnl6eqpFixYKCwvTyy+/LOn8m9YHDRqkpk2bqkmTJrr11luVnZ3t4LMA8HtBiAMAnX/Zdd++ffWnP/1JKSkpSklJ0ZkzZ/Tcc89Jkm677TZt2rRJSUlJmjVrln788UdNnjxZvr6++vnnn7Vt2zZNnz5dLi4uDj4TAL8Xzo4uAAAqg2+//VZVq1bV/fffL0lycnLSm2++qaCgIMXExNi3q1atmlq1aqU9e/YoPT1dQUFB9nWNGjWq8LoB/H4xEgcAkpKTk9WqVatCy6pXr6769etr9+7d9mXHjx/X+vXrFRERoQceeEBvvPGG2rVrpxdffFEpKSkVXTaA3zFG4gBA5y+nFjVh94VJbb7//nu1aNFCVapUUVxcnH1Kv7179+rrr7/WN998o9atW2vdunXMWwqgQhDiAEBSRESEvvjii0LLsrKydPjwYTVq1Ei33XabFi9efFk7d3d39e3bV3379lWVKlX01VdfEeIAVAgupwKApK5du+r06dP65JNPJEn5+fl65pln9Nhjj8nNza3INmvWrFFmZqYk6dy5c9q+fXuhe+QAoDwR4gBAks1m04IFCzRv3jyFhobK29tbVapU0ZgxY4pts2fPHnXq1ElNmzZVixYtFBUVpbvvvrsCqwbwe2YzF274AADYrV27VgMHDtT8+fMve+ABACoDQhwAAIAFcTkVAADAgghxAAAAFkSIAwAAsCBCHAAAgAUR4gAAACyIEAcAAGBB/w8ATarConr3ngAAAABJRU5ErkJggg==\n",
"text/plain": [
"<Figure size 720x504 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"fig = build_graph(\n",
" \"actual_qps\",\n",
" \"p99\",\n",
" {\n",
" \"TF2.7 GPU\": tf28_gpu_results,\n",
" \"TF opt GPU\": tf_opt_gpu_results,\n",
" \"TF opt GPU lossy\": tf_opt_lossy_gpu_results,\n",
" },\n",
" (0, 14, 0, 1000),\n",
")\n",
"fig.savefig(\"vit_p99_latency_32.png\", bbox_inches=\"tight\")"
]
},
{
"cell_type": "markdown",
"id": "791fcd11",
"metadata": {},
"source": [
"## Cleaning up of resources"
]
},
{
"cell_type": "code",
"execution_count": 58,
"id": "bf18cd91-29af-4bc3-8cb9-ded65aba40e6",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"running undeploy_model operation: projects/29880397572/locations/us-central1/endpoints/4481143201884798976/operations/1548701229104758784\n",
"\n",
"running delete_endpoint operation: projects/29880397572/locations/us-central1/operations/6826919992382980096\n",
"\n",
"running delete_model operation: projects/29880397572/locations/us-central1/operations/54350577747886080\n",
"\n",
"running undeploy_model operation: projects/29880397572/locations/us-central1/endpoints/4848186571515494400/operations/4032436423599587328\n",
"\n",
"running delete_endpoint operation: projects/29880397572/locations/us-central1/operations/6160387247532146688\n",
"\n",
"running delete_model operation: projects/29880397572/locations/us-central1/operations/3931949856913883136\n",
"\n",
"running undeploy_model operation: projects/29880397572/locations/us-central1/endpoints/7782176175631106048/operations/3733791473309581312\n",
"\n",
"running delete_endpoint operation: projects/29880397572/locations/us-central1/operations/5858646072498323456\n",
"\n",
"running delete_model operation: projects/29880397572/locations/us-central1/operations/4994799368973320192\n",
"\n"
]
}
],
"source": [
"def cleanup(endpoint, model_name, deployed_model_id):\n",
" response = endpoint_service_client.undeploy_model(\n",
" endpoint=endpoint, deployed_model_id=deployed_model_id\n",
" )\n",
" print(\"running undeploy_model operation:\", response.operation.name)\n",
" print(response.result())\n",
"\n",
" response = endpoint_service_client.delete_endpoint(name=endpoint)\n",
" print(\"running delete_endpoint operation:\", response.operation.name)\n",
" print(response.result())\n",
"\n",
" response = model_service_client.delete_model(name=model_name)\n",
" print(\"running delete_model operation:\", response.operation.name)\n",
" print(response.result())\n",
"\n",
"\n",
"cleanup(tf28_gpu_endpoint, tf28_gpu_model, tf28_gpu_deployed_model.deployed_model.id)\n",
"\n",
"cleanup(\n",
" tf_opt_gpu_endpoint, tf_opt_gpu_model, tf_opt_gpu_deployed_model.deployed_model.id\n",
")\n",
"cleanup(\n",
" tf_opt_lossy_gpu_endpoint,\n",
" tf_opt_lossy_gpu_model,\n",
" tf_opt_lossy_gpu_deployed_model.deployed_model.id,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 59,
"id": "d33a0d9f-4f56-408f-8fb5-d7e327b66029",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Removing gs://hf-tf-vision/keras_metadata.pb#1658220109884993...\n",
"Removing gs://hf-tf-vision/saved_model.pb#1658220109970261... \n",
"Removing gs://hf-tf-vision/assets.extra/tf_serving_warmup_requests#1658220107785769...\n",
"Removing gs://hf-tf-vision/variables/variables.data-00000-of-00001#1658220112658544...\n",
"/ [4 objects] \n",
"==> NOTE: You are performing a sequence of gsutil operations that may\n",
"run significantly faster if you instead use gsutil -m rm ... Please\n",
"see the -m section under \"gsutil help options\" for further information\n",
"about when gsutil -m can be advantageous.\n",
"\n",
"Removing gs://hf-tf-vision/variables/variables.index#1658220109910689...\n",
"/ [5 objects] \n",
"Operation completed over 5 objects. \n",
"Removing gs://hf-tf-vision/...\n"
]
}
],
"source": [
"!gsutil rm -r $GCS_BUCKET"
]
}
],
"metadata": {
"environment": {
"kernel": "python3",
"name": "tf2-gpu.2-9.m94",
"type": "gcloud",
"uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-9:m94"
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.12"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment