Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save NobuoTsukamoto/ba7028bd4e77c1431f4ff560ad61e8d7 to your computer and use it in GitHub Desktop.
Save NobuoTsukamoto/ba7028bd4e77c1431f4ff560ad61e8d7 to your computer and use it in GitHub Desktop.
Training TF1 Detection Model SSDLite Mobilenet V3
Display the source blob
Display the rendered blob
Raw
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "Training TF1 Detection Model SSDLite Mobilenet V3",
"provenance": [],
"collapsed_sections": [],
"toc_visible": true,
"machine_shape": "hm",
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"accelerator": "GPU"
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/NobuoTsukamoto/ba7028bd4e77c1431f4ff560ad61e8d7/training-tf1-detection-model-ssdlite-mobilenet-v3.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SEv4Nu2G8OFa"
},
"source": [
"# Training with TensorFlow 1 Detection Model\n",
"\n",
"This notebook provides steps for fine-tuning with the TensorFlow 1 Object detection API. Since the release of the TensorFlow 2 Object detection API, training with the TF 1 Object detection API has been unclear. Also, fine-tuning is not working with the latest commits([8454ded](https://github.com/tensorflow/models/commit/8454dedc4c0dc5eeea0f67c443fb9da32fcb7733)) as of 2021.05.02. \n",
"Work around the bug and use the Oxford-IIIT Pet Dataset to fine-tune the SSDLite MobileNet V3 Large. Finally, export the model and TF-Lite model.\n",
"\n",
"Reference:\n",
"- [Object Detection API with TensorFlow 1](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf1.md)\n",
"- [Training and Evaluation with TensorFlow 1](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf1_training_and_evaluation.md)\n",
"- [Running on mobile with TensorFlow Lite](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/running_on_mobile_tensorflowlite.md)\n",
"- [cant load pre-train models #8695](https://github.com/tensorflow/models/issues/8695)\n",
"- [Transfer Learning with the MobileDet checkpoints #9641](https://github.com/tensorflow/models/issues/9641)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "9TBgUmdwF8zw"
},
"source": [
"MIT License\n",
"\n",
"Copyright (c) 2019 Nobuo Tsukamoto\n",
"\n",
"Permission is hereby granted, free of charge, to any person obtaining a copy\n",
"of this software and associated documentation files (the \"Software\"), to deal\n",
"in the Software without restriction, including without limitation the rights\n",
"to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
"copies of the Software, and to permit persons to whom the Software is\n",
"furnished to do so, subject to the following conditions:\n",
"\n",
"The above copyright notice and this permission notice shall be included in all\n",
"copies or substantial portions of the Software.\n",
"\n",
"THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
"IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
"FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
"AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
"LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
"OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n",
"SOFTWARE."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "t67NpfjyBTnF"
},
"source": [
"# Install TensorFlow 1.x"
]
},
{
"cell_type": "code",
"metadata": {
"id": "45JZXAHmjdnf",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "f415c41a-9a19-4079-a41d-522fc3f77600"
},
"source": [
"%tensorflow_version 1.x"
],
"execution_count": 1,
"outputs": [
{
"output_type": "stream",
"text": [
"TensorFlow 1.x selected.\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "6QB2TaR4AKve"
},
"source": [
"# Check TensorFlow and NVIDIA CUDA version."
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
},
"id": "7zCUA1hiPlWV",
"outputId": "121c0ad7-a568-47b3-d843-72d0d2bfc58d"
},
"source": [
"import tensorflow as tf\n",
"tf.__version__"
],
"execution_count": 2,
"outputs": [
{
"output_type": "execute_result",
"data": {
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "string"
},
"text/plain": [
"'1.15.2'"
]
},
"metadata": {
"tags": []
},
"execution_count": 2
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "CnhOFVQTzoWD",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "62dfd8c8-3a77-4e6a-9208-c09a8f2b594d"
},
"source": [
"!nvidia-smi"
],
"execution_count": 3,
"outputs": [
{
"output_type": "stream",
"text": [
"Mon May 3 22:43:22 2021 \n",
"+-----------------------------------------------------------------------------+\n",
"| NVIDIA-SMI 465.19.01 Driver Version: 460.32.03 CUDA Version: 11.2 |\n",
"|-------------------------------+----------------------+----------------------+\n",
"| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |\n",
"| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |\n",
"| | | MIG M. |\n",
"|===============================+======================+======================|\n",
"| 0 Tesla P100-PCIE... Off | 00000000:00:04.0 Off | 0 |\n",
"| N/A 37C P0 27W / 250W | 0MiB / 16280MiB | 0% Default |\n",
"| | | N/A |\n",
"+-------------------------------+----------------------+----------------------+\n",
" \n",
"+-----------------------------------------------------------------------------+\n",
"| Processes: |\n",
"| GPU GI CI PID Type Process name GPU Memory |\n",
"| ID ID Usage |\n",
"|=============================================================================|\n",
"| No running processes found |\n",
"+-----------------------------------------------------------------------------+\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "y_owSZjMHN4-"
},
"source": [
"# Install TensorFlow 1 Object detection API\n",
"\n",
"Note:\n",
"- It is not included in [Python Package Installation](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf1.md) procedure, but you need to install the tensorflow-addons module.\n",
"- Modify model_lib.py to avoid a bug where fine_tune_checkpoint is not enabled. \n",
"See the comment on bug [#8695](https://github.com/tensorflow/models/issues/8695#issuecomment-775456367) for more details.\n",
"- If you specify `--use-feature=2020-resolver` during installation, you will get an error."
]
},
{
"cell_type": "code",
"metadata": {
"id": "KIue85E-uRe8",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "8d583b79-a38e-48b2-b55b-6c118616769c"
},
"source": [
"%%bash\n",
"\n",
"# Install tensorflow-addons.\n",
"# Note: tensorflow-addons have dependencies, \n",
"# but they are not installed by object_detection/packages/tf1/setup.py.\n",
"pip install tensorflow-addons\n",
"\n",
"# Clone tensorflow/models repository.\n",
"git clone https://github.com/tensorflow/models.git\n",
"cd models\n",
"git show"
],
"execution_count": 4,
"outputs": [
{
"output_type": "stream",
"text": [
"Collecting tensorflow-addons\n",
" Downloading https://files.pythonhosted.org/packages/74/e3/56d2fe76f0bb7c88ed9b2a6a557e25e83e252aec08f13de34369cd850a0b/tensorflow_addons-0.12.1-cp37-cp37m-manylinux2010_x86_64.whl (703kB)\n",
"Requirement already satisfied: typeguard>=2.7 in /usr/local/lib/python3.7/dist-packages (from tensorflow-addons) (2.7.1)\n",
"Installing collected packages: tensorflow-addons\n",
"Successfully installed tensorflow-addons-0.12.1\n",
"commit d7a784e6a85528292abf960be3a7cc643fa2b02c\n",
"Author: Yu-hui Chen <yuhuic@google.com>\n",
"Date: Mon May 3 14:54:11 2021 -0700\n",
"\n",
" Fixed the bug in the model builder of not setting the default values properly\n",
" for the object center head parameters in CenterNet.\n",
" \n",
" PiperOrigin-RevId: 371783912\n",
"\n",
"diff --git a/research/object_detection/builders/model_builder.py b/research/object_detection/builders/model_builder.py\n",
"index 949618f8..08c65e07 100644\n",
"--- a/research/object_detection/builders/model_builder.py\n",
"+++ b/research/object_detection/builders/model_builder.py\n",
"@@ -946,7 +946,7 @@ def object_center_proto_to_params(oc_config):\n",
" if oc_config.keypoint_weights_for_center:\n",
" keypoint_weights_for_center = list(oc_config.keypoint_weights_for_center)\n",
" \n",
"- if oc_config.center_head_params:\n",
"+ if oc_config.HasField('center_head_params'):\n",
" center_head_num_filters = list(oc_config.center_head_params.num_filters)\n",
" center_head_kernel_sizes = list(oc_config.center_head_params.kernel_sizes)\n",
" else:\n",
"diff --git a/research/object_detection/builders/model_builder_tf2_test.py b/research/object_detection/builders/model_builder_tf2_test.py\n",
"index 4bcc6dd6..9d386fdd 100644\n",
"--- a/research/object_detection/builders/model_builder_tf2_test.py\n",
"+++ b/research/object_detection/builders/model_builder_tf2_test.py\n",
"@@ -18,6 +18,7 @@\n",
" import os\n",
" import unittest\n",
" \n",
"+from absl.testing import parameterized\n",
" import tensorflow.compat.v1 as tf\n",
" \n",
" from google.protobuf import text_format\n",
"@@ -32,7 +33,8 @@ from object_detection.utils import tf_version\n",
" \n",
" \n",
" @unittest.skipIf(tf_version.is_tf1(), 'Skipping TF2.X only test.')\n",
"-class ModelBuilderTF2Test(model_builder_test.ModelBuilderTest):\n",
"+class ModelBuilderTF2Test(\n",
"+ model_builder_test.ModelBuilderTest, parameterized.TestCase):\n",
" \n",
" def default_ssd_feature_extractor(self):\n",
" return 'ssd_resnet50_v1_fpn_keras'\n",
"@@ -79,7 +81,7 @@ class ModelBuilderTF2Test(model_builder_test.ModelBuilderTest):\n",
" f.write(keypoint_spec_text)\n",
" return keypoint_label_map_path\n",
" \n",
"- def get_fake_keypoint_proto(self):\n",
"+ def get_fake_keypoint_proto(self, customize_head_params=False):\n",
" task_proto_txt = \"\"\"\n",
" task_name: \"human_pose\"\n",
" task_loss_weight: 0.9\n",
"@@ -120,18 +122,27 @@ class ModelBuilderTF2Test(model_builder_test.ModelBuilderTest):\n",
" predict_depth: true\n",
" per_keypoint_depth: true\n",
" keypoint_depth_loss_weight: 0.3\n",
"+ \"\"\"\n",
"+ if customize_head_params:\n",
"+ task_proto_txt += \"\"\"\n",
" heatmap_head_params {\n",
" num_filters: 64\n",
" num_filters: 32\n",
" kernel_sizes: 5\n",
" kernel_sizes: 3\n",
" }\n",
"- \"\"\"\n",
"+ offset_head_params {\n",
"+ num_filters: 128\n",
"+ num_filters: 64\n",
"+ kernel_sizes: 5\n",
"+ kernel_sizes: 3\n",
"+ }\n",
"+ \"\"\"\n",
" config = text_format.Merge(task_proto_txt,\n",
" center_net_pb2.CenterNet.KeypointEstimation())\n",
" return config\n",
" \n",
"- def get_fake_object_center_proto(self):\n",
"+ def get_fake_object_center_proto(self, customize_head_params=False):\n",
" proto_txt = \"\"\"\n",
" object_center_loss_weight: 0.5\n",
" heatmap_bias_init: 3.14\n",
"@@ -143,13 +154,16 @@ class ModelBuilderTF2Test(model_builder_test.ModelBuilderTest):\n",
" beta: 4.0\n",
" }\n",
" }\n",
"+ \"\"\"\n",
"+ if customize_head_params:\n",
"+ proto_txt += \"\"\"\n",
" center_head_params {\n",
" num_filters: 64\n",
" num_filters: 32\n",
" kernel_sizes: 5\n",
" kernel_sizes: 3\n",
" }\n",
"- \"\"\"\n",
"+ \"\"\"\n",
" return text_format.Merge(proto_txt,\n",
" center_net_pb2.CenterNet.ObjectCenterParams())\n",
" \n",
"@@ -222,7 +236,11 @@ class ModelBuilderTF2Test(model_builder_test.ModelBuilderTest):\n",
" return text_format.Merge(proto_txt,\n",
" center_net_pb2.CenterNet.DensePoseEstimation())\n",
" \n",
"- def test_create_center_net_model(self):\n",
"+ @parameterized.parameters(\n",
"+ {'customize_head_params': True},\n",
"+ {'customize_head_params': False}\n",
"+ )\n",
"+ def test_create_center_net_model(self, customize_head_params):\n",
" \"\"\"Test building a CenterNet model from proto txt.\"\"\"\n",
" proto_txt = \"\"\"\n",
" center_net {\n",
"@@ -244,11 +262,13 @@ class ModelBuilderTF2Test(model_builder_test.ModelBuilderTest):\n",
" # Set up the configuration proto.\n",
" config = text_format.Merge(proto_txt, model_pb2.DetectionModel())\n",
" config.center_net.object_center_params.CopyFrom(\n",
"- self.get_fake_object_center_proto())\n",
"+ self.get_fake_object_center_proto(\n",
"+ customize_head_params=customize_head_params))\n",
" config.center_net.object_detection_task.CopyFrom(\n",
" self.get_fake_object_detection_proto())\n",
" config.center_net.keypoint_estimation_task.append(\n",
"- self.get_fake_keypoint_proto())\n",
"+ self.get_fake_keypoint_proto(\n",
"+ customize_head_params=customize_head_params))\n",
" config.center_net.keypoint_label_map_path = (\n",
" self.get_fake_label_map_file_path())\n",
" config.center_net.mask_estimation_task.CopyFrom(\n",
"@@ -269,8 +289,12 @@ class ModelBuilderTF2Test(model_builder_test.ModelBuilderTest):\n",
" self.assertAlmostEqual(\n",
" model._center_params.heatmap_bias_init, 3.14, places=4)\n",
" self.assertEqual(model._center_params.max_box_predictions, 15)\n",
"- self.assertEqual(model._center_params.center_head_num_filters, [64, 32])\n",
"- self.assertEqual(model._center_params.center_head_kernel_sizes, [5, 3])\n",
"+ if customize_head_params:\n",
"+ self.assertEqual(model._center_params.center_head_num_filters, [64, 32])\n",
"+ self.assertEqual(model._center_params.center_head_kernel_sizes, [5, 3])\n",
"+ else:\n",
"+ self.assertEqual(model._center_params.center_head_num_filters, [256])\n",
"+ self.assertEqual(model._center_params.center_head_kernel_sizes, [3])\n",
" \n",
" # Check object detection related parameters.\n",
" self.assertAlmostEqual(model._od_params.offset_loss_weight, 0.1)\n",
"@@ -305,12 +329,18 @@ class ModelBuilderTF2Test(model_builder_test.ModelBuilderTest):\n",
" self.assertEqual(kp_params.predict_depth, True)\n",
" self.assertEqual(kp_params.per_keypoint_depth, True)\n",
" self.assertAlmostEqual(kp_params.keypoint_depth_loss_weight, 0.3)\n",
"- # Set by the config.\n",
"- self.assertEqual(kp_params.heatmap_head_num_filters, [64, 32])\n",
"- self.assertEqual(kp_params.heatmap_head_kernel_sizes, [5, 3])\n",
"- # Default values:\n",
"- self.assertEqual(kp_params.offset_head_num_filters, [256])\n",
"- self.assertEqual(kp_params.offset_head_kernel_sizes, [3])\n",
"+ if customize_head_params:\n",
"+ # Set by the config.\n",
"+ self.assertEqual(kp_params.heatmap_head_num_filters, [64, 32])\n",
"+ self.assertEqual(kp_params.heatmap_head_kernel_sizes, [5, 3])\n",
"+ self.assertEqual(kp_params.offset_head_num_filters, [128, 64])\n",
"+ self.assertEqual(kp_params.offset_head_kernel_sizes, [5, 3])\n",
"+ else:\n",
"+ # Default values:\n",
"+ self.assertEqual(kp_params.heatmap_head_num_filters, [256])\n",
"+ self.assertEqual(kp_params.heatmap_head_kernel_sizes, [3])\n",
"+ self.assertEqual(kp_params.offset_head_num_filters, [256])\n",
"+ self.assertEqual(kp_params.offset_head_kernel_sizes, [3])\n",
" \n",
" # Check mask related parameters.\n",
" self.assertAlmostEqual(model._mask_params.task_loss_weight, 0.7)\n"
],
"name": "stdout"
},
{
"output_type": "stream",
"text": [
"Cloning into 'models'...\n"
],
"name": "stderr"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "x8kQDJH55inl"
},
"source": [
"%%bash\n",
"\n",
"cat <<EOF > /content/model_lib.patch\n",
"diff --git a/research/object_detection/model_lib.py b/research/object_detection/model_lib.py\n",
"index 111be9cb..bd8ab197 100644\n",
"--- a/research/object_detection/model_lib.py\n",
"+++ b/research/object_detection/model_lib.py\n",
"@@ -497,7 +497,7 @@ def create_model_fn(detection_model_fn, configs, hparams=None, use_tpu=False,\n",
" features[fields.InputDataFields.true_image_shape]))\n",
" \n",
" if mode == tf.estimator.ModeKeys.TRAIN:\n",
"- load_pretrained = hparams.load_pretrained if hparams else False\n",
"+ load_pretrained = hparams.load_pretrained if hparams else True\n",
" if train_config.fine_tune_checkpoint and load_pretrained:\n",
" if not train_config.fine_tune_checkpoint_type:\n",
" # train_config.from_detection_checkpoint field is deprecated. For\n",
"@@ -507,6 +507,14 @@ def create_model_fn(detection_model_fn, configs, hparams=None, use_tpu=False,\n",
" train_config.fine_tune_checkpoint_type = 'detection'\n",
" else:\n",
" train_config.fine_tune_checkpoint_type = 'classification'\n",
"+\n",
"+ tf.logging.info('fine tuning parameter: fine_tune_checkpoint_type %s, '\n",
"+ 'from_detection_checkpoint %s fine_tune_checkpoint %s '\n",
"+ 'load_all_detection_checkpoint_vars %s',\n",
"+ train_config.fine_tune_checkpoint_type,\n",
"+ train_config.from_detection_checkpoint,\n",
"+ train_config.fine_tune_checkpoint,\n",
"+ train_config.load_all_detection_checkpoint_vars)\n",
" asg_map = detection_model.restore_map(\n",
" fine_tune_checkpoint_type=train_config.fine_tune_checkpoint_type,\n",
" load_all_detection_checkpoint_vars=(\n",
"EOF"
],
"execution_count": 5,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "ueEgAw_s53dG",
"outputId": "73b077f1-a2c7-441e-bbd2-af5f4ed0d8af"
},
"source": [
"%%bash\n",
"\n",
"cd models\n",
"patch -p1 < /content/model_lib.patch"
],
"execution_count": 6,
"outputs": [
{
"output_type": "stream",
"text": [
"patching file research/object_detection/model_lib.py\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "3d73ABl0hluK",
"outputId": "b37b6587-62f7-4692-f484-edcccad0e7f4"
},
"source": [
"%%bash\n",
"\n",
"cd models/research\n",
"# Compile protos.\n",
"protoc object_detection/protos/*.proto --python_out=.\n",
"\n",
"# Install TensorFlow Object Detection API.\n",
"cp object_detection/packages/tf1/setup.py .\n",
"python -m pip install ."
],
"execution_count": 7,
"outputs": [
{
"output_type": "stream",
"text": [
"Processing /content/models/research\n",
"Requirement already satisfied: pillow in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (7.1.2)\n",
"Requirement already satisfied: lxml in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (4.2.6)\n",
"Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (3.2.2)\n",
"Requirement already satisfied: Cython in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (0.29.22)\n",
"Requirement already satisfied: contextlib2 in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (0.5.5)\n",
"Collecting tf-slim\n",
" Downloading https://files.pythonhosted.org/packages/02/97/b0f4a64df018ca018cc035d44f2ef08f91e2e8aa67271f6f19633a015ff7/tf_slim-1.1.0-py2.py3-none-any.whl (352kB)\n",
"Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (1.15.0)\n",
"Requirement already satisfied: pycocotools in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (2.0.2)\n",
"Collecting lvis\n",
" Downloading https://files.pythonhosted.org/packages/72/b6/1992240ab48310b5360bfdd1d53163f43bb97d90dc5dc723c67d41c38e78/lvis-0.5.3-py3-none-any.whl\n",
"Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (1.4.1)\n",
"Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from object-detection==0.1) (1.1.5)\n",
"Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->object-detection==0.1) (0.10.0)\n",
"Requirement already satisfied: numpy>=1.11 in /usr/local/lib/python3.7/dist-packages (from matplotlib->object-detection==0.1) (1.19.5)\n",
"Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->object-detection==0.1) (1.3.1)\n",
"Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->object-detection==0.1) (2.4.7)\n",
"Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->object-detection==0.1) (2.8.1)\n",
"Requirement already satisfied: absl-py>=0.2.2 in /usr/local/lib/python3.7/dist-packages (from tf-slim->object-detection==0.1) (0.12.0)\n",
"Requirement already satisfied: setuptools>=18.0 in /usr/local/lib/python3.7/dist-packages (from pycocotools->object-detection==0.1) (56.0.0)\n",
"Requirement already satisfied: opencv-python>=4.1.0.25 in /usr/local/lib/python3.7/dist-packages (from lvis->object-detection==0.1) (4.1.2.30)\n",
"Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->object-detection==0.1) (2018.9)\n",
"Building wheels for collected packages: object-detection\n",
" Building wheel for object-detection (setup.py): started\n",
" Building wheel for object-detection (setup.py): finished with status 'done'\n",
" Created wheel for object-detection: filename=object_detection-0.1-cp37-none-any.whl size=1647527 sha256=b63dc1d0f49f7e0b003b63c3d6429c8fe45af0a96d3a54a5255898f6db568f44\n",
" Stored in directory: /tmp/pip-ephem-wheel-cache-jo2lg33i/wheels/94/49/4b/39b051683087a22ef7e80ec52152a27249d1a644ccf4e442ea\n",
"Successfully built object-detection\n",
"Installing collected packages: tf-slim, lvis, object-detection\n",
"Successfully installed lvis-0.5.3 object-detection-0.1 tf-slim-1.1.0\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "4F3kd4rEGIgo",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "90fc1deb-eb62-4b30-8595-397b4e8df4d9"
},
"source": [
"%cd /content/models/research"
],
"execution_count": 8,
"outputs": [
{
"output_type": "stream",
"text": [
"/content/models/research\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "XRfEVbHijRiq",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "20d5606d-4fa5-4848-b252-0d097ea73fc0"
},
"source": [
"import os\n",
"\n",
"os.environ['PYTHONPATH'] = '/content/models:' + os.environ['PYTHONPATH']\n",
"print(os.environ['PYTHONPATH'])"
],
"execution_count": 9,
"outputs": [
{
"output_type": "stream",
"text": [
"/content/models:/tensorflow-1.15.2/python3.7:/env/python\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "Y_uBeLzBHHVH",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "2a6a347e-2d50-4a09-b780-3ec8d9f388de"
},
"source": [
"!python object_detection/builders/model_builder_tf1_test.py"
],
"execution_count": 10,
"outputs": [
{
"output_type": "stream",
"text": [
"Running tests under Python 3.7.10: /usr/bin/python3\n",
"[ RUN ] ModelBuilderTF1Test.test_create_context_rcnn_from_config_with_params0 (True)\n",
"[ OK ] ModelBuilderTF1Test.test_create_context_rcnn_from_config_with_params0 (True)\n",
"[ RUN ] ModelBuilderTF1Test.test_create_context_rcnn_from_config_with_params1 (False)\n",
"[ OK ] ModelBuilderTF1Test.test_create_context_rcnn_from_config_with_params1 (False)\n",
"[ RUN ] ModelBuilderTF1Test.test_create_experimental_model\n",
"[ OK ] ModelBuilderTF1Test.test_create_experimental_model\n",
"[ RUN ] ModelBuilderTF1Test.test_create_faster_rcnn_from_config_with_crop_feature0 (True)\n",
"[ OK ] ModelBuilderTF1Test.test_create_faster_rcnn_from_config_with_crop_feature0 (True)\n",
"[ RUN ] ModelBuilderTF1Test.test_create_faster_rcnn_from_config_with_crop_feature1 (False)\n",
"[ OK ] ModelBuilderTF1Test.test_create_faster_rcnn_from_config_with_crop_feature1 (False)\n",
"[ RUN ] ModelBuilderTF1Test.test_create_faster_rcnn_model_from_config_with_example_miner\n",
"[ OK ] ModelBuilderTF1Test.test_create_faster_rcnn_model_from_config_with_example_miner\n",
"[ RUN ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_faster_rcnn_with_matmul\n",
"[ OK ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_faster_rcnn_with_matmul\n",
"[ RUN ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_faster_rcnn_without_matmul\n",
"[ OK ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_faster_rcnn_without_matmul\n",
"[ RUN ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_mask_rcnn_with_matmul\n",
"[ OK ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_mask_rcnn_with_matmul\n",
"[ RUN ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_mask_rcnn_without_matmul\n",
"[ OK ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_mask_rcnn_without_matmul\n",
"[ RUN ] ModelBuilderTF1Test.test_create_rfcn_model_from_config\n",
"[ OK ] ModelBuilderTF1Test.test_create_rfcn_model_from_config\n",
"[ RUN ] ModelBuilderTF1Test.test_create_ssd_fpn_model_from_config\n",
"[ OK ] ModelBuilderTF1Test.test_create_ssd_fpn_model_from_config\n",
"[ RUN ] ModelBuilderTF1Test.test_create_ssd_models_from_config\n",
"[ OK ] ModelBuilderTF1Test.test_create_ssd_models_from_config\n",
"[ RUN ] ModelBuilderTF1Test.test_invalid_faster_rcnn_batchnorm_update\n",
"[ OK ] ModelBuilderTF1Test.test_invalid_faster_rcnn_batchnorm_update\n",
"[ RUN ] ModelBuilderTF1Test.test_invalid_first_stage_nms_iou_threshold\n",
"[ OK ] ModelBuilderTF1Test.test_invalid_first_stage_nms_iou_threshold\n",
"[ RUN ] ModelBuilderTF1Test.test_invalid_model_config_proto\n",
"[ OK ] ModelBuilderTF1Test.test_invalid_model_config_proto\n",
"[ RUN ] ModelBuilderTF1Test.test_invalid_second_stage_batch_size\n",
"[ OK ] ModelBuilderTF1Test.test_invalid_second_stage_batch_size\n",
"[ RUN ] ModelBuilderTF1Test.test_session\n",
"[ SKIPPED ] ModelBuilderTF1Test.test_session\n",
"[ RUN ] ModelBuilderTF1Test.test_unknown_faster_rcnn_feature_extractor\n",
"[ OK ] ModelBuilderTF1Test.test_unknown_faster_rcnn_feature_extractor\n",
"[ RUN ] ModelBuilderTF1Test.test_unknown_meta_architecture\n",
"[ OK ] ModelBuilderTF1Test.test_unknown_meta_architecture\n",
"[ RUN ] ModelBuilderTF1Test.test_unknown_ssd_feature_extractor\n",
"[ OK ] ModelBuilderTF1Test.test_unknown_ssd_feature_extractor\n",
"----------------------------------------------------------------------\n",
"Ran 21 tests in 0.214s\n",
"\n",
"OK (skipped=1)\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "iKdLuAihHhFA"
},
"source": [
"# Download Pet dataset and create tf-record"
]
},
{
"cell_type": "code",
"metadata": {
"id": "rZG8f4LsnpNl",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "d2081229-049f-4b8e-ba02-0272ba4f0aea"
},
"source": [
"%set_env DATA_DIR=/content/data\n",
"%set_env TF_RECORD_DIR=/content/data/tf_record"
],
"execution_count": 11,
"outputs": [
{
"output_type": "stream",
"text": [
"env: DATA_DIR=/content/data\n",
"env: TF_RECORD_DIR=/content/data/tf_record\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "8ANpO7QmHjbo",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "defe6d85-c385-4879-8f86-c706085b11eb"
},
"source": [
"# Download dataset and extrac tar files.\n",
"!mkdir ${DATA_DIR}\n",
"!mkdir ${TF_RECORD_DIR}\n",
"!wget http://www.robots.ox.ac.uk/~vgg/data/pets/data/images.tar.gz -P ${DATA_DIR}\n",
"!wget http://www.robots.ox.ac.uk/~vgg/data/pets/data/annotations.tar.gz -P ${DATA_DIR}\n",
"!tar xf ${DATA_DIR}/images.tar.gz -C ${DATA_DIR}\n",
"!tar xf ${DATA_DIR}/annotations.tar.gz -C ${DATA_DIR}\n",
"\n",
"# Create TF-Record.\n",
"!python object_detection/dataset_tools/create_pet_tf_record.py \\\n",
" --label_map_path=./object_detection/data/pet_label_map.pbtxt \\\n",
" --data_dir=\"${DATA_DIR}\" \\\n",
" --output_dir=\"${TF_RECORD_DIR}\"\n",
"\n",
"!ls ${TF_RECORD_DIR}"
],
"execution_count": 12,
"outputs": [
{
"output_type": "stream",
"text": [
"--2021-05-03 22:43:57-- http://www.robots.ox.ac.uk/~vgg/data/pets/data/images.tar.gz\n",
"Resolving www.robots.ox.ac.uk (www.robots.ox.ac.uk)... 129.67.94.2\n",
"Connecting to www.robots.ox.ac.uk (www.robots.ox.ac.uk)|129.67.94.2|:80... connected.\n",
"HTTP request sent, awaiting response... 301 Moved Permanently\n",
"Location: https://www.robots.ox.ac.uk/~vgg/data/pets/data/images.tar.gz [following]\n",
"--2021-05-03 22:43:57-- https://www.robots.ox.ac.uk/~vgg/data/pets/data/images.tar.gz\n",
"Connecting to www.robots.ox.ac.uk (www.robots.ox.ac.uk)|129.67.94.2|:443... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: 791918971 (755M) [application/x-gzip]\n",
"Saving to: ‘/content/data/images.tar.gz’\n",
"\n",
"images.tar.gz 100%[===================>] 755.23M 68.9MB/s in 11s \n",
"\n",
"2021-05-03 22:44:08 (68.1 MB/s) - ‘/content/data/images.tar.gz’ saved [791918971/791918971]\n",
"\n",
"--2021-05-03 22:44:08-- http://www.robots.ox.ac.uk/~vgg/data/pets/data/annotations.tar.gz\n",
"Resolving www.robots.ox.ac.uk (www.robots.ox.ac.uk)... 129.67.94.2\n",
"Connecting to www.robots.ox.ac.uk (www.robots.ox.ac.uk)|129.67.94.2|:80... connected.\n",
"HTTP request sent, awaiting response... 301 Moved Permanently\n",
"Location: https://www.robots.ox.ac.uk/~vgg/data/pets/data/annotations.tar.gz [following]\n",
"--2021-05-03 22:44:08-- https://www.robots.ox.ac.uk/~vgg/data/pets/data/annotations.tar.gz\n",
"Connecting to www.robots.ox.ac.uk (www.robots.ox.ac.uk)|129.67.94.2|:443... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: 19173078 (18M) [application/x-gzip]\n",
"Saving to: ‘/content/data/annotations.tar.gz’\n",
"\n",
"annotations.tar.gz 100%[===================>] 18.28M 58.9MB/s in 0.3s \n",
"\n",
"2021-05-03 22:44:09 (58.9 MB/s) - ‘/content/data/annotations.tar.gz’ saved [19173078/19173078]\n",
"\n",
"I0503 22:44:17.412478 139960508098432 create_pet_tf_record.py:273] Reading from Pet dataset.\n",
"I0503 22:44:17.422560 139960508098432 create_pet_tf_record.py:288] 2576 training and 1104 validation examples.\n",
"I0503 22:44:17.423127 139960508098432 create_pet_tf_record.py:241] On image 0 of 2576\n",
"/usr/local/lib/python3.7/dist-packages/object_detection/utils/dataset_util.py:83: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(elem)' or 'elem is not None' test instead.\n",
" if not xml:\n",
"I0503 22:44:17.630365 139960508098432 create_pet_tf_record.py:241] On image 100 of 2576\n",
"I0503 22:44:17.820542 139960508098432 create_pet_tf_record.py:241] On image 200 of 2576\n",
"I0503 22:44:18.008630 139960508098432 create_pet_tf_record.py:241] On image 300 of 2576\n",
"I0503 22:44:18.212171 139960508098432 create_pet_tf_record.py:241] On image 400 of 2576\n",
"I0503 22:44:18.400474 139960508098432 create_pet_tf_record.py:241] On image 500 of 2576\n",
"W0503 22:44:18.538891 139960508098432 create_pet_tf_record.py:246] Could not find /content/data/annotations/xmls/Egyptian_Mau_14.xml, ignoring example.\n",
"W0503 22:44:18.570832 139960508098432 create_pet_tf_record.py:246] Could not find /content/data/annotations/xmls/saint_bernard_15.xml, ignoring example.\n",
"I0503 22:44:18.592516 139960508098432 create_pet_tf_record.py:241] On image 600 of 2576\n",
"W0503 22:44:18.600499 139960508098432 create_pet_tf_record.py:246] Could not find /content/data/annotations/xmls/Ragdoll_199.xml, ignoring example.\n",
"I0503 22:44:18.784520 139960508098432 create_pet_tf_record.py:241] On image 700 of 2576\n",
"I0503 22:44:18.980668 139960508098432 create_pet_tf_record.py:241] On image 800 of 2576\n",
"I0503 22:44:19.164678 139960508098432 create_pet_tf_record.py:241] On image 900 of 2576\n",
"I0503 22:44:19.353859 139960508098432 create_pet_tf_record.py:241] On image 1000 of 2576\n",
"I0503 22:44:19.542048 139960508098432 create_pet_tf_record.py:241] On image 1100 of 2576\n",
"I0503 22:44:19.754319 139960508098432 create_pet_tf_record.py:241] On image 1200 of 2576\n",
"I0503 22:44:19.959057 139960508098432 create_pet_tf_record.py:241] On image 1300 of 2576\n",
"I0503 22:44:20.199476 139960508098432 create_pet_tf_record.py:241] On image 1400 of 2576\n",
"W0503 22:44:20.300813 139960508098432 create_pet_tf_record.py:246] Could not find /content/data/annotations/xmls/Egyptian_Mau_186.xml, ignoring example.\n",
"I0503 22:44:20.392260 139960508098432 create_pet_tf_record.py:241] On image 1500 of 2576\n",
"W0503 22:44:20.521817 139960508098432 create_pet_tf_record.py:246] Could not find /content/data/annotations/xmls/Bengal_175.xml, ignoring example.\n",
"I0503 22:44:20.583704 139960508098432 create_pet_tf_record.py:241] On image 1600 of 2576\n",
"I0503 22:44:20.766173 139960508098432 create_pet_tf_record.py:241] On image 1700 of 2576\n",
"W0503 22:44:20.880429 139960508098432 create_pet_tf_record.py:246] Could not find /content/data/annotations/xmls/samoyed_10.xml, ignoring example.\n",
"I0503 22:44:20.946202 139960508098432 create_pet_tf_record.py:241] On image 1800 of 2576\n",
"I0503 22:44:21.145046 139960508098432 create_pet_tf_record.py:241] On image 1900 of 2576\n",
"I0503 22:44:21.332493 139960508098432 create_pet_tf_record.py:241] On image 2000 of 2576\n",
"I0503 22:44:21.528210 139960508098432 create_pet_tf_record.py:241] On image 2100 of 2576\n",
"W0503 22:44:21.596982 139960508098432 create_pet_tf_record.py:246] Could not find /content/data/annotations/xmls/Egyptian_Mau_156.xml, ignoring example.\n",
"I0503 22:44:21.728883 139960508098432 create_pet_tf_record.py:241] On image 2200 of 2576\n",
"I0503 22:44:21.926898 139960508098432 create_pet_tf_record.py:241] On image 2300 of 2576\n",
"I0503 22:44:22.108330 139960508098432 create_pet_tf_record.py:241] On image 2400 of 2576\n",
"I0503 22:44:22.298947 139960508098432 create_pet_tf_record.py:241] On image 2500 of 2576\n",
"W0503 22:44:22.395591 139960508098432 create_pet_tf_record.py:246] Could not find /content/data/annotations/xmls/Abyssinian_104.xml, ignoring example.\n",
"I0503 22:44:22.450354 139960508098432 create_pet_tf_record.py:241] On image 0 of 1104\n",
"I0503 22:44:22.641016 139960508098432 create_pet_tf_record.py:241] On image 100 of 1104\n",
"I0503 22:44:22.834721 139960508098432 create_pet_tf_record.py:241] On image 200 of 1104\n",
"I0503 22:44:23.016927 139960508098432 create_pet_tf_record.py:241] On image 300 of 1104\n",
"I0503 22:44:23.217114 139960508098432 create_pet_tf_record.py:241] On image 400 of 1104\n",
"I0503 22:44:23.400382 139960508098432 create_pet_tf_record.py:241] On image 500 of 1104\n",
"I0503 22:44:23.596589 139960508098432 create_pet_tf_record.py:241] On image 600 of 1104\n",
"I0503 22:44:23.786553 139960508098432 create_pet_tf_record.py:241] On image 700 of 1104\n",
"I0503 22:44:23.993257 139960508098432 create_pet_tf_record.py:241] On image 800 of 1104\n",
"W0503 22:44:24.078453 139960508098432 create_pet_tf_record.py:246] Could not find /content/data/annotations/xmls/Bengal_111.xml, ignoring example.\n",
"I0503 22:44:24.195823 139960508098432 create_pet_tf_record.py:241] On image 900 of 1104\n",
"I0503 22:44:24.379019 139960508098432 create_pet_tf_record.py:241] On image 1000 of 1104\n",
"I0503 22:44:24.605327 139960508098432 create_pet_tf_record.py:241] On image 1100 of 1104\n",
"pet_faces_train.record-00000-of-00010 pet_faces_val.record-00000-of-00010\n",
"pet_faces_train.record-00001-of-00010 pet_faces_val.record-00001-of-00010\n",
"pet_faces_train.record-00002-of-00010 pet_faces_val.record-00002-of-00010\n",
"pet_faces_train.record-00003-of-00010 pet_faces_val.record-00003-of-00010\n",
"pet_faces_train.record-00004-of-00010 pet_faces_val.record-00004-of-00010\n",
"pet_faces_train.record-00005-of-00010 pet_faces_val.record-00005-of-00010\n",
"pet_faces_train.record-00006-of-00010 pet_faces_val.record-00006-of-00010\n",
"pet_faces_train.record-00007-of-00010 pet_faces_val.record-00007-of-00010\n",
"pet_faces_train.record-00008-of-00010 pet_faces_val.record-00008-of-00010\n",
"pet_faces_train.record-00009-of-00010 pet_faces_val.record-00009-of-00010\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Ndb6N895JwJ6"
},
"source": [
"# Get the pretrained model for transfer learning\n",
"\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "at-wzrEXkDIe",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "d3a1fb15-ede1-4240-b36e-bfbe1107bef1"
},
"source": [
"!wget http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v3_large_coco_2020_01_14.tar.gz -P ${DATA_DIR}\n",
"!tar xf ${DATA_DIR}/ssd_mobilenet_v3_large_coco_2020_01_14.tar.gz -C ${DATA_DIR}"
],
"execution_count": 13,
"outputs": [
{
"output_type": "stream",
"text": [
"--2021-05-03 22:44:25-- http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v3_large_coco_2020_01_14.tar.gz\n",
"Resolving download.tensorflow.org (download.tensorflow.org)... 173.194.79.128, 2a00:1450:4013:c00::80\n",
"Connecting to download.tensorflow.org (download.tensorflow.org)|173.194.79.128|:80... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: 49318766 (47M) [application/gzip]\n",
"Saving to: ‘/content/data/ssd_mobilenet_v3_large_coco_2020_01_14.tar.gz’\n",
"\n",
"ssd_mobilenet_v3_la 100%[===================>] 47.03M 31.6MB/s in 1.5s \n",
"\n",
"2021-05-03 22:44:27 (31.6 MB/s) - ‘/content/data/ssd_mobilenet_v3_large_coco_2020_01_14.tar.gz’ saved [49318766/49318766]\n",
"\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "KlCKFYLmBvI6"
},
"source": [
"# Create config file"
]
},
{
"cell_type": "code",
"metadata": {
"id": "pDwELNPC-Sqh"
},
"source": [
"!cp ./object_detection/samples/configs/ssdlite_mobilenet_v3_large_320x320_coco.config ${DATA_DIR}/ssdlite_mobilenet_v3_large_320x320_pet.config"
],
"execution_count": 14,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "wpRIcKWn_NnN"
},
"source": [
"%%bash\n",
"\n",
"cat <<EOF > ${DATA_DIR}/config.patch\n",
"--- ./object_detection/samples/configs/ssdlite_mobilenet_v3_large_320x320_coco.config\t2021-05-03 08:34:03.934459490 +0000\n",
"+++ /content/data/ssdlite_mobilenet_v3_large_320x320_pet.config\t2021-05-03 09:01:56.235655642 +0000\n",
"@@ -11,7 +11,7 @@\n",
" ssd {\n",
" inplace_batchnorm_update: true\n",
" freeze_batchnorm: false\n",
"- num_classes: 90\n",
"+ num_classes: 37\n",
" box_coder {\n",
" faster_rcnn_box_coder {\n",
" y_scale: 10.0\n",
"@@ -147,11 +147,14 @@\n",
" }\n",
" \n",
" train_config: {\n",
"- batch_size: 512\n",
"+ batch_size: 128\n",
" sync_replicas: true\n",
" startup_delay_steps: 0\n",
" replicas_to_aggregate: 32\n",
" num_steps: 400000\n",
"+ fine_tune_checkpoint: \"/content/data/ssd_mobilenet_v3_large_coco_2020_01_14/model.ckpt\"\n",
"+ load_all_detection_checkpoint_vars: true\n",
"+ fine_tune_checkpoint_type: \"detection\"\n",
" data_augmentation_options {\n",
" random_horizontal_flip {\n",
" }\n",
"@@ -164,9 +167,9 @@\n",
" momentum_optimizer: {\n",
" learning_rate: {\n",
" cosine_decay_learning_rate {\n",
"- learning_rate_base: 0.4\n",
"+ learning_rate_base: 0.04\n",
" total_steps: 400000\n",
"- warmup_learning_rate: 0.13333\n",
"+ warmup_learning_rate: 0.013333\n",
" warmup_steps: 2000\n",
" }\n",
" }\n",
"@@ -180,20 +183,20 @@\n",
" \n",
" train_input_reader: {\n",
" tf_record_input_reader {\n",
"- input_path: \"PATH_TO_BE_CONFIGURED/mscoco_train.record-?????-of-00100\"\n",
"+ input_path: \"/content/data/tf_record/pet_faces_train.record-?????-of-00010\"\n",
" }\n",
"- label_map_path: \"PATH_TO_BE_CONFIGURED/mscoco_label_map.pbtxt\"\n",
"+ label_map_path: \"/content/models/research/object_detection/data/pet_label_map.pbtxt\"\n",
" }\n",
" \n",
" eval_config: {\n",
"- num_examples: 8000\n",
"+ num_examples: 1104\n",
" }\n",
" \n",
" eval_input_reader: {\n",
" tf_record_input_reader {\n",
"- input_path: \"PATH_TO_BE_CONFIGURED/mscoco_val.record-?????-of-00010\"\n",
"+ input_path: \"/content/data/tf_record/pet_faces_train.record-?????-of-00010\"\n",
" }\n",
"- label_map_path: \"PATH_TO_BE_CONFIGURED/mscoco_label_map.pbtxt\"\n",
"+ label_map_path: \"/content/models/research/object_detection/data/pet_label_map.pbtxt\"\n",
" shuffle: false\n",
" num_readers: 1\n",
" }\n",
"EOF"
],
"execution_count": 15,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "kmptYYT6-PyE",
"outputId": "45ee3bbe-4286-4e0d-bfb5-1b58b18a09fb"
},
"source": [
"!patch -u ${DATA_DIR}/ssdlite_mobilenet_v3_large_320x320_pet.config < ${DATA_DIR}/config.patch"
],
"execution_count": 16,
"outputs": [
{
"output_type": "stream",
"text": [
"patching file /content/data/ssdlite_mobilenet_v3_large_320x320_pet.config\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "SwbAVryqA5t7",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "3f75d300-d25f-4bf8-d26a-cb9eccf9d6b6"
},
"source": [
"!cat ${DATA_DIR}/ssdlite_mobilenet_v3_large_320x320_pet.config"
],
"execution_count": 17,
"outputs": [
{
"output_type": "stream",
"text": [
"# SSDLite with Mobilenet v3 large feature extractor.\n",
"# Trained on COCO14, initialized from scratch.\n",
"# 3.22M parameters, 1.02B FLOPs\n",
"# TPU-compatible.\n",
"# Users should configure the fine_tune_checkpoint field in the train config as\n",
"# well as the label_map_path and input_path fields in the train_input_reader and\n",
"# eval_input_reader. Search for \"PATH_TO_BE_CONFIGURED\" to find the fields that\n",
"# should be configured.\n",
"\n",
"model {\n",
" ssd {\n",
" inplace_batchnorm_update: true\n",
" freeze_batchnorm: false\n",
" num_classes: 37\n",
" box_coder {\n",
" faster_rcnn_box_coder {\n",
" y_scale: 10.0\n",
" x_scale: 10.0\n",
" height_scale: 5.0\n",
" width_scale: 5.0\n",
" }\n",
" }\n",
" matcher {\n",
" argmax_matcher {\n",
" matched_threshold: 0.5\n",
" unmatched_threshold: 0.5\n",
" ignore_thresholds: false\n",
" negatives_lower_than_unmatched: true\n",
" force_match_for_each_row: true\n",
" use_matmul_gather: true\n",
" }\n",
" }\n",
" similarity_calculator {\n",
" iou_similarity {\n",
" }\n",
" }\n",
" encode_background_as_zeros: true\n",
" anchor_generator {\n",
" ssd_anchor_generator {\n",
" num_layers: 6\n",
" min_scale: 0.2\n",
" max_scale: 0.95\n",
" aspect_ratios: 1.0\n",
" aspect_ratios: 2.0\n",
" aspect_ratios: 0.5\n",
" aspect_ratios: 3.0\n",
" aspect_ratios: 0.3333\n",
" }\n",
" }\n",
" image_resizer {\n",
" fixed_shape_resizer {\n",
" height: 320\n",
" width: 320\n",
" }\n",
" }\n",
" box_predictor {\n",
" convolutional_box_predictor {\n",
" min_depth: 0\n",
" max_depth: 0\n",
" num_layers_before_predictor: 0\n",
" use_dropout: false\n",
" dropout_keep_probability: 0.8\n",
" kernel_size: 3\n",
" use_depthwise: true\n",
" box_code_size: 4\n",
" apply_sigmoid_to_scores: false\n",
" class_prediction_bias_init: -4.6\n",
" conv_hyperparams {\n",
" activation: RELU_6,\n",
" regularizer {\n",
" l2_regularizer {\n",
" weight: 0.00004\n",
" }\n",
" }\n",
" initializer {\n",
" random_normal_initializer {\n",
" stddev: 0.03\n",
" mean: 0.0\n",
" }\n",
" }\n",
" batch_norm {\n",
" train: true,\n",
" scale: true,\n",
" center: true,\n",
" decay: 0.97,\n",
" epsilon: 0.001,\n",
" }\n",
" }\n",
" }\n",
" }\n",
" feature_extractor {\n",
" type: 'ssd_mobilenet_v3_large'\n",
" min_depth: 16\n",
" depth_multiplier: 1.0\n",
" use_depthwise: true\n",
" conv_hyperparams {\n",
" activation: RELU_6,\n",
" regularizer {\n",
" l2_regularizer {\n",
" weight: 0.00004\n",
" }\n",
" }\n",
" initializer {\n",
" truncated_normal_initializer {\n",
" stddev: 0.03\n",
" mean: 0.0\n",
" }\n",
" }\n",
" batch_norm {\n",
" train: true,\n",
" scale: true,\n",
" center: true,\n",
" decay: 0.97,\n",
" epsilon: 0.001,\n",
" }\n",
" }\n",
" override_base_feature_extractor_hyperparams: true\n",
" }\n",
" loss {\n",
" classification_loss {\n",
" weighted_sigmoid_focal {\n",
" alpha: 0.75,\n",
" gamma: 2.0\n",
" }\n",
" }\n",
" localization_loss {\n",
" weighted_smooth_l1 {\n",
" delta: 1.0\n",
" }\n",
" }\n",
" classification_weight: 1.0\n",
" localization_weight: 1.0\n",
" }\n",
" normalize_loss_by_num_matches: true\n",
" normalize_loc_loss_by_codesize: true\n",
" post_processing {\n",
" batch_non_max_suppression {\n",
" score_threshold: 1e-8\n",
" iou_threshold: 0.6\n",
" max_detections_per_class: 100\n",
" max_total_detections: 100\n",
" use_static_shapes: true\n",
" }\n",
" score_converter: SIGMOID\n",
" }\n",
" }\n",
"}\n",
"\n",
"train_config: {\n",
" batch_size: 128\n",
" sync_replicas: true\n",
" startup_delay_steps: 0\n",
" replicas_to_aggregate: 32\n",
" num_steps: 400000\n",
" fine_tune_checkpoint: \"/content/data/ssd_mobilenet_v3_large_coco_2020_01_14/model.ckpt\"\n",
" load_all_detection_checkpoint_vars: true\n",
" fine_tune_checkpoint_type: \"detection\"\n",
" data_augmentation_options {\n",
" random_horizontal_flip {\n",
" }\n",
" }\n",
" data_augmentation_options {\n",
" ssd_random_crop {\n",
" }\n",
" }\n",
" optimizer {\n",
" momentum_optimizer: {\n",
" learning_rate: {\n",
" cosine_decay_learning_rate {\n",
" learning_rate_base: 0.04\n",
" total_steps: 400000\n",
" warmup_learning_rate: 0.013333\n",
" warmup_steps: 2000\n",
" }\n",
" }\n",
" momentum_optimizer_value: 0.9\n",
" }\n",
" use_moving_average: false\n",
" }\n",
" max_number_of_boxes: 100\n",
" unpad_groundtruth_tensors: false\n",
"}\n",
"\n",
"train_input_reader: {\n",
" tf_record_input_reader {\n",
" input_path: \"/content/data/tf_record/pet_faces_train.record-?????-of-00010\"\n",
" }\n",
" label_map_path: \"/content/models/research/object_detection/data/pet_label_map.pbtxt\"\n",
"}\n",
"\n",
"eval_config: {\n",
" num_examples: 1104\n",
"}\n",
"\n",
"eval_input_reader: {\n",
" tf_record_input_reader {\n",
" input_path: \"/content/data/tf_record/pet_faces_train.record-?????-of-00010\"\n",
" }\n",
" label_map_path: \"/content/models/research/object_detection/data/pet_label_map.pbtxt\"\n",
" shuffle: false\n",
" num_readers: 1\n",
"}\n",
"\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rF7J2C_aUb0n"
},
"source": [
"# Train model"
]
},
{
"cell_type": "code",
"metadata": {
"id": "gs57oAxVUery",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "d630b65c-d947-4fcd-dd20-849914a53466"
},
"source": [
"!python ./object_detection/model_main.py \\\n",
" --alsologtostderr \\\n",
" --pipeline_config_path=${DATA_DIR}/ssdlite_mobilenet_v3_large_320x320_pet.config \\\n",
" --num_train_steps=5000 \\\n",
" --sample_1_of_n_eval_examples=1 \\\n",
" --model_dir=${DATA_DIR}/train_ssdlite_mobilenet_v3_large_320x320_pet"
],
"execution_count": 18,
"outputs": [
{
"output_type": "stream",
"text": [
"WARNING:tensorflow:Forced number of epochs for all eval validations to be 1.\n",
"W0503 22:44:31.944674 140215059400576 model_lib.py:820] Forced number of epochs for all eval validations to be 1.\n",
"INFO:tensorflow:Maybe overwriting train_steps: 5000\n",
"I0503 22:44:31.944860 140215059400576 config_util.py:552] Maybe overwriting train_steps: 5000\n",
"INFO:tensorflow:Maybe overwriting use_bfloat16: False\n",
"I0503 22:44:31.944965 140215059400576 config_util.py:552] Maybe overwriting use_bfloat16: False\n",
"INFO:tensorflow:Maybe overwriting sample_1_of_n_eval_examples: 1\n",
"I0503 22:44:31.945029 140215059400576 config_util.py:552] Maybe overwriting sample_1_of_n_eval_examples: 1\n",
"INFO:tensorflow:Maybe overwriting eval_num_epochs: 1\n",
"I0503 22:44:31.945102 140215059400576 config_util.py:552] Maybe overwriting eval_num_epochs: 1\n",
"WARNING:tensorflow:Expected number of evaluation epochs is 1, but instead encountered `eval_on_train_input_config.num_epochs` = 0. Overwriting `num_epochs` to 1.\n",
"W0503 22:44:31.945193 140215059400576 model_lib.py:836] Expected number of evaluation epochs is 1, but instead encountered `eval_on_train_input_config.num_epochs` = 0. Overwriting `num_epochs` to 1.\n",
"INFO:tensorflow:create_estimator_and_inputs: use_tpu False, export_to_tpu None\n",
"I0503 22:44:31.945266 140215059400576 model_lib.py:873] create_estimator_and_inputs: use_tpu False, export_to_tpu None\n",
"INFO:tensorflow:Using config: {'_model_dir': '/content/data/train_ssdlite_mobilenet_v3_large_320x320_pet', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true\n",
"graph_options {\n",
" rewrite_options {\n",
" meta_optimizer_iterations: ONE\n",
" }\n",
"}\n",
", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f85c7012d90>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
"I0503 22:44:31.945604 140215059400576 estimator.py:212] Using config: {'_model_dir': '/content/data/train_ssdlite_mobilenet_v3_large_320x320_pet', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true\n",
"graph_options {\n",
" rewrite_options {\n",
" meta_optimizer_iterations: ONE\n",
" }\n",
"}\n",
", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f85c7012d90>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
"WARNING:tensorflow:Estimator's model_fn (<function create_model_fn.<locals>.model_fn at 0x7f85c7013a70>) includes params argument, but params are not passed to Estimator.\n",
"W0503 22:44:31.945781 140215059400576 model_fn.py:630] Estimator's model_fn (<function create_model_fn.<locals>.model_fn at 0x7f85c7013a70>) includes params argument, but params are not passed to Estimator.\n",
"INFO:tensorflow:Not using Distribute Coordinator.\n",
"I0503 22:44:31.946280 140215059400576 estimator_training.py:186] Not using Distribute Coordinator.\n",
"INFO:tensorflow:Running training and evaluation locally (non-distributed).\n",
"I0503 22:44:31.946429 140215059400576 training.py:612] Running training and evaluation locally (non-distributed).\n",
"INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600.\n",
"I0503 22:44:31.946604 140215059400576 training.py:700] Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600.\n",
"WARNING:tensorflow:From /tensorflow-1.15.2/python3.7/tensorflow_core/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.\n",
"W0503 22:44:31.950980 140215059400576 deprecation.py:323] From /tensorflow-1.15.2/python3.7/tensorflow_core/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.\n",
"INFO:tensorflow:Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0503 22:44:31.978471 140215059400576 dataset_builder.py:163] Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0503 22:44:31.979441 140215059400576 dataset_builder.py:80] Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Number of filenames to read: 10\n",
"I0503 22:44:31.979539 140215059400576 dataset_builder.py:81] Number of filenames to read: 10\n",
"WARNING:tensorflow:num_readers has been reduced to 10 to match input file shards.\n",
"W0503 22:44:31.979594 140215059400576 dataset_builder.py:88] num_readers has been reduced to 10 to match input file shards.\n",
"WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/object_detection/builders/dataset_builder.py:105: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_determinstic`.\n",
"W0503 22:44:31.984447 140215059400576 deprecation.py:323] From /usr/local/lib/python3.7/dist-packages/object_detection/builders/dataset_builder.py:105: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_determinstic`.\n",
"WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/object_detection/builders/dataset_builder.py:237: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"Use `tf.data.Dataset.map()\n",
"W0503 22:44:32.002814 140215059400576 deprecation.py:323] From /usr/local/lib/python3.7/dist-packages/object_detection/builders/dataset_builder.py:237: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"Use `tf.data.Dataset.map()\n",
"WARNING:tensorflow:Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f85c7027dd0>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"W0503 22:44:32.031176 140215059400576 ag_logging.py:146] Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f85c7027dd0>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"WARNING:tensorflow:Entity <function train_input.<locals>.transform_and_pad_input_data_fn at 0x7f85c7013e60> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"W0503 22:44:32.213216 140215059400576 ag_logging.py:146] Entity <function train_input.<locals>.transform_and_pad_input_data_fn at 0x7f85c7013e60> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/object_detection/inputs.py:111: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
"W0503 22:44:32.214441 140215059400576 deprecation.py:323] From /usr/local/lib/python3.7/dist-packages/object_detection/inputs.py:111: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
"WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/object_detection/inputs.py:97: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead.\n",
"W0503 22:44:32.223712 140215059400576 deprecation.py:323] From /usr/local/lib/python3.7/dist-packages/object_detection/inputs.py:97: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead.\n",
"WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/object_detection/core/preprocessor.py:199: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"`seed2` arg is deprecated.Use sample_distorted_bounding_box_v2 instead.\n",
"W0503 22:44:32.318740 140215059400576 deprecation.py:323] From /usr/local/lib/python3.7/dist-packages/object_detection/core/preprocessor.py:199: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"`seed2` arg is deprecated.Use sample_distorted_bounding_box_v2 instead.\n",
"WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/object_detection/inputs.py:284: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"Use `tf.cast` instead.\n",
"W0503 22:44:32.987949 140215059400576 deprecation.py:323] From /usr/local/lib/python3.7/dist-packages/object_detection/inputs.py:284: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"Use `tf.cast` instead.\n",
"INFO:tensorflow:Calling model_fn.\n",
"I0503 22:44:33.405897 140215059400576 estimator.py:1148] Calling model_fn.\n",
"WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tf_slim/layers/layers.py:1089: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"Please use `layer.__call__` method instead.\n",
"W0503 22:44:33.543333 140215059400576 deprecation.py:323] From /usr/local/lib/python3.7/dist-packages/tf_slim/layers/layers.py:1089: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"Please use `layer.__call__` method instead.\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 22:44:35.805704 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 22:44:35.882828 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 22:44:35.962100 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 22:44:36.040409 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 22:44:36.118508 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 22:44:36.195902 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:fine tuning parameter: fine_tune_checkpoint_type detection, from_detection_checkpoint False fine_tune_checkpoint /content/data/ssd_mobilenet_v3_large_coco_2020_01_14/model.ckpt load_all_detection_checkpoint_vars True\n",
"I0503 22:44:36.389824 140215059400576 model_lib.py:517] fine tuning parameter: fine_tune_checkpoint_type detection, from_detection_checkpoint False fine_tune_checkpoint /content/data/ssd_mobilenet_v3_large_coco_2020_01_14/model.ckpt load_all_detection_checkpoint_vars True\n",
"W0503 22:44:36.397223 140215059400576 variables_helper.py:153] Variable [BoxPredictor_0/ClassPredictor/biases] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[273]], model variable shape: [[114]]. This variable will not be initialized from the checkpoint.\n",
"W0503 22:44:36.397351 140215059400576 variables_helper.py:153] Variable [BoxPredictor_0/ClassPredictor/weights] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[1, 1, 672, 273]], model variable shape: [[1, 1, 672, 114]]. This variable will not be initialized from the checkpoint.\n",
"W0503 22:44:36.397506 140215059400576 variables_helper.py:153] Variable [BoxPredictor_1/ClassPredictor/biases] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[546]], model variable shape: [[228]]. This variable will not be initialized from the checkpoint.\n",
"W0503 22:44:36.397562 140215059400576 variables_helper.py:153] Variable [BoxPredictor_1/ClassPredictor/weights] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[1, 1, 480, 546]], model variable shape: [[1, 1, 480, 228]]. This variable will not be initialized from the checkpoint.\n",
"W0503 22:44:36.397706 140215059400576 variables_helper.py:153] Variable [BoxPredictor_2/ClassPredictor/biases] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[546]], model variable shape: [[228]]. This variable will not be initialized from the checkpoint.\n",
"W0503 22:44:36.397794 140215059400576 variables_helper.py:153] Variable [BoxPredictor_2/ClassPredictor/weights] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[1, 1, 512, 546]], model variable shape: [[1, 1, 512, 228]]. This variable will not be initialized from the checkpoint.\n",
"W0503 22:44:36.397950 140215059400576 variables_helper.py:153] Variable [BoxPredictor_3/ClassPredictor/biases] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[546]], model variable shape: [[228]]. This variable will not be initialized from the checkpoint.\n",
"W0503 22:44:36.398005 140215059400576 variables_helper.py:153] Variable [BoxPredictor_3/ClassPredictor/weights] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[1, 1, 256, 546]], model variable shape: [[1, 1, 256, 228]]. This variable will not be initialized from the checkpoint.\n",
"W0503 22:44:36.398206 140215059400576 variables_helper.py:153] Variable [BoxPredictor_4/ClassPredictor/biases] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[546]], model variable shape: [[228]]. This variable will not be initialized from the checkpoint.\n",
"W0503 22:44:36.398305 140215059400576 variables_helper.py:153] Variable [BoxPredictor_4/ClassPredictor/weights] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[1, 1, 256, 546]], model variable shape: [[1, 1, 256, 228]]. This variable will not be initialized from the checkpoint.\n",
"W0503 22:44:36.398443 140215059400576 variables_helper.py:153] Variable [BoxPredictor_5/ClassPredictor/biases] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[546]], model variable shape: [[228]]. This variable will not be initialized from the checkpoint.\n",
"W0503 22:44:36.398492 140215059400576 variables_helper.py:153] Variable [BoxPredictor_5/ClassPredictor/weights] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[1, 1, 128, 546]], model variable shape: [[1, 1, 128, 228]]. This variable will not be initialized from the checkpoint.\n",
"W0503 22:44:36.400852 140215059400576 variables_helper.py:156] Variable [global_step] is not available in checkpoint\n",
"INFO:tensorflow:Done calling model_fn.\n",
"I0503 22:44:57.526339 140215059400576 estimator.py:1150] Done calling model_fn.\n",
"INFO:tensorflow:Create CheckpointSaverHook.\n",
"I0503 22:44:57.527533 140215059400576 basic_session_run_hooks.py:541] Create CheckpointSaverHook.\n",
"INFO:tensorflow:Graph was finalized.\n",
"I0503 22:45:02.287679 140215059400576 monitored_session.py:240] Graph was finalized.\n",
"2021-05-03 22:45:02.288169: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX512F\n",
"2021-05-03 22:45:02.299423: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2000179999 Hz\n",
"2021-05-03 22:45:02.299789: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x555b93d53800 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n",
"2021-05-03 22:45:02.299819: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version\n",
"2021-05-03 22:45:02.304706: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1\n",
"2021-05-03 22:45:02.464597: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 22:45:02.465331: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x555b93d53480 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:\n",
"2021-05-03 22:45:02.465366: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0\n",
"2021-05-03 22:45:02.466466: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 22:45:02.467000: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: \n",
"name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285\n",
"pciBusID: 0000:00:04.0\n",
"2021-05-03 22:45:02.481921: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
"2021-05-03 22:45:02.649101: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n",
"2021-05-03 22:45:02.748976: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10\n",
"2021-05-03 22:45:02.786580: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10\n",
"2021-05-03 22:45:02.973976: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10\n",
"2021-05-03 22:45:02.992988: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10\n",
"2021-05-03 22:45:03.339491: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
"2021-05-03 22:45:03.339677: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 22:45:03.340388: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 22:45:03.340903: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0\n",
"2021-05-03 22:45:03.344743: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
"2021-05-03 22:45:03.346960: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
"2021-05-03 22:45:03.346999: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 \n",
"2021-05-03 22:45:03.347007: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N \n",
"2021-05-03 22:45:03.347882: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 22:45:03.348510: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 22:45:03.349029: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
"2021-05-03 22:45:03.349062: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15224 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0)\n",
"INFO:tensorflow:Running local_init_op.\n",
"I0503 22:45:10.319791 140215059400576 session_manager.py:500] Running local_init_op.\n",
"INFO:tensorflow:Done running local_init_op.\n",
"I0503 22:45:10.910343 140215059400576 session_manager.py:502] Done running local_init_op.\n",
"INFO:tensorflow:Saving checkpoints for 0 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"I0503 22:45:25.765369 140215059400576 basic_session_run_hooks.py:606] Saving checkpoints for 0 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"2021-05-03 22:45:50.528150: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n",
"Corrupt JPEG data: premature end of data segment\n",
"2021-05-03 22:45:57.377577: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
"INFO:tensorflow:loss = 535.09656, step = 0\n",
"I0503 22:46:02.159381 140215059400576 basic_session_run_hooks.py:262] loss = 535.09656, step = 0\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.654843\n",
"I0503 22:48:34.867180 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.654843\n",
"INFO:tensorflow:loss = 1.5590442, step = 100 (152.709 sec)\n",
"I0503 22:48:34.868344 140215059400576 basic_session_run_hooks.py:260] loss = 1.5590442, step = 100 (152.709 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.740994\n",
"I0503 22:50:49.821016 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.740994\n",
"INFO:tensorflow:loss = 1.3520994, step = 200 (134.954 sec)\n",
"I0503 22:50:49.822021 140215059400576 basic_session_run_hooks.py:260] loss = 1.3520994, step = 200 (134.954 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.734331\n",
"I0503 22:53:05.999416 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.734331\n",
"INFO:tensorflow:loss = 1.2393731, step = 300 (136.178 sec)\n",
"I0503 22:53:06.000502 140215059400576 basic_session_run_hooks.py:260] loss = 1.2393731, step = 300 (136.178 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.737303\n",
"I0503 22:55:21.628954 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.737303\n",
"INFO:tensorflow:loss = 1.1287594, step = 400 (135.629 sec)\n",
"I0503 22:55:21.629845 140215059400576 basic_session_run_hooks.py:260] loss = 1.1287594, step = 400 (135.629 sec)\n",
"INFO:tensorflow:Saving checkpoints for 407 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"I0503 22:55:29.701190 140215059400576 basic_session_run_hooks.py:606] Saving checkpoints for 407 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"INFO:tensorflow:Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0503 22:55:31.790008 140215059400576 dataset_builder.py:163] Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0503 22:55:31.790931 140215059400576 dataset_builder.py:80] Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Number of filenames to read: 10\n",
"I0503 22:55:31.791066 140215059400576 dataset_builder.py:81] Number of filenames to read: 10\n",
"WARNING:tensorflow:Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f85ac22f090>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"W0503 22:55:31.832900 140215059400576 ag_logging.py:146] Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f85ac22f090>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"WARNING:tensorflow:Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7f85ba63f710> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"W0503 22:55:31.993654 140215059400576 ag_logging.py:146] Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7f85ba63f710> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"INFO:tensorflow:Calling model_fn.\n",
"I0503 22:55:32.463019 140215059400576 estimator.py:1148] Calling model_fn.\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 22:55:34.454580 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 22:55:34.524233 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 22:55:34.592488 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 22:55:34.662521 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 22:55:34.730876 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 22:55:34.799815 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/object_detection/eval_util.py:929: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"Use `tf.cast` instead.\n",
"W0503 22:55:35.840647 140215059400576 deprecation.py:323] From /usr/local/lib/python3.7/dist-packages/object_detection/eval_util.py:929: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"Use `tf.cast` instead.\n",
"WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/object_detection/utils/visualization_utils.py:618: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"tf.py_func is deprecated in TF V2. Instead, there are two\n",
" options available in V2.\n",
" - tf.py_function takes a python function which manipulates tf eager\n",
" tensors instead of numpy arrays. It's easy to convert a tf eager tensor to\n",
" an ndarray (just call tensor.numpy()) but having access to eager tensors\n",
" means `tf.py_function`s can use accelerators such as GPUs as well as\n",
" being differentiable using a gradient tape.\n",
" - tf.numpy_function maintains the semantics of the deprecated tf.py_func\n",
" (it is not differentiable, and manipulates numpy arrays). It drops the\n",
" stateful argument making all functions stateful.\n",
" \n",
"W0503 22:55:36.018670 140215059400576 deprecation.py:323] From /usr/local/lib/python3.7/dist-packages/object_detection/utils/visualization_utils.py:618: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"tf.py_func is deprecated in TF V2. Instead, there are two\n",
" options available in V2.\n",
" - tf.py_function takes a python function which manipulates tf eager\n",
" tensors instead of numpy arrays. It's easy to convert a tf eager tensor to\n",
" an ndarray (just call tensor.numpy()) but having access to eager tensors\n",
" means `tf.py_function`s can use accelerators such as GPUs as well as\n",
" being differentiable using a gradient tape.\n",
" - tf.numpy_function maintains the semantics of the deprecated tf.py_func\n",
" (it is not differentiable, and manipulates numpy arrays). It drops the\n",
" stateful argument making all functions stateful.\n",
" \n",
"INFO:tensorflow:Done calling model_fn.\n",
"I0503 22:55:36.534003 140215059400576 estimator.py:1150] Done calling model_fn.\n",
"INFO:tensorflow:Starting evaluation at 2021-05-03T22:55:36Z\n",
"I0503 22:55:36.548862 140215059400576 evaluation.py:255] Starting evaluation at 2021-05-03T22:55:36Z\n",
"INFO:tensorflow:Graph was finalized.\n",
"I0503 22:55:37.000953 140215059400576 monitored_session.py:240] Graph was finalized.\n",
"2021-05-03 22:55:37.002135: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 22:55:37.002428: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: \n",
"name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285\n",
"pciBusID: 0000:00:04.0\n",
"2021-05-03 22:55:37.002548: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
"2021-05-03 22:55:37.002585: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n",
"2021-05-03 22:55:37.002615: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10\n",
"2021-05-03 22:55:37.002646: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10\n",
"2021-05-03 22:55:37.002668: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10\n",
"2021-05-03 22:55:37.002691: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10\n",
"2021-05-03 22:55:37.002714: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
"2021-05-03 22:55:37.002806: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 22:55:37.003114: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 22:55:37.003346: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0\n",
"2021-05-03 22:55:37.003395: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
"2021-05-03 22:55:37.003411: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 \n",
"2021-05-03 22:55:37.003421: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N \n",
"2021-05-03 22:55:37.003530: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 22:55:37.003797: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 22:55:37.004029: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15224 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0)\n",
"INFO:tensorflow:Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-407\n",
"I0503 22:55:37.004954 140215059400576 saver.py:1284] Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-407\n",
"INFO:tensorflow:Running local_init_op.\n",
"I0503 22:55:37.990526 140215059400576 session_manager.py:500] Running local_init_op.\n",
"INFO:tensorflow:Done running local_init_op.\n",
"I0503 22:55:38.125984 140215059400576 session_manager.py:502] Done running local_init_op.\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:Performing evaluation on 2568 images.\n",
"I0503 22:56:40.951148 140210432521984 coco_evaluation.py:293] Performing evaluation on 2568 images.\n",
"creating index...\n",
"index created!\n",
"INFO:tensorflow:Loading and preparing annotation results...\n",
"I0503 22:56:40.959702 140210432521984 coco_tools.py:116] Loading and preparing annotation results...\n",
"INFO:tensorflow:DONE (t=0.19s)\n",
"I0503 22:56:41.151035 140210432521984 coco_tools.py:138] DONE (t=0.19s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=19.53s).\n",
"Accumulating evaluation results...\n",
"DONE (t=7.02s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.072\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.096\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.086\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.049\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.080\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.760\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.773\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.773\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.645\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.799\n",
"INFO:tensorflow:Finished evaluation at 2021-05-03-22:57:08\n",
"I0503 22:57:08.682882 140215059400576 evaluation.py:275] Finished evaluation at 2021-05-03-22:57:08\n",
"INFO:tensorflow:Saving dict for global step 407: DetectionBoxes_Precision/mAP = 0.07237052, DetectionBoxes_Precision/mAP (large) = 0.07999087, DetectionBoxes_Precision/mAP (medium) = 0.049325153, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.09640334, DetectionBoxes_Precision/mAP@.75IOU = 0.08647537, DetectionBoxes_Recall/AR@1 = 0.75975764, DetectionBoxes_Recall/AR@10 = 0.77336717, DetectionBoxes_Recall/AR@100 = 0.77336717, DetectionBoxes_Recall/AR@100 (large) = 0.7987, DetectionBoxes_Recall/AR@100 (medium) = 0.6453445, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/classification_loss = 0.8944688, Loss/localization_loss = 0.067470625, Loss/regularization_loss = 0.2306503, Loss/total_loss = 1.1925858, global_step = 407, learning_rate = 0.018759735, loss = 1.1925858\n",
"I0503 22:57:08.683235 140215059400576 estimator.py:2049] Saving dict for global step 407: DetectionBoxes_Precision/mAP = 0.07237052, DetectionBoxes_Precision/mAP (large) = 0.07999087, DetectionBoxes_Precision/mAP (medium) = 0.049325153, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.09640334, DetectionBoxes_Precision/mAP@.75IOU = 0.08647537, DetectionBoxes_Recall/AR@1 = 0.75975764, DetectionBoxes_Recall/AR@10 = 0.77336717, DetectionBoxes_Recall/AR@100 = 0.77336717, DetectionBoxes_Recall/AR@100 (large) = 0.7987, DetectionBoxes_Recall/AR@100 (medium) = 0.6453445, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/classification_loss = 0.8944688, Loss/localization_loss = 0.067470625, Loss/regularization_loss = 0.2306503, Loss/total_loss = 1.1925858, global_step = 407, learning_rate = 0.018759735, loss = 1.1925858\n",
"INFO:tensorflow:Saving 'checkpoint_path' summary for global step 407: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-407\n",
"I0503 22:57:09.511198 140215059400576 estimator.py:2109] Saving 'checkpoint_path' summary for global step 407: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-407\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.42602\n",
"I0503 22:59:16.359457 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.42602\n",
"INFO:tensorflow:loss = 1.1076175, step = 500 (234.731 sec)\n",
"I0503 22:59:16.360794 140215059400576 basic_session_run_hooks.py:260] loss = 1.1076175, step = 500 (234.731 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.741202\n",
"I0503 23:01:31.275370 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.741202\n",
"INFO:tensorflow:loss = 1.0399542, step = 600 (134.916 sec)\n",
"I0503 23:01:31.276308 140215059400576 basic_session_run_hooks.py:260] loss = 1.0399542, step = 600 (134.916 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.738895\n",
"I0503 23:03:46.612538 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.738895\n",
"INFO:tensorflow:loss = 0.92504525, step = 700 (135.337 sec)\n",
"I0503 23:03:46.613550 140215059400576 basic_session_run_hooks.py:260] loss = 0.92504525, step = 700 (135.337 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:Saving checkpoints for 778 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"I0503 23:05:30.540197 140215059400576 basic_session_run_hooks.py:606] Saving checkpoints for 778 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"INFO:tensorflow:Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0503 23:05:32.588105 140215059400576 dataset_builder.py:163] Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0503 23:05:32.589048 140215059400576 dataset_builder.py:80] Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Number of filenames to read: 10\n",
"I0503 23:05:32.589216 140215059400576 dataset_builder.py:81] Number of filenames to read: 10\n",
"WARNING:tensorflow:Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f85ac180510>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"W0503 23:05:32.626588 140215059400576 ag_logging.py:146] Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f85ac180510>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"WARNING:tensorflow:Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7f805cd06d40> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"W0503 23:05:32.788226 140215059400576 ag_logging.py:146] Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7f805cd06d40> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"INFO:tensorflow:Calling model_fn.\n",
"I0503 23:05:33.322505 140215059400576 estimator.py:1148] Calling model_fn.\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:05:35.335118 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:05:35.403026 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:05:35.469755 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:05:35.536586 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:05:35.605880 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:05:35.673256 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:Done calling model_fn.\n",
"I0503 23:05:37.381840 140215059400576 estimator.py:1150] Done calling model_fn.\n",
"INFO:tensorflow:Starting evaluation at 2021-05-03T23:05:37Z\n",
"I0503 23:05:37.396681 140215059400576 evaluation.py:255] Starting evaluation at 2021-05-03T23:05:37Z\n",
"INFO:tensorflow:Graph was finalized.\n",
"I0503 23:05:37.850116 140215059400576 monitored_session.py:240] Graph was finalized.\n",
"2021-05-03 23:05:37.850790: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:05:37.851180: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: \n",
"name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285\n",
"pciBusID: 0000:00:04.0\n",
"2021-05-03 23:05:37.851323: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
"2021-05-03 23:05:37.851346: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n",
"2021-05-03 23:05:37.851362: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10\n",
"2021-05-03 23:05:37.851377: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10\n",
"2021-05-03 23:05:37.851392: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10\n",
"2021-05-03 23:05:37.851405: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10\n",
"2021-05-03 23:05:37.851423: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
"2021-05-03 23:05:37.851498: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:05:37.851760: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:05:37.851960: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0\n",
"2021-05-03 23:05:37.852001: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
"2021-05-03 23:05:37.852011: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 \n",
"2021-05-03 23:05:37.852018: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N \n",
"2021-05-03 23:05:37.852111: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:05:37.852376: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:05:37.852594: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15224 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0)\n",
"INFO:tensorflow:Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-778\n",
"I0503 23:05:37.853563 140215059400576 saver.py:1284] Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-778\n",
"INFO:tensorflow:Running local_init_op.\n",
"I0503 23:05:38.816956 140215059400576 session_manager.py:500] Running local_init_op.\n",
"INFO:tensorflow:Done running local_init_op.\n",
"I0503 23:05:38.954433 140215059400576 session_manager.py:502] Done running local_init_op.\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:Performing evaluation on 2568 images.\n",
"I0503 23:06:42.640245 140210432521984 coco_evaluation.py:293] Performing evaluation on 2568 images.\n",
"creating index...\n",
"index created!\n",
"INFO:tensorflow:Loading and preparing annotation results...\n",
"I0503 23:06:42.647192 140210432521984 coco_tools.py:116] Loading and preparing annotation results...\n",
"INFO:tensorflow:DONE (t=0.20s)\n",
"I0503 23:06:42.843179 140210432521984 coco_tools.py:138] DONE (t=0.20s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=20.69s).\n",
"Accumulating evaluation results...\n",
"DONE (t=6.94s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.281\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.354\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.339\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.107\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.315\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.804\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.809\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.809\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.700\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.829\n",
"INFO:tensorflow:Finished evaluation at 2021-05-03-23:07:11\n",
"I0503 23:07:11.459625 140215059400576 evaluation.py:275] Finished evaluation at 2021-05-03-23:07:11\n",
"INFO:tensorflow:Saving dict for global step 778: DetectionBoxes_Precision/mAP = 0.2810173, DetectionBoxes_Precision/mAP (large) = 0.3148433, DetectionBoxes_Precision/mAP (medium) = 0.10657664, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.35377043, DetectionBoxes_Precision/mAP@.75IOU = 0.33944526, DetectionBoxes_Recall/AR@1 = 0.80354273, DetectionBoxes_Recall/AR@10 = 0.8086807, DetectionBoxes_Recall/AR@100 = 0.8086807, DetectionBoxes_Recall/AR@100 (large) = 0.8292294, DetectionBoxes_Recall/AR@100 (medium) = 0.69966906, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/classification_loss = 0.63184404, Loss/localization_loss = 0.049853478, Loss/regularization_loss = 0.229921, Loss/total_loss = 0.9116245, global_step = 778, learning_rate = 0.023706462, loss = 0.9116245\n",
"I0503 23:07:11.459895 140215059400576 estimator.py:2049] Saving dict for global step 778: DetectionBoxes_Precision/mAP = 0.2810173, DetectionBoxes_Precision/mAP (large) = 0.3148433, DetectionBoxes_Precision/mAP (medium) = 0.10657664, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.35377043, DetectionBoxes_Precision/mAP@.75IOU = 0.33944526, DetectionBoxes_Recall/AR@1 = 0.80354273, DetectionBoxes_Recall/AR@10 = 0.8086807, DetectionBoxes_Recall/AR@100 = 0.8086807, DetectionBoxes_Recall/AR@100 (large) = 0.8292294, DetectionBoxes_Recall/AR@100 (medium) = 0.69966906, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/classification_loss = 0.63184404, Loss/localization_loss = 0.049853478, Loss/regularization_loss = 0.229921, Loss/total_loss = 0.9116245, global_step = 778, learning_rate = 0.023706462, loss = 0.9116245\n",
"INFO:tensorflow:Saving 'checkpoint_path' summary for global step 778: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-778\n",
"I0503 23:07:11.463823 140215059400576 estimator.py:2109] Saving 'checkpoint_path' summary for global step 778: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-778\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.423704\n",
"I0503 23:07:42.626347 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.423704\n",
"INFO:tensorflow:loss = 0.8872799, step = 800 (236.014 sec)\n",
"I0503 23:07:42.627664 140215059400576 basic_session_run_hooks.py:260] loss = 0.8872799, step = 800 (236.014 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.741439\n",
"I0503 23:09:57.499171 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.741439\n",
"INFO:tensorflow:loss = 0.7907349, step = 900 (134.873 sec)\n",
"I0503 23:09:57.500277 140215059400576 basic_session_run_hooks.py:260] loss = 0.7907349, step = 900 (134.873 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.74041\n",
"I0503 23:12:12.559424 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.74041\n",
"INFO:tensorflow:loss = 0.7546335, step = 1000 (135.060 sec)\n",
"I0503 23:12:12.560322 140215059400576 basic_session_run_hooks.py:260] loss = 0.7546335, step = 1000 (135.060 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.740813\n",
"I0503 23:14:27.546211 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.740813\n",
"INFO:tensorflow:loss = 0.67965263, step = 1100 (134.987 sec)\n",
"I0503 23:14:27.547178 140215059400576 basic_session_run_hooks.py:260] loss = 0.67965263, step = 1100 (134.987 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:Saving checkpoints for 1148 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"I0503 23:15:30.629634 140215059400576 basic_session_run_hooks.py:606] Saving checkpoints for 1148 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"INFO:tensorflow:Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0503 23:15:32.691810 140215059400576 dataset_builder.py:163] Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0503 23:15:32.692809 140215059400576 dataset_builder.py:80] Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Number of filenames to read: 10\n",
"I0503 23:15:32.692945 140215059400576 dataset_builder.py:81] Number of filenames to read: 10\n",
"WARNING:tensorflow:Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f806463d610>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"W0503 23:15:32.730887 140215059400576 ag_logging.py:146] Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f806463d610>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"WARNING:tensorflow:Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7f8064a3a320> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"W0503 23:15:32.891924 140215059400576 ag_logging.py:146] Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7f8064a3a320> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"INFO:tensorflow:Calling model_fn.\n",
"I0503 23:15:33.377007 140215059400576 estimator.py:1148] Calling model_fn.\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:15:35.388021 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:15:35.458101 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:15:35.525646 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:15:35.593148 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:15:35.662152 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:15:35.730339 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:Done calling model_fn.\n",
"I0503 23:15:37.442188 140215059400576 estimator.py:1150] Done calling model_fn.\n",
"INFO:tensorflow:Starting evaluation at 2021-05-03T23:15:37Z\n",
"I0503 23:15:37.457147 140215059400576 evaluation.py:255] Starting evaluation at 2021-05-03T23:15:37Z\n",
"INFO:tensorflow:Graph was finalized.\n",
"I0503 23:15:37.906858 140215059400576 monitored_session.py:240] Graph was finalized.\n",
"2021-05-03 23:15:37.907585: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:15:37.907897: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: \n",
"name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285\n",
"pciBusID: 0000:00:04.0\n",
"2021-05-03 23:15:37.908022: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
"2021-05-03 23:15:37.908077: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n",
"2021-05-03 23:15:37.908126: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10\n",
"2021-05-03 23:15:37.908152: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10\n",
"2021-05-03 23:15:37.908168: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10\n",
"2021-05-03 23:15:37.908182: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10\n",
"2021-05-03 23:15:37.908204: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
"2021-05-03 23:15:37.908287: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:15:37.908553: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:15:37.908759: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0\n",
"2021-05-03 23:15:37.908828: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
"2021-05-03 23:15:37.908840: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 \n",
"2021-05-03 23:15:37.908847: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N \n",
"2021-05-03 23:15:37.908932: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:15:37.909208: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:15:37.909428: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15224 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0)\n",
"INFO:tensorflow:Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-1148\n",
"I0503 23:15:37.910608 140215059400576 saver.py:1284] Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-1148\n",
"INFO:tensorflow:Running local_init_op.\n",
"I0503 23:15:38.892279 140215059400576 session_manager.py:500] Running local_init_op.\n",
"INFO:tensorflow:Done running local_init_op.\n",
"I0503 23:15:39.031018 140215059400576 session_manager.py:502] Done running local_init_op.\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:Performing evaluation on 2568 images.\n",
"I0503 23:16:44.290100 140210424129280 coco_evaluation.py:293] Performing evaluation on 2568 images.\n",
"creating index...\n",
"index created!\n",
"INFO:tensorflow:Loading and preparing annotation results...\n",
"I0503 23:16:44.298785 140210424129280 coco_tools.py:116] Loading and preparing annotation results...\n",
"INFO:tensorflow:DONE (t=0.19s)\n",
"I0503 23:16:44.486017 140210424129280 coco_tools.py:138] DONE (t=0.19s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=19.51s).\n",
"Accumulating evaluation results...\n",
"DONE (t=7.22s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.490\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.601\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.587\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.145\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.547\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.815\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.819\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.819\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.703\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.842\n",
"INFO:tensorflow:Finished evaluation at 2021-05-03-23:17:12\n",
"I0503 23:17:12.198430 140215059400576 evaluation.py:275] Finished evaluation at 2021-05-03-23:17:12\n",
"INFO:tensorflow:Saving dict for global step 1148: DetectionBoxes_Precision/mAP = 0.48982677, DetectionBoxes_Precision/mAP (large) = 0.5474836, DetectionBoxes_Precision/mAP (medium) = 0.14508818, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.6014761, DetectionBoxes_Precision/mAP@.75IOU = 0.58707434, DetectionBoxes_Recall/AR@1 = 0.81457376, DetectionBoxes_Recall/AR@10 = 0.819219, DetectionBoxes_Recall/AR@100 = 0.819219, DetectionBoxes_Recall/AR@100 (large) = 0.8421357, DetectionBoxes_Recall/AR@100 (medium) = 0.7030473, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/classification_loss = 0.4837387, Loss/localization_loss = 0.045580048, Loss/regularization_loss = 0.22921352, Loss/total_loss = 0.7585366, global_step = 1148, learning_rate = 0.028639857, loss = 0.7585366\n",
"I0503 23:17:12.198681 140215059400576 estimator.py:2049] Saving dict for global step 1148: DetectionBoxes_Precision/mAP = 0.48982677, DetectionBoxes_Precision/mAP (large) = 0.5474836, DetectionBoxes_Precision/mAP (medium) = 0.14508818, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.6014761, DetectionBoxes_Precision/mAP@.75IOU = 0.58707434, DetectionBoxes_Recall/AR@1 = 0.81457376, DetectionBoxes_Recall/AR@10 = 0.819219, DetectionBoxes_Recall/AR@100 = 0.819219, DetectionBoxes_Recall/AR@100 (large) = 0.8421357, DetectionBoxes_Recall/AR@100 (medium) = 0.7030473, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/classification_loss = 0.4837387, Loss/localization_loss = 0.045580048, Loss/regularization_loss = 0.22921352, Loss/total_loss = 0.7585366, global_step = 1148, learning_rate = 0.028639857, loss = 0.7585366\n",
"INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1148: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-1148\n",
"I0503 23:17:12.202373 140215059400576 estimator.py:2109] Saving 'checkpoint_path' summary for global step 1148: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-1148\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.422406\n",
"I0503 23:18:24.285328 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.422406\n",
"INFO:tensorflow:loss = 0.6914617, step = 1200 (236.739 sec)\n",
"I0503 23:18:24.286469 140215059400576 basic_session_run_hooks.py:260] loss = 0.6914617, step = 1200 (236.739 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.739784\n",
"I0503 23:20:39.459912 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.739784\n",
"INFO:tensorflow:loss = 0.6413725, step = 1300 (135.175 sec)\n",
"I0503 23:20:39.461007 140215059400576 basic_session_run_hooks.py:260] loss = 0.6413725, step = 1300 (135.175 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.741462\n",
"I0503 23:22:54.328642 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.741462\n",
"INFO:tensorflow:loss = 0.64289474, step = 1400 (134.869 sec)\n",
"I0503 23:22:54.329862 140215059400576 basic_session_run_hooks.py:260] loss = 0.64289474, step = 1400 (134.869 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.741569\n",
"I0503 23:25:09.177831 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.741569\n",
"INFO:tensorflow:loss = 0.6105086, step = 1500 (134.849 sec)\n",
"I0503 23:25:09.178897 140215059400576 basic_session_run_hooks.py:260] loss = 0.6105086, step = 1500 (134.849 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:Saving checkpoints for 1517 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"I0503 23:25:30.690894 140215059400576 basic_session_run_hooks.py:606] Saving checkpoints for 1517 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"INFO:tensorflow:Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0503 23:25:32.766442 140215059400576 dataset_builder.py:163] Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0503 23:25:32.767466 140215059400576 dataset_builder.py:80] Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Number of filenames to read: 10\n",
"I0503 23:25:32.767613 140215059400576 dataset_builder.py:81] Number of filenames to read: 10\n",
"WARNING:tensorflow:Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f852809bad0>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"W0503 23:25:32.805515 140215059400576 ag_logging.py:146] Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f852809bad0>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"WARNING:tensorflow:Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7f85ac1f1d40> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"W0503 23:25:32.967390 140215059400576 ag_logging.py:146] Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7f85ac1f1d40> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"INFO:tensorflow:Calling model_fn.\n",
"I0503 23:25:33.443732 140215059400576 estimator.py:1148] Calling model_fn.\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:25:35.480487 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:25:35.550337 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:25:35.618407 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:25:35.687046 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:25:35.756820 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:25:35.826197 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:Done calling model_fn.\n",
"I0503 23:25:37.560714 140215059400576 estimator.py:1150] Done calling model_fn.\n",
"INFO:tensorflow:Starting evaluation at 2021-05-03T23:25:37Z\n",
"I0503 23:25:37.575657 140215059400576 evaluation.py:255] Starting evaluation at 2021-05-03T23:25:37Z\n",
"INFO:tensorflow:Graph was finalized.\n",
"I0503 23:25:38.028902 140215059400576 monitored_session.py:240] Graph was finalized.\n",
"2021-05-03 23:25:38.029615: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:25:38.029984: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: \n",
"name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285\n",
"pciBusID: 0000:00:04.0\n",
"2021-05-03 23:25:38.030103: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
"2021-05-03 23:25:38.030129: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n",
"2021-05-03 23:25:38.030146: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10\n",
"2021-05-03 23:25:38.030161: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10\n",
"2021-05-03 23:25:38.030174: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10\n",
"2021-05-03 23:25:38.030188: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10\n",
"2021-05-03 23:25:38.030202: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
"2021-05-03 23:25:38.030274: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:25:38.030521: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:25:38.030711: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0\n",
"2021-05-03 23:25:38.030750: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
"2021-05-03 23:25:38.030769: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 \n",
"2021-05-03 23:25:38.030775: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N \n",
"2021-05-03 23:25:38.030852: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:25:38.031104: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:25:38.031307: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15224 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0)\n",
"INFO:tensorflow:Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-1517\n",
"I0503 23:25:38.032435 140215059400576 saver.py:1284] Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-1517\n",
"INFO:tensorflow:Running local_init_op.\n",
"I0503 23:25:39.018480 140215059400576 session_manager.py:500] Running local_init_op.\n",
"INFO:tensorflow:Done running local_init_op.\n",
"I0503 23:25:39.154753 140215059400576 session_manager.py:502] Done running local_init_op.\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:Performing evaluation on 2568 images.\n",
"I0503 23:26:44.479240 140210432521984 coco_evaluation.py:293] Performing evaluation on 2568 images.\n",
"creating index...\n",
"index created!\n",
"INFO:tensorflow:Loading and preparing annotation results...\n",
"I0503 23:26:44.485937 140210432521984 coco_tools.py:116] Loading and preparing annotation results...\n",
"INFO:tensorflow:DONE (t=0.18s)\n",
"I0503 23:26:44.670150 140210432521984 coco_tools.py:138] DONE (t=0.18s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=19.66s).\n",
"Accumulating evaluation results...\n",
"DONE (t=6.90s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.632\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.796\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.756\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.195\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.703\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.799\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.810\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.810\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.684\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.835\n",
"INFO:tensorflow:Finished evaluation at 2021-05-03-23:27:12\n",
"I0503 23:27:12.192707 140215059400576 evaluation.py:275] Finished evaluation at 2021-05-03-23:27:12\n",
"INFO:tensorflow:Saving dict for global step 1517: DetectionBoxes_Precision/mAP = 0.6323059, DetectionBoxes_Precision/mAP (large) = 0.70349455, DetectionBoxes_Precision/mAP (medium) = 0.19478361, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.79594624, DetectionBoxes_Precision/mAP@.75IOU = 0.7563676, DetectionBoxes_Recall/AR@1 = 0.7993223, DetectionBoxes_Recall/AR@10 = 0.80954814, DetectionBoxes_Recall/AR@100 = 0.80954814, DetectionBoxes_Recall/AR@100 (large) = 0.83493155, DetectionBoxes_Recall/AR@100 (medium) = 0.6838918, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/classification_loss = 0.39249372, Loss/localization_loss = 0.051634457, Loss/regularization_loss = 0.22847962, Loss/total_loss = 0.6726066, global_step = 1517, learning_rate = 0.03355992, loss = 0.6726066\n",
"I0503 23:27:12.192950 140215059400576 estimator.py:2049] Saving dict for global step 1517: DetectionBoxes_Precision/mAP = 0.6323059, DetectionBoxes_Precision/mAP (large) = 0.70349455, DetectionBoxes_Precision/mAP (medium) = 0.19478361, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.79594624, DetectionBoxes_Precision/mAP@.75IOU = 0.7563676, DetectionBoxes_Recall/AR@1 = 0.7993223, DetectionBoxes_Recall/AR@10 = 0.80954814, DetectionBoxes_Recall/AR@100 = 0.80954814, DetectionBoxes_Recall/AR@100 (large) = 0.83493155, DetectionBoxes_Recall/AR@100 (medium) = 0.6838918, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/classification_loss = 0.39249372, Loss/localization_loss = 0.051634457, Loss/regularization_loss = 0.22847962, Loss/total_loss = 0.6726066, global_step = 1517, learning_rate = 0.03355992, loss = 0.6726066\n",
"INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1517: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-1517\n",
"I0503 23:27:12.196601 140215059400576 estimator.py:2109] Saving 'checkpoint_path' summary for global step 1517: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-1517\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.422346\n",
"I0503 23:29:05.950512 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.422346\n",
"INFO:tensorflow:loss = 0.5302361, step = 1600 (236.772 sec)\n",
"I0503 23:29:05.951356 140215059400576 basic_session_run_hooks.py:260] loss = 0.5302361, step = 1600 (236.772 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.741342\n",
"I0503 23:31:20.840980 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.741342\n",
"INFO:tensorflow:loss = 0.5171362, step = 1700 (134.891 sec)\n",
"I0503 23:31:20.841886 140215059400576 basic_session_run_hooks.py:260] loss = 0.5171362, step = 1700 (134.891 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.740093\n",
"I0503 23:33:35.959067 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.740093\n",
"INFO:tensorflow:loss = 0.50612265, step = 1800 (135.118 sec)\n",
"I0503 23:33:35.960165 140215059400576 basic_session_run_hooks.py:260] loss = 0.50612265, step = 1800 (135.118 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:Saving checkpoints for 1887 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"I0503 23:35:31.814470 140215059400576 basic_session_run_hooks.py:606] Saving checkpoints for 1887 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"WARNING:tensorflow:From /tensorflow-1.15.2/python3.7/tensorflow_core/python/training/saver.py:963: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"Use standard file APIs to delete files with this prefix.\n",
"W0503 23:35:31.851176 140215059400576 deprecation.py:323] From /tensorflow-1.15.2/python3.7/tensorflow_core/python/training/saver.py:963: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"Use standard file APIs to delete files with this prefix.\n",
"INFO:tensorflow:Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0503 23:35:33.899321 140215059400576 dataset_builder.py:163] Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0503 23:35:33.900343 140215059400576 dataset_builder.py:80] Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Number of filenames to read: 10\n",
"I0503 23:35:33.900473 140215059400576 dataset_builder.py:81] Number of filenames to read: 10\n",
"WARNING:tensorflow:Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f8061e9df10>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"W0503 23:35:33.938002 140215059400576 ag_logging.py:146] Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f8061e9df10>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"WARNING:tensorflow:Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7f85a0369830> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"W0503 23:35:34.098869 140215059400576 ag_logging.py:146] Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7f85a0369830> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"INFO:tensorflow:Calling model_fn.\n",
"I0503 23:35:34.580175 140215059400576 estimator.py:1148] Calling model_fn.\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:35:36.596133 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:35:36.664330 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:35:36.732290 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:35:36.799439 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:35:36.868535 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:35:36.937592 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:Done calling model_fn.\n",
"I0503 23:35:38.644671 140215059400576 estimator.py:1150] Done calling model_fn.\n",
"INFO:tensorflow:Starting evaluation at 2021-05-03T23:35:38Z\n",
"I0503 23:35:38.659392 140215059400576 evaluation.py:255] Starting evaluation at 2021-05-03T23:35:38Z\n",
"INFO:tensorflow:Graph was finalized.\n",
"I0503 23:35:39.110249 140215059400576 monitored_session.py:240] Graph was finalized.\n",
"2021-05-03 23:35:39.110821: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:35:39.111126: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: \n",
"name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285\n",
"pciBusID: 0000:00:04.0\n",
"2021-05-03 23:35:39.111218: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
"2021-05-03 23:35:39.111240: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n",
"2021-05-03 23:35:39.111255: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10\n",
"2021-05-03 23:35:39.111273: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10\n",
"2021-05-03 23:35:39.111313: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10\n",
"2021-05-03 23:35:39.111326: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10\n",
"2021-05-03 23:35:39.111341: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
"2021-05-03 23:35:39.111410: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:35:39.111657: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:35:39.111844: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0\n",
"2021-05-03 23:35:39.111918: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
"2021-05-03 23:35:39.111930: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 \n",
"2021-05-03 23:35:39.111936: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N \n",
"2021-05-03 23:35:39.112023: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:35:39.112268: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:35:39.112467: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15224 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0)\n",
"INFO:tensorflow:Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-1887\n",
"I0503 23:35:39.113397 140215059400576 saver.py:1284] Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-1887\n",
"INFO:tensorflow:Running local_init_op.\n",
"I0503 23:35:40.097286 140215059400576 session_manager.py:500] Running local_init_op.\n",
"INFO:tensorflow:Done running local_init_op.\n",
"I0503 23:35:40.235534 140215059400576 session_manager.py:502] Done running local_init_op.\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:Performing evaluation on 2568 images.\n",
"I0503 23:36:45.215229 140210415736576 coco_evaluation.py:293] Performing evaluation on 2568 images.\n",
"creating index...\n",
"index created!\n",
"INFO:tensorflow:Loading and preparing annotation results...\n",
"I0503 23:36:45.220566 140210415736576 coco_tools.py:116] Loading and preparing annotation results...\n",
"INFO:tensorflow:DONE (t=0.19s)\n",
"I0503 23:36:45.412241 140210415736576 coco_tools.py:138] DONE (t=0.19s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=19.16s).\n",
"Accumulating evaluation results...\n",
"DONE (t=7.61s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.672\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.826\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.799\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.189\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.747\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.822\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.829\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.829\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.714\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.852\n",
"INFO:tensorflow:Finished evaluation at 2021-05-03-23:37:13\n",
"I0503 23:37:13.162960 140215059400576 evaluation.py:275] Finished evaluation at 2021-05-03-23:37:13\n",
"INFO:tensorflow:Saving dict for global step 1887: DetectionBoxes_Precision/mAP = 0.67242706, DetectionBoxes_Precision/mAP (large) = 0.74694633, DetectionBoxes_Precision/mAP (medium) = 0.18869188, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.82617134, DetectionBoxes_Precision/mAP@.75IOU = 0.79909503, DetectionBoxes_Recall/AR@1 = 0.82155824, DetectionBoxes_Recall/AR@10 = 0.82916605, DetectionBoxes_Recall/AR@100 = 0.8292433, DetectionBoxes_Recall/AR@100 (large) = 0.85227907, DetectionBoxes_Recall/AR@100 (medium) = 0.7142042, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/classification_loss = 0.35495389, Loss/localization_loss = 0.044486213, Loss/regularization_loss = 0.22757879, Loss/total_loss = 0.62702036, global_step = 1887, learning_rate = 0.038493313, loss = 0.62702036\n",
"I0503 23:37:13.163253 140215059400576 estimator.py:2049] Saving dict for global step 1887: DetectionBoxes_Precision/mAP = 0.67242706, DetectionBoxes_Precision/mAP (large) = 0.74694633, DetectionBoxes_Precision/mAP (medium) = 0.18869188, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.82617134, DetectionBoxes_Precision/mAP@.75IOU = 0.79909503, DetectionBoxes_Recall/AR@1 = 0.82155824, DetectionBoxes_Recall/AR@10 = 0.82916605, DetectionBoxes_Recall/AR@100 = 0.8292433, DetectionBoxes_Recall/AR@100 (large) = 0.85227907, DetectionBoxes_Recall/AR@100 (medium) = 0.7142042, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/classification_loss = 0.35495389, Loss/localization_loss = 0.044486213, Loss/regularization_loss = 0.22757879, Loss/total_loss = 0.62702036, global_step = 1887, learning_rate = 0.038493313, loss = 0.62702036\n",
"INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1887: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-1887\n",
"I0503 23:37:13.166974 140215059400576 estimator.py:2109] Saving 'checkpoint_path' summary for global step 1887: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-1887\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.42351\n",
"I0503 23:37:32.081185 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.42351\n",
"INFO:tensorflow:loss = 0.5166437, step = 1900 (236.122 sec)\n",
"I0503 23:37:32.082311 140215059400576 basic_session_run_hooks.py:260] loss = 0.5166437, step = 1900 (236.122 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.739704\n",
"I0503 23:39:47.270425 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.739704\n",
"INFO:tensorflow:loss = 0.50334144, step = 2000 (135.189 sec)\n",
"I0503 23:39:47.271392 140215059400576 basic_session_run_hooks.py:260] loss = 0.50334144, step = 2000 (135.189 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.738424\n",
"I0503 23:42:02.694048 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.738424\n",
"INFO:tensorflow:loss = 0.50477946, step = 2100 (135.424 sec)\n",
"I0503 23:42:02.695218 140215059400576 basic_session_run_hooks.py:260] loss = 0.50477946, step = 2100 (135.424 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.740113\n",
"I0503 23:44:17.808477 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.740113\n",
"INFO:tensorflow:loss = 0.47339472, step = 2200 (135.114 sec)\n",
"I0503 23:44:17.809370 140215059400576 basic_session_run_hooks.py:260] loss = 0.47339472, step = 2200 (135.114 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:Saving checkpoints for 2256 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"I0503 23:45:32.084156 140215059400576 basic_session_run_hooks.py:606] Saving checkpoints for 2256 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"INFO:tensorflow:Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0503 23:45:34.181569 140215059400576 dataset_builder.py:163] Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0503 23:45:34.182606 140215059400576 dataset_builder.py:80] Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Number of filenames to read: 10\n",
"I0503 23:45:34.182740 140215059400576 dataset_builder.py:81] Number of filenames to read: 10\n",
"WARNING:tensorflow:Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f8064e651d0>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"W0503 23:45:34.220583 140215059400576 ag_logging.py:146] Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f8064e651d0>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"WARNING:tensorflow:Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7f8063b79200> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"W0503 23:45:34.381741 140215059400576 ag_logging.py:146] Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7f8063b79200> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"INFO:tensorflow:Calling model_fn.\n",
"I0503 23:45:34.859106 140215059400576 estimator.py:1148] Calling model_fn.\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:45:36.859054 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:45:36.928016 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:45:36.996630 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:45:37.065469 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:45:37.133673 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:45:37.203128 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:Done calling model_fn.\n",
"I0503 23:45:38.929836 140215059400576 estimator.py:1150] Done calling model_fn.\n",
"INFO:tensorflow:Starting evaluation at 2021-05-03T23:45:38Z\n",
"I0503 23:45:38.944777 140215059400576 evaluation.py:255] Starting evaluation at 2021-05-03T23:45:38Z\n",
"INFO:tensorflow:Graph was finalized.\n",
"I0503 23:45:39.392005 140215059400576 monitored_session.py:240] Graph was finalized.\n",
"2021-05-03 23:45:39.392753: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:45:39.393127: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: \n",
"name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285\n",
"pciBusID: 0000:00:04.0\n",
"2021-05-03 23:45:39.393243: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
"2021-05-03 23:45:39.393275: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n",
"2021-05-03 23:45:39.393300: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10\n",
"2021-05-03 23:45:39.393325: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10\n",
"2021-05-03 23:45:39.393353: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10\n",
"2021-05-03 23:45:39.393376: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10\n",
"2021-05-03 23:45:39.393400: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
"2021-05-03 23:45:39.393493: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:45:39.393766: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:45:39.393977: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0\n",
"2021-05-03 23:45:39.394021: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
"2021-05-03 23:45:39.394036: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 \n",
"2021-05-03 23:45:39.394046: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N \n",
"2021-05-03 23:45:39.394161: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:45:39.394432: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:45:39.394647: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15224 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0)\n",
"INFO:tensorflow:Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-2256\n",
"I0503 23:45:39.395557 140215059400576 saver.py:1284] Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-2256\n",
"INFO:tensorflow:Running local_init_op.\n",
"I0503 23:45:40.371191 140215059400576 session_manager.py:500] Running local_init_op.\n",
"INFO:tensorflow:Done running local_init_op.\n",
"I0503 23:45:40.509582 140215059400576 session_manager.py:502] Done running local_init_op.\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:Performing evaluation on 2568 images.\n",
"I0503 23:46:45.338707 140210432521984 coco_evaluation.py:293] Performing evaluation on 2568 images.\n",
"creating index...\n",
"index created!\n",
"INFO:tensorflow:Loading and preparing annotation results...\n",
"I0503 23:46:45.344563 140210432521984 coco_tools.py:116] Loading and preparing annotation results...\n",
"INFO:tensorflow:DONE (t=0.19s)\n",
"I0503 23:46:45.535475 140210432521984 coco_tools.py:138] DONE (t=0.19s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=19.42s).\n",
"Accumulating evaluation results...\n",
"DONE (t=6.71s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.720\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.864\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.841\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.241\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.788\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.840\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.845\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.845\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.713\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.870\n",
"INFO:tensorflow:Finished evaluation at 2021-05-03-23:47:12\n",
"I0503 23:47:12.657825 140215059400576 evaluation.py:275] Finished evaluation at 2021-05-03-23:47:12\n",
"INFO:tensorflow:Saving dict for global step 2256: DetectionBoxes_Precision/mAP = 0.7203577, DetectionBoxes_Precision/mAP (large) = 0.78831357, DetectionBoxes_Precision/mAP (medium) = 0.2406104, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.86428666, DetectionBoxes_Precision/mAP@.75IOU = 0.84071743, DetectionBoxes_Recall/AR@1 = 0.83993936, DetectionBoxes_Recall/AR@10 = 0.8446778, DetectionBoxes_Recall/AR@100 = 0.8446778, DetectionBoxes_Recall/AR@100 (large) = 0.87, DetectionBoxes_Recall/AR@100 (medium) = 0.71340513, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/classification_loss = 0.31298685, Loss/localization_loss = 0.036284216, Loss/regularization_loss = 0.22650018, Loss/total_loss = 0.57576823, global_step = 2256, learning_rate = 0.039999958, loss = 0.57576823\n",
"I0503 23:47:12.658100 140215059400576 estimator.py:2049] Saving dict for global step 2256: DetectionBoxes_Precision/mAP = 0.7203577, DetectionBoxes_Precision/mAP (large) = 0.78831357, DetectionBoxes_Precision/mAP (medium) = 0.2406104, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.86428666, DetectionBoxes_Precision/mAP@.75IOU = 0.84071743, DetectionBoxes_Recall/AR@1 = 0.83993936, DetectionBoxes_Recall/AR@10 = 0.8446778, DetectionBoxes_Recall/AR@100 = 0.8446778, DetectionBoxes_Recall/AR@100 (large) = 0.87, DetectionBoxes_Recall/AR@100 (medium) = 0.71340513, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/classification_loss = 0.31298685, Loss/localization_loss = 0.036284216, Loss/regularization_loss = 0.22650018, Loss/total_loss = 0.57576823, global_step = 2256, learning_rate = 0.039999958, loss = 0.57576823\n",
"INFO:tensorflow:Saving 'checkpoint_path' summary for global step 2256: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-2256\n",
"I0503 23:47:12.661755 140215059400576 estimator.py:2109] Saving 'checkpoint_path' summary for global step 2256: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-2256\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.425083\n",
"I0503 23:48:13.056861 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.425083\n",
"INFO:tensorflow:loss = 0.48110726, step = 2300 (235.248 sec)\n",
"I0503 23:48:13.057757 140215059400576 basic_session_run_hooks.py:260] loss = 0.48110726, step = 2300 (235.248 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.740782\n",
"I0503 23:50:28.049423 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.740782\n",
"INFO:tensorflow:loss = 0.4774669, step = 2400 (134.993 sec)\n",
"I0503 23:50:28.050292 140215059400576 basic_session_run_hooks.py:260] loss = 0.4774669, step = 2400 (134.993 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.74331\n",
"I0503 23:52:42.582801 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.74331\n",
"INFO:tensorflow:loss = 0.51193935, step = 2500 (134.533 sec)\n",
"I0503 23:52:42.583755 140215059400576 basic_session_run_hooks.py:260] loss = 0.51193935, step = 2500 (134.533 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.741872\n",
"I0503 23:54:57.376957 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.741872\n",
"INFO:tensorflow:loss = 0.4634264, step = 2600 (134.794 sec)\n",
"I0503 23:54:57.377939 140215059400576 basic_session_run_hooks.py:260] loss = 0.4634264, step = 2600 (134.794 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:Saving checkpoints for 2627 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"I0503 23:55:32.360968 140215059400576 basic_session_run_hooks.py:606] Saving checkpoints for 2627 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"INFO:tensorflow:Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0503 23:55:34.447839 140215059400576 dataset_builder.py:163] Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0503 23:55:34.448860 140215059400576 dataset_builder.py:80] Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Number of filenames to read: 10\n",
"I0503 23:55:34.448984 140215059400576 dataset_builder.py:81] Number of filenames to read: 10\n",
"WARNING:tensorflow:Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f8536566e90>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"W0503 23:55:34.486161 140215059400576 ag_logging.py:146] Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f8536566e90>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"WARNING:tensorflow:Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7f805fc77170> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"W0503 23:55:34.647705 140215059400576 ag_logging.py:146] Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7f805fc77170> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"INFO:tensorflow:Calling model_fn.\n",
"I0503 23:55:35.134610 140215059400576 estimator.py:1148] Calling model_fn.\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:55:37.151859 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:55:37.220672 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:55:37.288931 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:55:37.356766 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:55:37.427735 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0503 23:55:37.498246 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:Done calling model_fn.\n",
"I0503 23:55:39.250312 140215059400576 estimator.py:1150] Done calling model_fn.\n",
"INFO:tensorflow:Starting evaluation at 2021-05-03T23:55:39Z\n",
"I0503 23:55:39.264970 140215059400576 evaluation.py:255] Starting evaluation at 2021-05-03T23:55:39Z\n",
"INFO:tensorflow:Graph was finalized.\n",
"I0503 23:55:39.714994 140215059400576 monitored_session.py:240] Graph was finalized.\n",
"2021-05-03 23:55:39.715629: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:55:39.715926: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: \n",
"name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285\n",
"pciBusID: 0000:00:04.0\n",
"2021-05-03 23:55:39.716017: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
"2021-05-03 23:55:39.716039: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n",
"2021-05-03 23:55:39.716054: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10\n",
"2021-05-03 23:55:39.716071: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10\n",
"2021-05-03 23:55:39.716099: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10\n",
"2021-05-03 23:55:39.716118: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10\n",
"2021-05-03 23:55:39.716130: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
"2021-05-03 23:55:39.716193: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:55:39.716434: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:55:39.716623: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0\n",
"2021-05-03 23:55:39.716697: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
"2021-05-03 23:55:39.716709: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 \n",
"2021-05-03 23:55:39.716715: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N \n",
"2021-05-03 23:55:39.716795: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:55:39.717024: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-03 23:55:39.717234: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15224 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0)\n",
"INFO:tensorflow:Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-2627\n",
"I0503 23:55:39.718320 140215059400576 saver.py:1284] Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-2627\n",
"INFO:tensorflow:Running local_init_op.\n",
"I0503 23:55:40.710749 140215059400576 session_manager.py:500] Running local_init_op.\n",
"INFO:tensorflow:Done running local_init_op.\n",
"I0503 23:55:40.846814 140215059400576 session_manager.py:502] Done running local_init_op.\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:Performing evaluation on 2568 images.\n",
"I0503 23:56:46.608262 140210432521984 coco_evaluation.py:293] Performing evaluation on 2568 images.\n",
"creating index...\n",
"index created!\n",
"INFO:tensorflow:Loading and preparing annotation results...\n",
"I0503 23:56:46.615258 140210432521984 coco_tools.py:116] Loading and preparing annotation results...\n",
"INFO:tensorflow:DONE (t=0.19s)\n",
"I0503 23:56:46.803236 140210432521984 coco_tools.py:138] DONE (t=0.19s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=19.62s).\n",
"Accumulating evaluation results...\n",
"DONE (t=6.87s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.767\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.901\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.887\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.003\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.309\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.826\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.860\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.862\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.862\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.200\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.740\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.883\n",
"INFO:tensorflow:Finished evaluation at 2021-05-03-23:57:14\n",
"I0503 23:57:14.283967 140215059400576 evaluation.py:275] Finished evaluation at 2021-05-03-23:57:14\n",
"INFO:tensorflow:Saving dict for global step 2627: DetectionBoxes_Precision/mAP = 0.76652586, DetectionBoxes_Precision/mAP (large) = 0.82630664, DetectionBoxes_Precision/mAP (medium) = 0.30866796, DetectionBoxes_Precision/mAP (small) = 0.0033663367, DetectionBoxes_Precision/mAP@.50IOU = 0.9011963, DetectionBoxes_Precision/mAP@.75IOU = 0.88666576, DetectionBoxes_Recall/AR@1 = 0.85985476, DetectionBoxes_Recall/AR@10 = 0.8622456, DetectionBoxes_Recall/AR@100 = 0.8622456, DetectionBoxes_Recall/AR@100 (large) = 0.8833723, DetectionBoxes_Recall/AR@100 (medium) = 0.74026436, DetectionBoxes_Recall/AR@100 (small) = 0.2, Loss/classification_loss = 0.2612513, Loss/localization_loss = 0.031032389, Loss/regularization_loss = 0.22521982, Loss/total_loss = 0.5175049, global_step = 2627, learning_rate = 0.039999757, loss = 0.5175049\n",
"I0503 23:57:14.284255 140215059400576 estimator.py:2049] Saving dict for global step 2627: DetectionBoxes_Precision/mAP = 0.76652586, DetectionBoxes_Precision/mAP (large) = 0.82630664, DetectionBoxes_Precision/mAP (medium) = 0.30866796, DetectionBoxes_Precision/mAP (small) = 0.0033663367, DetectionBoxes_Precision/mAP@.50IOU = 0.9011963, DetectionBoxes_Precision/mAP@.75IOU = 0.88666576, DetectionBoxes_Recall/AR@1 = 0.85985476, DetectionBoxes_Recall/AR@10 = 0.8622456, DetectionBoxes_Recall/AR@100 = 0.8622456, DetectionBoxes_Recall/AR@100 (large) = 0.8833723, DetectionBoxes_Recall/AR@100 (medium) = 0.74026436, DetectionBoxes_Recall/AR@100 (small) = 0.2, Loss/classification_loss = 0.2612513, Loss/localization_loss = 0.031032389, Loss/regularization_loss = 0.22521982, Loss/total_loss = 0.5175049, global_step = 2627, learning_rate = 0.039999757, loss = 0.5175049\n",
"INFO:tensorflow:Saving 'checkpoint_path' summary for global step 2627: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-2627\n",
"I0503 23:57:14.287927 140215059400576 estimator.py:2109] Saving 'checkpoint_path' summary for global step 2627: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-2627\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.423392\n",
"I0503 23:58:53.564939 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.423392\n",
"INFO:tensorflow:loss = 0.44238347, step = 2700 (236.188 sec)\n",
"I0503 23:58:53.566258 140215059400576 basic_session_run_hooks.py:260] loss = 0.44238347, step = 2700 (236.188 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.738639\n",
"I0504 00:01:08.949173 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.738639\n",
"INFO:tensorflow:loss = 0.44358426, step = 2800 (135.384 sec)\n",
"I0504 00:01:08.950550 140215059400576 basic_session_run_hooks.py:260] loss = 0.44358426, step = 2800 (135.384 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.739956\n",
"I0504 00:03:24.092235 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.739956\n",
"INFO:tensorflow:loss = 0.4026372, step = 2900 (135.143 sec)\n",
"I0504 00:03:24.093274 140215059400576 basic_session_run_hooks.py:260] loss = 0.4026372, step = 2900 (135.143 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:Saving checkpoints for 2996 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"I0504 00:05:32.992605 140215059400576 basic_session_run_hooks.py:606] Saving checkpoints for 2996 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"INFO:tensorflow:Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0504 00:05:35.185099 140215059400576 dataset_builder.py:163] Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0504 00:05:35.186123 140215059400576 dataset_builder.py:80] Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Number of filenames to read: 10\n",
"I0504 00:05:35.186289 140215059400576 dataset_builder.py:81] Number of filenames to read: 10\n",
"WARNING:tensorflow:Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f85c66c8410>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"W0504 00:05:35.223627 140215059400576 ag_logging.py:146] Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f85c66c8410>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"WARNING:tensorflow:Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7f85ba63fa70> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"W0504 00:05:35.383057 140215059400576 ag_logging.py:146] Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7f85ba63fa70> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"INFO:tensorflow:Calling model_fn.\n",
"I0504 00:05:35.860261 140215059400576 estimator.py:1148] Calling model_fn.\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:05:37.884728 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:05:37.954850 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:05:38.026676 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:05:38.096376 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:05:38.166676 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:05:38.238041 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:Done calling model_fn.\n",
"I0504 00:05:39.966716 140215059400576 estimator.py:1150] Done calling model_fn.\n",
"INFO:tensorflow:Starting evaluation at 2021-05-04T00:05:39Z\n",
"I0504 00:05:39.981636 140215059400576 evaluation.py:255] Starting evaluation at 2021-05-04T00:05:39Z\n",
"INFO:tensorflow:Graph was finalized.\n",
"I0504 00:05:40.430765 140215059400576 monitored_session.py:240] Graph was finalized.\n",
"2021-05-04 00:05:40.431482: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:05:40.431842: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: \n",
"name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285\n",
"pciBusID: 0000:00:04.0\n",
"2021-05-04 00:05:40.431951: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
"2021-05-04 00:05:40.431975: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n",
"2021-05-04 00:05:40.431991: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10\n",
"2021-05-04 00:05:40.432008: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10\n",
"2021-05-04 00:05:40.432033: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10\n",
"2021-05-04 00:05:40.432046: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10\n",
"2021-05-04 00:05:40.432061: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
"2021-05-04 00:05:40.432152: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:05:40.432400: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:05:40.432589: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0\n",
"2021-05-04 00:05:40.432628: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
"2021-05-04 00:05:40.432639: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 \n",
"2021-05-04 00:05:40.432654: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N \n",
"2021-05-04 00:05:40.432742: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:05:40.432973: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:05:40.433184: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15224 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0)\n",
"INFO:tensorflow:Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-2996\n",
"I0504 00:05:40.434189 140215059400576 saver.py:1284] Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-2996\n",
"INFO:tensorflow:Running local_init_op.\n",
"I0504 00:05:41.416208 140215059400576 session_manager.py:500] Running local_init_op.\n",
"INFO:tensorflow:Done running local_init_op.\n",
"I0504 00:05:41.550564 140215059400576 session_manager.py:502] Done running local_init_op.\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:Performing evaluation on 2568 images.\n",
"I0504 00:06:46.620033 140210432521984 coco_evaluation.py:293] Performing evaluation on 2568 images.\n",
"creating index...\n",
"index created!\n",
"INFO:tensorflow:Loading and preparing annotation results...\n",
"I0504 00:06:46.625624 140210432521984 coco_tools.py:116] Loading and preparing annotation results...\n",
"INFO:tensorflow:DONE (t=0.21s)\n",
"I0504 00:06:46.832377 140210432521984 coco_tools.py:138] DONE (t=0.21s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=19.79s).\n",
"Accumulating evaluation results...\n",
"DONE (t=6.77s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.773\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.920\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.903\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.001\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.322\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.830\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.851\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.855\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.855\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.100\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.730\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.877\n",
"INFO:tensorflow:Finished evaluation at 2021-05-04-00:07:14\n",
"I0504 00:07:14.388152 140215059400576 evaluation.py:275] Finished evaluation at 2021-05-04-00:07:14\n",
"INFO:tensorflow:Saving dict for global step 2996: DetectionBoxes_Precision/mAP = 0.77297103, DetectionBoxes_Precision/mAP (large) = 0.82952887, DetectionBoxes_Precision/mAP (medium) = 0.3219404, DetectionBoxes_Precision/mAP (small) = 0.00050495047, DetectionBoxes_Precision/mAP@.50IOU = 0.9201254, DetectionBoxes_Precision/mAP@.75IOU = 0.903216, DetectionBoxes_Recall/AR@1 = 0.8508792, DetectionBoxes_Recall/AR@10 = 0.85464895, DetectionBoxes_Recall/AR@100 = 0.85472006, DetectionBoxes_Recall/AR@100 (large) = 0.8771587, DetectionBoxes_Recall/AR@100 (medium) = 0.7297572, DetectionBoxes_Recall/AR@100 (small) = 0.1, Loss/classification_loss = 0.2320339, Loss/localization_loss = 0.033271253, Loss/regularization_loss = 0.22389893, Loss/total_loss = 0.4892042, global_step = 2996, learning_rate = 0.03999938, loss = 0.4892042\n",
"I0504 00:07:14.388409 140215059400576 estimator.py:2049] Saving dict for global step 2996: DetectionBoxes_Precision/mAP = 0.77297103, DetectionBoxes_Precision/mAP (large) = 0.82952887, DetectionBoxes_Precision/mAP (medium) = 0.3219404, DetectionBoxes_Precision/mAP (small) = 0.00050495047, DetectionBoxes_Precision/mAP@.50IOU = 0.9201254, DetectionBoxes_Precision/mAP@.75IOU = 0.903216, DetectionBoxes_Recall/AR@1 = 0.8508792, DetectionBoxes_Recall/AR@10 = 0.85464895, DetectionBoxes_Recall/AR@100 = 0.85472006, DetectionBoxes_Recall/AR@100 (large) = 0.8771587, DetectionBoxes_Recall/AR@100 (medium) = 0.7297572, DetectionBoxes_Recall/AR@100 (small) = 0.1, Loss/classification_loss = 0.2320339, Loss/localization_loss = 0.033271253, Loss/regularization_loss = 0.22389893, Loss/total_loss = 0.4892042, global_step = 2996, learning_rate = 0.03999938, loss = 0.4892042\n",
"INFO:tensorflow:Saving 'checkpoint_path' summary for global step 2996: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-2996\n",
"I0504 00:07:14.392197 140215059400576 estimator.py:2109] Saving 'checkpoint_path' summary for global step 2996: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-2996\n",
"INFO:tensorflow:global_step/sec: 0.421709\n",
"I0504 00:07:21.222511 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.421709\n",
"INFO:tensorflow:loss = 0.40506738, step = 3000 (237.130 sec)\n",
"I0504 00:07:21.223703 140215059400576 basic_session_run_hooks.py:260] loss = 0.40506738, step = 3000 (237.130 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.738826\n",
"I0504 00:09:36.572391 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.738826\n",
"INFO:tensorflow:loss = 0.4035138, step = 3100 (135.350 sec)\n",
"I0504 00:09:36.573441 140215059400576 basic_session_run_hooks.py:260] loss = 0.4035138, step = 3100 (135.350 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.742076\n",
"I0504 00:11:51.329473 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.742076\n",
"INFO:tensorflow:loss = 0.3989345, step = 3200 (134.757 sec)\n",
"I0504 00:11:51.330450 140215059400576 basic_session_run_hooks.py:260] loss = 0.3989345, step = 3200 (134.757 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.740713\n",
"I0504 00:14:06.334585 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.740713\n",
"INFO:tensorflow:loss = 0.39785632, step = 3300 (135.005 sec)\n",
"I0504 00:14:06.335711 140215059400576 basic_session_run_hooks.py:260] loss = 0.39785632, step = 3300 (135.005 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:Saving checkpoints for 3366 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"I0504 00:15:33.556672 140215059400576 basic_session_run_hooks.py:606] Saving checkpoints for 3366 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"INFO:tensorflow:Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0504 00:15:35.672658 140215059400576 dataset_builder.py:163] Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0504 00:15:35.673603 140215059400576 dataset_builder.py:80] Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Number of filenames to read: 10\n",
"I0504 00:15:35.673736 140215059400576 dataset_builder.py:81] Number of filenames to read: 10\n",
"WARNING:tensorflow:Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f8061a0bf50>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"W0504 00:15:35.711191 140215059400576 ag_logging.py:146] Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f8061a0bf50>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"WARNING:tensorflow:Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7f805cd06d40> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"W0504 00:15:35.870846 140215059400576 ag_logging.py:146] Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7f805cd06d40> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"INFO:tensorflow:Calling model_fn.\n",
"I0504 00:15:36.724589 140215059400576 estimator.py:1148] Calling model_fn.\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:15:38.753186 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:15:38.822665 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:15:38.899943 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:15:38.970214 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:15:39.044627 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:15:39.116199 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:Done calling model_fn.\n",
"I0504 00:15:40.858367 140215059400576 estimator.py:1150] Done calling model_fn.\n",
"INFO:tensorflow:Starting evaluation at 2021-05-04T00:15:40Z\n",
"I0504 00:15:40.873203 140215059400576 evaluation.py:255] Starting evaluation at 2021-05-04T00:15:40Z\n",
"INFO:tensorflow:Graph was finalized.\n",
"I0504 00:15:41.351674 140215059400576 monitored_session.py:240] Graph was finalized.\n",
"2021-05-04 00:15:41.352276: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:15:41.352571: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: \n",
"name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285\n",
"pciBusID: 0000:00:04.0\n",
"2021-05-04 00:15:41.352665: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
"2021-05-04 00:15:41.352688: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n",
"2021-05-04 00:15:41.352702: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10\n",
"2021-05-04 00:15:41.352720: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10\n",
"2021-05-04 00:15:41.352739: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10\n",
"2021-05-04 00:15:41.352752: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10\n",
"2021-05-04 00:15:41.352767: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
"2021-05-04 00:15:41.352834: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:15:41.353077: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:15:41.353303: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0\n",
"2021-05-04 00:15:41.353392: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
"2021-05-04 00:15:41.353408: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 \n",
"2021-05-04 00:15:41.353417: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N \n",
"2021-05-04 00:15:41.353534: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:15:41.353810: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:15:41.354072: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15224 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0)\n",
"INFO:tensorflow:Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-3366\n",
"I0504 00:15:41.355482 140215059400576 saver.py:1284] Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-3366\n",
"INFO:tensorflow:Running local_init_op.\n",
"I0504 00:15:42.373521 140215059400576 session_manager.py:500] Running local_init_op.\n",
"INFO:tensorflow:Done running local_init_op.\n",
"I0504 00:15:42.529315 140215059400576 session_manager.py:502] Done running local_init_op.\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:Performing evaluation on 2568 images.\n",
"I0504 00:16:48.795337 140210424129280 coco_evaluation.py:293] Performing evaluation on 2568 images.\n",
"creating index...\n",
"index created!\n",
"INFO:tensorflow:Loading and preparing annotation results...\n",
"I0504 00:16:48.813607 140210424129280 coco_tools.py:116] Loading and preparing annotation results...\n",
"INFO:tensorflow:DONE (t=0.21s)\n",
"I0504 00:16:49.026471 140210424129280 coco_tools.py:138] DONE (t=0.21s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=19.41s).\n",
"Accumulating evaluation results...\n",
"DONE (t=7.61s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.714\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.885\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.843\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.001\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.307\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.768\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.815\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.827\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.827\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.100\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.698\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.852\n",
"INFO:tensorflow:Finished evaluation at 2021-05-04-00:17:17\n",
"I0504 00:17:17.009499 140215059400576 evaluation.py:275] Finished evaluation at 2021-05-04-00:17:17\n",
"INFO:tensorflow:Saving dict for global step 3366: DetectionBoxes_Precision/mAP = 0.7135667, DetectionBoxes_Precision/mAP (large) = 0.7683116, DetectionBoxes_Precision/mAP (medium) = 0.30692887, DetectionBoxes_Precision/mAP (small) = 0.00068236556, DetectionBoxes_Precision/mAP@.50IOU = 0.8847008, DetectionBoxes_Precision/mAP@.75IOU = 0.8431751, DetectionBoxes_Recall/AR@1 = 0.8154388, DetectionBoxes_Recall/AR@10 = 0.8272542, DetectionBoxes_Recall/AR@100 = 0.8273293, DetectionBoxes_Recall/AR@100 (large) = 0.8517159, DetectionBoxes_Recall/AR@100 (medium) = 0.6981567, DetectionBoxes_Recall/AR@100 (small) = 0.1, Loss/classification_loss = 0.29136416, Loss/localization_loss = 0.047467694, Loss/regularization_loss = 0.2224056, Loss/total_loss = 0.561232, global_step = 3366, learning_rate = 0.039998837, loss = 0.561232\n",
"I0504 00:17:17.009752 140215059400576 estimator.py:2049] Saving dict for global step 3366: DetectionBoxes_Precision/mAP = 0.7135667, DetectionBoxes_Precision/mAP (large) = 0.7683116, DetectionBoxes_Precision/mAP (medium) = 0.30692887, DetectionBoxes_Precision/mAP (small) = 0.00068236556, DetectionBoxes_Precision/mAP@.50IOU = 0.8847008, DetectionBoxes_Precision/mAP@.75IOU = 0.8431751, DetectionBoxes_Recall/AR@1 = 0.8154388, DetectionBoxes_Recall/AR@10 = 0.8272542, DetectionBoxes_Recall/AR@100 = 0.8273293, DetectionBoxes_Recall/AR@100 (large) = 0.8517159, DetectionBoxes_Recall/AR@100 (medium) = 0.6981567, DetectionBoxes_Recall/AR@100 (small) = 0.1, Loss/classification_loss = 0.29136416, Loss/localization_loss = 0.047467694, Loss/regularization_loss = 0.2224056, Loss/total_loss = 0.561232, global_step = 3366, learning_rate = 0.039998837, loss = 0.561232\n",
"INFO:tensorflow:Saving 'checkpoint_path' summary for global step 3366: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-3366\n",
"I0504 00:17:17.013588 140215059400576 estimator.py:2109] Saving 'checkpoint_path' summary for global step 3366: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-3366\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.419808\n",
"I0504 00:18:04.538462 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.419808\n",
"INFO:tensorflow:loss = 0.37604678, step = 3400 (238.204 sec)\n",
"I0504 00:18:04.539533 140215059400576 basic_session_run_hooks.py:260] loss = 0.37604678, step = 3400 (238.204 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.736851\n",
"I0504 00:20:20.251132 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.736851\n",
"INFO:tensorflow:loss = 0.37406266, step = 3500 (135.713 sec)\n",
"I0504 00:20:20.252358 140215059400576 basic_session_run_hooks.py:260] loss = 0.37406266, step = 3500 (135.713 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.742703\n",
"I0504 00:22:34.894427 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.742703\n",
"INFO:tensorflow:loss = 0.39673847, step = 3600 (134.643 sec)\n",
"I0504 00:22:34.895339 140215059400576 basic_session_run_hooks.py:260] loss = 0.39673847, step = 3600 (134.643 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.741718\n",
"I0504 00:24:49.716670 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.741718\n",
"INFO:tensorflow:loss = 0.39229053, step = 3700 (134.822 sec)\n",
"I0504 00:24:49.717634 140215059400576 basic_session_run_hooks.py:260] loss = 0.39229053, step = 3700 (134.822 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:Saving checkpoints for 3734 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"I0504 00:25:34.346060 140215059400576 basic_session_run_hooks.py:606] Saving checkpoints for 3734 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"INFO:tensorflow:Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0504 00:25:36.445817 140215059400576 dataset_builder.py:163] Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0504 00:25:36.446804 140215059400576 dataset_builder.py:80] Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Number of filenames to read: 10\n",
"I0504 00:25:36.446929 140215059400576 dataset_builder.py:81] Number of filenames to read: 10\n",
"WARNING:tensorflow:Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f8047904f10>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"W0504 00:25:36.484265 140215059400576 ag_logging.py:146] Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f8047904f10>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"WARNING:tensorflow:Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7f85285340e0> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"W0504 00:25:36.645600 140215059400576 ag_logging.py:146] Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7f85285340e0> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"INFO:tensorflow:Calling model_fn.\n",
"I0504 00:25:37.124161 140215059400576 estimator.py:1148] Calling model_fn.\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:25:39.123061 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:25:39.192206 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:25:39.262681 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:25:39.331016 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:25:39.399062 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:25:39.472804 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:Done calling model_fn.\n",
"I0504 00:25:41.188759 140215059400576 estimator.py:1150] Done calling model_fn.\n",
"INFO:tensorflow:Starting evaluation at 2021-05-04T00:25:41Z\n",
"I0504 00:25:41.203554 140215059400576 evaluation.py:255] Starting evaluation at 2021-05-04T00:25:41Z\n",
"INFO:tensorflow:Graph was finalized.\n",
"I0504 00:25:41.659526 140215059400576 monitored_session.py:240] Graph was finalized.\n",
"2021-05-04 00:25:41.660275: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:25:41.660651: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: \n",
"name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285\n",
"pciBusID: 0000:00:04.0\n",
"2021-05-04 00:25:41.660769: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
"2021-05-04 00:25:41.660802: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n",
"2021-05-04 00:25:41.660827: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10\n",
"2021-05-04 00:25:41.660851: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10\n",
"2021-05-04 00:25:41.660874: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10\n",
"2021-05-04 00:25:41.660895: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10\n",
"2021-05-04 00:25:41.660917: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
"2021-05-04 00:25:41.661007: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:25:41.661342: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:25:41.661565: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0\n",
"2021-05-04 00:25:41.661610: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
"2021-05-04 00:25:41.661623: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 \n",
"2021-05-04 00:25:41.661633: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N \n",
"2021-05-04 00:25:41.661729: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:25:41.662015: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:25:41.662275: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15224 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0)\n",
"INFO:tensorflow:Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-3734\n",
"I0504 00:25:41.663229 140215059400576 saver.py:1284] Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-3734\n",
"INFO:tensorflow:Running local_init_op.\n",
"I0504 00:25:42.661723 140215059400576 session_manager.py:500] Running local_init_op.\n",
"INFO:tensorflow:Done running local_init_op.\n",
"I0504 00:25:42.803506 140215059400576 session_manager.py:502] Done running local_init_op.\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:Performing evaluation on 2568 images.\n",
"I0504 00:26:48.375295 140210415736576 coco_evaluation.py:293] Performing evaluation on 2568 images.\n",
"creating index...\n",
"index created!\n",
"INFO:tensorflow:Loading and preparing annotation results...\n",
"I0504 00:26:48.382606 140210415736576 coco_tools.py:116] Loading and preparing annotation results...\n",
"INFO:tensorflow:DONE (t=0.19s)\n",
"I0504 00:26:48.575759 140210415736576 coco_tools.py:138] DONE (t=0.19s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=19.74s).\n",
"Accumulating evaluation results...\n",
"DONE (t=6.98s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.834\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.957\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.947\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.006\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.457\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.874\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.885\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.889\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.889\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.400\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.782\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.907\n",
"INFO:tensorflow:Finished evaluation at 2021-05-04-00:27:16\n",
"I0504 00:27:16.270267 140215059400576 evaluation.py:275] Finished evaluation at 2021-05-04-00:27:16\n",
"INFO:tensorflow:Saving dict for global step 3734: DetectionBoxes_Precision/mAP = 0.83393395, DetectionBoxes_Precision/mAP (large) = 0.8735613, DetectionBoxes_Precision/mAP (medium) = 0.4568373, DetectionBoxes_Precision/mAP (small) = 0.006100533, DetectionBoxes_Precision/mAP@.50IOU = 0.9569519, DetectionBoxes_Precision/mAP@.75IOU = 0.9465632, DetectionBoxes_Recall/AR@1 = 0.8853066, DetectionBoxes_Recall/AR@10 = 0.8890051, DetectionBoxes_Recall/AR@100 = 0.8890051, DetectionBoxes_Recall/AR@100 (large) = 0.90670025, DetectionBoxes_Recall/AR@100 (medium) = 0.7820043, DetectionBoxes_Recall/AR@100 (small) = 0.4, Loss/classification_loss = 0.17915349, Loss/localization_loss = 0.023984391, Loss/regularization_loss = 0.22084594, Loss/total_loss = 0.42398, global_step = 3734, learning_rate = 0.039998125, loss = 0.42398\n",
"I0504 00:27:16.270514 140215059400576 estimator.py:2049] Saving dict for global step 3734: DetectionBoxes_Precision/mAP = 0.83393395, DetectionBoxes_Precision/mAP (large) = 0.8735613, DetectionBoxes_Precision/mAP (medium) = 0.4568373, DetectionBoxes_Precision/mAP (small) = 0.006100533, DetectionBoxes_Precision/mAP@.50IOU = 0.9569519, DetectionBoxes_Precision/mAP@.75IOU = 0.9465632, DetectionBoxes_Recall/AR@1 = 0.8853066, DetectionBoxes_Recall/AR@10 = 0.8890051, DetectionBoxes_Recall/AR@100 = 0.8890051, DetectionBoxes_Recall/AR@100 (large) = 0.90670025, DetectionBoxes_Recall/AR@100 (medium) = 0.7820043, DetectionBoxes_Recall/AR@100 (small) = 0.4, Loss/classification_loss = 0.17915349, Loss/localization_loss = 0.023984391, Loss/regularization_loss = 0.22084594, Loss/total_loss = 0.42398, global_step = 3734, learning_rate = 0.039998125, loss = 0.42398\n",
"INFO:tensorflow:Saving 'checkpoint_path' summary for global step 3734: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-3734\n",
"I0504 00:27:16.274342 140215059400576 estimator.py:2109] Saving 'checkpoint_path' summary for global step 3734: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-3734\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.421316\n",
"I0504 00:28:47.068329 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.421316\n",
"INFO:tensorflow:loss = 0.39515993, step = 3800 (237.352 sec)\n",
"I0504 00:28:47.069396 140215059400576 basic_session_run_hooks.py:260] loss = 0.39515993, step = 3800 (237.352 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.740153\n",
"I0504 00:31:02.175557 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.740153\n",
"INFO:tensorflow:loss = 0.37139446, step = 3900 (135.107 sec)\n",
"I0504 00:31:02.176748 140215059400576 basic_session_run_hooks.py:260] loss = 0.37139446, step = 3900 (135.107 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.740383\n",
"I0504 00:33:17.240847 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.740383\n",
"INFO:tensorflow:loss = 0.40161628, step = 4000 (135.065 sec)\n",
"I0504 00:33:17.241988 140215059400576 basic_session_run_hooks.py:260] loss = 0.40161628, step = 4000 (135.065 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.740292\n",
"I0504 00:35:32.322628 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.740292\n",
"INFO:tensorflow:loss = 0.383332, step = 4100 (135.082 sec)\n",
"I0504 00:35:32.323656 140215059400576 basic_session_run_hooks.py:260] loss = 0.383332, step = 4100 (135.082 sec)\n",
"INFO:tensorflow:Saving checkpoints for 4103 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"I0504 00:35:35.057940 140215059400576 basic_session_run_hooks.py:606] Saving checkpoints for 4103 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"INFO:tensorflow:Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0504 00:35:37.166390 140215059400576 dataset_builder.py:163] Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0504 00:35:37.167422 140215059400576 dataset_builder.py:80] Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Number of filenames to read: 10\n",
"I0504 00:35:37.167551 140215059400576 dataset_builder.py:81] Number of filenames to read: 10\n",
"WARNING:tensorflow:Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f8063900f50>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"W0504 00:35:37.204989 140215059400576 ag_logging.py:146] Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f8063900f50>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"WARNING:tensorflow:Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7f80646d0170> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"W0504 00:35:37.365764 140215059400576 ag_logging.py:146] Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7f80646d0170> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"INFO:tensorflow:Calling model_fn.\n",
"I0504 00:35:37.850458 140215059400576 estimator.py:1148] Calling model_fn.\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:35:39.881611 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:35:39.950798 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:35:40.020510 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:35:40.088593 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:35:40.159373 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:35:40.228047 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:Done calling model_fn.\n",
"I0504 00:35:41.970594 140215059400576 estimator.py:1150] Done calling model_fn.\n",
"INFO:tensorflow:Starting evaluation at 2021-05-04T00:35:41Z\n",
"I0504 00:35:41.985589 140215059400576 evaluation.py:255] Starting evaluation at 2021-05-04T00:35:41Z\n",
"INFO:tensorflow:Graph was finalized.\n",
"I0504 00:35:42.442586 140215059400576 monitored_session.py:240] Graph was finalized.\n",
"2021-05-04 00:35:42.443217: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:35:42.443514: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: \n",
"name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285\n",
"pciBusID: 0000:00:04.0\n",
"2021-05-04 00:35:42.443611: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
"2021-05-04 00:35:42.443630: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n",
"2021-05-04 00:35:42.443643: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10\n",
"2021-05-04 00:35:42.443658: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10\n",
"2021-05-04 00:35:42.443672: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10\n",
"2021-05-04 00:35:42.443686: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10\n",
"2021-05-04 00:35:42.443700: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
"2021-05-04 00:35:42.443777: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:35:42.444014: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:35:42.444218: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0\n",
"2021-05-04 00:35:42.444297: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
"2021-05-04 00:35:42.444308: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 \n",
"2021-05-04 00:35:42.444315: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N \n",
"2021-05-04 00:35:42.444407: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:35:42.444642: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:35:42.444849: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15224 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0)\n",
"INFO:tensorflow:Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-4103\n",
"I0504 00:35:42.448815 140215059400576 saver.py:1284] Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-4103\n",
"INFO:tensorflow:Running local_init_op.\n",
"I0504 00:35:43.439375 140215059400576 session_manager.py:500] Running local_init_op.\n",
"INFO:tensorflow:Done running local_init_op.\n",
"I0504 00:35:43.582256 140215059400576 session_manager.py:502] Done running local_init_op.\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:Performing evaluation on 2568 images.\n",
"I0504 00:36:50.223706 140210440914688 coco_evaluation.py:293] Performing evaluation on 2568 images.\n",
"creating index...\n",
"index created!\n",
"INFO:tensorflow:Loading and preparing annotation results...\n",
"I0504 00:36:50.232547 140210440914688 coco_tools.py:116] Loading and preparing annotation results...\n",
"INFO:tensorflow:DONE (t=0.19s)\n",
"I0504 00:36:50.418372 140210440914688 coco_tools.py:138] DONE (t=0.19s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=19.73s).\n",
"Accumulating evaluation results...\n",
"DONE (t=6.79s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.832\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.962\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.948\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.034\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.484\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.871\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.881\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.884\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.884\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.100\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.758\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.906\n",
"INFO:tensorflow:Finished evaluation at 2021-05-04-00:37:17\n",
"I0504 00:37:17.920741 140215059400576 evaluation.py:275] Finished evaluation at 2021-05-04-00:37:17\n",
"INFO:tensorflow:Saving dict for global step 4103: DetectionBoxes_Precision/mAP = 0.8317826, DetectionBoxes_Precision/mAP (large) = 0.87101597, DetectionBoxes_Precision/mAP (medium) = 0.4841115, DetectionBoxes_Precision/mAP (small) = 0.033663366, DetectionBoxes_Precision/mAP@.50IOU = 0.96235496, DetectionBoxes_Precision/mAP@.75IOU = 0.94771767, DetectionBoxes_Recall/AR@1 = 0.88121945, DetectionBoxes_Recall/AR@10 = 0.88393545, DetectionBoxes_Recall/AR@100 = 0.88393545, DetectionBoxes_Recall/AR@100 (large) = 0.90570116, DetectionBoxes_Recall/AR@100 (medium) = 0.7580175, DetectionBoxes_Recall/AR@100 (small) = 0.1, Loss/classification_loss = 0.16863383, Loss/localization_loss = 0.025558911, Loss/regularization_loss = 0.21923961, Loss/total_loss = 0.4134348, global_step = 4103, learning_rate = 0.039997242, loss = 0.4134348\n",
"I0504 00:37:17.921006 140215059400576 estimator.py:2049] Saving dict for global step 4103: DetectionBoxes_Precision/mAP = 0.8317826, DetectionBoxes_Precision/mAP (large) = 0.87101597, DetectionBoxes_Precision/mAP (medium) = 0.4841115, DetectionBoxes_Precision/mAP (small) = 0.033663366, DetectionBoxes_Precision/mAP@.50IOU = 0.96235496, DetectionBoxes_Precision/mAP@.75IOU = 0.94771767, DetectionBoxes_Recall/AR@1 = 0.88121945, DetectionBoxes_Recall/AR@10 = 0.88393545, DetectionBoxes_Recall/AR@100 = 0.88393545, DetectionBoxes_Recall/AR@100 (large) = 0.90570116, DetectionBoxes_Recall/AR@100 (medium) = 0.7580175, DetectionBoxes_Recall/AR@100 (small) = 0.1, Loss/classification_loss = 0.16863383, Loss/localization_loss = 0.025558911, Loss/regularization_loss = 0.21923961, Loss/total_loss = 0.4134348, global_step = 4103, learning_rate = 0.039997242, loss = 0.4134348\n",
"INFO:tensorflow:Saving 'checkpoint_path' summary for global step 4103: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-4103\n",
"I0504 00:37:17.925047 140215059400576 estimator.py:2109] Saving 'checkpoint_path' summary for global step 4103: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-4103\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.418877\n",
"I0504 00:39:31.056219 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.418877\n",
"INFO:tensorflow:loss = 0.41872814, step = 4200 (238.734 sec)\n",
"I0504 00:39:31.057484 140215059400576 basic_session_run_hooks.py:260] loss = 0.41872814, step = 4200 (238.734 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.740372\n",
"I0504 00:41:46.123488 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.740372\n",
"INFO:tensorflow:loss = 0.38020697, step = 4300 (135.067 sec)\n",
"I0504 00:41:46.124709 140215059400576 basic_session_run_hooks.py:260] loss = 0.38020697, step = 4300 (135.067 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.738541\n",
"I0504 00:44:01.525508 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.738541\n",
"INFO:tensorflow:loss = 0.34623438, step = 4400 (135.402 sec)\n",
"I0504 00:44:01.526607 140215059400576 basic_session_run_hooks.py:260] loss = 0.34623438, step = 4400 (135.402 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:Saving checkpoints for 4471 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"I0504 00:45:36.171920 140215059400576 basic_session_run_hooks.py:606] Saving checkpoints for 4471 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"INFO:tensorflow:Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0504 00:45:38.285873 140215059400576 dataset_builder.py:163] Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0504 00:45:38.286848 140215059400576 dataset_builder.py:80] Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Number of filenames to read: 10\n",
"I0504 00:45:38.286989 140215059400576 dataset_builder.py:81] Number of filenames to read: 10\n",
"WARNING:tensorflow:Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f805d93c6d0>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"W0504 00:45:38.325303 140215059400576 ag_logging.py:146] Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f805d93c6d0>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"WARNING:tensorflow:Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7f805cd06d40> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"W0504 00:45:38.486684 140215059400576 ag_logging.py:146] Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7f805cd06d40> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"INFO:tensorflow:Calling model_fn.\n",
"I0504 00:45:38.964005 140215059400576 estimator.py:1148] Calling model_fn.\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:45:40.985460 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:45:41.054814 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:45:41.123134 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:45:41.191340 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:45:41.262410 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:45:41.331383 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:Done calling model_fn.\n",
"I0504 00:45:43.070206 140215059400576 estimator.py:1150] Done calling model_fn.\n",
"INFO:tensorflow:Starting evaluation at 2021-05-04T00:45:43Z\n",
"I0504 00:45:43.085019 140215059400576 evaluation.py:255] Starting evaluation at 2021-05-04T00:45:43Z\n",
"INFO:tensorflow:Graph was finalized.\n",
"I0504 00:45:43.531547 140215059400576 monitored_session.py:240] Graph was finalized.\n",
"2021-05-04 00:45:43.532221: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:45:43.532569: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: \n",
"name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285\n",
"pciBusID: 0000:00:04.0\n",
"2021-05-04 00:45:43.532679: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
"2021-05-04 00:45:43.532699: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n",
"2021-05-04 00:45:43.532717: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10\n",
"2021-05-04 00:45:43.532733: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10\n",
"2021-05-04 00:45:43.532747: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10\n",
"2021-05-04 00:45:43.532763: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10\n",
"2021-05-04 00:45:43.532781: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
"2021-05-04 00:45:43.532852: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:45:43.533145: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:45:43.533352: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0\n",
"2021-05-04 00:45:43.533390: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
"2021-05-04 00:45:43.533414: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 \n",
"2021-05-04 00:45:43.533420: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N \n",
"2021-05-04 00:45:43.533513: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:45:43.533737: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:45:43.533935: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15224 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0)\n",
"INFO:tensorflow:Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-4471\n",
"I0504 00:45:43.534970 140215059400576 saver.py:1284] Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-4471\n",
"INFO:tensorflow:Running local_init_op.\n",
"I0504 00:45:44.609776 140215059400576 session_manager.py:500] Running local_init_op.\n",
"INFO:tensorflow:Done running local_init_op.\n",
"I0504 00:45:44.770944 140215059400576 session_manager.py:502] Done running local_init_op.\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:Performing evaluation on 2568 images.\n",
"I0504 00:46:51.560856 140210424129280 coco_evaluation.py:293] Performing evaluation on 2568 images.\n",
"creating index...\n",
"index created!\n",
"INFO:tensorflow:Loading and preparing annotation results...\n",
"I0504 00:46:51.567034 140210424129280 coco_tools.py:116] Loading and preparing annotation results...\n",
"INFO:tensorflow:DONE (t=0.20s)\n",
"I0504 00:46:51.771404 140210424129280 coco_tools.py:138] DONE (t=0.20s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=19.81s).\n",
"Accumulating evaluation results...\n",
"DONE (t=6.68s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.857\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.980\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.969\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.017\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.599\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.886\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.892\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.894\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.894\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.200\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.792\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.912\n",
"INFO:tensorflow:Finished evaluation at 2021-05-04-00:47:19\n",
"I0504 00:47:19.259883 140215059400576 evaluation.py:275] Finished evaluation at 2021-05-04-00:47:19\n",
"INFO:tensorflow:Saving dict for global step 4471: DetectionBoxes_Precision/mAP = 0.85685056, DetectionBoxes_Precision/mAP (large) = 0.8855686, DetectionBoxes_Precision/mAP (medium) = 0.5988093, DetectionBoxes_Precision/mAP (small) = 0.016831683, DetectionBoxes_Precision/mAP@.50IOU = 0.9797586, DetectionBoxes_Precision/mAP@.75IOU = 0.96924186, DetectionBoxes_Recall/AR@1 = 0.8917457, DetectionBoxes_Recall/AR@10 = 0.8937107, DetectionBoxes_Recall/AR@100 = 0.8937107, DetectionBoxes_Recall/AR@100 (large) = 0.91176426, DetectionBoxes_Recall/AR@100 (medium) = 0.7915283, DetectionBoxes_Recall/AR@100 (small) = 0.2, Loss/classification_loss = 0.13611425, Loss/localization_loss = 0.02159942, Loss/regularization_loss = 0.21761687, Loss/total_loss = 0.37532955, global_step = 4471, learning_rate = 0.039996196, loss = 0.37532955\n",
"I0504 00:47:19.260176 140215059400576 estimator.py:2049] Saving dict for global step 4471: DetectionBoxes_Precision/mAP = 0.85685056, DetectionBoxes_Precision/mAP (large) = 0.8855686, DetectionBoxes_Precision/mAP (medium) = 0.5988093, DetectionBoxes_Precision/mAP (small) = 0.016831683, DetectionBoxes_Precision/mAP@.50IOU = 0.9797586, DetectionBoxes_Precision/mAP@.75IOU = 0.96924186, DetectionBoxes_Recall/AR@1 = 0.8917457, DetectionBoxes_Recall/AR@10 = 0.8937107, DetectionBoxes_Recall/AR@100 = 0.8937107, DetectionBoxes_Recall/AR@100 (large) = 0.91176426, DetectionBoxes_Recall/AR@100 (medium) = 0.7915283, DetectionBoxes_Recall/AR@100 (small) = 0.2, Loss/classification_loss = 0.13611425, Loss/localization_loss = 0.02159942, Loss/regularization_loss = 0.21761687, Loss/total_loss = 0.37532955, global_step = 4471, learning_rate = 0.039996196, loss = 0.37532955\n",
"INFO:tensorflow:Saving 'checkpoint_path' summary for global step 4471: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-4471\n",
"I0504 00:47:19.263885 140215059400576 estimator.py:2109] Saving 'checkpoint_path' summary for global step 4471: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-4471\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.419648\n",
"I0504 00:47:59.820444 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.419648\n",
"INFO:tensorflow:loss = 0.32304123, step = 4500 (238.295 sec)\n",
"I0504 00:47:59.821493 140215059400576 basic_session_run_hooks.py:260] loss = 0.32304123, step = 4500 (238.295 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.738003\n",
"I0504 00:50:15.321199 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.738003\n",
"INFO:tensorflow:loss = 0.38807705, step = 4600 (135.501 sec)\n",
"I0504 00:50:15.322352 140215059400576 basic_session_run_hooks.py:260] loss = 0.38807705, step = 4600 (135.501 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.739737\n",
"I0504 00:52:30.504318 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.739737\n",
"INFO:tensorflow:loss = 0.33238897, step = 4700 (135.183 sec)\n",
"I0504 00:52:30.505529 140215059400576 basic_session_run_hooks.py:260] loss = 0.33238897, step = 4700 (135.183 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.735586\n",
"I0504 00:54:46.450327 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.735586\n",
"INFO:tensorflow:loss = 0.39697814, step = 4800 (135.946 sec)\n",
"I0504 00:54:46.451421 140215059400576 basic_session_run_hooks.py:260] loss = 0.39697814, step = 4800 (135.946 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:Saving checkpoints for 4838 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"I0504 00:55:36.339880 140215059400576 basic_session_run_hooks.py:606] Saving checkpoints for 4838 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"INFO:tensorflow:Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0504 00:55:38.447259 140215059400576 dataset_builder.py:163] Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0504 00:55:38.448234 140215059400576 dataset_builder.py:80] Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Number of filenames to read: 10\n",
"I0504 00:55:38.448386 140215059400576 dataset_builder.py:81] Number of filenames to read: 10\n",
"WARNING:tensorflow:Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f8060f32f50>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"W0504 00:55:38.485988 140215059400576 ag_logging.py:146] Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f8060f32f50>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"WARNING:tensorflow:Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7f805d37d830> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"W0504 00:55:38.655124 140215059400576 ag_logging.py:146] Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7f805d37d830> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"INFO:tensorflow:Calling model_fn.\n",
"I0504 00:55:39.201994 140215059400576 estimator.py:1148] Calling model_fn.\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:55:41.364953 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:55:41.434043 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:55:41.502981 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:55:41.571713 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:55:41.639482 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 00:55:41.711484 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:Done calling model_fn.\n",
"I0504 00:55:43.432304 140215059400576 estimator.py:1150] Done calling model_fn.\n",
"INFO:tensorflow:Starting evaluation at 2021-05-04T00:55:43Z\n",
"I0504 00:55:43.447510 140215059400576 evaluation.py:255] Starting evaluation at 2021-05-04T00:55:43Z\n",
"INFO:tensorflow:Graph was finalized.\n",
"I0504 00:55:43.906028 140215059400576 monitored_session.py:240] Graph was finalized.\n",
"2021-05-04 00:55:43.906670: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:55:43.906977: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: \n",
"name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285\n",
"pciBusID: 0000:00:04.0\n",
"2021-05-04 00:55:43.907096: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
"2021-05-04 00:55:43.907132: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n",
"2021-05-04 00:55:43.907159: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10\n",
"2021-05-04 00:55:43.907183: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10\n",
"2021-05-04 00:55:43.907205: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10\n",
"2021-05-04 00:55:43.907226: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10\n",
"2021-05-04 00:55:43.907248: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
"2021-05-04 00:55:43.907335: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:55:43.907614: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:55:43.907829: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0\n",
"2021-05-04 00:55:43.907917: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
"2021-05-04 00:55:43.907932: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 \n",
"2021-05-04 00:55:43.907942: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N \n",
"2021-05-04 00:55:43.908043: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:55:43.908316: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 00:55:43.908537: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15224 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0)\n",
"INFO:tensorflow:Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-4838\n",
"I0504 00:55:43.909493 140215059400576 saver.py:1284] Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-4838\n",
"INFO:tensorflow:Running local_init_op.\n",
"I0504 00:55:44.964654 140215059400576 session_manager.py:500] Running local_init_op.\n",
"INFO:tensorflow:Done running local_init_op.\n",
"I0504 00:55:45.104933 140215059400576 session_manager.py:502] Done running local_init_op.\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:Performing evaluation on 2568 images.\n",
"I0504 00:56:52.510566 140210415736576 coco_evaluation.py:293] Performing evaluation on 2568 images.\n",
"creating index...\n",
"index created!\n",
"INFO:tensorflow:Loading and preparing annotation results...\n",
"I0504 00:56:52.518196 140210415736576 coco_tools.py:116] Loading and preparing annotation results...\n",
"INFO:tensorflow:DONE (t=0.19s)\n",
"I0504 00:56:52.708378 140210415736576 coco_tools.py:138] DONE (t=0.19s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=19.40s).\n",
"Accumulating evaluation results...\n",
"DONE (t=6.77s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.842\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.971\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.954\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.252\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.578\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.874\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.881\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.885\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.885\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.250\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.765\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.905\n",
"INFO:tensorflow:Finished evaluation at 2021-05-04-00:57:19\n",
"I0504 00:57:19.849217 140215059400576 evaluation.py:275] Finished evaluation at 2021-05-04-00:57:19\n",
"INFO:tensorflow:Saving dict for global step 4838: DetectionBoxes_Precision/mAP = 0.8417523, DetectionBoxes_Precision/mAP (large) = 0.87389356, DetectionBoxes_Precision/mAP (medium) = 0.5784151, DetectionBoxes_Precision/mAP (small) = 0.25247526, DetectionBoxes_Precision/mAP@.50IOU = 0.9710778, DetectionBoxes_Precision/mAP@.75IOU = 0.9540878, DetectionBoxes_Recall/AR@1 = 0.88101786, DetectionBoxes_Recall/AR@10 = 0.88519293, DetectionBoxes_Recall/AR@100 = 0.88519293, DetectionBoxes_Recall/AR@100 (large) = 0.9049079, DetectionBoxes_Recall/AR@100 (medium) = 0.7649073, DetectionBoxes_Recall/AR@100 (small) = 0.25, Loss/classification_loss = 0.1509938, Loss/localization_loss = 0.025918532, Loss/regularization_loss = 0.21599802, Loss/total_loss = 0.39290625, global_step = 4838, learning_rate = 0.03999498, loss = 0.39290625\n",
"I0504 00:57:19.849468 140215059400576 estimator.py:2049] Saving dict for global step 4838: DetectionBoxes_Precision/mAP = 0.8417523, DetectionBoxes_Precision/mAP (large) = 0.87389356, DetectionBoxes_Precision/mAP (medium) = 0.5784151, DetectionBoxes_Precision/mAP (small) = 0.25247526, DetectionBoxes_Precision/mAP@.50IOU = 0.9710778, DetectionBoxes_Precision/mAP@.75IOU = 0.9540878, DetectionBoxes_Recall/AR@1 = 0.88101786, DetectionBoxes_Recall/AR@10 = 0.88519293, DetectionBoxes_Recall/AR@100 = 0.88519293, DetectionBoxes_Recall/AR@100 (large) = 0.9049079, DetectionBoxes_Recall/AR@100 (medium) = 0.7649073, DetectionBoxes_Recall/AR@100 (small) = 0.25, Loss/classification_loss = 0.1509938, Loss/localization_loss = 0.025918532, Loss/regularization_loss = 0.21599802, Loss/total_loss = 0.39290625, global_step = 4838, learning_rate = 0.03999498, loss = 0.39290625\n",
"INFO:tensorflow:Saving 'checkpoint_path' summary for global step 4838: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-4838\n",
"I0504 00:57:19.853150 140215059400576 estimator.py:2109] Saving 'checkpoint_path' summary for global step 4838: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-4838\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:global_step/sec: 0.41944\n",
"I0504 00:58:44.863583 140215059400576 basic_session_run_hooks.py:692] global_step/sec: 0.41944\n",
"INFO:tensorflow:loss = 0.35798532, step = 4900 (238.413 sec)\n",
"I0504 00:58:44.864837 140215059400576 basic_session_run_hooks.py:260] loss = 0.35798532, step = 4900 (238.413 sec)\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:Saving checkpoints for 5000 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"I0504 01:00:58.754750 140215059400576 basic_session_run_hooks.py:606] Saving checkpoints for 5000 into /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt.\n",
"INFO:tensorflow:Skip the current checkpoint eval due to throttle secs (600 secs).\n",
"I0504 01:01:00.836331 140215059400576 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).\n",
"INFO:tensorflow:Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0504 01:01:00.861598 140215059400576 dataset_builder.py:163] Reading unweighted datasets: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"I0504 01:01:00.862543 140215059400576 dataset_builder.py:80] Reading record datasets for input file: ['/content/data/tf_record/pet_faces_train.record-?????-of-00010']\n",
"INFO:tensorflow:Number of filenames to read: 10\n",
"I0504 01:01:00.862663 140215059400576 dataset_builder.py:81] Number of filenames to read: 10\n",
"WARNING:tensorflow:Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f806346ff10>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"W0504 01:01:00.899821 140215059400576 ag_logging.py:146] Entity <bound method TfExampleDecoder.decode of <object_detection.data_decoders.tf_example_decoder.TfExampleDecoder object at 0x7f806346ff10>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
"WARNING:tensorflow:Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7f852836dcb0> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"W0504 01:01:01.059637 140215059400576 ag_logging.py:146] Entity <function eval_input.<locals>.transform_and_pad_input_data_fn at 0x7f852836dcb0> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Bad argument number for Name: 3, expecting 4\n",
"INFO:tensorflow:Calling model_fn.\n",
"I0504 01:01:01.534383 140215059400576 estimator.py:1148] Calling model_fn.\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 01:01:03.650071 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 01:01:03.720171 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 01:01:03.787953 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 01:01:03.856566 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 01:01:03.924641 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 01:01:03.994073 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:Done calling model_fn.\n",
"I0504 01:01:06.120358 140215059400576 estimator.py:1150] Done calling model_fn.\n",
"INFO:tensorflow:Starting evaluation at 2021-05-04T01:01:06Z\n",
"I0504 01:01:06.135504 140215059400576 evaluation.py:255] Starting evaluation at 2021-05-04T01:01:06Z\n",
"INFO:tensorflow:Graph was finalized.\n",
"I0504 01:01:06.595707 140215059400576 monitored_session.py:240] Graph was finalized.\n",
"2021-05-04 01:01:06.596335: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 01:01:06.596643: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: \n",
"name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285\n",
"pciBusID: 0000:00:04.0\n",
"2021-05-04 01:01:06.596733: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
"2021-05-04 01:01:06.596753: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n",
"2021-05-04 01:01:06.596766: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10\n",
"2021-05-04 01:01:06.596781: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10\n",
"2021-05-04 01:01:06.596795: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10\n",
"2021-05-04 01:01:06.596810: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10\n",
"2021-05-04 01:01:06.596825: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
"2021-05-04 01:01:06.596890: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 01:01:06.597178: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 01:01:06.597396: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0\n",
"2021-05-04 01:01:06.597438: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
"2021-05-04 01:01:06.597449: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 \n",
"2021-05-04 01:01:06.597456: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N \n",
"2021-05-04 01:01:06.597543: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 01:01:06.597787: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 01:01:06.597999: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15224 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0)\n",
"INFO:tensorflow:Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-5000\n",
"I0504 01:01:06.601448 140215059400576 saver.py:1284] Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-5000\n",
"INFO:tensorflow:Running local_init_op.\n",
"I0504 01:01:07.712863 140215059400576 session_manager.py:500] Running local_init_op.\n",
"INFO:tensorflow:Done running local_init_op.\n",
"I0504 01:01:07.899731 140215059400576 session_manager.py:502] Done running local_init_op.\n",
"Corrupt JPEG data: premature end of data segment\n",
"INFO:tensorflow:Performing evaluation on 2568 images.\n",
"I0504 01:02:14.895640 140210432521984 coco_evaluation.py:293] Performing evaluation on 2568 images.\n",
"creating index...\n",
"index created!\n",
"INFO:tensorflow:Loading and preparing annotation results...\n",
"I0504 01:02:14.906803 140210432521984 coco_tools.py:116] Loading and preparing annotation results...\n",
"INFO:tensorflow:DONE (t=0.21s)\n",
"I0504 01:02:15.115651 140210432521984 coco_tools.py:138] DONE (t=0.21s)\n",
"creating index...\n",
"index created!\n",
"Running per image evaluation...\n",
"Evaluate annotation type *bbox*\n",
"DONE (t=19.63s).\n",
"Accumulating evaluation results...\n",
"DONE (t=6.65s).\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.844\n",
" Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.965\n",
" Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.957\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.217\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.594\n",
" Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.877\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.891\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.893\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.893\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.250\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.787\n",
" Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.913\n",
"INFO:tensorflow:Finished evaluation at 2021-05-04-01:02:42\n",
"I0504 01:02:42.384744 140215059400576 evaluation.py:275] Finished evaluation at 2021-05-04-01:02:42\n",
"INFO:tensorflow:Saving dict for global step 5000: DetectionBoxes_Precision/mAP = 0.84409505, DetectionBoxes_Precision/mAP (large) = 0.87721324, DetectionBoxes_Precision/mAP (medium) = 0.59417856, DetectionBoxes_Precision/mAP (small) = 0.21749175, DetectionBoxes_Precision/mAP@.50IOU = 0.9647228, DetectionBoxes_Precision/mAP@.75IOU = 0.95705926, DetectionBoxes_Recall/AR@1 = 0.8907396, DetectionBoxes_Recall/AR@10 = 0.89296573, DetectionBoxes_Recall/AR@100 = 0.89296573, DetectionBoxes_Recall/AR@100 (large) = 0.91329783, DetectionBoxes_Recall/AR@100 (medium) = 0.7871724, DetectionBoxes_Recall/AR@100 (small) = 0.25, Loss/classification_loss = 0.14837511, Loss/localization_loss = 0.02272812, Loss/regularization_loss = 0.2152484, Loss/total_loss = 0.3863538, global_step = 5000, learning_rate = 0.039994393, loss = 0.3863538\n",
"I0504 01:02:42.385046 140215059400576 estimator.py:2049] Saving dict for global step 5000: DetectionBoxes_Precision/mAP = 0.84409505, DetectionBoxes_Precision/mAP (large) = 0.87721324, DetectionBoxes_Precision/mAP (medium) = 0.59417856, DetectionBoxes_Precision/mAP (small) = 0.21749175, DetectionBoxes_Precision/mAP@.50IOU = 0.9647228, DetectionBoxes_Precision/mAP@.75IOU = 0.95705926, DetectionBoxes_Recall/AR@1 = 0.8907396, DetectionBoxes_Recall/AR@10 = 0.89296573, DetectionBoxes_Recall/AR@100 = 0.89296573, DetectionBoxes_Recall/AR@100 (large) = 0.91329783, DetectionBoxes_Recall/AR@100 (medium) = 0.7871724, DetectionBoxes_Recall/AR@100 (small) = 0.25, Loss/classification_loss = 0.14837511, Loss/localization_loss = 0.02272812, Loss/regularization_loss = 0.2152484, Loss/total_loss = 0.3863538, global_step = 5000, learning_rate = 0.039994393, loss = 0.3863538\n",
"INFO:tensorflow:Saving 'checkpoint_path' summary for global step 5000: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-5000\n",
"I0504 01:02:42.388940 140215059400576 estimator.py:2109] Saving 'checkpoint_path' summary for global step 5000: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-5000\n",
"INFO:tensorflow:Performing the final export in the end of training.\n",
"I0504 01:02:42.389613 140215059400576 exporter.py:410] Performing the final export in the end of training.\n",
"INFO:tensorflow:Calling model_fn.\n",
"I0504 01:02:42.629983 140215059400576 estimator.py:1148] Calling model_fn.\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 01:02:44.649070 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 01:02:44.721361 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 01:02:44.788778 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 01:02:44.856194 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 01:02:44.928780 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:depth of additional conv before box predictor: 0\n",
"I0504 01:02:45.000893 140215059400576 convolutional_box_predictor.py:156] depth of additional conv before box predictor: 0\n",
"INFO:tensorflow:Done calling model_fn.\n",
"I0504 01:02:46.088902 140215059400576 estimator.py:1150] Done calling model_fn.\n",
"WARNING:tensorflow:From /tensorflow-1.15.2/python3.7/tensorflow_core/python/saved_model/signature_def_utils_impl.py:201: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.\n",
"W0504 01:02:46.089168 140215059400576 deprecation.py:323] From /tensorflow-1.15.2/python3.7/tensorflow_core/python/saved_model/signature_def_utils_impl.py:201: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.\n",
"Instructions for updating:\n",
"This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.\n",
"INFO:tensorflow:Signatures INCLUDED in export for Classify: None\n",
"I0504 01:02:46.089719 140215059400576 export_utils.py:170] Signatures INCLUDED in export for Classify: None\n",
"INFO:tensorflow:Signatures INCLUDED in export for Regress: None\n",
"I0504 01:02:46.089831 140215059400576 export_utils.py:170] Signatures INCLUDED in export for Regress: None\n",
"INFO:tensorflow:Signatures INCLUDED in export for Predict: ['tensorflow/serving/predict', 'serving_default']\n",
"I0504 01:02:46.089886 140215059400576 export_utils.py:170] Signatures INCLUDED in export for Predict: ['tensorflow/serving/predict', 'serving_default']\n",
"INFO:tensorflow:Signatures INCLUDED in export for Train: None\n",
"I0504 01:02:46.089935 140215059400576 export_utils.py:170] Signatures INCLUDED in export for Train: None\n",
"INFO:tensorflow:Signatures INCLUDED in export for Eval: None\n",
"I0504 01:02:46.089978 140215059400576 export_utils.py:170] Signatures INCLUDED in export for Eval: None\n",
"2021-05-04 01:02:46.090425: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 01:02:46.090723: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: \n",
"name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285\n",
"pciBusID: 0000:00:04.0\n",
"2021-05-04 01:02:46.090814: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
"2021-05-04 01:02:46.090834: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10\n",
"2021-05-04 01:02:46.090849: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10\n",
"2021-05-04 01:02:46.090864: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10\n",
"2021-05-04 01:02:46.090878: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10\n",
"2021-05-04 01:02:46.090892: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10\n",
"2021-05-04 01:02:46.090906: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7\n",
"2021-05-04 01:02:46.090973: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 01:02:46.091218: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 01:02:46.091406: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0\n",
"2021-05-04 01:02:46.091445: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
"2021-05-04 01:02:46.091475: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186] 0 \n",
"2021-05-04 01:02:46.091481: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0: N \n",
"2021-05-04 01:02:46.091562: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 01:02:46.091784: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
"2021-05-04 01:02:46.091988: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15224 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0)\n",
"INFO:tensorflow:Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-5000\n",
"I0504 01:02:46.094195 140215059400576 saver.py:1284] Restoring parameters from /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/model.ckpt-5000\n",
"INFO:tensorflow:Assets added to graph.\n",
"I0504 01:02:46.613681 140215059400576 builder_impl.py:665] Assets added to graph.\n",
"INFO:tensorflow:No assets to write.\n",
"I0504 01:02:46.613869 140215059400576 builder_impl.py:460] No assets to write.\n",
"INFO:tensorflow:SavedModel written to: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/export/Servo/temp-b'1620090162'/saved_model.pb\n",
"I0504 01:02:47.470509 140215059400576 builder_impl.py:425] SavedModel written to: /content/data/train_ssdlite_mobilenet_v3_large_320x320_pet/export/Servo/temp-b'1620090162'/saved_model.pb\n",
"INFO:tensorflow:Loss for final step: 0.3643322.\n",
"I0504 01:02:47.987567 140215059400576 estimator.py:371] Loss for final step: 0.3643322.\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"colab": {
"resources": {
"http://localhost:6006/": {
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment