changtimwu/openvino_quantization_example.md

## openvino_quantization_example.md

      
    Raw
  

              openvino_quantization_example.md
            
          
    Accuracy Checker for TBH, Yolo v4 (MR1)


Setup
Check the accuracy of original Tensorflow model
Check the accuracy of OpenVino FP32 IR
Check the accuracy of OpenVino INT8 IR
Use POT for accuracy aware quantization

Setup


Install OpenVINO
Please refer “Compile IR to blobs for TBH (MR1)” to download and install public OpenVINO 2021.3.394 https://docs.google.com/document/d/1BK_7PTMN8Y7miyTAVbdoiBrKnPrrBEraOsHl75-S4hQ/edit?usp=sharing

Prepare Pre-Trained Models

Follow Scenario 2: Build blobs from Open Model Zoo IRs (Yolo v4 TensorFlow) to download and convert Yolo v4.  You should see Yolo v4 models in the following directory.

TensorFlow FP32 protobuf:

/opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/downloader/public/yolo-v4-tf/yolo-v4.pb


OpenVINO FP32 IR:

/opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/downloader/public/yolo-v4-tf/FP32/yolo-v4-tf.xml
/opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/downloader/public/yolo-v4-tf/FP32/yolo-v4-tf.bin


OpenVINO INT8 IR (quantized by POT):

/opt/intel/openvino_2021.3.394/deployment_tools/tools/post_training_optimization_toolkit/results/yolo-v4-tf_DefaultQuantization/2022-04-07_17-00-41/optimized/yolo-v4-tf.xml 
/opt/intel/openvino_2021.3.394/deployment_tools/tools/post_training_optimization_toolkit/results/yolo-v4-tf_DefaultQuantization/2022-04-07_17-00-41/optimized/yolo-v4-tf.bin

Prepare Dataset

Yolo v4 is pretrained on COCO dataset. see link
We need to download COCO dataset for accuracy checker. To reduce checking time, we download the smallest dataset (5000 images).
https://cocodataset.org/#download

Images: 2017 Val images [5K/1GB]

Download http://images.cocodataset.org/zips/val2017.zip
Decompress val2017.zip under your working directory (can be changed later in .json). Here we decompress it under /home/intel/data,all the .jpg files will appear under /home/intel/data/val2017


Annotations: 2017 Train/Val annotations [241MB]

Download http://images.cocodataset.org/annotations/annotations_trainval2017.zip
Decompress it then put “instances_val2017.json” into the directory contains the .jpg files.  Here we put it in /home/intel/data/val2017


Scenario 1: Check the accuracy of original TensorFlow model

Prepare yolo-v4-tf_pb_fp32_ac.yml
The content of yolo-v4-tf_pb_fp32_ac.yml is as follows
models:
  - name: yolo-v4-tf
	launchers:
#下面指定运行的框架 dlsdk就是openvino， 也可以是tensorflow或者其他框架
#model/weights是要测试的模型文件名
#adapter是告诉accuracy checker模型的输出是目标分类还是目标识别或者其他的输出
  	- framework: tf
    	model:   /home/intel/models/yolo-v4.pb
    	adapter:
      	type: yolo_v3
      	anchors: 12,16,19,36,40,28,36,75,76,55,72,146,142,110,192,243,459,401
      	num: 9
      	coords: 4
      	classes: 80
      	threshold: 0.001
      	anchor_masks: [[0, 1, 2], [3, 4, 5], [6, 7, 8]]
      	raw_output: True
      	output_format: HWB
      	do_reshape: True
      	cells: [76, 38, 19]
      	outputs:
        	- conv2d_93/BiasAdd
        	- conv2d_101/BiasAdd
        	- conv2d_109/BiasAdd
    	inputs:
      	- name: 'image_input'
        	type: INPUT
        	shape: [1, 608, 608, 3]
    	outputs:
      	- conv2d_93/BiasAdd
      	- conv2d_101/BiasAdd
      	- conv2d_109/BiasAdd
	datasets:
#下面这部分是有关数据集的路径，数据集的格式(imagenet，VOC或者其他)以及标注文件的名字
  	- name: ms_coco_detection_80_class_without_background
    	data_source: /home/intel/data/val2017
    	annotation_conversion:
      	converter: mscoco_detection
      	annotation_file: "/home/intel/data/val2017/instances_val2017.json"
#预处理告诉ac_checker工具在把数据集图片送进模型前要做那些预处理，比如缩放，剪切或者调整RGB/BGR顺序之类
    	preprocessing:
      	- type: bgr_to_rgb
      	- type: normalization
        	std: 255.0
      	- type: resize
        	size: 608
    	postprocessing:
      	- type: resize_prediction_boxes
      	- type: filter
        	apply_to: prediction
        	min_confidence: 0.001
        	remove_filtered: true
      	- type: nms
        	overlap: 0.5
      	- type: clip_boxes
        	apply_to: prediction
    	metrics:
      	- type: map
        	integral: 11point
        	ignore_difficult: true
        	presenter: print_scalar
      	- name: AP@0.5
        	type: coco_precision
        	max_detections: 100
        	threshold: 0.5
      	- name: AP@0.5:0.05:95
        	type: coco_precision
        	max_detections: 100
        	threshold: '0.5:0.05:0.95'
accuracy_check command session
intel@intel-ATX-Q370A:/opt/intel/openvino_2021.3.394/deployment_tools/tools/post_training_optimization_toolkit$ cd /opt/intel/thb_openvino_mr1/models
intel@intel-ATX-Q370A:~/models$ accuracy_check -c yolo-v4-tf_pb_fp32_ac.yml -s ./ -td CPU
10:16:28 accuracy_checker WARNING: /usr/local/lib/python3.8/dist-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning: 0.23ubuntu1 is an invalid version and will not be supported in a future release
  warnings.warn( 
10:16:28 accuracy_checker WARNING: /usr/local/lib/python3.8/dist-packages/pkg_resources/_vendor/packaging/version.py:111: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release
  warnings.warn(
10:16:28 accuracy_checker WARNING: /usr/local/lib/python3.8/dist-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning: 0.1.36ubuntu1 is an invalid version and will not be supported in a future release
  warnings.warn(
10:16:28 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/python/python3.8/ngraph/utils/types.py:37: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  (NgraphType.boolean, np.bool),

Processing info:
model: yolo-v4-tf
launcher: tf
device: CPU
dataset: ms_coco_detection_80_class_without_background
OpenCV version: 4.5.2-openvino
Annotation conversion for ms_coco_detection_80_class_without_background dataset has been started
Parameters to be used for conversion:
converter: mscoco_detection
annotation_file: /home/intel/data/val2017/instances_val2017.json
Total annotations size: 5000
100 / 5000 processed in 0.539s
200 / 5000 processed in 0.540s
300 / 5000 processed in 0.542s
400 / 5000 processed in 0.540s
500 / 5000 processed in 0.544s
600 / 5000 processed in 0.536s
700 / 5000 processed in 0.535s
800 / 5000 processed in 0.547s
900 / 5000 processed in 0.556s
1000 / 5000 processed in 0.558s
1100 / 5000 processed in 0.556s
1200 / 5000 processed in 0.557s
1300 / 5000 processed in 0.557s
1400 / 5000 processed in 0.558s
1500 / 5000 processed in 0.561s
1600 / 5000 processed in 0.624s
1700 / 5000 processed in 0.563s
1800 / 5000 processed in 0.554s
1900 / 5000 processed in 0.562s
2000 / 5000 processed in 0.567s
2100 / 5000 processed in 0.568s
2200 / 5000 processed in 0.568s
2300 / 5000 processed in 0.571s
2400 / 5000 processed in 0.568s
2500 / 5000 processed in 0.570s
2600 / 5000 processed in 0.567s
2700 / 5000 processed in 0.572s
2800 / 5000 processed in 0.566s
2900 / 5000 processed in 0.569s
3000 / 5000 processed in 0.569s
3100 / 5000 processed in 0.575s
3200 / 5000 processed in 0.569s
3300 / 5000 processed in 0.563s
3400 / 5000 processed in 0.554s
3500 / 5000 processed in 0.556s
3600 / 5000 processed in 0.559s
3700 / 5000 processed in 0.561s
3800 / 5000 processed in 0.548s
3900 / 5000 processed in 0.554s
4000 / 5000 processed in 0.555s
4100 / 5000 processed in 0.557s
4200 / 5000 processed in 0.552s
4300 / 5000 processed in 0.549s
4400 / 5000 processed in 0.541s
4500 / 5000 processed in 0.543s
4600 / 5000 processed in 0.541s
4700 / 5000 processed in 0.541s
4800 / 5000 processed in 0.534s
4900 / 5000 processed in 0.529s
5000 / 5000 processed in 0.530s
5000 objects processed in 27.793 seconds
Annotation conversion for ms_coco_detection_80_class_without_background dataset has been finished
10:16:57 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:224: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
  'NEAREST': Image.NEAREST,
10:16:57 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:225: DeprecationWarning: NONE is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
  'NONE': Image.NONE,
10:16:57 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:226: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
  'BILINEAR': Image.BILINEAR,
10:16:57 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:227: DeprecationWarning: LINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
  'LINEAR': Image.LINEAR,
10:16:57 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:228: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
  'BICUBIC': Image.BICUBIC,
10:16:57 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:229: DeprecationWarning: CUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
  'CUBIC': Image.CUBIC,
10:16:57 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:230: DeprecationWarning: ANTIALIAS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
  'ANTIALIAS': Image.ANTIALIAS
10:16:57 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:234: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.
  'BOX': Image.BOX,
10:16:57 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:235: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
  'LANCZOS': Image.LANCZOS,
10:16:57 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:236: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.
  'HAMMING': Image.HAMMING,
2022-04-20 10:16:57.700551: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/intel/openvino_2021.3.394/data_processing/dl_streamer/lib:/opt/intel/openvino_2021.3.394/data_processing/gstreamer/lib:/opt/intel/openvino_2021.3.394/opencv/lib:/opt/intel/openvino_2021.3.394/deployment_tools/ngraph/lib:/opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/external/tbb/lib::/opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/external/hddl/lib:/opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/external/omp/lib:/opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/external/gna/lib:/opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/external/mkltiny_lnx/lib:/opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/lib/intel64:/opt/intel/openvino_2021.3.394/data_processing/dl_streamer/lib:/opt/intel/openvino_2021.3.394/data_processing/gstreamer/lib:/opt/intel/openvino_2021.3.394/opencv/lib:/opt/intel/openvino_2021.3.394/deployment_tools/ngraph/lib:/opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/external/tbb/lib::/opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/external/hddl/lib:/opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/external/omp/lib:/opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/external/gna/lib:/opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/external/mkltiny_lnx/lib:/opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/lib/intel64
2022-04-20 10:16:57.700630: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
10:16:59 accuracy_checker WARNING: /home/intel/.local/lib/python3.8/site-packages/flatbuffers/compat.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
10:17:00 accuracy_checker WARNING: /home/intel/.local/lib/python3.8/site-packages/pandas/compat/__init__.py:437: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if LooseVersion(dateutil.__version__) < LooseVersion('2.5'):
10:17:00 accuracy_checker WARNING: /home/intel/.local/lib/python3.8/site-packages/pandas/compat/numpy/__init__.py:11: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  _nlv = LooseVersion(_np_version)
10:17:00 accuracy_checker WARNING: /usr/local/lib/python3.8/dist-packages/setuptools/_distutils/version.py:351: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  other = LooseVersion(other)
10:17:00 accuracy_checker WARNING: /home/intel/.local/lib/python3.8/site-packages/pandas/util/testing.py:20: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  from pandas._libs import testing as _testing
10:17:00 accuracy_checker WARNING: /home/intel/.local/lib/python3.8/site-packages/keras_preprocessing/image/utils.py:23: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
  'nearest': pil_image.NEAREST,
10:17:00 accuracy_checker WARNING: /home/intel/.local/lib/python3.8/site-packages/keras_preprocessing/image/utils.py:24: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
  'bilinear': pil_image.BILINEAR,
10:17:00 accuracy_checker WARNING: /home/intel/.local/lib/python3.8/site-packages/keras_preprocessing/image/utils.py:25: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
  'bicubic': pil_image.BICUBIC,
10:17:00 accuracy_checker WARNING: /home/intel/.local/lib/python3.8/site-packages/keras_preprocessing/image/utils.py:28: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.
  if hasattr(pil_image, 'HAMMING'):
10:17:00 accuracy_checker WARNING: /home/intel/.local/lib/python3.8/site-packages/keras_preprocessing/image/utils.py:30: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.
  if hasattr(pil_image, 'BOX'):
10:17:00 accuracy_checker WARNING: /home/intel/.local/lib/python3.8/site-packages/keras_preprocessing/image/utils.py:33: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
  if hasattr(pil_image, 'LANCZOS'):
10:17:00 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/config/config_validator.py:164: UserWarning: launcher.tf specifies unknown options: ['outputs']
  warnings.warn(message)
  0%|                                                                                                                                                                                    | 0/5000 [00:00<?]2022-04-20 10:17:03.484453: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/intel/openvino_2021.3.394/data_processing/dl_streamer/lib:/opt/intel/openvino_2021.3.394/data_processing/gstreamer/lib:/opt/intel/openvino_2021.3.394/opencv/lib:/opt/intel/openvino_2021.3.394/deployment_tools/ngraph/lib:/opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/external/tbb/lib::/opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/external/hddl/lib:/opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/external/omp/lib:/opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/external/gna/lib:/opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/external/mkltiny_lnx/lib:/opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/lib/intel64:/opt/intel/openvino_2021.3.394/data_processing/dl_streamer/lib:/opt/intel/openvino_2021.3.394/data_processing/gstreamer/lib:/opt/intel/openvino_2021.3.394/opencv/lib:/opt/intel/openvino_2021.3.394/deployment_tools/ngraph/lib:/opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/external/tbb/lib::/opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/external/hddl/lib:/opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/external/omp/lib:/opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/external/gna/lib:/opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/external/mkltiny_lnx/lib:/opt/intel/openvino_2021.3.394/deployment_tools/inference_engine/lib/intel64
2022-04-20 10:17:03.484488: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
2022-04-20 10:17:03.484510: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (intel-ATX-Q370A): /proc/driver/nvidia/version does not exist
2022-04-20 10:17:03.485646: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
10:17:05 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/adapters/yolo.py:373: RuntimeWarning: overflow encountered in exp
  prob_correct=lambda x: 1.0 / (1.0 + np.exp(-x)))
10:17:05 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/metrics/coco_metrics.py:344: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  tp_sum = np.cumsum(tps, axis=1).astype(dtype=np.float)
5000 objects processed in 8090.572 seconds                                                                                                                                                                 
map: 71.16%
AP@0.5: 75.04%
AP@0.5:0.05:95: 49.21%
intel@intel-ATX-Q370A:~/models$

Scenario 2: Check the accuracy of OpenVINO FP32 IR

Prepare yolo-v4-tf_fp32_ac.yml
The content of yolo-v4-tf_fp32_ac.yml is as follows
models:
  - name: yolo-v4-tf
	launchers:
#下面指定运行的框架 dlsdk就是openvino， 也可以是tensorflow或者其他框架
#model/weights是要测试的模型文件名
#adapter是告诉accuracy checker模型的输出是目标分类还是目标识别或者其他的输出
  	- framework: dlsdk
    	model:   /home/intel/models/yolo-v4-tf.xml
    	weights: /home/intel/models/yolo-v4-tf.bin
    	adapter:
      	type: yolo_v3
      	anchors: 12,16,19,36,40,28,36,75,76,55,72,146,142,110,192,243,459,401
      	num: 3
      	coords: 4
      	classes: 80
      	threshold: 0.001
      	anchor_masks: [[0, 1, 2], [3, 4, 5], [6, 7, 8]]
      	raw_output: True
      	outputs:
        	- conv2d_93/BiasAdd/Add
        	- conv2d_101/BiasAdd/Add
        	- conv2d_109/BiasAdd/Add
	datasets:
#下面这部分是有关数据集的路径，数据集的格式(imagenet，VOC或者其他)以及标注文件的名字
  	- name: ms_coco_detection_80_class_without_background
    	data_source: /home/intel/data/val2017
    	annotation_conversion:
      	converter: mscoco_detection
      	annotation_file: "/home/intel/data/val2017/instances_val2017.json"
#预处理告诉ac_checker工具在把数据集图片送进模型前要做那些预处理，比如缩放，剪切或者调整RGB/BGR顺序之类
    	preprocessing:
      	- type: resize
        	size: 608
    	postprocessing:
      	- type: resize_prediction_boxes
      	- type: filter
        	apply_to: prediction
        	min_confidence: 0.001
        	remove_filtered: true
      	- type: nms
        	overlap: 0.5
      	- type: clip_boxes
        	apply_to: prediction
    	metrics:
      	- type: map
        	integral: 11point
        	ignore_difficult: true
        	presenter: print_scalar
      	- name: AP@0.5
        	type: coco_precision
        	max_detections: 100
        	threshold: 0.5
      	- name: AP@0.5:0.05:95
        	type: coco_precision
        	max_detections: 100
        	threshold: '0.5:0.05:0.95'
the accuracy_check command session
intel@intel-ATX-Q370A:~/models$ accuracy_check -c yolo-v4-tf_fp32_ac.yml -s ./ -td CPU
18:05:14 accuracy_checker WARNING: /usr/local/lib/python3.8/dist-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning: 0.23ubuntu1 is an invalid version and will not be supported in a future release
  warnings.warn(
18:05:14 accuracy_checker WARNING: /usr/local/lib/python3.8/dist-packages/pkg_resources/_vendor/packaging/version.py:111: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release
  warnings.warn(
18:05:14 accuracy_checker WARNING: /usr/local/lib/python3.8/dist-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning: 0.1.36ubuntu1 is an invalid version and will not be supported in a future release
  warnings.warn(
18:05:14 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/python/python3.8/ngraph/utils/types.py:37: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  (NgraphType.boolean, np.bool),
Processing info:
model: yolo-v4-tf
launcher: dlsdk
device: CPU
dataset: ms_coco_detection_80_class_without_background
OpenCV version: 4.5.2-openvino
Annotation conversion for ms_coco_detection_80_class_without_background dataset has been started
Parameters to be used for conversion:
converter: mscoco_detection
annotation_file: /home/intel/data/val2017/instances_val2017.json
Total annotations size: 5000
100 / 5000 processed in 0.529s
200 / 5000 processed in 0.532s
300 / 5000 processed in 0.536s
400 / 5000 processed in 0.535s
500 / 5000 processed in 0.533s
600 / 5000 processed in 0.534s
700 / 5000 processed in 0.535s
800 / 5000 processed in 0.542s
900 / 5000 processed in 0.542s
1000 / 5000 processed in 0.544s
1100 / 5000 processed in 0.542s
1200 / 5000 processed in 0.548s
1300 / 5000 processed in 0.548s
1400 / 5000 processed in 0.549s
1500 / 5000 processed in 0.548s
1600 / 5000 processed in 0.540s
1700 / 5000 processed in 0.548s
1800 / 5000 processed in 0.606s
1900 / 5000 processed in 0.545s
2000 / 5000 processed in 0.556s
2100 / 5000 processed in 0.559s
2200 / 5000 processed in 0.561s
2300 / 5000 processed in 0.560s
2400 / 5000 processed in 0.558s
2500 / 5000 processed in 0.556s
2600 / 5000 processed in 0.556s
2700 / 5000 processed in 0.558s
2800 / 5000 processed in 0.557s
2900 / 5000 processed in 0.559s
3000 / 5000 processed in 0.552s
3100 / 5000 processed in 0.543s
3200 / 5000 processed in 0.546s
3300 / 5000 processed in 0.547s
3400 / 5000 processed in 0.552s
3500 / 5000 processed in 0.554s
3600 / 5000 processed in 0.549s
3700 / 5000 processed in 0.552s
3800 / 5000 processed in 0.548s
3900 / 5000 processed in 0.537s
4000 / 5000 processed in 0.536s
4100 / 5000 processed in 0.543s
4200 / 5000 processed in 0.546s
4300 / 5000 processed in 0.539s
4400 / 5000 processed in 0.539s
4500 / 5000 processed in 0.536s
4600 / 5000 processed in 0.532s
4700 / 5000 processed in 0.529s
4800 / 5000 processed in 0.535s
4900 / 5000 processed in 0.522s
5000 / 5000 processed in 0.532s
5000 objects processed in 27.286 seconds
Annotation conversion for ms_coco_detection_80_class_without_background dataset has been finished
18:05:42 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:224: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
  'NEAREST': Image.NEAREST,
18:05:42 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:225: DeprecationWarning: NONE is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
  'NONE': Image.NONE,
18:05:42 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:226: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
  'BILINEAR': Image.BILINEAR,
18:05:42 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:227: DeprecationWarning: LINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
  'LINEAR': Image.LINEAR,
18:05:42 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:228: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
  'BICUBIC': Image.BICUBIC,
18:05:42 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:229: DeprecationWarning: CUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
  'CUBIC': Image.CUBIC,
18:05:42 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:230: DeprecationWarning: ANTIALIAS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
  'ANTIALIAS': Image.ANTIALIAS
18:05:42 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:234: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.
  'BOX': Image.BOX,
18:05:42 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:235: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
  'LANCZOS': Image.LANCZOS,
18:05:42 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:236: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.
  'HAMMING': Image.HAMMING,
IE version: 2.1.2021.3.0-2787-60059f2c755-releases/2021/3
Loaded CPU plugin version:
	CPU - MKLDNNPlugin: 2.1.2021.3.0-2787-60059f2c755-releases/2021/3
Input info:
              Layer name: image_input
              precision: FP32
              shape [1, 3, 608, 608]
Output info
              Layer name: conv2d_101/BiasAdd/Add
              precision: FP32
              shape: [1, 255, 38, 38]
 
              Layer name: conv2d_109/BiasAdd/Add
              precision: FP32
              shape: [1, 255, 19, 19]
 
              Layer name: conv2d_93/BiasAdd/Add
              precision: FP32
              shape: [1, 255, 76, 76]
 
  0%|                                                                                                                                                                                    | 0/5000 [00:00<?]18:05:43 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/adapters/yolo.py:373: RuntimeWarning: overflow encountered in exp
  prob_correct=lambda x: 1.0 / (1.0 + np.exp(-x)))
 
18:05:43 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/metrics/coco_metrics.py:344: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  tp_sum = np.cumsum(tps, axis=1).astype(dtype=np.float) 
5000 objects processed in 4135.110 seconds                                                                                                                                                                
map: 71.17%
AP@0.5: 75.02%
AP@0.5:0.05:95: 49.20%

Scenario 3: Check the accuracy of OpenVINO INT8 IR

Prepare yolo-v4-tf_int8_ac.yml,
The content of yolo-v4-tf_int8_ac.yml is as follows
models:
  - name: yolo-v4-tf
	launchers:
#下面指定运行的框架 dlsdk就是openvino， 也可以是tensorflow或者其他框架
#model/weights是要测试的模型文件名
#adapter是告诉accuracy checker模型的输出是目标分类还是目标识别或者其他的输出
  	- framework: dlsdk
    	model:   /home/intel/models/optimized/yolo-v4-tf.xml
    	weights: /home/intel/models/optimized/yolo-v4-tf.bin
    	adapter:
      	type: yolo_v3
      	anchors: 12,16,19,36,40,28,36,75,76,55,72,146,142,110,192,243,459,401
      	num: 3
      	coords: 4
      	classes: 80
      	threshold: 0.001
      	anchor_masks: [[0, 1, 2], [3, 4, 5], [6, 7, 8]]
      	raw_output: True
      	outputs:
        	- conv2d_93/BiasAdd/Add
        	- conv2d_101/BiasAdd/Add
        	- conv2d_109/BiasAdd/Add
	datasets:
#下面这部分是有关数据集的路径，数据集的格式(imagenet，VOC或者其他)以及标注文件的名字
  	- name: ms_coco_detection_80_class_without_background
    	data_source: /home/intel/data/val2017
    	annotation_conversion:
      	converter: mscoco_detection
      	annotation_file: "/home/intel/data/val2017/instances_val2017.json"
#预处理告诉ac_checker工具在把数据集图片送进模型前要做那些预处理，比如缩放，剪切或者调整RGB/BGR顺序之类
    	preprocessing:
      	- type: resize
        	size: 608
    	postprocessing:
      	- type: resize_prediction_boxes
      	- type: filter
        	apply_to: prediction
        	min_confidence: 0.001
        	remove_filtered: true
      	- type: nms
        	overlap: 0.5
      	- type: clip_boxes
        	apply_to: prediction
    	metrics:
      	- type: map
        	integral: 11point
        	ignore_difficult: true
        	presenter: print_scalar
      	- name: AP@0.5
        	type: coco_precision
        	max_detections: 100
        	threshold: 0.5
      	- name: AP@0.5:0.05:95
        	type: coco_precision
        	max_detections: 100
        	threshold: '0.5:0.05:0.95'

accuracy_check command session
intel@intel-ATX-Q370A:~/models$ accuracy_check -c yolo-v4-tf_int8_ac.yml -s ./ -td CPU
12:41:03 accuracy_checker WARNING: /usr/local/lib/python3.8/dist-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning: 0.23ubuntu1 is an invalid version and will not be supported in a future release
  warnings.warn(
12:41:03 accuracy_checker WARNING: /usr/local/lib/python3.8/dist-packages/pkg_resources/_vendor/packaging/version.py:111: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release
  warnings.warn(
12:41:03 accuracy_checker WARNING: /usr/local/lib/python3.8/dist-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning: 0.1.36ubuntu1 is an invalid version and will not be supported in a future release
  warnings.warn(
12:41:03 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/python/python3.8/ngraph/utils/types.py:37: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  (NgraphType.boolean, np.bool),
Processing info:
model: yolo-v4-tf
launcher: dlsdk
device: CPU
dataset: ms_coco_detection_80_class_without_background
OpenCV version: 4.5.2-openvino
Annotation conversion for ms_coco_detection_80_class_without_background dataset has been started
Parameters to be used for conversion:
converter: mscoco_detection
annotation_file: /home/intel/data/val2017/instances_val2017.json
Total annotations size: 5000
100 / 5000 processed in 0.545s
200 / 5000 processed in 0.545s
300 / 5000 processed in 0.547s
400 / 5000 processed in 0.548s
500 / 5000 processed in 0.551s
600 / 5000 processed in 0.551s
700 / 5000 processed in 0.550s
800 / 5000 processed in 0.547s
900 / 5000 processed in 0.557s
1000 / 5000 processed in 0.561s
1100 / 5000 processed in 0.566s
1200 / 5000 processed in 0.561s
1300 / 5000 processed in 0.564s
1400 / 5000 processed in 0.566s
1500 / 5000 processed in 0.566s
1600 / 5000 processed in 0.566s
1700 / 5000 processed in 0.567s
1800 / 5000 processed in 0.617s
1900 / 5000 processed in 0.560s
2000 / 5000 processed in 0.568s
2100 / 5000 processed in 0.569s
2200 / 5000 processed in 0.572s
2300 / 5000 processed in 0.575s
2400 / 5000 processed in 0.575s
2500 / 5000 processed in 0.574s
2600 / 5000 processed in 0.572s
2700 / 5000 processed in 0.571s
2800 / 5000 processed in 0.569s
2900 / 5000 processed in 0.564s
3000 / 5000 processed in 0.568s
3100 / 5000 processed in 0.571s
3200 / 5000 processed in 0.574s
3300 / 5000 processed in 0.567s
3400 / 5000 processed in 0.565s
3500 / 5000 processed in 0.569s
3600 / 5000 processed in 0.562s
3700 / 5000 processed in 0.555s
3800 / 5000 processed in 0.553s
3900 / 5000 processed in 0.560s
4000 / 5000 processed in 0.565s
4100 / 5000 processed in 0.557s
4200 / 5000 processed in 0.555s
4300 / 5000 processed in 0.551s
4400 / 5000 processed in 0.551s
4500 / 5000 processed in 0.551s
4600 / 5000 processed in 0.548s
4700 / 5000 processed in 0.545s
4800 / 5000 processed in 0.536s
4900 / 5000 processed in 0.538s
5000 / 5000 processed in 0.535s
5000 objects processed in 28.020 seconds
Annotation conversion for ms_coco_detection_80_class_without_background dataset has been finished
12:41:32 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:224: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
  'NEAREST': Image.NEAREST,
12:41:32 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:225: DeprecationWarning: NONE is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
  'NONE': Image.NONE,
12:41:32 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:226: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
  'BILINEAR': Image.BILINEAR,
12:41:32 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:227: DeprecationWarning: LINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
  'LINEAR': Image.LINEAR,
12:41:32 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:228: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
  'BICUBIC': Image.BICUBIC,
12:41:32 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:229: DeprecationWarning: CUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
  'CUBIC': Image.CUBIC,
12:41:32 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:230: DeprecationWarning: ANTIALIAS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
  'ANTIALIAS': Image.ANTIALIAS
12:41:32 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:234: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.
  'BOX': Image.BOX,
12:41:32 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:235: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
  'LANCZOS': Image.LANCZOS,
12:41:32 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:236: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.
  'HAMMING': Image.HAMMING,
IE version: 2.1.2021.3.0-2787-60059f2c755-releases/2021/3
Loaded CPU plugin version:
	CPU - MKLDNNPlugin: 2.1.2021.3.0-2787-60059f2c755-releases/2021/3
Input info:
              Layer name: image_input
              precision: FP32
              shape [1, 3, 608, 608]
 
Output info
              Layer name: conv2d_101/BiasAdd/Add
              precision: FP32
              shape: [1, 255, 38, 38]
 
              Layer name: conv2d_109/BiasAdd/Add
              precision: FP32
              shape: [1, 255, 19, 19]
 
              Layer name: conv2d_93/BiasAdd/Add
              precision: FP32
              shape: [1, 255, 76, 76]
 
  0%|                                                                                                                                                                                    | 0/5000 [00:00<?]12:41:33 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/metrics/coco_metrics.py:344: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  tp_sum = np.cumsum(tps, axis=1).astype(dtype=np.float)
 
5000 objects processed in 3548.627 seconds                                                                                                                                                                
map: 70.35%
AP@0.5: 74.62%
AP@0.5:0.05:95: 47.31%

Comparison accuracy of quanitized model


model format
map (%)
AP@0.5 (%)
AP@0.5:0.05:95 (%)


TensorFlow FP32
71.16
75.04
49.21


OpenVINO FP32 IR
71.17
75.02
49.2


OpenVINO INT8 IR
70.35
74.62
47.31


Use POT for accuracy aware quantization

Prepare yolo-v4-tf_ac_model.json
The content of yolo-v4-tf_ac_model.json is as follows. The criteria we set here is to allow max 2.1% accuracy drop. POT will randomly revert some layers from INT8 to FP32 then use accuracy checker to check if the accuracy can meet the criteria.
{
	"model": {
    	"model_name": "yolo-v4-tf",
    	"model": "/home/intel/models/yolo-v4-tf.xml",
    	"weights": "/home/intel/models/yolo-v4-tf.bin"
	},
 
	"engine": {
    	"type": "accuracy_checker",
    	"config": "/home/intel/models/yolo-v4-tf_fp32_ac.yml"
	},
 
	"compression": {
    	"target_device": "CPU",
    	"algorithms": [
        	{
            	"name": "AccuracyAwareQuantization",
            	"params": {
                	"preset": "performance",
                	"stat_subset_size": 300,
                	"maximal_drop": 0.021
            	}
        	}
 
 
    	]
	}
}
pot command session

intel@intel-ATX-Q370A:/opt/intel/openvino_2021.3.394/deployment_tools/tools/post_training_optimization_toolkit$ pot -c /home/intel/models/yolo-v4-tf_ac_model.json
13:10:45 accuracy_checker WARNING: /usr/local/lib/python3.8/dist-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning: 0.23ubuntu1 is an invalid version and will not be supported in a future release
  warnings.warn(
13:10:45 accuracy_checker WARNING: /usr/local/lib/python3.8/dist-packages/pkg_resources/_vendor/packaging/version.py:111: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release
  warnings.warn(
13:10:45 accuracy_checker WARNING: /usr/local/lib/python3.8/dist-packages/pkg_resources/__init__.py:123: PkgResourcesDeprecationWarning: 0.1.36ubuntu1 is an invalid version and will not be supported in a future release
  warnings.warn(
13:10:45 accuracy_checker WARNING: /home/intel/.local/lib/python3.8/site-packages/networkx/classes/graph.py:23: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
  from collections import Mapping
13:10:45 accuracy_checker WARNING: /home/intel/.local/lib/python3.8/site-packages/networkx/classes/reportviews.py:95: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working
  from collections import Mapping, Set, Iterable
13:10:45 accuracy_checker WARNING: /home/intel/.local/lib/python3.8/site-packages/pandas/compat/__init__.py:437: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if LooseVersion(dateutil.__version__) < LooseVersion('2.5'):
13:10:45 accuracy_checker WARNING: /home/intel/.local/lib/python3.8/site-packages/pandas/compat/numpy/__init__.py:11: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  _nlv = LooseVersion(_np_version)
13:10:45 accuracy_checker WARNING: /usr/local/lib/python3.8/dist-packages/setuptools/_distutils/version.py:351: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  other = LooseVersion(other)
13:10:45 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/tools/post_training_optimization_toolkit/compression/algorithms/quantization/optimization/algorithm.py:41: UserWarning: Nevergrad package could not be imported. If you are planning to useany hyperparameter optimization algo, consider installing itusing pip. This implies advanced usage of the tool.Note that nevergrad is compatible only with Python 3.6+
  warnings.warn(
INFO:app.run:Output log dir: ./results/yolo-v4-tf_AccuracyAwareQuantization/2022-04-27_13-10-45
INFO:app.run:Creating pipeline:
Algorithm: AccuracyAwareQuantization
Parameters:
              preset                     : performance
              stat_subset_size       	: 300
              maximal_drop               : 0.021
              target_device              : CPU
              model_type                 : None
              dump_intermediate_model	: False
              exec_log_dir               : ./results/yolo-v4-tf_AccuracyAwareQuantization/2022-04-27_13-10-45
===========================================================================
IE version: 2.1.2021.3.0-2787-60059f2c755-releases/2021/3
Loaded CPU plugin version:
	CPU - MKLDNNPlugin: 2.1.2021.3.0-2787-60059f2c755-releases/2021/3
Annotation conversion for ms_coco_detection_80_class_without_background dataset has been started
Parameters to be used for conversion:
converter: mscoco_detection
annotation_file: /home/intel/data/val2017/instances_val2017.json
Total annotations size: 5000
100 / 5000 processed in 0.590s
200 / 5000 processed in 0.595s
300 / 5000 processed in 0.591s
400 / 5000 processed in 0.595s
500 / 5000 processed in 0.595s
600 / 5000 processed in 0.593s
700 / 5000 processed in 0.596s
800 / 5000 processed in 0.602s
900 / 5000 processed in 0.602s
1000 / 5000 processed in 0.605s
1100 / 5000 processed in 0.611s
1200 / 5000 processed in 0.610s
1300 / 5000 processed in 0.603s
1400 / 5000 processed in 0.595s
1500 / 5000 processed in 0.592s
1600 / 5000 processed in 0.598s
1700 / 5000 processed in 0.606s
1800 / 5000 processed in 0.595s
1900 / 5000 processed in 0.599s
2000 / 5000 processed in 0.600s
2100 / 5000 processed in 0.609s
2200 / 5000 processed in 0.609s
2300 / 5000 processed in 0.601s
2400 / 5000 processed in 0.599s
2500 / 5000 processed in 0.600s
2600 / 5000 processed in 0.597s
2700 / 5000 processed in 0.605s
2800 / 5000 processed in 0.610s
2900 / 5000 processed in 0.592s
3000 / 5000 processed in 0.603s
3100 / 5000 processed in 0.599s
3200 / 5000 processed in 0.604s
3300 / 5000 processed in 0.592s
3400 / 5000 processed in 0.599s
3500 / 5000 processed in 0.591s
3600 / 5000 processed in 0.607s
3700 / 5000 processed in 0.601s
3800 / 5000 processed in 0.595s
3900 / 5000 processed in 0.591s
4000 / 5000 processed in 0.594s
4100 / 5000 processed in 0.587s
4200 / 5000 processed in 0.592s
4300 / 5000 processed in 0.588s
4400 / 5000 processed in 0.593s
4500 / 5000 processed in 0.589s
4600 / 5000 processed in 0.594s
4700 / 5000 processed in 0.592s
4800 / 5000 processed in 0.593s
4900 / 5000 processed in 0.589s
5000 / 5000 processed in 0.583s
5000 objects processed in 29.870 seconds
Annotation conversion for ms_coco_detection_80_class_without_background dataset has been finished
13:11:17 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/tools/post_training_optimization_toolkit/libs/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:224: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
  'NEAREST': Image.NEAREST,
13:11:17 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/tools/post_training_optimization_toolkit/libs/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:225: DeprecationWarning: NONE is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
  'NONE': Image.NONE,
13:11:17 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/tools/post_training_optimization_toolkit/libs/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:226: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
  'BILINEAR': Image.BILINEAR,
13:11:17 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/tools/post_training_optimization_toolkit/libs/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:227: DeprecationWarning: LINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
  'LINEAR': Image.LINEAR,
13:11:17 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/tools/post_training_optimization_toolkit/libs/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:228: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
  'BICUBIC': Image.BICUBIC,
13:11:17 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/tools/post_training_optimization_toolkit/libs/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:229: DeprecationWarning: CUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
  'CUBIC': Image.CUBIC,
13:11:17 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/tools/post_training_optimization_toolkit/libs/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:230: DeprecationWarning: ANTIALIAS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
  'ANTIALIAS': Image.ANTIALIAS
13:11:17 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/tools/post_training_optimization_toolkit/libs/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:234: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.
  'BOX': Image.BOX,
13:11:17 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/tools/post_training_optimization_toolkit/libs/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:235: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
  'LANCZOS': Image.LANCZOS,
13:11:17 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/tools/post_training_optimization_toolkit/libs/open_model_zoo/tools/accuracy_checker/accuracy_checker/preprocessor/resize.py:236: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.
  'HAMMING': Image.HAMMING,
INFO:compression.statistics.collector:Start computing statistics for algorithms : AccuracyAwareQuantization
INFO:compression.statistics.collector:Computing statistics finished
INFO:compression.pipeline.pipeline:Start algorithm: AccuracyAwareQuantization
INFO:compression.algorithms.quantization.accuracy_aware.algorithm:Start original model inference
13:11:35 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/model_optimizer/mo/back/ie_ir_ver_2/emitter.py:243: DeprecationWarning: This method will be removed in future versions.  Use 'list(elem)' or iteration over elem instead.
  if len(element.attrib) == 0 and len(element.getchildren()) == 0:
INFO:compression.engines.ac_engine:Start inference of 5000 images
Total dataset size: 5000
13:12:06 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/tools/post_training_optimization_toolkit/libs/open_model_zoo/tools/accuracy_checker/accuracy_checker/adapters/yolo.py:373: RuntimeWarning: overflow encountered in exp
  prob_correct=lambda x: 1.0 / (1.0 + np.exp(-x)))
1000 / 5000 processed in 904.201s
2000 / 5000 processed in 903.452s
3000 / 5000 processed in 903.376s
4000 / 5000 processed in 903.313s
5000 / 5000 processed in 902.878s
5000 objects processed in 4517.220 seconds
INFO:compression.engines.ac_engine:Inference finished
14:28:17 accuracy_checker WARNING: /opt/intel/openvino_2021.3.394/deployment_tools/tools/post_training_optimization_toolkit/compression/algorithms/quantization/accuracy_aware/utils.py:334: RuntimeWarning: Mean of empty slice
  result_value = np.nanmean(value['result'])
 
INFO:compression.algorithms.quantization.accuracy_aware.algorithm:Baseline metrics: {'map': 0.7117293964776917, 'AP@0.5': 0.7502475077100084, 'AP@0.5:0.05:95': 0.49196698535103006}
INFO:compression.algorithms.quantization.accuracy_aware.algorithm:Start quantization
INFO:compression.algorithms.quantization.default.algorithm:Start computing statistics for algorithm : ActivationChannelAlignment
INFO:compression.algorithms.quantization.default.algorithm:Computing statistics finished
INFO:compression.algorithms.quantization.default.algorithm:Start computing statistics for algorithms : MinMaxQuantization,FastBiasCorrection
INFO:compression.algorithms.quantization.default.algorithm:Computing statistics finished
INFO:compression.algorithms.quantization.accuracy_aware.algorithm:Start compressed model inference
INFO:compression.engines.ac_engine:Start inference of 5000 images
Total dataset size: 5000
1000 / 5000 processed in 746.465s
2000 / 5000 processed in 750.161s
3000 / 5000 processed in 744.248s
4000 / 5000 processed in 746.456s
5000 / 5000 processed in 745.793s
5000 objects processed in 3733.124 seconds
INFO:compression.engines.ac_engine:Inference finished
INFO:compression.algorithms.quantization.accuracy_aware.algorithm:Fully quantized metrics: {'map': 0.7057140315810843, 'AP@0.5': 0.7491440795501776, 'AP@0.5:0.05:95': 0.4679447290237501}
INFO:compression.algorithms.quantization.accuracy_aware.algorithm:Accuracy drop: {'map': 0.006015364896607478, 'AP@0.5': 0.0011034281598307771, 'AP@0.5:0.05:95': 0.024022256327279945}
INFO:compression.algorithms.quantization.accuracy_aware.algorithm:The total number of quantized operations in the graph: 135
INFO:compression.algorithms.quantization.accuracy_aware.algorithm:Removing fake quantize nodes
INFO:compression.algorithms.quantization.accuracy_aware.algorithm:Optimizing AP@0.5:0.05:95 metric
``

The generated IR will be located under `./results/yolo-v4-tf_AccuracyAwareQuantization/2022-04-27_13-10-45`
model format	map (%)	AP@0.5 (%)	AP@0.5:0.05:95 (%)
TensorFlow FP32	71.16	75.04	49.21
OpenVINO FP32 IR	71.17	75.02	49.2
OpenVINO INT8 IR	70.35	74.62	47.31