This document provides a guide to using OpenVino for optimizing Maximo Visual Inspection (MVI) and other open source models. Since MVI segmentation model Detectron is Caffe2 based which is not yet supported on OpenVino, for image segmentation we have referred to Mask R-CNN models.
Kartikeya Vats https://w3.ibm.com/bluepages/profile.html?uid=01931Z744 Sujoy Roychowdhury https://w3.ibm.com/bluepages/profile.html?uid=01918P744
Sincere thanks to Ashish Rao https://w3.ibm.com/bluepages/profile.html?uid=00135O744 for review and comments.
Open Visual Inference and Neural Network abbreviated as OpenVINO is a toolkit provided by Intel to facilitate faster inference on Neural Networks on Intel devices where power of the device is a constraint. This documentation assumes that openVino is already installed in the system. If not, please install that first from here: https://docs.openvinotoolkit.org/2020.4/openvino_docs_install_guides_installing_openvino_linux.html
The field of Deep learning has taken up speed like it had never before. With models capable of a lot many things with great accuracy they need high powered machines to be deployed upon. But we don't always have these high powered machines at our disposal all the time now, do we? The possibilities are endless if we find a way to optimize these models so that they can be deployed on low powered edge devices. This has been an active area of research and now has taken a great boom in the industry with techniques like quantization, pruning etc. typically used to optimize the models along with various frameworks in place suited for a particular device. Our focus in this documentation would be on intel devices and optimization of models through openVINO so that they can be deployed on Intel devices.
OpenVINO toolkit provides various kinds of models that can be optimized for inference through openVINO on various Intel devices. In order to perform inference on these devices we need to convert the model into an Intermediate Representation(IR) consisting of an xml file(architecture), bin file(weights) and the mapping file. The IR can be made for FP32, FP16 and INT8 precision points. OpenVINO provides a model optimizer(mo) and it's variants for the conversion of model from various frameworks to IR. Following is the flow for the general process:
There is a huge boom in the field of AI on edge and Intel provides openVINO toolkit to the end user in order to leverage power of AI on Intel devices. However, there is a huge gap in documentation for the same and the user may end up doing a lot of ‘hit and trial’ to get their model to work. To ease the pain of doing this, we have compiled a few steps in order to make life simpler for people using the toolkit for some commonly used models.
Here, we trained a GoogleNet model to do cats and dogs classification. We used IBM Maximo Visual Inspection to do the same which gives us the GoogleNet model as a caffe model with relevant prototxt and caffemodel file. You can download it from here: https://ibm.ent.box.com/folder/129581607752?s=ctyqnv2e4hkqedl79ov95bvvaswaqkba
python3 /opt/intel/openvino_2020.4.287/deployment_tools/model_optimizer/mo.py --input_model <path_to_caffemodel> --input_proto <path_to_prototxt>
python3 /opt/intel/openvino_2020.4.287/deployment_tools/model_optimizer/mo.py -input_model <path_to_caffemodel> --input_proto <path_to_prototxt> --data_type FP16
(Assuming openvino is installed at /opt/intel/openvino otherwise choose the appropriate path)
The xml, bin and the mapping files will be created in the same directory from which the run command was fired.
For 8 bit conversions we are using the post optimization tool kit provided by openVino. Unfortunately, unlike 32bit and 16bit IR representations, this is not as straightforward. First things first, install the post optimization tool kit (pot) from https://docs.openvinotoolkit.org/2020.4/pot_README.html
To keep things simple, we will use the default quantization algorithm provided by open Vino. OpenVino provides a lot of config files with a lot of parameters but it can get cumbersome to filter out and keep only the relevant parameters. For googleNet, thankfully, we have done the job for you.
If you want to read up on various kind of quantizations provided by open vino, please refer to: https://docs.openvinotoolkit.org/2020.4/pot_compression_algorithms_quantization_README.html
Create a file “quant_config.json” with the following content:
https://gist.github.com/bdfaf34cd725e74aca599520ec1f4acb
The "model.xml", "model.bin" files are the 32bit IR(preferebly, though 16bit can also be given) generated in the previous steps.
Create a file “gnet.yml” with the following content: https://gist.github.com/0c84f02ff86cefa60a6cf2177ec976bc
You will need to provide the path for the data source key with a path to a calibration dataset in your system. Create a folder with atleast 2 images and provide the appropriate path.
Run the following command: pot -c quant_config.json
A folder with the name “results” will be created in the same directory and inside the folder there would be another folder with the timestamp of the created model. Navigate the folder to reach to the corresponding xml, bin and mapping files.
Same code can be used to perform inference using the intermediate representation for all the three “precisions”. You will just need to provide the appropriate “xml” and “bin” file paths.
Create a file “infer_gnet.py” with the following content:
https://gist.github.com/b52ebe00212ff540ff8777160af50471
Note: the above code is a minimalistic code for inference on a sample image. Please change the paths to relevant paths for your system and add the functionalities on top of it as you desire.
Here, we are using a pretrained yolov3 model trained on coco dataset. We take the route of converting the model to a saved graph (.pb format). Convert the saved graph to Intermediate Representation (32bit, 16bit or 8bit) and perform inference on IR.
Convert the yolov3.weights file to a frozen graph file. We have pre-converted(frozen_darknet_yolov3_model.pb) it and you can download it from here: https://ibm.ent.box.com/folder/129582446937?s=qlt3wgxv9ynp9vr41b7nt4yw7nf0darz
First things first, we need to create a custom operations config file. Let’s name it “yolo_v3.json” Contents of this file are:
https://gist.github.com/2cd8835120695f0891541a3de6cf7da7
python3 /opt/intel/openvino_2020.4.287/deployment_tools/model_optimizer/mo_tf.py --input_model frozen_darknet_yolov3_model.pb --tensorflow_use_custom_operations_config yolo_v3.json --batch 1
python3 /opt/intel/openvino_2020.4.287/deployment_tools/model_optimizer/mo_tf.py --input_model frozen_darknet_yolov3_model.pb --tensorflow_use_custom_operations_config yolo_v3.json --batch 1 --data_type FP16
(Assuming openvino is installed at /opt/intel/openvino otherwise choose the appropriate path)
The xml, bin and the mapping files will be created in the same directory from which the run command was fired.
Similar to googLENet we will use post optimization toolkit(pot) for the same with default quantization.
Create a file “quant_config.json” with the following content:
https://gist.github.com/b97332b8f9dd29f997e875971ddf34cd
The "frozen_darknet_yolov3_model.xml", "frozen_darknet_yolov3_model.bin" files are the 32bit IR(preferebly, though 16bit can also be given) generated in the previous steps.
Create a file “yolo.yml” with the following content:
https://gist.github.com/f89fc1a7d1433e858019432052d0e524
Of course, you need to provide the appropriate paths for "data_source", "annotatations_dir", "images_dir" and "imageset_file" keys. Given below are examples for sample voc annotation file and imageset file.
https://gist.github.com/c6c0d94e0dcb881653b9c95af8d10712
https://gist.github.com/6611e4b39211064deaa23ef1e4256c19
Run the following command: pot -c quant_config.json
A folder with the name “results” will be created in the same directory and inside the folder there would be another folder with the timestamp of the created model. Navigate the folder to reach to the corresponding xml, bin and mapping files.
Same code can be used to perform inference using the intermediate representation for all the three “precisions”. You will just need to provide the appropriate “xml” file path.
Run the following command
python3 <path_to_openvino_2020.4.287>/deployment_tools/open_model_zoo/demos/python_demos/object_detection_demo_yolov3_async/object_detection_demo_yolov3_async.py -i dog.4001.jpg -m /home/kartvat3/OpenVinoBlog/yolov3/frozen_darknet_yolov3_model.xml -d CPU
We are running the demo provided by the open vino toolkit. It expects access to webCam on your system.
Here, we take a caffe model trained on PascalVOC dataset and perform inference on it through the cpp demo script. You can download the prototxt file(test.protoxt) from: https://raw.githubusercontent.com/rbgirshick/py-faster-rcnn/master/models/pascal_voc/VGG16/faster_rcnn_end2end/test.prototxt. You can also download the caffemodel file from: https://dl.dropboxusercontent.com/s/o6ii098bu51d139/faster_rcnn_models.tgz?dl=0(You'll need to unzip the archive)
python3 /opt/intel/openvino_2020.4.287/deployment_tools/model_optimizer/mo_caffe.py --input_model faster_rcnn_models/VGG16_faster_rcnn_final.caffemodel --input_proto test.prototxt
python3 /opt/intel/openvino_2020.4.287/deployment_tools/model_optimizer/mo_caffe.py --input_model faster_rcnn_models/VGG16_faster_rcnn_final.caffemodel --input_proto test.prototxt --data_type FP16
(Assuming openvino is installed at /opt/intel/openvino otherwise choose the appropriate path)
The xml, bin and the mapping files will be created in the same directory from which the run command was fired.
Similar to models we have already seen(GoogLENet and Yolov3) we'd need to do a little work and more to get the 8-bit model running.
Create a file "quantize_frcnn.json” with the following content:
https://gist.github.com/f8b1b3cba5c8b7cc08085260a5599fc2
The "VGG16_faster_rcnn_final.xml", "VGG16_faster_rcnn_final.bin" files are the 32bit IR(preferably, though 16bit can also be given) generated in the previous steps.
Create a file "frcnn.yaml" with the following content:
https://gist.github.com/9faf4a9a06007bd6fb60c9d9aaa30c23
Please refer to the yolov3 documentation in case of doubt regarding values and format for the relevant keys in the yaml file.
We would need to alter the source code a bit for the code to work properly. Else, if defaults to an error. The change has to be made in the "accuracy_checker" module and the module needs to be recompiled. Firstly, open the following file in the editor of your choice. We prefer vi.
vi <path_to_open_vino>/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker/adapters/ssd.py
In the "process" function make the following changes on the loop over "identifiers":
Recompile the accuracy checker through the following command:
python3 path_to_open_vino>/deployment_tools/open_model_zoo/tools/accuracy_checker/accuracy_checker setup.py install
Run the following command: pot -c quantize_frcnn.json
A folder with the name “results” will be created in the same directory and inside the folder there would be another folder with the timestamp of the created model. Navigate the folder to reach to the corresponding xml, bin and mapping files.
The openVINO toolkit provides inference option for fasterRCNN through IR. Let's do it on picture of a dog. For that you will need to build those demos first. It is pretty straightforward. For our convenience we'll add a few statements and rebuild the example.
We can find the demos at: <path_to_openvino_installation>/deployment_tools/open_model_zoo/demos/
Open "object_detection_demo_faster_rcnn/main.cpp" from this directory.
Make the following changes in "main.cpp":
Alternatively, you can download the updated "main.cpp" from here: https://ibm.ent.box.com/folder/129583129217?s=as2xnxw4f2f5yz57i6xxa40bddlogpba
This will help us to know which is the output class
Rebuild the demos through "./build_demos.sh"(from the demos directory mentioned above)
The logs will output the built examples path. Navigate to that directory and run the following command.
./object_detection_demo_faster_rcnn -i <path_to_input_image> -m <path_to_IR_xml> -d CPU
The output would be the confidence scores and labels of various detections. Thanks to our change we'll know the label with high confidence score and it'll dump the detected image with the name out_0(default) in the same folder.
Output log:
As it can be seen the detected label is label number 12 which represents "dog" in the pascal_voc classes.
For your reference this is the pascal voc classes order with index starting from zero. You can verify the your labels through it.
Output image result with bounding box:
That is it! FasterRCNN through openVINO :)
For maskRCNN demo we will use the training extensions provided in the open vino toolkit. In our experience, in the official documentation, model and scripts they provide seemed to fail when we train our own model and even to run the default model there were unmentioned changes. Instead we found the training extensions route that is provided by the openVINO tookit for instance segmentation to do our job. We trained our own pytorch model(instructions are given in the toolkit) on a dataset that we converted to coco format(must be done). You can download it from here: https://ibm.ent.box.com/folder/129589552017?s=71nojc5yqeqixn91773i4seq85y0fmom
We treat this as a "defect detection" on an object task. Let's assume that the sticker on top of the object is "defect_on_top" and the mark on the side of the object is "defect on side"
Note - You will need to train your pytorch model(based on instructions given in the toolkit, link to which is given below)
We will convert it the pytorch model to onnx and then to open vino. Make sure to have a GPU system with cuda(9) and cudnn(7). We had problems with other configurations while onnx conversion trhough the toolkit. We recommend that you use the following docker image.
cmhi/cuda9.0-cudnn7:latest
Source: https://hub.docker.com/r/cmhi/cuda9.0-cudnn7
Other requirements: Ubuntu >= 16.04( we have used 18.04) Python 3.5.2 Pytorch 0.4.1 OpenVINO™ 2019 R1 with Python API
You will get python 3.5.2 with the docker image. You'll need to install install pytorch 0.4.1 inside the image. You can install "OpenVINO™ 2019 R1 with Python API" inside or outside the docker, however you wish but will need to make sure that the onnx file is accessible to it.
Yes, these are the configurations that worked for us. For the following commands to work you'd need to install the above mentioned version of openVINO unlike the one we have been using for other models. Please install that.
Now, first things first. Attach to the container you have created for the above mentioned image and ensure that the above dependencies are met.
Then simply git clone the "training_extenstions" through the following command.
git clone https://github.com/openvinotoolkit/training_extensions.git
Navigate to the "instance segmentation" directory
cd openvino_training_extensions/pytorch_toolkit/instance_segmentation
Run the following commands in the following order:
cat requirements.txt | xargs -n 1 -L 1 pip3 install python3 setup.py install
(refer to https://github.com/openvinotoolkit/training_extensions/tree/develop/pytorch_toolkit/instance_segmentation to train your model or else download the model we have trained from https://ibm.ent.box.com/folder/129603811266?s=wsxy29eq55fw761cq6k5454b6keijv0m)
Next, save the coco dataset images and annotations under this directory: "<path_to_openvino_training_extensions>/pytorch_toolkit/instance_segmentation/data/"
Make sure that the name of the dataset is "coco" and is a subdirectory inside "<path_to_openvino_training_extensions>/pytorch_toolkit/instance_segmentation/data/" and has the following format:
Assuming all the following commands are run from the "instance segmentation" directory.
ONNX(Open Neural Network Exchange) is useful for conversion from various machine learning frameworks to another. Since, openVINO provides the option to convert a model on onnx to IR. We will convert our pytorch model to onnx. You can learn more about onnx from here: https://onnx.ai/
Open the following file: "/usr/local/lib/python3.5/dist-packages/pycocotools/coco.py" and make the following changes to it.
Yep, you read that right! We need to add the path to the instances_val2017.json manually. Make sure to provide the appropriate file path based on your system.
We are ready to convert the ".pth" file to a ".onnx" file. Run the following command to do it:
python3 tools/convert_to_onnx.py --model segmentoly.rcnn.model_zoo.resnet_fpn_mask_rcnn.ResNet50FPNMaskRCNN --ckpt <path_to_downloaded/trained_pth_file> --input_size 800 1344 --dataset coco_2017_val --show_flops --output_file data/converted_onnx_models/<path_to_converted_onnx_file>
Run the following command: python3 <relevant_path_to_openvino_installation>/openvino_2019.1.144/deployment_tools/model_optimizer/mo.py --framework onnx --input_model <path_to_converted_onnx_file> --output_dir <path_to_IR_files> --input "im_data,im_info" --output "boxes,scores,classes,batch_ids,raw_masks" --mean_values "im_data[102.9801,115.9465,122.7717],im_info[0,0,0]"
The xml, bin and the mapping files will be created in the specified output directory.
Create a file named "infer.py" with the following contents:
https://gist.github.com/4a881b1b73a2cb0db57538a1976bd787
The above script is a manipulated version of "tools/test.py".
Next you'd need to locate a file named "libcpu_extension.so" in your system. You should be able to get it using the following command: locate libcpu_extension.so
Keep the path shown in the terminal handy.
Next, run the following command:
python3 infer.py --dataset coco_2017_val --ckpt <path_to_IR_bin_file> --fit_window 800 1333 --image <path_to_input_image> openvino --model <path_to_IR_xml_file> -l <path_to_libcpu_extension.so>
A file named out.png will be created with detection done through IR. For our input image we got an output like this(with the segmentation mask):
Viola! It's done.