zihaomu/OpenCV_with_TimVX.md

## OpenCV_with_TimVX.md

      
    Raw
  

              OpenCV_with_TimVX.md
            
          
    Compiling OpenCV with TIM-VX backend

OpenCV with TIM-VX is developped to run deep learning models on NPUs (on Khadas VIM3 etc.) using OpenCV with TIM-VX integrated. Code is under review at opencv/opencv#21036. Inference performance on the NPU on Khadas VIM3 can be found in OpenCV Zoo: https://github.com/opencv/opencv_zoo#models--benchmark-results.
We provide two ways compiling OpenCV with TIM-VX backend:

(Recommanded) Compile OpenCV together with TIM-VX.
Compile OpenCV with TIM-VX library installed previously.

Requirements

Before compiling, make sure you have the following dependencies installed:

git
cmake (3.14 or above)
(optional) python3 (3.6 or above), python3-numpy

Install packages on Ubuntu:
sudo apt install git cmake

# python3 is needed if you want to run examples from OpenCV Zoo
sudo apt install python3 python3-dev
pip3 install numpy
NOTE: If running on Khadas VIM3, you will need to replace galcore.ko with the one extracted from aarch64_A311D_6.4.9.tgz, which can be downloaded from https://github.com/VeriSilicon/TIM-VX/releases.
# replace everytime system is rebooted
sudo rmmod galcore
sudo insmod /path/to/aarch64_A311D_6.4.9/galcore.ko

# replace once for all
sudo find /usr/lib -name galcore.ko
mv /path/to/aarch64_A311D_6.4.9/galcore.ko <found_galcore_path_in_usr_lib>
sudo reboot # reboot to take effect
Compile OpenCV together with TIM-VX

To compile OpenCV together with TIM-VX, all you need to do is adding the option -D WITH_TIMVX=ON into your cmake command:
git clone https://github.com/fengyuentau/opencv.git
cd opencv
git checkout timvx_backend_support

mkdir build & cd build
# '-D BUILD_opencv_python3=ON' is needed for OpenCV's Python interface
cmake -D CMAKE_BUILD_TYPE=RELEASE \
      -D CMAKE_INSTALL_PREFIX=./install \
      -D WITH_TIMVX=ON ..
# NOTE: Make sure you see `TIMVX: YES` in the CMake output.
#       Copy the value of `VIVANTE SDK path` for running on TIM-VX backend.
make -j8
make install
LD_LIBRARY_PATH and VIVANTE_SDK_DIR is needed to run OpenCV with TIM-VX backend:
export LD_LIBRARY_PATH=/path/to/<VIVANTE_SDK_path>/lib:$LD_LIBRARY_PATH
export VIVANTE_SDK_DIR=/path/to/<VIVANTE_SDK_path>
# On x86_64 Linux, a typical <VIVANTE_SDK_path> is /home/user/opencv/build/3rdparty/libtim-vx/TIM-VX-c1f6866b697000934d1d61ca40937248c262fa9d/prebuilt-sdk/x86_64_linux
# On Khadas VIM3, a typical <VIVANTE_SDK_path> is /home/user/opencv/build/3rdparty/libtim-vx/VIVANTE_SDK/vim3_aarch64
Our cmake scripts detect system architecture automatically to determine which SDK is used. For x86_64 Linux, SDK for x86_64 simulation is used; For Ubuntu on Khadas VIM3, the aarch64_A311D_6.4.8.tgz SDK is used instead. Otherwise, building OpenCV with TIM-VX will be turned off.
Compile OpenCV with TIM-VX library installed previously

Step 1: Compile and install TIM-VX

git clone https://github.com/VeriSilicon/TIM-VX.git
cd TIM-VX # lets say ${TIM-VX} is the root of the cloned TIM-VX repositry
git checkout -b v1.1.37 v1.1.37
mkdir build & cd build

# build for x86_64 Linux
cmake -D BUILD_SHARED_LIBS=OFF ..

# build on Khadas VIM3
# Download aarch64_A311D_6.4.9.tgz from https://github.com/VeriSilicon/TIM-VX/releases
# You will need to replace 'drivers' with 'lib' in file ${TIM-VX}/cmake/local_sdk.cmake
cmake -D EXTERNAL_VIV_SDK=/path/to/aarch64_A311D_6.4.9  -D BUILD_SHARED_LIBS=OFF ..

make -j8
make install # installs to ./install by default
Step2: Compile OpenCV with TimVX

git clone https://github.com/fengyuentau/opencv.git
cd opencv 
git checkout timvx_backend_support
mkdir build & cd build

# VIVANTE_SDK_DIR is /path/to/TIM-VX/prebuilt-sdk/x86_64_linux for x86_64
#  or /path/to/aarch64_A311D_6.4.9 for Khadas VIM3.
# '-D BUILD_opencv_python3=ON' is needed for OpenCV's Python interface.
cmake -D CMAKE_BUILD_TYPE=RELEASE \
     -D CMAKE_INSTALL_PREFIX=./install \
     -D WITH_TIMVX=ON \
     -D TIMVX_INSTALL_DIR=/path/to/TIM-VX/build/install \
     -D VIVANTE_SDK_DIR=/path/to/VIVANTE-SDK
     ..

make -j8
make install
LD_LIBRARY_PATH and VIVANTE_SDK_DIR is needed as well to run OpenCV with TIM-VX backend:
export LD_LIBRARY_PATH=/path/to/<VIVANTE_SDK_path>/lib:$LD_LIBRARY_PATH
export VIVANTE_SDK_DIR=/path/to/<VIVANTE_SDK_path>
Performance Test on OpenCV Zoo

We have done performance tests on OpenCV Zoo: https://github.com/opencv/opencv_zoo#models--benchmark-results. You can reproduce the results as follows:
# Install git-lfs from https://git-lfs.github.com/
git clone https://github.com/opencv/opencv_zoo && cd opencv_zoo
git lfs install
git lfs pull

cd benchmark

# edit the config of the model you want to run, switch backend to TIM-VX and target to npu
vim config/face_detection_yunet.yaml
# in vim, press i to edit:
backend: "default" -> backend: "timvx"
target: "cpu" -> target: "npu"
# in vim, press esc and type :wq to save & exit

# run benchmarking
PYTHONPATH=.. python benchmark.py --cfg ./config/face_detection_yunet.yaml
Please ensure you have set LD_LIBRARY_PATH and VIVANTE_SDK_DIR properly as noted above.
FAQ

Q1: I cannot get OpenCV's Python interface compiled.

Please make sure you have installed python3, python3-dev and numpy:
sudo apt install python3 python3-dev
pip3 install numpy
Remove cmake cache under build:
# lets say ${opencv} is the root of the cloned OpenCV repositry
cd ${opencv}/build
rm CMakeCache.txt
Add the following options to your cmake command:
-D PYTHON3_INCLUDE_DIR=/path/to/python/include # A typical one is /usr/include/python3.8, where you can find Python.h
-D PYTHON3_EXECUTABLE=/path/to/python_binary   # A typical one is /usr/bin/python3.8, where you can locate the binary
-D PYTHON3_LIBRARY=/path/to/libpython3.x.so    # A typical one is /usr/lib/aarch64-linux-gnu/libpython3.8.so.1.0, where you can locate the shared library
Then run make -j 4 and make install to re-build OpenCV. The built Python interface cv2.x.so should be located under install/lib/python3.8/site-packages/cv2/python-3.8/.
Q2: How do I install the built Python interface of OpenCV?

By default, you can locate the OpenCV's Python interface cv2.x.so under ${opencv}/build/install/lib/python3.8/site-packages/cv2/python-3.8 once compilation is finished.
You may create a symbolic link to cv2.x.so under python3's package directory. For example,
# '/usr/lib/python3/dist-packages/cv2.so': you will need to find your own location to replace in your system
ln -sf ${opencv}/build/install/lib/python3.8/site-packages/cv2/python-3.8/cv2.cpython-38-aarch64-linux-gnu.so /usr/lib/python3/dist-packages/cv2.so
OpenCV should be ready for Python:
python3 -c "import cv2; print(cv2.__version__)"
Q3: Why do I have to set VIVANTE_SDK_DIR and LD_LIBRARY_PATH for TIM-VX backend?

VIVANTE_SDK_DIR is set for TIM-VX to find VIVANTE_SDK_DIR/include/CL/cl_viv_vx_ext.h. A workaround to avoid setting VIVANTE_SDK_DIR is put cl_viv_vx_ext.h along with executable binary files.
LD_LIBRARY_PATH is set for TIM-VX to find OpenVX and other dependencies. TIM-VX is built on top of OpenVX and other shared libraries. Integrating TIM-VX into OpenCV DNN also brings the dependencies for OpenCV DNN.