Run OpenCV Zoo (demo and benchmarks) on Allwinner D1 (RISC-V-based)

a_d1_with_opencv_zoo.md

Running OpenCV Zoo on Allwinner D1 (RISC-V-based)

This is a guide to run YuNet demo and benchmark from OpenCV Zoo on the Allwinnner D1 (RISC-V-based). Since the Python support is very limited, demo and benchmark for YuNet is rewritten in C++ and attached.

Tested environment:

• OS: Ubuntu 20.04
• SHELL: bash

Get the RISC-V toolchain and initialize environments

Visit this link to get the RISC-V toolchain from Allwinner: [CN] [EN]. Follow the official guide to download the toolchain, then you will have the following directories:

$ tree repo # repo is the root directory of TinaLinux-SDK
repo
|-- ...
|-- tina-d1-open
|-- ...

Run the following command to initialize environments before any cross-compilings:

cd $repo/tina-d1-open
source build/envsetup.sh
lunch d1_nezha-tina

Build OpenCV for RISC-V Arch

Follow this guide (Chinese) step by step and you can compile successfully. Assume you have install prefix set to $opencv/build/install, you need to manually copy libatomic from the SDK directory to $opencv/build/install/lib.

Build and run the C++ demo and benchmark for YuNet from OpenCV Zoo

The C++ demo detect.cpp and benchmark benchmark.cpp for YuNet from OpenCV Zoo are attached. CMakeLists.txt for building is also attached. Create a directory $yunet and save these files.

You can download the model face_detection_yunet_2021sep.onnx from opencv_zoo/face_detection_yunet, and the test data from opencv_zoo/benchmarks. NOTICE: Since the above guide compiling OpenCV turns off openjpeg when compiling, you need to port jpg images into png.

Ensure you have already initialized the environment for the RISC-V toolchain. Then run the following command to build the demo and benchmark:

cd $yunet
mkdir build && cd build
# replace $repo and $opencv with your actual absolute paths
cmake -D CMAKE_C_COMPILER=$repo/tina-d1-open/prebuilt/gcc/linux-x86/riscv/toolchain-thead-glibc/riscv64-glibc-gcc-thead_20200702/bin/riscv64-unknown-linux-gnu-gcc \
      -D CMAKE_CXX_COMPILER=$repo/tina-d1-open/prebuilt/gcc/linux-x86/riscv/toolchain-thead-glibc/riscv64-glibc-gcc-thead_20200702/bin/riscv64-unknown-linux-gnu-g++ \
      -D CMAKE_BUILD_TYPE=RELEASE \
      -D CMAKE_PREFIX_PATH=$opencv/build/install ..
make

Push $opencv/build/install, $yunet, face_detection_yunet_2021sep.onnx and test data to D1 using ADB:

# make sure you replace $opencv and $yunet with your actual absolute path
adb push $opencv/build/install/. /root/opencv-install
adb push $yunet/. /root/opencv_zoo/yunet
adb push face_detection_yunet_2021sep.onnx /root/opencv_zoo/yunet
adb push /path/to/test_data /root/opencv_zoo/data

Run demo and benchmark on D1:

export LD_LIBRARY_PATH=/root/opencv-install/lib

cd /root/opencv_zoo/yunet/build
# run demo
./detect -m=/root/opencv_zoo/yunet/face_detection_yunet_2021sep.onnx -i=/root/opencv_zoo/data/image_name.png
# run benchmark
./benchmark -m=/root/opencv_zoo/yunet/face_detection_yunet_2021sep.onnx -i=/root/opencv_zoo/data/image_name.png

benchmark.cpp

#include <opencv2/dnn.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/objdetect.hpp>

#include <iostream>

// using namespace cv;
using namespace std;

int main(int argc, char ** argv)
{
    cv::CommandLineParser parser(argc, argv,
        "{help  h           |            | Print this message.}"
        "{input i           |            | Path to the input image. Omit for detecting on default camera.}"
        "{backend_id        | 0          | Backend to run on. 0: default, 1: Halide, 2: Intel's Inference Engine, 3: OpenCV, 4: VKCOM, 5: CUDA}"
        "{target_id         | 0          | Target to run on. 0: CPU, 1: OpenCL, 2: OpenCL FP16, 3: Myriad, 4: Vulkan, 5: FPGA, 6: CUDA, 7: CUDA FP16, 8: HDDL}"
        "{model m           | yunet.onnx | Path to the model. Download yunet.onnx in https://github.com/ShiqiYu/libfacedetection.train/tree/master/tasks/task1/onnx.}"
        "{score_threshold   | 0.9        | Filter out faces of score < score_threshold.}"
        "{nms_threshold     | 0.3        | Suppress bounding boxes of iou >= nms_threshold.}"
        "{top_k             | 5000       | Keep top_k bounding boxes before NMS.}"
        "{warmup            | 30         | Run given times to warmup before benchmark.}"
        "{repeat            | 10         | Repeat given times for benchmark.}"
    );
    if (argc == 1 || parser.has("help"))
    {
        parser.printMessage();
        return -1;
    }

    cv::String modelPath = parser.get<cv::String>("model");
    int backendId = parser.get<int>("backend_id");
    int targetId = parser.get<int>("target_id");

    float scoreThreshold = parser.get<float>("score_threshold");
    float nmsThreshold = parser.get<float>("nms_threshold");
    int topK = parser.get<int>("top_k");

    int warmup = parser.get<int>("warmup");
    int repeat = parser.get<int>("repeat");

    // Initialize FaceDetectorYN
    cv::Ptr<cv::FaceDetectorYN> detector = cv::FaceDetectorYN::create(modelPath, "", cv::Size(320, 320), scoreThreshold, nmsThreshold, topK, backendId, targetId);

    // If input is an image
    if (parser.has("input"))
    {
        cv::String input = parser.get<cv::String>("input");
        cv::Mat image = cv::imread(input);
        cv::resize(image, image, cv::Size(160, 120));

        detector->setInputSize(image.size());
        cv::Mat faces;

        // warmup
        for (int i = 0; i < warmup; i++)
        {
            detector->detect(image, faces);
        }
        // repeat
        cv::TickMeter tickmeter;
        std::vector<double> times;
        for (int i = 0; i < repeat; i++)
        {
            tickmeter.reset();
            tickmeter.start();
            detector->detect(image, faces);
            tickmeter.stop();
            times.push_back(tickmeter.getTimeMilli());
        }
        // print results (median)
        int l = times.size();
        int mid = int(l / 2);
        if (l % 2 == 0)
        {
            std::cout << (times[mid] + times[mid - 1]) / 2 << std::endl;
        }
        else
        {
            std::cout << times[mid] << std::endl;
        }
    }
    else
    {
        std::cout << "must have input" << std::endl; 
    }
}

CMakeLists.txt

cmake_minimum_required(VERSION 2.8.12)

project(libfacedetection_opencvdnn)

# OpenCV
find_package(OpenCV 4.5.4 REQUIRED)
include_directories(${OpenCV_INCLUDE_DIRS})

add_executable(detect detect.cpp)
target_link_libraries(detect ${OpenCV_LIBS})

add_executable(benchmark benchmark.cpp)
target_link_libraries(benchmark ${OpenCV_LIBS})

detect.cpp

#include <opencv2/dnn.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/objdetect.hpp>

#include <iostream>

// using namespace cv;
using namespace std;

static cv::Mat visualize(cv::Mat input, cv::Mat faces, bool print_flag=false, double fps=-1, int thickness=2)
{
    cv::Mat output = input.clone();

    if (fps > 0) {
        cv::putText(output, cv::format("FPS: %.2f", fps), cv::Point2i(0, 15), cv::FONT_HERSHEY_SIMPLEX, 0.5, cv::Scalar(0, 255, 0));
    }

    for (int i = 0; i < faces.rows; i++)
    {
        if (print_flag) {
            cout << "Face " << i
                 << ", top-left coordinates: (" << faces.at<float>(i, 0) << ", " << faces.at<float>(i, 1) << "), "
                 << "box width: " << faces.at<float>(i, 2)  << ", box height: " << faces.at<float>(i, 3) << ", "
                 << "score: " << faces.at<float>(i, 14) << "\n";
        }

        // Draw bounding box
        cv::rectangle(output, cv::Rect2i(int(faces.at<float>(i, 0)), int(faces.at<float>(i, 1)), int(faces.at<float>(i, 2)), int(faces.at<float>(i, 3))), cv::Scalar(0, 255, 0), thickness);
        // Draw landmarks
        cv::circle(output, cv::Point2i(int(faces.at<float>(i, 4)),  int(faces.at<float>(i, 5))),  2, cv::Scalar(255,   0,   0), thickness);
        cv::circle(output, cv::Point2i(int(faces.at<float>(i, 6)),  int(faces.at<float>(i, 7))),  2, cv::Scalar(  0,   0, 255), thickness);
        cv::circle(output, cv::Point2i(int(faces.at<float>(i, 8)),  int(faces.at<float>(i, 9))),  2, cv::Scalar(  0, 255,   0), thickness);
        cv::circle(output, cv::Point2i(int(faces.at<float>(i, 10)), int(faces.at<float>(i, 11))), 2, cv::Scalar(255,   0, 255), thickness);
        cv::circle(output, cv::Point2i(int(faces.at<float>(i, 12)), int(faces.at<float>(i, 13))), 2, cv::Scalar(  0, 255, 255), thickness);
        // Put score
        cv::putText(output, cv::format("%.4f", faces.at<float>(i, 14)), cv::Point2i(int(faces.at<float>(i, 0)), int(faces.at<float>(i, 1))+15), cv::FONT_HERSHEY_SIMPLEX, 0.5, cv::Scalar(0, 255, 0));
    }
    return output;
}

int main(int argc, char ** argv)
{
    cv::CommandLineParser parser(argc, argv,
        "{help  h           |            | Print this message.}"
        "{input i           |            | Path to the input image. Omit for detecting on default camera.}"
        "{backend_id        | 0          | Backend to run on. 0: default, 1: Halide, 2: Intel's Inference Engine, 3: OpenCV, 4: VKCOM, 5: CUDA}"
        "{target_id         | 0          | Target to run on. 0: CPU, 1: OpenCL, 2: OpenCL FP16, 3: Myriad, 4: Vulkan, 5: FPGA, 6: CUDA, 7: CUDA FP16, 8: HDDL}"
        "{model m           | yunet.onnx | Path to the model. Download yunet.onnx in https://github.com/ShiqiYu/libfacedetection.train/tree/master/tasks/task1/onnx.}"
        "{score_threshold   | 0.9        | Filter out faces of score < score_threshold.}"
        "{nms_threshold     | 0.3        | Suppress bounding boxes of iou >= nms_threshold.}"
        "{top_k             | 5000       | Keep top_k bounding boxes before NMS.}"
        "{save  s           | false      | Set true to save results. This flag is invalid when using camera.}"
        "{vis   v           | true       | Set true to open a window for result visualization. This flag is invalid when using camera.}"
    );
    if (argc == 1 || parser.has("help"))
    {
        parser.printMessage();
        return -1;
    }

    cv::String modelPath = parser.get<cv::String>("model");
    int backendId = parser.get<int>("backend_id");
    int targetId = parser.get<int>("target_id");

    float scoreThreshold = parser.get<float>("score_threshold");
    float nmsThreshold = parser.get<float>("nms_threshold");
    int topK = parser.get<int>("top_k");

    bool save = parser.get<bool>("save");
    bool vis = parser.get<bool>("vis");

    // Initialize FaceDetectorYN
    cv::Ptr<cv::FaceDetectorYN> detector = cv::FaceDetectorYN::create(modelPath, "", cv::Size(320, 320), scoreThreshold, nmsThreshold, topK, backendId, targetId);

    // If input is an image
    if (parser.has("input"))
    {
        cv::String input = parser.get<cv::String>("input");
        cv::Mat image = cv::imread(input);

        detector->setInputSize(image.size());
        cv::Mat faces;
        detector->detect(image, faces);

        cv::Mat vis_image = visualize(image, faces, true);
        if(save)
        {
            cout << "result.png saved.\n";
            cv::imwrite("result.png", vis_image);
        }
        if (vis)
        {
            cv::namedWindow(input, cv::WINDOW_AUTOSIZE);
            cv::imshow(input, vis_image);
            cv::waitKey(0);
        }
    }
    else
    {
        std::cout << "must have input" << std::endl; 
    }
}