Skip to content

Instantly share code, notes, and snippets.

@pancx pancx/GSoC_report.md
Last active Feb 8, 2019

Embed
What would you like to do?
[GSoC 2017] Improve and Extend the JavaScript Bindings for OpenCV

[GSoC 2017] Improve and Extend the JavaScript Bindings for OpenCV

Congxiang Pan | github | twitter | email

Overview

Introduction

OpenCV.js is an open source project initiated in Parallel Architectures and Systems Group at UC Irvine. OpenCV.js extends the OpenCV language binding by providing a JavaScript interface. It leverages Emscripten to compile OpenCV functions into asm.js and WebAssembly. It allows emerging web applications with multimedia processing to benefit from the wide variety of vision functions available in OpenCV.

OpenCV.js was the first step to prove OpenCV can be exposed to web with high performance. However, to make OpenCV an attractive option for web developers, there were several shortcomings that need to be addressed and the code need to be optimized. , some change are needed to the existing code. What need to do are listed in my proposal, and what I have done are listed below.

As a collaborative and open source project, there has been contributors outside of GSoC:

  • Ningxin Hu (Intel Corporation),
  • Wenyao Gan (Intel intern, Shanghai Jiao Tong University).

Progress

Restructuring the OpenCV.js source code

Compared tp OpenCV, OpenCV.js project repository was using a different structure for maintaining the codes and building the project. To follow OpenCV convention, I created a new OpenCV module named js to keep all OpenCV.js related source files, tests and documenations. Ningxin adjusted the CMake build scripts to build OpenCV.js using cmake and make command.

Improve and extend binding generator and OpenCV.js APIs

We use a binding generator embindgen.py to control which functions will be exposed to JavaScript. Currently, functions from core, imgproc, objdetect, and video modules are exposed.To make OpenCV.js API easier to use for developers, we imporved it in the following aspects:

  • Implementing function overloading by the number of parameters or different functions names.
  • Implementing default parameters for the functions.
  • Implementing Enums with constants.

Also, we added a JavaScript utility in opencv/modules/js/src. We defined the basic data types as js object here, which makes the basic data types more easy-to-use and avoid to be deleted manually.

The GUI

Native OpenCV GUI opetions that are offered by HighGUI module are not available on the web. I created a web replacement for HighGUI using HTML5 components.

  • Use HTML Canvas element to transfer images between cv.Mat and the web.
  • Use WebRTC, HTML canvas, and video element to implement video input from various sources such as live cameras and video files.
  • Use HTML Range objects to implement trackbars. I also created a tutorial on how to use GUI Features.

Also, I implemented some helper functions in helpers.js, like imread, imshow, VideoCaptue. They make the code of image i/o and video capture more similar to OpenCV, just like below.

// image i/o
//imageSource is the canvas/img element or id you want to input.
let img = cv.imread(imageSource);
//canvasOutput is the canvas element or id where you want to display the image.
cv.imshow(canvasOutput, img);
img.delete();

// video capture
// the video id or element you want to capture.
let cap = new cv.VideoCapture(videoSource);
let src = new cv.Mat(height, width, cv.CV_8UC4);
cap.read(src);

The tests

Because the bindings are improved and extended, tests are adjusted to fit the new API and to extend the coverage. Also, to support tests on Node.js, I add tests.js to require qunit which is necessary to execute tests. And add some code in every test file to detect whether the environment is Node.js.

if (typeof module !== 'undefined' && module.exports) {
    // The envrionment is Node.js
    var cv = require('./opencv.js');
}

And I added package.json to install dependencies(e.g. qunit) automatically.

The web-based interactive tutorials for OpenCV.js

OpenCV.js is very fresh to the web developers. Good documentations and interactive tutorials could help the developers to adopt it easier. Just the fact tat computer vision functions will be running on the web interactively, is very valuable for educational purposes. I found a method to implement interactive OpenCV tutorials on the web. I have used htmlonly and endhtmlonly commands to embed OpenCV HTML programs in markdown-tutorials.

\htmlonly
<!-- the html elements are embedded here -->
\endhtmlonly

In the embedded HTML pages, native web elements such as button, textarea and canvas for image I/O are used. For each tutorial, textarea is used to contain OpenCV sample program. The readers can change any part of the code or even replace it with their own code and run it to see the results immediately, just like the picture below.
tutorial demo

We followed the existing OpenCV and OpenCV-Python tutorials that we found to be well taught, and re-implemented them in OpenCV.js/JavaScript.

I implemented tutorials of GUI Features and Video Analysis. Thanks to Ningxin for implementing Introduction to OpenCV.js, and thanks to Wenyao for implementing Core Operations, Image Processing and Object Detection.

All the tutorials source codes and assests are packaged in OpenCV js module. Tutorials will be generated by invoking make doxygen command.

Future Work

We will continue our work to further improve OpenCV.js. We plan to implement missing functions from other modules such as features2d, ml, and DNN. We also plan to better support it on other platforms such as Node.js.

OpenCV.js Demos

OpenCV.js Demos
OpenCV.js Tutorials Demos
My report for GSoC on Youtube

Commits List

The PR
The list of my commits

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.