- Proposal: Improve and Extend the JavaScript Bindings for OpenCV
- Mentor: Sajjad Taheri
- Organization: OpenCV
OpenCV.js is an open source project initiated in Parallel Architectures and Systems Group at UC Irvine. OpenCV.js extends the OpenCV language binding by providing a JavaScript interface. It leverages Emscripten to compile OpenCV functions into asm.js and WebAssembly. It allows emerging web applications with multimedia processing to benefit from the wide variety of vision functions available in OpenCV.
OpenCV.js was the first step to prove OpenCV can be exposed to web with high performance. However, to make OpenCV an attractive option for web developers, there were several shortcomings that need to be addressed and the code need to be optimized. , some change are needed to the existing code. What need to do are listed in my proposal, and what I have done are listed below.
As a collaborative and open source project, there has been contributors outside of GSoC:
- Ningxin Hu (Intel Corporation),
- Wenyao Gan (Intel intern, Shanghai Jiao Tong University).
Compared tp OpenCV, OpenCV.js project repository was using a different structure for maintaining the codes and building the project. To follow OpenCV convention, I created a new OpenCV module named js to keep all OpenCV.js related source files, tests and documenations. Ningxin adjusted the CMake build scripts to build OpenCV.js using cmake and make command.
We use a binding generator embindgen.py to control which functions will be exposed to JavaScript. Currently, functions from core, imgproc, objdetect, and video modules are exposed.To make OpenCV.js API easier to use for developers, we imporved it in the following aspects:
- Implementing function overloading by the number of parameters or different functions names.
- Implementing default parameters for the functions.
- Implementing Enums with constants.
Also, we added a JavaScript utility in opencv/modules/js/src. We defined the basic data types as js object here, which makes the basic data types more easy-to-use and avoid to be deleted manually.
Native OpenCV GUI opetions that are offered by HighGUI module are not available on the web. I created a web replacement for HighGUI using HTML5 components.
- Use HTML Canvas element to transfer images between cv.Mat and the web.
- Use WebRTC, HTML canvas, and video element to implement video input from various sources such as live cameras and video files.
- Use HTML Range objects to implement trackbars. I also created a tutorial on how to use GUI Features.
Also, I implemented some helper functions in helpers.js, like imread, imshow, VideoCaptue. They make the code of image i/o and video capture more similar to OpenCV, just like below.
// image i/o
//imageSource is the canvas/img element or id you want to input.
let img = cv.imread(imageSource);
//canvasOutput is the canvas element or id where you want to display the image.
cv.imshow(canvasOutput, img);
img.delete();
// video capture
// the video id or element you want to capture.
let cap = new cv.VideoCapture(videoSource);
let src = new cv.Mat(height, width, cv.CV_8UC4);
cap.read(src);
Because the bindings are improved and extended, tests are adjusted to fit the new API and to extend the coverage. Also, to support tests on Node.js, I add tests.js to require qunit which is necessary to execute tests. And add some code in every test file to detect whether the environment is Node.js.
if (typeof module !== 'undefined' && module.exports) {
// The envrionment is Node.js
var cv = require('./opencv.js');
}
And I added package.json to install dependencies(e.g. qunit) automatically.
OpenCV.js is very fresh to the web developers. Good documentations and interactive tutorials could help the developers to adopt it easier. Just the fact tat computer vision functions will be running on the web interactively, is very valuable for educational purposes. I found a method to implement interactive OpenCV tutorials on the web. I have used htmlonly and endhtmlonly commands to embed OpenCV HTML programs in markdown-tutorials.
\htmlonly
<!-- the html elements are embedded here -->
\endhtmlonly
In the embedded HTML pages, native web elements such as button, textarea and canvas for image I/O are used. For each tutorial, textarea is used to contain OpenCV sample program. The readers can change any part of the code or even replace it with their own code and run it to see the results immediately, just like the picture below.
We followed the existing OpenCV and OpenCV-Python tutorials that we found to be well taught, and re-implemented them in OpenCV.js/JavaScript.
I implemented tutorials of GUI Features and Video Analysis. Thanks to Ningxin for implementing Introduction to OpenCV.js, and thanks to Wenyao for implementing Core Operations, Image Processing and Object Detection.
All the tutorials source codes and assests are packaged in OpenCV js module. Tutorials will be generated by invoking make doxygen command.
We will continue our work to further improve OpenCV.js. We plan to implement missing functions from other modules such as features2d, ml, and DNN. We also plan to better support it on other platforms such as Node.js.
OpenCV.js Demos
OpenCV.js Tutorials Demos
My report for GSoC on Youtube