Report for building a new renderer backend for the matplotlib library based on the APIs of HTML5 Canvas Element
This gist is my final submission for GSoC, and it contains a description of my work and the journey of the project.
Majority of my time went in studying the internals of the matplotlib library in this period. I looked up the classes that I need to extend and the functions I had to re-implement. These included the following classes:
- Extending the FigureCanvasBase class
- Extending the NavigationToolbar2 class
- Extending the GraphicsContextBase class
- Extending the RendererBase class
- Extending the FigureManagerBase class
One of the first tasks that I did was to make 2 new subclasses.
- The class GraphicsContextHTMLCanvas is extended from the GraphicsContextBase class
- The class RendererHTMLCanvas is extended from the RendererBase class
The extended GraphicsContextHTMLCanvas was used to set the values linewidth, cap-style, join style, dashes using the APIs of the <canvas> element.
The extended RendererHTMLCanvas was used to use the gc object from the above created GraphicsContextHTMLCanvas class.
The matplotlib library was also configured to use this new incomplete renderer as its new backend.
The existing Agg backend used through WebAssembly contained a lot of reusable components such as the Navigation Toolbar, functions to zoom, pan around, etc. All this functionality could be reused for the new renderer as well.
Thus, the existing backend was refactored into 3 files. These were:
- browser_backend.py: contained the generic code that can be reused for both the backends
- wasm_backend.py: contained Agg specific code
- html5_canvas_backend.py : contained the HTML5 <canvas> element-specific code
Functions draw_path and draw_markers were implemented, and the DPI was set to the default value instead of hardcoded 72.
This enabled us to draw plots on-screen -- all sorts of plots (lines, sine curves, etc.)
The previous code path to saving a PNG file used the Agg renderer. Thus, the saved PNG wouldn't reflect what was drawn by our new backend. The drawn plots were by the canvas-based renderer, BUT the saved plot was by the Agg renderer. The above was changed and fixed so that the saved PNG is in accordance with what is drawn on the screen.
With essential support for drawing all kinds of plots (ignoring the text and a few graphical glitches), I started implementing support for rendering images and setting up the testing infrastructure. Both of them would be completed in Phase - II.
Out of the 2 pending incomplete tasks -- Testing Infrastructure and drawing Images -- we (Mike and I) prioritized the testing infrastructure since we could make sure that adding a new feature didn't break a previous one. This took much time and required fixing a lot of weird bugs. Over 1 month, we were finally able to land it in.
However, this was well worth it since we could then actually see tests failing if a new feature changed the plot in some way.
While the work on drawing images was done before, we held it in the queue and finished the Testing Infrastructure first. Then, when we resumed the work on drawing Images and we noticed a few bugs -- 2 in particular.
One was related to passing the correct height. The other was related to the introduction of transparent pixels instead of white pixels.
The 2nd bug leads us to use the concept of an in-memory canvas to make sure that those white pixels weren't rendered as transparent.
With support for rendering plots + images along with having our testing infrastructure in place, the next obvious task was to add support for rendering text. This was begun in Phase - II but continued till the first week into Phase - III.
The basic pipeline from getting the font name and properties from the matplotlib library to using them with the <canvas> element was done in the 2nd week of Phase - II. This served as a proof of concept that text could be rendered + it's variations such as rotated text. Work on text resumed after about 10/11 days after the PRs for Testing Infrastructure, and Drawing Images were merged.
Math-text is different from regular text and usually contains mathematical symbols and scientific notation. We used the in-built bitmap - mathtextparser of the matplotlib library to get the math-text to be rendered as an image. This image was then passed onto the already implemented draw_image function (done in Phase-II). A small bug was fixed where we didn't account for the descent parameter before.
Once both regular text and math-text were working, the problem of custom fonts was next. It turned out that although the matplotlib library gave us the name and the properties of the font we should use, it didn't give us the font itself. Fonts used by matplotlib were in the virtual filesystem of matplotlib and weren't accessible to the browser.
To overcome this, we used the FontFace API to load the fonts we require -- on-demand dynamically. This task was not as easy as it sounds. Numerous problems like Infinite loops, Concurrent Invocations for loading fonts, redrawing the plot once the correct font was loaded (using a counter approach), etc. occurred. Lastly, we also modified the testing infrastructure a bit to wait for the redraw after the dynamic fonts were loaded before we made comparisons with reference images.
With support for rendering plots, images and text, there was one significant feature yet to be implemented -- clipping. It was necessary to implement clipping since only then we could support features like panning and zooming. If not done, the zoomed-in / panned part would go out of the bounding box. This feature turned out to be easier than expected, and we were done with this milestone too.
The renderer was now functionally complete except for 2 graphical glitches -- one of them was that colours didn't appear to be transparent even when their transparency was set explicitly. The other bug was the <canvas> element always ignored the value 0 for linewidth and thus, it even drew stuff in places where it shouldn't be. This bug was fixed using a boolean variable that was set if the linewidth was not 0 and the stroke was called according to the state of this boolean variable.
After we were done with implementing the renderer and fixing all bugs -- the next job was to benchmark it against the already available Agg renderer. This task was done on 5 plots, and the slowdown factor for each on both chrome and firefox was calculated. We concluded that the new renderer was 2x to 2.5x slower than before.
Repository with gsoc branch: https://github.com/iodide-project/pyodide/tree/gsoc
Commits : https://github.com/iodide-project/pyodide/commits/gsoc
Branch diff: https://github.com/iodide-project/pyodide/compare/gsoc
Space for Blog Posts: https://medium.com/gsoc-2k19-with-mozilla
Write a Mozilla Hacks Post
- Optimization (already in exploration)
- Implementing support for
Gouraud shading.