Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
GSoC with Mozilla 2019 Final Report

Report for building a new renderer backend for the matplotlib library based on the APIs of HTML5 Canvas Element

This gist is my final submission for GSoC, and it contains a description of my work and the journey of the project.

Community Bonding


Majority of my time went in studying the internals of the matplotlib library in this period. I looked up the classes that I need to extend and the functions I had to re-implement. These included the following classes:

  1. Extending the FigureCanvasBase class
  2. Extending the NavigationToolbar2 class
  3. Extending the GraphicsContextBase class
  4. Extending the RendererBase class
  5. Extending the FigureManagerBase class

Phase - I


Basic Setup of the new Backend

One of the first tasks that I did was to make 2 new subclasses.

  • The class GraphicsContextHTMLCanvas is extended from the GraphicsContextBase class
  • The class RendererHTMLCanvas is extended from the RendererBase class

The extended GraphicsContextHTMLCanvas was used to set the values linewidth, cap-style, join style, dashes using the APIs of the <canvas> element. The extended RendererHTMLCanvas was used to use the gc object from the above created GraphicsContextHTMLCanvas class.

The matplotlib library was also configured to use this new incomplete renderer as its new backend.

Refactoring existing backend into 3 components

The existing Agg backend used through WebAssembly contained a lot of reusable components such as the Navigation Toolbar, functions to zoom, pan around, etc. All this functionality could be reused for the new renderer as well.

Thus, the existing backend was refactored into 3 files. These were:

  1. browser_backend.py: contained the generic code that can be reused for both the backends
  2. wasm_backend.py: contained Agg specific code
  3. html5_canvas_backend.py : contained the HTML5 <canvas> element-specific code

Support Rendering

Functions draw_path and draw_markers were implemented, and the DPI was set to the default value instead of hardcoded 72. This enabled us to draw plots on-screen -- all sorts of plots (lines, sine curves, etc.)

Support for saving the plot as a PNG file

The previous code path to saving a PNG file used the Agg renderer. Thus, the saved PNG wouldn't reflect what was drawn by our new backend. The drawn plots were by the canvas-based renderer, BUT the saved plot was by the Agg renderer. The above was changed and fixed so that the saved PNG is in accordance with what is drawn on the screen.

Work on Testing Infrastructure and drawing Images begins

With essential support for drawing all kinds of plots (ignoring the text and a few graphical glitches), I started implementing support for rendering images and setting up the testing infrastructure. Both of them would be completed in Phase - II.

Phase - II


Testing Infrastructure

Out of the 2 pending incomplete tasks -- Testing Infrastructure and drawing Images -- we (Mike and I) prioritized the testing infrastructure since we could make sure that adding a new feature didn't break a previous one. This took much time and required fixing a lot of weird bugs. Over 1 month, we were finally able to land it in.

However, this was well worth it since we could then actually see tests failing if a new feature changed the plot in some way.

Drawing Images

While the work on drawing images was done before, we held it in the queue and finished the Testing Infrastructure first. Then, when we resumed the work on drawing Images and we noticed a few bugs -- 2 in particular.

One was related to passing the correct height. The other was related to the introduction of transparent pixels instead of white pixels.

The 2nd bug leads us to use the concept of an in-memory canvas to make sure that those white pixels weren't rendered as transparent.

Work on Rendering Text begins

With support for rendering plots + images along with having our testing infrastructure in place, the next obvious task was to add support for rendering text. This was begun in Phase - II but continued till the first week into Phase - III.

Phase - III


Support for Rendering Text

The basic pipeline from getting the font name and properties from the matplotlib library to using them with the <canvas> element was done in the 2nd week of Phase - II. This served as a proof of concept that text could be rendered + it's variations such as rotated text. Work on text resumed after about 10/11 days after the PRs for Testing Infrastructure, and Drawing Images were merged.

Support for Rendering Math Text

Math-text is different from regular text and usually contains mathematical symbols and scientific notation. We used the in-built bitmap - mathtextparser of the matplotlib library to get the math-text to be rendered as an image. This image was then passed onto the already implemented draw_image function (done in Phase-II). A small bug was fixed where we didn't account for the descent parameter before.

Support for having custom fonts

Once both regular text and math-text were working, the problem of custom fonts was next. It turned out that although the matplotlib library gave us the name and the properties of the font we should use, it didn't give us the font itself. Fonts used by matplotlib were in the virtual filesystem of matplotlib and weren't accessible to the browser.

To overcome this, we used the FontFace API to load the fonts we require -- on-demand dynamically. This task was not as easy as it sounds. Numerous problems like Infinite loops, Concurrent Invocations for loading fonts, redrawing the plot once the correct font was loaded (using a counter approach), etc. occurred. Lastly, we also modified the testing infrastructure a bit to wait for the redraw after the dynamic fonts were loaded before we made comparisons with reference images.

Support for Clipping

With support for rendering plots, images and text, there was one significant feature yet to be implemented -- clipping. It was necessary to implement clipping since only then we could support features like panning and zooming. If not done, the zoomed-in / panned part would go out of the bounding box. This feature turned out to be easier than expected, and we were done with this milestone too.

Fixing Transparency and a bug for 0 linewidth

The renderer was now functionally complete except for 2 graphical glitches -- one of them was that colours didn't appear to be transparent even when their transparency was set explicitly. The other bug was the <canvas> element always ignored the value 0 for linewidth and thus, it even drew stuff in places where it shouldn't be. This bug was fixed using a boolean variable that was set if the linewidth was not 0 and the stroke was called according to the state of this boolean variable.

Benchmarking

After we were done with implementing the renderer and fixing all bugs -- the next job was to benchmark it against the already available Agg renderer. This task was done on 5 plots, and the slowdown factor for each on both chrome and firefox was calculated. We concluded that the new renderer was 2x to 2.5x slower than before.

Links to Stuff that is completed


Repository with gsoc branch: https://github.com/iodide-project/pyodide/tree/gsoc

Commits : https://github.com/iodide-project/pyodide/commits/gsoc

Merged PRs: https://github.com/iodide-project/pyodide/pulls?q=is%3Apr+author%3Amadhur-tandon+is%3Aclosed+label%3A%22GSoC+2019%22

Open PRs: https://github.com/iodide-project/pyodide/pulls?q=is%3Apr+author%3Amadhur-tandon+label%3A%22GSoC+2019%22+is%3Aopen

Branch diff: https://github.com/iodide-project/pyodide/compare/gsoc

Space for Blog Posts: https://medium.com/gsoc-2k19-with-mozilla

Things to do

Write a Mozilla Hacks Post

Scope of Improvement

  1. Optimization (already in exploration)
  2. Implementing support for Gouraud shading.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.