The best practice for pinning a Python library (or any kernel-side library) to a specific JupyterLab extension is to use a versioned MIME type or media type. A good example of this is the Altair Python library which supports multiple versions of Vega and Vega-lite and vega4-extension, vega3-extension, and vega2-extension which render Altair output.
In Jupyter notebooks, cell output areas contain one or more outputs of a given MIME type. The MIME type describes the data type of the output. The default MIME type is text/plain. JupyterLab provides a set of mimerender extensions that render common MIME types such as text/plain or application/json. There are many third-party extensions that render other MIME types, such as geojson-extension which renders application/geo+json and plotly-extension that renders application/vnd.plotly.v1+json.
As you can see in the Plotly example, MIME types can include versions. This allows a mimerender extension to target a specific version of a MIME type. The vega2-extension targets application/vnd.vega.v2+json and vega3-extension targets application/vnd.vega.v3+json, etc. In the case of Vega (which is currently at version 5), this is a useful pattern because vega2-extension cannot render vega 5 specs and vice-versa.
The data representation a Jupyter cell output area is mimebundle. A mimebundle is a map/dictionary containing one or more outputs of a given MIME type. A simple mimebundle looks like:
{
"text/plain": ["foo"]
}A mimebundle containing multiple outputs looks like:
{
"application/geo+json": {
"type": "Feature",
"geometry": {
"type": "Point",
"coordinates": [-118.4563712, 34.0163116]
}
},
"text/plain": ["<GeoJSON object>"]
}Kernel-side libraries can provide mimebundles to inform Jupyter clients how to render specific objects/data. Python libraries tend to include Jupyter-specific methods (e.g. repr_mimebundle, ipython_display , or Python's repr) on their classes to inform ipython how to render them. In Altair's case, it provides a repr_mimebundle method on it's Chart class so that when an Altair chart is returned, ipython will call it's repr_mimebundle method and return the resulting mimebundle to the client. An Altair mimebundle looks like:
{
"application/vnd.vegalite.v2+json": {
"$schema": "https://vega.github.io/schema/vega-lite/v2.3.0.json",
"config": {
"view": {
"height": 300,
"width": 400
}
},
"data": "cars.json",
"encoding": {
"color": {
"field": "Origin",
"type": "nominal"
},
"x": {
"field": "Horsepower",
"type": "quantitative"
},
"y": {
"field": "Miles_per_Gallon",
"type": "quantitative"
}
},
"mark": "point"
},
"image/png": "BASE64_DATA",
"text/plain": [
"<VegaLite 2 object>\n",
"\n",
"If you see this message, it means the renderer has not been properly enabled\n",
"for the frontend that you are using. For more information, see\n",
"https://altair-viz.github.io/user_guide/display.html\n"
]
}Jupyter clients receive a mimebundle on execute_reply and display_data
messages. When one of those messages is received, the client will select the
richest MIME type provided in the mimebundle and pass the mimebundle to the
mimerender extension that targets that MIME type. In the Altair example above,
the richest MIME type is application/vnd.vegalite.v2+json and the mimerender
extension associated with that MIME type is vega3-extension (vega3-extension
renders Vega 3 and Vega-lite 2), so the mimebundle will be rendered by
vega3-extension.
Altair can target multiple versions of Vega-lite. In its current version (3), it defaults to rendering Vega-lite 4. It can also target Vega-lite 3 and 2 when configured. It uses a versioned MIME type to inform the client-side of which extension to use to render the mimebundle.
Now that we understand how Altair uses versioned MIME types to target specific mimerender extensions and enforce compatibility, we can look at another common use case: Jupyter widgets.
Jupyter widget libraries require a kernel-side library and a client-side extension because they require that the kernel and client communicate. A good example of this is the ipywidgets Python library. The ipywidgets lifecycle looks like:
- A widget is created in Python (
IntSlider) - IPython returns a mimebundle containing a library-specific MIME type (
application/vnd.jupyter.widget-view+json) - The Jupyter notebook client renders the mimebundle using the mimerender extension for that MIME type (@jupyter-widgets/jupyterlab-manager)
- The mimerender extensions opens a comm channel for the client-side and kernel-side to communicate over
- A user interacts with the widget (slides the slider)
- The updated state of the widget is sent over the comm channel to the kernel
- The widget model on the kernel is recomputed to reflect the new state
- Any changes are synced with the client-side.
Jupyter widget libraries define model schemas on both the kernel-side and client-side and those schemas often change over time. As a result, kernel-side and client-side versions can get out-of-sync and become incompatible. ipywidgets addresses this problem by performing version checks to confirm that both the Python library and client-side extension are compatible. If they aren't, the client-side extension renders an error instead. This works but many of us have encountered this by surprise and found it difficult to debug because it isn't clear which Python version is compatible with which extension version. Vega's approach of maintaining a versioned spec (Vega 3, Vega 4) and client-side extensions that render a specific version of the spec (vega3-extension, vega4-extension) is more explicit and offers broader compatibility because kernel-side libraries can support multiple versions. Widget libraries can also maintain versioned specs and respective client-side extensions for each version. When a backward-incompatible change is introduced to the spec, a new major version of the spec and a new client-side extension can be introduced.
This approach involves:
- A versioned spec
- A versioned MIME type
- A kernel-side library that provides mimebundles with output of a specific version MIME type and data that conforms to the specific version spec
- One or more client-side extensions that render data of a specific version MIME type
Altair supports multiple versions of Vega-lite (e.g. from altair.vegalite.v3 import alt) and delivers the Vega-lite output
in a mimebundle with the appropriate versioned MIME type (e.g.
application/vnd.vegalite.v4+json). vega-extension, which is bundled with
JupyterLab, supports the current version of Vega and Vega-lite, vega4-extension
supports Vega 4 and Vega-lite 3, vega3-extension supports Vega 3 and Vega-lite
2, etc. Each of these extensions is published as an individual package on npm.
This creates some maintenance overhead that can be avoided using a unique
feature of yarn. yarn alias allows a
Javascript application to alias different versions (or version ranges) of the same Javascript
package so that each may be consumed separately in different parts
of the application. We can use yarn alias to install different versions of the
same JupyterLab extension package as separate extensions, thereby allowing us to
maintain a single package on npm.
Let's imagine that we have a JupyterLab extension called test-extension and
that the current version is 1.0.0. We recently introduced some breaking
changes to the spec that the kernel-side (e.g. Python) library outputs to the client-side
library. We have updated our test Python library and published a 2.0.0
version to PyPI. Now we must update our JupyterLab extension to support the new
spec. If our JupyterLab extension depends on an external library to render the
output from our Python library, then we update that library to support the new spec, publish a
2.0.0 version to npm, and bump the dependency version in our JupyterLab extension's
package.json. Otherwise, we make the necessary changes to the JuypyterLab
extension's source to support the new spec. Next, we publish a new 2.0.0 version of the
JupyterLab extension to npm. Lastly, we install the 2.0.0
version of the JupyterLab extension as a separate extension using yarn alias:
jupyter labextension install test2-extension@npm:test-extension@2.xBy appending @npm:test-extension@2.x to the npm package name, we inform yarn
to install test-extension at version >=2.0.0 and <= 3.0.0 and alias it as
test2-extension. This allows us to install different semver ranges of
test-extension as separate JupyterLab extensions and in turn support multiple
versions of the Python library and/or spec without the overhead of maintaining
multiple repos or npm packages for the respective JupyterLab extensions.