Using %store magic to capture meta within/across Jupyter notebooks

store-magic-for-meta-pattern.md

Overview

The %store magic allows data to be stored in the ipython database, outside of a notebook's particular kernel, so it can be accessed from other notebooks.

Instructions

Initialization

If you're executing only one notebook, or this is the first notebook in a pipeline of notebooks.

First, clear any pre-existing meta objects from the database:

%store -d meta

Second, use a SimpleNamespace instance as the meta object:

from types import SimpleNamespace
meta = SimpleNamespace()

When writing to meta across notebooks, I will often create a more specific namespace. This affords a more structured final meta object (e.g. all the metadata captured across the notebook pipeline) and affords shorter/cleaner attribute names.

meta.specific_namespace = SimpleNamespace()  # Create a more specific namespace for this notebook

Loading

Now, regardless of whether where this notebook is in your pipeline, load the meta object into the notebook context:

%store meta

Recording

Record meta values throughout rest of notebook execution:

meta.count = 1000

Making meta available to other notebooks

At the end, make the meta object available to other notebooks:

%store meta

Persisting meta outside of the ipython database

# TODO

Add reading:
%store -r meta

Where am I missing the -r exactly?