Cell magic for IPython/Jupyter notebooks - %%module

ModuleMagic Demo

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The ModuleMagic cell magic\n",
    "Gordon Bean, April 2015"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Motivation\n",
    "- %%file magic is nice for creating files - it is nice to see the contents of the file in the notebook.\n",
    "- %%cython allows you to create cython modules directly from notebook cells\n",
    "\n",
    "I want to create python modules directly in the notebook. Like %%file, I want to be able to see the contents of the module, rather than having an additional editor for maintaining the file. \n",
    "\n",
    "However, rather than just using %%file, I want the code to be automatically imported into my namespace as a module, much like %%cython. New versions of the cell should override existing modules. Such functionality could be obtained using %%file, import, and importlib.reload, but would require two input cells. \n",
    "\n",
    "So, here I introduce the %%module cell magic. It saves the cell contents in a temporary file (unique to the underlying python kernel) and imports that file to the `__main__` namespace as a module."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the magic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from beans.magics import ModuleMagics\n",
    "ip = get_ipython()\n",
    "ip.register_magics(ModuleMagics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using the magic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<module 'foobar' from '/tmp/.tmp-modules/kernel-24228926-336d-441b-bb58-f83c55cf7bf0/foobar.py'>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%module foobar\n",
    "a = 1\n",
    "b = 2\n",
    "\n",
    "def baz(c, d):\n",
    "    return c + d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 2)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "foobar.a, foobar.b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "foobar.baz(4,5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Under the hood\n",
    "The ultimate look under the hood is to examine the source code.\n",
    "\n",
    "Here is the \"cliff-notes\" version:\n",
    "\n",
    "- Cell contents are saved in a kernel-specific directory in `/tmp/.tmp-modules/`. \n",
    "- This module directory is added to `sys.path`, so they can be imported again by the same kernel (including in other modules created by %%module). \n",
    "- The first word after %%module is taken as the name of the file, and therefore becomes the name of the module. \n",
    " - It should conform to python syntax requirements, but this is not enforced. As the module is added to sys.modules and the global namespace via `__dict__` interfaces, any name will probably work, but can't be accessed using normal syntax - I advise avoiding pathology.\n",
    "- The module is reloaded everytime the cell is run. Existing entries in sys.modules are deleted before the module is loaded again.\n",
    "- The module is loaded into the global namespace (i.e. `sys.modules['__main__'].__dict__`). The ModuleMagics class can be instantiated with the keyword `namespace` and a dictionary representing the namespace %%module modules will be loaded into. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Future directions\n",
    "- Add second argument to specify the package. \n",
    "- Use ArgParse to handle magic command parsing.\n",
    "- Perhaps there is a better way to get a kernel-specific ID than parsing the connection_file path."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The current source code\n",
    "This is likely to change, but this is the current version:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## beans.magics\n",
    "# Gordon Bean, April 2015\n",
    "\n",
    "# To use these magics, you must register them with IPython\n",
    "# e.g:\n",
    "# from beans.magics import ModuleMagics\n",
    "# ip = get_ipython()\n",
    "# ip.register_magics(ModuleMagics)\n",
    "\n",
    "from IPython.core.magic import Magics, magics_class, cell_magic\n",
    "import os, sys, importlib\n",
    "\n",
    "@magics_class\n",
    "class ModuleMagics(Magics):\n",
    "    '''Magics for creating modules in IPython.'''\n",
    "    \n",
    "    def __init__(self, shell=None, namespace=None):\n",
    "        if shell is None:\n",
    "            shell = get_ipython()\n",
    "            \n",
    "        super(ModuleMagics, self).__init__(shell)\n",
    "        \n",
    "        self.namespace = namespace\n",
    "        \n",
    "        # Get the kernel id\n",
    "        self.kernelID = os.path.basename(shell.kernel.config['IPKernelApp']['connection_file'])[:-5]\n",
    "        \n",
    "        # Create kernel-specific tmp-module directory\n",
    "        self.module_dir = os.path.join('/tmp/.tmp-modules', self.kernelID)\n",
    "        os.makedirs(self.module_dir, exist_ok=True)\n",
    "    \n",
    "    def __del__(self):\n",
    "        # Remove module_dir from file system and sys.path\n",
    "        # I'm not sure this works - evidence so far says no...\n",
    "        tmpfiles = os.listdir(self.module_dir)\n",
    "        for file in tmpfiles:\n",
    "            os.remove(os.path.join(self.module_dir, file))\n",
    "            \n",
    "        os.rmdir(self.module_dir)\n",
    "        \n",
    "        sys.path.remove(self.module_dir)\n",
    "    \n",
    "    @cell_magic\n",
    "    def module(self, line, cell):\n",
    "        '''Import the cell as a module.'''\n",
    "        # Parse module name\n",
    "        tokens = line.split()\n",
    "        name = tokens[0]\n",
    "\n",
    "        # Save to file\n",
    "        filename = os.path.join(self.module_dir, name + '.py')\n",
    "        with open(filename, 'w') as f:\n",
    "            f.write(cell)\n",
    "\n",
    "        # Import module\n",
    "        if self.module_dir not in sys.path:\n",
    "            sys.path.insert(0, self.module_dir)\n",
    "\n",
    "        namespace = self.namespace if self.namespace else sys.modules['__main__'].__dict__\n",
    "        \n",
    "        if name in namespace:\n",
    "            # Always reload\n",
    "            del namespace[name]\n",
    "\n",
    "        module = importlib.import_module(name)\n",
    "        namespace[name] = module\n",
    "\n",
    "        return namespace[name]\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}