Interactive feedback for long-running jobs in IPython

LongRunningJobs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Interactive feedback for long-running jobs in IPython\n",
    "\n",
    "Here we illustrate a couple of techniques we can use to provide feedback in the IPython Notebook for the status of potentially long-running jobs.\n",
    "\n",
    "The basic idea is to put on the page, either in a widget or in a standard output area, content that reflects the in-progress status of a job, and later update that content from Javascript. The JS code may be getting new status info from the kernel, or it may be receiving it from another channel (e.g. polling a separate server directly), but it is free to update the page regardless of the state of the IPython kernel.\n",
    "\n",
    "None of this is production-ready code, just an illustration of the various available tools with very simple examples.  In production, we'll need to decide on which specific approach to use based on the constraints of the problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## First approach: using a widget\n",
    "\n",
    "Here we make a custom IPython Widget, along the lines of [the examples](http://nbviewer.ipython.org/github/ipython/ipython/blob/2.x/examples/Interactive%20Widgets/Custom%20Widgets.ipynb) in the official IPython docs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from IPython.html import widgets\n",
    "from IPython.display import display, HTML\n",
    "from IPython.utils.traitlets import link, Bool, Unicode"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A simple `LaterWidget` will update its status after it's been created.  The Python side code has only a `_view_name` attribute to link it to its JS view, and a boolean `done` flag that represents completion.  The JS code will provide timer-based status updates either until the timer runs out, or until the `done` status is set to True kernel-side."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class LaterWidget(widgets.DOMWidget):\n",
    "    _view_name = Unicode('LaterView', sync=True)\n",
    "    done = Bool(False, sync=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we add JS code that sets up a timer function. In this case it simply counts until exhaustion, but it could obviously be polling a separate service, etc.  Note that the timer *also* checks against the kernel to see if the status has been updated:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "require([\"widgets/js/widget\", \"widgets/js/manager\"], function(widget, manager){\n",
       "    \n",
       "    var LaterView = widget.DOMWidgetView.extend({\n",
       "        counter : 0,\n",
       "        maxcount : 5,\n",
       "        \n",
       "        render: function(){\n",
       "            var that = this;\n",
       "            this.$el.text(\"Starting execution...\");\n",
       "            var c = function() {that.checkme(that)};\n",
       "            this.checker = setInterval(c, 1000);\n",
       "            this.model.on('change:done', c, this);\n",
       "        },\n",
       "        \n",
       "        checkme : function(that){\n",
       "            if (that.model.get('done')){\n",
       "                that.$el.html(\"<h2>DONE. Step: \" + that.counter + \"</h2> \");\n",
       "                clearInterval(that.checker);\n",
       "                return;\n",
       "            }\n",
       "            if (that.counter >= that.maxcount) {\n",
       "                that.$el.html(\"<h1>Timeout!!! Step: \" + that.counter + \"</h1> \");\n",
       "                clearInterval(that.checker);\n",
       "                return;\n",
       "            }\n",
       "            that.$el.html(\"<em>Working...</em> \" + that.counter + \"/\" + that.maxcount);\n",
       "            that.counter += 1;\n",
       "        },\n",
       "    });   \n",
       "    manager.WidgetManager.register_widget_view('LaterView', LaterView);\n",
       "});"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%javascript\n",
    "require([\"widgets/js/widget\", \"widgets/js/manager\"], function(widget, manager){\n",
    "    \n",
    "    var LaterView = widget.DOMWidgetView.extend({\n",
    "        counter : 0,\n",
    "        maxcount : 5,\n",
    "        \n",
    "        render: function(){\n",
    "            var that = this;\n",
    "            this.$el.text(\"Starting execution...\");\n",
    "            var c = function() {that.checkme(that)};\n",
    "            this.checker = setInterval(c, 1000);\n",
    "            this.model.on('change:done', c, this);\n",
    "        },\n",
    "        \n",
    "        checkme : function(that){\n",
    "            if (that.model.get('done')){\n",
    "                that.$el.html(\"<h2>DONE. Step: \" + that.counter + \"</h2> \");\n",
    "                clearInterval(that.checker);\n",
    "                return;\n",
    "            }\n",
    "            if (that.counter >= that.maxcount) {\n",
    "                that.$el.html(\"<h1>Timeout!!! Step: \" + that.counter + \"</h1> \");\n",
    "                clearInterval(that.checker);\n",
    "                return;\n",
    "            }\n",
    "            that.$el.html(\"<em>Working...</em> \" + that.counter + \"/\" + that.maxcount);\n",
    "            that.counter += 1;\n",
    "        },\n",
    "    });   \n",
    "    manager.WidgetManager.register_widget_view('LaterView', LaterView);\n",
    "});"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we display the counter, but also create a separate Checkbox control that we link to our `LaterWidget`, allowing us to toggle its `done` state before completion.\n",
    "\n",
    "Try running the following cell and toggle the checkbox to stop execution or let it run to completion to see the different behaviors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "later = LaterWidget()\n",
    "checkbox = widgets.Checkbox()\n",
    "link((checkbox, 'value'), (later, 'done'));\n",
    "\n",
    "display(later, checkbox)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Second approach: displaying HTML directly\n",
    "\n",
    "IPython can put HTML directly on the page, much like a print statement but with rich text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<h3>Hi, I am IPython!</h3>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import  display_html, Javascript\n",
    "display(HTML('<h3>Hi, I am IPython!</h3>'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And since it can also put JavaScript, we can put the HTML in a named div that we can update afterwards from elsewhere... First, let's put some text out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"panel\">Initial data ready...</div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "h = '<div id=\"panel\">Initial data ready...</div>'\n",
    "display(HTML(h))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now, let's call some JS code to update the above div (named `panel`) with new data, using a little JS fade effect:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "$(\"#panel\").hide().html(\"<h2>New data has arrived!</h2>\").fadeIn(\"fast\");"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "j = '$(\"#panel\").hide().html(\"<h2>New data has arrived!</h2>\").fadeIn(\"fast\");'\n",
    "display(Javascript(j))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This can obviously be encapsulated in a convenience class for easy use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"cdea0ad9-b2a5-4199-93d1-48d24f614ebf\">Long running job...</div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import uuid\n",
    "\n",
    "class LongRunningJob(object):\n",
    "    def __init__(self):\n",
    "        self.uuid = uuid.uuid4()\n",
    "        \n",
    "    def _ipython_display_(self):\n",
    "        h = '<div id=\"%s\">Long running job...</div>' % self.uuid\n",
    "        display(HTML(h))\n",
    "    \n",
    "    def update(self, text=\"new data\"):\n",
    "        j = '$(\"#%s\").hide().html(\"%s\").fadeIn(\"fast\");' % (self.uuid, text)\n",
    "        display(Javascript(j))\n",
    "    \n",
    "j = LongRunningJob()\n",
    "j"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can call `update` anytime we want to put new text in that area:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "$(\"#cdea0ad9-b2a5-4199-93d1-48d24f614ebf\").hide().html(\"new data\").fadeIn(\"fast\");"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "j.update()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "$(\"#cdea0ad9-b2a5-4199-93d1-48d24f614ebf\").hide().html(\"More info!\").fadeIn(\"fast\");"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "j.update('More info!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussion\n",
    "\n",
    "The widgets above have full bi-directional linking between the JS side and the kernel, and their output goes automatically to the Widget output area, which has its own closing control and can even be floated into a PopUp window.  The simpler HTML example puts only a static div in the rest of the output area of the cell where it was created.\n",
    "\n",
    "Both can be obviously combined: we can use bidirectionally-linked models with output in regular divs instead of the widget output area. Which will work better will depend on the end application."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "IPython (Python 2)",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}