simple illustration of jupyter kernel messaging

Python_Jupyter_Interaction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Demonstration of Python - Jupyter kernel interaction\n",
    "\n",
    "Jupyter provides capabilities to launch a _compute kernel_ in a subprocess that can execute user computations. This notebook demonstrates how this works, using Python to launch and manage the compute kernel."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#starting-a-jupyter-kernel-using-the-KernelManager-class\" data-toc-modified-id=\"starting-a-jupyter-kernel-using-the-KernelManager-class-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>starting a jupyter kernel using the <code>KernelManager</code> class</a></span></li><li><span><a href=\"#interactions-with-the-jupyter-kernel-using-the-BlockingKernelClient-class\" data-toc-modified-id=\"interactions-with-the-jupyter-kernel-using-the-BlockingKernelClient-class-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>interactions with the jupyter kernel using the <code>BlockingKernelClient</code> class</a></span></li><li><span><a href=\"#sending-serialized-Python-objects-to-jupyter\" data-toc-modified-id=\"sending-serialized-Python-objects-to-jupyter-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>sending serialized Python objects to jupyter</a></span></li></ul></div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import jupyter_client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## starting a jupyter kernel using the `KernelManager` class\n",
    "\n",
    "Jupyter kernels are managed by a `KernelManager` class. This class is responsible to start, stop and manage a single compute kernel in a subprocess. It can be instantiated with default arguments with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "manager = jupyter_client.KernelManager()\n",
    "manager.is_alive()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Until now, this kernel has not been started. A _compute kernel_ can have different properties. For example, it could be a Python2 or Python3, a Haskell kernel or some other exotic specification. Jupyter stores the kernel specifications in a `KernelSpec` object. Let's have a look at it for the kernel that we have just instantiated:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'argv': ['/home/matto/miniconda3/bin/python',\n",
       "  '-m',\n",
       "  'ipykernel_launcher',\n",
       "  '-f',\n",
       "  '{connection_file}'],\n",
       " 'display_name': 'Python 3',\n",
       " 'env': {},\n",
       " 'interrupt_mode': 'signal',\n",
       " 'language': 'python',\n",
       " 'metadata': {}}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "manager.kernel_spec.to_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This dictionary describes in detail the kernel that the _KernelManager_ manages. In our case, it is a Python3 kernel. Let's start the kernel with the obvious `start_kernel()` method and check again if it is alive:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "manager.start_kernel()\n",
    "manager.is_alive()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `KernelManager` provides different methods to extract information about the kernel. Communication with the kernel is done through a message interface (http://jupyter-client.readthedocs.io/en/stable/messaging.html). The ports that are used for these messages can be seen:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'control_port': 48159,\n",
       " 'hb_port': 44763,\n",
       " 'iopub_port': 59763,\n",
       " 'ip': '127.0.0.1',\n",
       " 'key': b'2830e36f-86c120ead1c005e5797396c1',\n",
       " 'shell_port': 41363,\n",
       " 'signature_scheme': 'hmac-sha256',\n",
       " 'stdin_port': 51377,\n",
       " 'transport': 'tcp'}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "manager.get_connection_info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## interactions with the jupyter kernel using the `BlockingKernelClient` class\n",
    "\n",
    "Interaction with the kernel is handled by a `Client` class. We are going to use the default `BlockingKernelClient` here. Such a client that is connected to the appropriate kernel can be obtained from the manager:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<jupyter_client.blocking.client.BlockingKernelClient at 0x7fc9643e4550>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client = manager.client()\n",
    "client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try to communicate with our kernel. We have different channels available. For our purposes the most important one is the iopub channel, that is used to send code and receive results. Checking if the kernel has sent any messages on this channel throws an `Empty` Exception:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "empty queue\n",
      "<class 'queue.Empty'>\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    client.get_iopub_msg(timeout=0)\n",
    "except Exception as err:\n",
    "    print('empty queue')\n",
    "    print(type(err))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we would expect, there is no message from the kernel, because we didn't start sending any requests to it so far. A simple computation can be send through the `.execute()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'fea281ed-77987d7a89692479f3c93088'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.execute('3+2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This triggers a cascade of message responses from the jupyter kernel that we can inspect one by one using the `.get_iopub_msg`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'buffers': [],\n",
       " 'content': {'execution_state': 'starting'},\n",
       " 'header': {'date': datetime.datetime(2018, 5, 22, 11, 30, 23, 3695, tzinfo=tzutc()),\n",
       "  'msg_id': '16f7a9a3-7df46aea379ad558a48a8ed9',\n",
       "  'msg_type': 'status',\n",
       "  'session': 'c9b86ad9-e4f01c7571f295fe001b9b08',\n",
       "  'username': 'matto',\n",
       "  'version': '5.3'},\n",
       " 'metadata': {},\n",
       " 'msg_id': '16f7a9a3-7df46aea379ad558a48a8ed9',\n",
       " 'msg_type': 'status',\n",
       " 'parent_header': {}}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.get_iopub_msg()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above message is just telling us that the kernel is now _busy_ with our request ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'buffers': [],\n",
       " 'content': {'execution_state': 'busy'},\n",
       " 'header': {'date': datetime.datetime(2018, 5, 22, 11, 30, 23, 4407, tzinfo=tzutc()),\n",
       "  'msg_id': '9f1441d8-c44ac9476becd84ee0c147b6',\n",
       "  'msg_type': 'status',\n",
       "  'session': 'c9b86ad9-e4f01c7571f295fe001b9b08',\n",
       "  'username': 'matto',\n",
       "  'version': '5.3'},\n",
       " 'metadata': {},\n",
       " 'msg_id': '9f1441d8-c44ac9476becd84ee0c147b6',\n",
       " 'msg_type': 'status',\n",
       " 'parent_header': {'date': datetime.datetime(2018, 5, 22, 11, 30, 22, 363864, tzinfo=tzutc()),\n",
       "  'msg_id': 'fea281ed-77987d7a89692479f3c93088',\n",
       "  'msg_type': 'execute_request',\n",
       "  'session': 'af27da15-a5a7241d50cd60c5d20a783e',\n",
       "  'username': 'matto',\n",
       "  'version': '5.3'}}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.get_iopub_msg()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "the next message shows the statement `'3+2'` that is executed on the kernel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'buffers': [],\n",
       " 'content': {'code': '3+2', 'execution_count': 1},\n",
       " 'header': {'date': datetime.datetime(2018, 5, 22, 11, 30, 23, 4612, tzinfo=tzutc()),\n",
       "  'msg_id': '7a9c3f88-df5b892f6f5760c47fa31428',\n",
       "  'msg_type': 'execute_input',\n",
       "  'session': 'c9b86ad9-e4f01c7571f295fe001b9b08',\n",
       "  'username': 'matto',\n",
       "  'version': '5.3'},\n",
       " 'metadata': {},\n",
       " 'msg_id': '7a9c3f88-df5b892f6f5760c47fa31428',\n",
       " 'msg_type': 'execute_input',\n",
       " 'parent_header': {'date': datetime.datetime(2018, 5, 22, 11, 30, 22, 363864, tzinfo=tzutc()),\n",
       "  'msg_id': 'fea281ed-77987d7a89692479f3c93088',\n",
       "  'msg_type': 'execute_request',\n",
       "  'session': 'af27da15-a5a7241d50cd60c5d20a783e',\n",
       "  'username': 'matto',\n",
       "  'version': '5.3'}}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.get_iopub_msg()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "after code execution, a return value (stored in `'content': {'data': ....}'`) is sent to the client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'buffers': [],\n",
       " 'content': {'data': {'text/plain': '5'},\n",
       "  'execution_count': 1,\n",
       "  'metadata': {}},\n",
       " 'header': {'date': datetime.datetime(2018, 5, 22, 11, 30, 23, 6003, tzinfo=tzutc()),\n",
       "  'msg_id': '28b19685-167db8d658ce45f6f17b88cb',\n",
       "  'msg_type': 'execute_result',\n",
       "  'session': 'c9b86ad9-e4f01c7571f295fe001b9b08',\n",
       "  'username': 'matto',\n",
       "  'version': '5.3'},\n",
       " 'metadata': {},\n",
       " 'msg_id': '28b19685-167db8d658ce45f6f17b88cb',\n",
       " 'msg_type': 'execute_result',\n",
       " 'parent_header': {'date': datetime.datetime(2018, 5, 22, 11, 30, 22, 363864, tzinfo=tzutc()),\n",
       "  'msg_id': 'fea281ed-77987d7a89692479f3c93088',\n",
       "  'msg_type': 'execute_request',\n",
       "  'session': 'af27da15-a5a7241d50cd60c5d20a783e',\n",
       "  'username': 'matto',\n",
       "  'version': '5.3'}}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.get_iopub_msg()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "finally, the kernel goes back into idle mode again, and this is the end of the messages that follow a single execution, as can be seen by checking:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    client.get_iopub_msg(timeout=0)\n",
    "except Exception as err:\n",
    "    print('empty queue')\n",
    "    print(type(err))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All of this can be done in a much compressed and safer form using the `.execute_interactive()` method that immediately returns the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "{'buffers': [],\n",
       " 'content': {'execution_count': 2,\n",
       "  'payload': [],\n",
       "  'status': 'ok',\n",
       "  'user_expressions': {}},\n",
       " 'header': {'date': datetime.datetime(2018, 5, 22, 11, 30, 23, 46009, tzinfo=tzutc()),\n",
       "  'msg_id': '8aa270e2-b9cd67c9971a398ae45d91ca',\n",
       "  'msg_type': 'execute_reply',\n",
       "  'session': 'c9b86ad9-e4f01c7571f295fe001b9b08',\n",
       "  'username': 'matto',\n",
       "  'version': '5.3'},\n",
       " 'metadata': {'dependencies_met': True,\n",
       "  'engine': '7adc130c-cb3c-4783-bdf4-2e0eaf5015b3',\n",
       "  'started': '2018-05-22T11:30:23.042660Z',\n",
       "  'status': 'ok'},\n",
       " 'msg_id': '8aa270e2-b9cd67c9971a398ae45d91ca',\n",
       " 'msg_type': 'execute_reply',\n",
       " 'parent_header': {'date': datetime.datetime(2018, 5, 22, 11, 30, 23, 41239, tzinfo=tzutc()),\n",
       "  'msg_id': 'e649c9ff-d66ab94382f2621896b3c30c',\n",
       "  'msg_type': 'execute_request',\n",
       "  'session': 'af27da15-a5a7241d50cd60c5d20a783e',\n",
       "  'username': 'matto',\n",
       "  'version': '5.3'}}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client.execute_interactive('3+2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "manager.shutdown_kernel()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## sending serialized Python objects to jupyter\n",
    "\n",
    "Python objects can be serialized and send as a byte string to a jupyter kernel through the message interface. Python provides the pickle library to serialize objects. The following cells demonstrate how this can be done. First we open a kernel with a specific ID name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "manager = jupyter_client.KernelManager(connection_file='mykernel.json')\n",
    "manager.start_kernel()\n",
    "manager.is_alive()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can connect to this kernel with an external console using `jupyter console --existing mykernel.json`. We are now going to send a serialized Python dictionary object to this kernel:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'buffers': [],\n",
       " 'content': {'execution_count': 2,\n",
       "  'payload': [],\n",
       "  'status': 'ok',\n",
       "  'user_expressions': {}},\n",
       " 'header': {'date': datetime.datetime(2018, 5, 22, 11, 30, 24, 121976, tzinfo=tzutc()),\n",
       "  'msg_id': '45835c44-99384663671faf70ee518a9c',\n",
       "  'msg_type': 'execute_reply',\n",
       "  'session': 'bd6fdd1a-05b8e7149cae8fb4d6c8dc6d',\n",
       "  'username': 'matto',\n",
       "  'version': '5.3'},\n",
       " 'metadata': {'dependencies_met': True,\n",
       "  'engine': 'a20e8588-fc8d-4e5e-ba30-fe9fb1f943e0',\n",
       "  'started': '2018-05-22T11:30:24.115847Z',\n",
       "  'status': 'ok'},\n",
       " 'msg_id': '45835c44-99384663671faf70ee518a9c',\n",
       " 'msg_type': 'execute_reply',\n",
       " 'parent_header': {'date': datetime.datetime(2018, 5, 22, 11, 30, 23, 408879, tzinfo=tzutc()),\n",
       "  'msg_id': 'bd2a0eb5-f81c8f423d76dced76ab4a9b',\n",
       "  'msg_type': 'execute_request',\n",
       "  'session': 'a4202442-1a62d25bf330ab9810b861e1',\n",
       "  'username': 'matto',\n",
       "  'version': '5.3'}}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pickle\n",
    "client = manager.client()\n",
    "dictionary = {'a': 5, 'b': 3}\n",
    "client.execute('import pickle')\n",
    "client.execute_interactive('a=pickle.loads({})'.format(pickle.dumps(dictionary)), timeout=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "typing `a` in the external jupyter console should now show the dictionary. We can also retrieve it's content through the message interface:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'a': 5, 'b': 3}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = client.execute_interactive('a', timeout=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "even though we store the results of this command in `result`, we see a line `Out[5]` which is the actual output line of the jupyter kernel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "manager.shutdown_kernel()\n",
    "manager.is_alive()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}