Jupyter Orbit

Jupyter Orbit.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Jupyter Orbit\n",
    "> Legos for Interactive Computing\n",
    "\n",
    "**tl;dr: See the [Architecture](#Architecture) diagram**\n",
    "\n",
    "![](http://i.imgur.com/BaI7cDd.jpg)\n",
    "> Source: [NASA](http://www.jpl.nasa.gov/news/news.php?release=2011-241)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Motivation\n",
    "The _Single User Desktop_ Jupyter story is presently:\n",
    "\n",
    "> _use whatever your OS (doesn't) offers or, like, maybe, conda, then use that to install Jupyter, and maybe remember what you did last time, and install some extensions (after you install npm) and kernels (with another package manager, with some instructions in a gist). When it breaks, hit the mailing list with a traceback... or use the cloud... or just go back to matlab/mathematica_\n",
    "\n",
    "This proposes a native client installer for desktop OSes which tackles the _Other Hard Things_ that so far have been outside the Jupyter definition of _Interactive Computing_:\n",
    "- Cross-platform installation\n",
    "- Profile management\n",
    "- Polyglot package management\n",
    "- Sane browser environment (Chromium)\n",
    "\n",
    "The user story would be:\n",
    "\n",
    "> _Download an Installer. Double click it. When it starts, from a menu of possible compute kernels, click one and go get a coffee. When you get back, there's a fresh, local running Jupyter Hub and Kernel of your choice. While in the Kernel, add some notebook extensions, or even change how Hub works. _"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Enablers and Challenges\n",
    "The primary enabler is `docker`, which gives us (with VirtualBox) a cross-platform environment for reproducible compute baselines. We handle the few outliers of hardware integration (potentially in a poly-kernel manner), and stop worrying about windows/osx install problems.\n",
    "\n",
    "Docker images can get ludicrously big. Ideally, standardized on a minimal Linux distribution (i.e. debian ~150mb), all containers (Hub, kernel) would be bringing close the miminum additional layers.\n",
    "\n",
    "Maybe don't, by default, enable the hub image with full latex and pandoc, but offer an upgraded, derivative image."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Distribution and Applications\n",
    "\n",
    "- In Atom \"store\"\n",
    "    - search, click to install\n",
    "        - everything else installed on demand\n",
    "        - fine for people with big pipes\n",
    "- 8GB DVD iso (hosted, torrent, out-sourced on-demand duplication and shipping)\n",
    "    - All the OS installers\n",
    "    - maybe bootable!\n",
    "    - hub image\n",
    "    - as many kernel images as is reasonable (see baseline image)\n",
    "    - a data set image (a la try.j.o)\n",
    "    - some curated extensions, t/f local impersonations of various registries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Proposed Stack\n",
    "- _native_ browser\n",
    "    - never abandon the mainline browser experience!\n",
    "- _native_ [atom.io](http://atom.io)\n",
    "    - provides OSX, Linux and Windows:\n",
    "      - pluggable extension framework with hot updates\n",
    "      - sane browser environment (Chromium) as a highly automateable, testable baseline\n",
    "      - already built installers\n",
    "    - gonna need a text editor anyway!\n",
    "- atom extensions a la [nuclide](http://nuclide.io/) (holding back here, this is just the MVP components)\n",
    "    - `atom-jupyter-engine`\n",
    "        - manage a `docker-compose` confection (per profile? project?)\n",
    "    - `atom-jupyter-bazaar`\n",
    "        - pluggable package manager wrapper\n",
    "            - kernel/server-aware search of _n_ package managers\n",
    "            - doesn't reinvent anything (no new hosted registries) but uses channels/scopes or at worst, metadata tags, not magic names\n",
    "        - manipulates metadata files (rather than running commands):\n",
    "            - as a baseline\n",
    "                - `atom-jupyter-bazaar-npm` for `package.json`\n",
    "                - `atom-jupyter-bazaar-docker` for `docker-compose.yml`\n",
    "                - `atom-jupyter-bazaar-pip` for `requirements.txt`\n",
    "            - however, in addition to these files, this data can all be stored in the notebook itself, making a completely reproducible environment\n",
    "    - `atom-jupyter-hardware`\n",
    "        - attempt to standardize common hardware, two patterns:\n",
    "            - device mapping (through vbox/docker): GPU\n",
    "            - C/Python/Node wrappers speaking kernel messages over ZMQ: just about everything else\n",
    "    - `atom-jupyter-notebook`\n",
    "        - sandboxed Chromium for running the actual notebook\n",
    "            - without giving the arbitrary JS access to, you know, all your files.\n",
    "- virtualbox\n",
    "    - if needed (as with [kitematic](https://kitematic.com/))\n",
    "    - on linux, just use docker\n",
    "    - shares local files into the docker environment\n",
    "- bundled [`docker-compose`](https://docs.docker.com/compose/), (and [machine](https://github.com/docker/machine) and [swarm](https://github.com/docker/swarm)?)\n",
    "    - manages the scut work of getting and configuring containers\n",
    "    - kitematic doesn't do this yet\n",
    "    - requires a python environment\n",
    "    - maps shared guest files to volemes\n",
    "- `docker`\n",
    "- `jupyter-hub`\n",
    "- `docker-spawner`\n",
    "- `docker-jupyter-<kernel>`\n",
    "    - a family of containers that provide just their kernel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notional Future Directions\n",
    "- RaspberryPi/Raspbian distro!\n",
    "- Mixed local/cloud compute\n",
    "- one-click publishing of a project as a standalone docker container\n",
    "- `atom-juptyer-data-torrent`\n",
    "    - share large datasets effortlessly with colleagues, immediately installed and up-to-date into kernels\n",
    "- Jupyter Maker\n",
    "    - Full 3D-accelerated computable CAD/engineering environment\n",
    "        - CAD kernel: OCC\n",
    "        - FEA\n",
    "        - CFD: [OpenFOAM](http://www.openfoam.com/)\n",
    "    - `aj-hardware` plugins for common 3D printers, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Architecture\n",
    "\n",
    "> Please wait for the embedded diagrams to appear... it won't be pretty, but it's better than pure text :)\n",
    ">\n",
    "> _Legend_\n",
    "- Red box: doesn't exist\n",
    "- Dotted-line Arrow: source creates/initializes/downloads target\n",
    "- Solid Line: endpoints are connected over an evented system: websockets, share files, REST, etc.\n",
    "- Grouping: logical environment\n",
    "\n",
    "\n",
    "<pre>\n",
    "<div class=\"mermaid\">\n",
    "\n",
    "\n",
    "    graph LR\n",
    "        subgraph The Internet\n",
    "            PackageManagers\n",
    "            atom.io/download\n",
    "        end\n",
    "        \n",
    "        subgraph Host FS\n",
    "            PackageManagerMetadata\n",
    "            c:\\notebooks\n",
    "        end\n",
    "        \n",
    "        subgraph Atom\n",
    "            apm\n",
    "            apm -.-> aj-hardware\n",
    "            apm -.-> aj-engine\n",
    "            apm -.-> aj-bazaar\n",
    "            apm -.-> aj-notebook\n",
    "            MyProject\n",
    "        end\n",
    "        \n",
    "        subgraph Guest OS\n",
    "            docker-compose\n",
    "            linux-guest\n",
    "            guest:8888\n",
    "            guest:/var/notebooks\n",
    "        end\n",
    "        \n",
    "        subgraph Hub Container\n",
    "            jupyter-hub\n",
    "            docker-spawner\n",
    "            npm\n",
    "            jupyter-notebook\n",
    "            conda\n",
    "            pip\n",
    "            hub:/var/notebooks\n",
    "            hub:8888\n",
    "        end\n",
    "        \n",
    "        subgraph Kernel Container\n",
    "            docker\n",
    "            example-kernel\n",
    "            kernel-package-manager\n",
    "            kernel:/var/notebooks\n",
    "        end\n",
    "    \n",
    "        subgraph Native Applications\n",
    "            Chrome.exe\n",
    "            VBoxHeadless.exe\n",
    "            Atom.exe\n",
    "        end\n",
    "        \n",
    "        subgraph Host HW\n",
    "            GPU\n",
    "            Leapmotion\n",
    "            Oscilloscope\n",
    "            host:8888\n",
    "        end\n",
    "        \n",
    "        atom.io/download -.-> Atom.exe\n",
    "        \n",
    "        Atom.exe -.-> apm\n",
    "        \n",
    "        host:8888 --- guest:8888\n",
    "        guest:8888 --- hub:8888\n",
    "        \n",
    "        aj-notebook --- host:8888\n",
    "        Chrome.exe --- host:8888\n",
    "        hub:8888 --- jupyter-hub\n",
    "        \n",
    "        jupyter-hub -.-> docker-spawner\n",
    "        docker-spawner -.-> docker\n",
    "        docker -.-> kernel-package-manager\n",
    "        kernel-package-manager -.-> example-kernel\n",
    "        example-kernel --- hub:8888\n",
    "        npm -.-> jupyter-notebook\n",
    "        \n",
    "        linux-guest -.-> docker-compose\n",
    "        docker-compose -.-> jupyter-hub\n",
    "        \n",
    "        aj-engine -.-> VBoxHeadless.exe\n",
    "        VBoxHeadless.exe -.-> linux-guest\n",
    "        VBoxHeadless.exe -.-> guest:8888\n",
    "        \n",
    "        Leapmotion --- aj-hardware\n",
    "        GPU --- aj-hardware\n",
    "        Oscilloscope --- aj-hardware\n",
    "        aj-hardware --- host:8888\n",
    "        \n",
    "        c:\\notebooks --- guest:/var/notebooks\n",
    "        guest:/var/notebooks --- hub:/var/notebooks\n",
    "        guest:/var/notebooks --- kernel:/var/notebooks\n",
    "        c:\\notebooks --- MyProject\n",
    "        \n",
    "        PackageManagers --- apm\n",
    "        PackageManagers --- aj-bazaar\n",
    "        \n",
    "        \n",
    "        aj-bazaar -.-> PackageManagerMetadata\n",
    "        PackageManagerMetadata -.-> docker-compose\n",
    "        PackageManagerMetadata -.-> docker\n",
    "        PackageManagerMetadata -.-> npm\n",
    "        PackageManagerMetadata -.-> conda\n",
    "        PackageManagerMetadata -.-> pip\n",
    "        PackageManagerMetadata -.-> kernel-package-manager\n",
    "        \n",
    "        classDef missing stroke:#f00;\n",
    "        class aj-bazaar,aj-hardware,aj-engine,aj-notebook missing;\n",
    "\n",
    "</div>\n",
    "</pre>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<script src=\"//cdn.rawgit.com/bollwyvl/53e64cdafba38461943b/raw/0815758d591dfaf0f4918b388aed1bf11d82160d/mermaid.full.js\"></script>\n",
       "<style>\n",
       "#notebook-container {\n",
       "    width: 99%;\n",
       "}\n",
       ".mermaid *{font-family: sans-serif; }\n",
       ".mermaid .node, .mermaid .cluster{\n",
       "  fill: white !important;\n",
       "  stroke: black !important;\n",
       "  stroke-width: 1px !important;\n",
       "}\n",
       ".mermaid div{\n",
       "  text-align: center;\n",
       "}\n",
       ".mermaid .label{\n",
       "  color: black;\n",
       "}\n",
       ".mermaid .cluster > rect {\n",
       "  stroke: #666;\n",
       "}\n",
       ".mermaid .missing rect{\n",
       "  stroke: red;\n",
       "}\n",
       "\n",
       "</style>\n",
       "<script>$(function(){\n",
       "    // mermaid load a touch weirdly: try immediately, but try again later if it's not available\n",
       "    var initMermaid = function(){\n",
       "        return (window.mermaid && mermaid.init()) || setTimeout(initMermaid, 50);         \n",
       "    }\n",
       "    initMermaid();\n",
       "\n",
       "    // for live editing, re-render only the text of the current cell\n",
       "    window.IPython && $(IPython.events).on(\"rendered.MarkdownCell\", function(evt, data){\n",
       "        // this is using a hacked mermaid that accepts some nodes!\n",
       "        mermaid.init(undefined, data.cell.element.find(\".mermaid\"));\n",
       "    });\n",
       "});</script>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<script src=\"//cdn.rawgit.com/bollwyvl/53e64cdafba38461943b/raw/0815758d591dfaf0f4918b388aed1bf11d82160d/mermaid.full.js\"></script>\n",
    "<style>\n",
    "#notebook-container {\n",
    "    width: 99%;\n",
    "}\n",
    ".mermaid *{font-family: sans-serif; }\n",
    ".mermaid .node, .mermaid .cluster{\n",
    "  fill: white !important;\n",
    "  stroke: black !important;\n",
    "  stroke-width: 1px !important;\n",
    "}\n",
    ".mermaid div{\n",
    "  text-align: center;\n",
    "}\n",
    ".mermaid .label{\n",
    "  color: black;\n",
    "}\n",
    ".mermaid .cluster > rect {\n",
    "  stroke: #666;\n",
    "}\n",
    ".mermaid .missing rect{\n",
    "  stroke: red;\n",
    "}\n",
    "\n",
    "</style>\n",
    "<script>$(function(){\n",
    "    // mermaid load a touch weirdly: try immediately, but try again later if it's not available\n",
    "    var initMermaid = function(){\n",
    "        return (window.mermaid && mermaid.init()) || setTimeout(initMermaid, 50);         \n",
    "    }\n",
    "    initMermaid();\n",
    "\n",
    "    // for live editing, re-render only the text of the current cell\n",
    "    window.IPython && $(IPython.events).on(\"rendered.MarkdownCell\", function(evt, data){\n",
    "        // this is using a hacked mermaid that accepts some nodes!\n",
    "        mermaid.init(undefined, data.cell.element.find(\".mermaid\"));\n",
    "    });\n",
    "});</script>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}