PycroManager-ImJoy-Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building GUI with Pycro-Manager with ImJoy\n",
    "\n",
    "Author: [Wei OUYANG](https://oeway.github.io/)\n",
    "\n",
    "[ImJoy](https://imjoy.io) is a web framework for building interactive analysis tools. You can also use it to build easy-to-use and interactive data acquisition tool together with pycro-manager.\n",
    "\n",
    "In this tutorial notebook, we will go through the steps for developing interactive GUI using ImJoy and Jupyter notebooks. Here are the steps:\n",
    "1. build a simple ImJoy plugin for visualizing images\n",
    "2. acquire images with pycro-manager and visualize it\n",
    "3. add buttons to acquire and process images\n",
    "4. deploy and publish the plugin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparation\n",
    "\n",
    "You will be able to follow this tutorial in a local installation of Jupyter notebook. To use Pycro-Manager(which connects to Micro-Manager), you need to run the jupyter notebook server (typically with the `jupyter notebook` command) on the computer with Micro-Manager.\n",
    "\n",
    "Importantly, Pycro-Manager exposes full access of your microscope to the python scripting interface, please be careful that some commands (e.g. moving the stage) may damage your hardware. Although this tutorial only involves camera control which is safe, we still recommend to disconnect your hardware and start Micro-Manager with the simulated demo devices for exploration, and only connect the hardware when you fully understand the scripts.\n",
    "\n",
    "\n",
    "1. Install Pycro-Manager, ImJoy and [ImJoy Jupyter Extension](https://github.com/imjoy-team/imjoy-jupyter-extension) by run `pip install pycromanager imjoy imjoy-jupyter-extension`, then start or restart your Jupyter notebook server by using `jupyter notebook` command.\n",
    "2. Create an empty notebook, or download and run [this one](https://github.com/micro-manager/pycro-manager/blob/master/docs/source/pycro_manager_imjoy_tutorial.ipynb). Make sure you see an ImJoy icon in the toolbar in opened notebooks.\n",
    "3. If you don't have Micro-Manager installed, download the lastest version of [micro-manager 2.0](https://micro-manager.org/wiki/Micro-Manager_Nightly_Builds)\n",
    "4. Run Micro-Manager, select tools-options, and check the box that says Run server on port 4827 (you only need to do this once)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# if you don't see an ImJoy icon (blue) in the toolbar, run this cell\n",
    "!pip install -U pycromanager imjoy imjoy-jupyter-extension\n",
    "\n",
    "# And, restart your jupyter notebook after running the above command"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To quickly verify whether you have everything ready, you should see something like `'MMCore version 10.1.0'` without error after running the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pycromanager import Bridge\n",
    "from imjoy import api\n",
    "bridge = Bridge()\n",
    "core = bridge.get_core()\n",
    "core.get_version_info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build your first ImJoy plugin\n",
    "\n",
    "Let's start by making a \"hello world\" plugin example with ImJoy.\n",
    "\n",
    "An ImJoy plugin is a class defines at least two functions `setup` and `run`. In the `setup` function we put preparation or initialization code and the `run` function is an entrypoint when the user starts the plugin. As an example, we do nothing in the `setup` function and popup a hello world message in the `run` function.\n",
    "\n",
    "Importantly, you need to export your plugin by running `api.export(ImJoyPlugin())` to register the plugin to the ImJoy core (running in the browser with the notebook page).\n",
    "\n",
    "Now run the following cell.\n",
    "\n",
    "If you see a popup message saying \"hello world\", congrats that you have build your first ImJoy plugin!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from imjoy import api\n",
    "\n",
    "class ImJoyPlugin():\n",
    "    '''Defines an ImJoy plugin'''\n",
    "    async def setup(self):\n",
    "        '''for initialization'''\n",
    "        pass\n",
    "\n",
    "    async def run(self, ctx):\n",
    "        '''called when the user run this plugin'''\n",
    "        \n",
    "        # show a popup message\n",
    "        await api.alert(\"hello world\")\n",
    "\n",
    "# register the plugin to the imjoy core\n",
    "api.export(ImJoyPlugin())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: if the `async` and `await` keywords are new to you, you may want to learn about an imporant programing style called \"asynchronous programming\". It's basically a cheap way to achieve parallelizatin in a single thread, and Python3 provides [asyncio API](https://docs.python.org/3/library/asyncio-task.html) for it. With the async/await syntax, you can write async code as you usually do with your other synchronous code.\n",
    "\n",
    "Don't worry if you don't fully understand asynchronous programming. For now you can treat it the same as regular python programming, but remember the following simplified rules:\n",
    "1. it is recommended to add `await` before every ImJoy api call except `api.export`, e.g.: do `await api.alert(\"hello\")`.\n",
    "2. if you used `await` in a function, then you have to also add `async def` to define the function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Snap an image in the plugin\n",
    "\n",
    "Now let's define a function for acquire images with Pycro-Manager and call it `snap_image()`. Add this function into the plugin class and use it in the `run` function.\n",
    "\n",
    "Run the fullowing cell, you should see a message if you acquired an image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from imjoy import api\n",
    "from pycromanager import Bridge\n",
    "\n",
    "class MyMicroscope():\n",
    "    '''Defines a Microscope plugin'''\n",
    "    async def setup(self):\n",
    "        '''initialize the pycro-manager bridge'''\n",
    "        bridge = Bridge()\n",
    "        self._core = bridge.get_core()\n",
    "    \n",
    "    def snap_image(self):\n",
    "        '''snape an image with the pycro-manager bridge and return it as a numpy array'''\n",
    "        self._core.snap_image()\n",
    "        tagged_image = self._core.get_tagged_image()\n",
    "        # get the pixels in numpy array and reshape it according to its height and width\n",
    "        image_array = np.reshape(tagged_image.pix, newshape=[-1, tagged_image.tags['Height'], tagged_image.tags['Width']])\n",
    "        # for display, we can scale the image into the range of 0~255\n",
    "        image_array = (image_array/image_array.max()*255).astype('uint8')\n",
    "        return image_array\n",
    "\n",
    "    async def run(self, ctx):\n",
    "        '''acquire one image and notify the user'''\n",
    "        img = self.snap_image()\n",
    "        # show a popup message\n",
    "        await api.alert(\"Acquired an image (size={}) with Pycro-Manager\".format(img.shape))\n",
    "\n",
    "# register the plugin to the imjoy core\n",
    "api.export(MyMicroscope())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Display image with an ImJoy window plugin\n",
    "\n",
    "In the following example, we will wrap the `MyMicroscope` class as an ImJoy plugin (by adding a `run` function). In the `run` function, we create a viewer by using the `itk-vtk-viewer` hosted on https://oeway.github.io/itk-vtk-viewer/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from imjoy import api\n",
    "from pycromanager import Bridge\n",
    "\n",
    "class MyMicroscope():\n",
    "    '''Defines a Microscope plugin'''\n",
    "    async def setup(self):\n",
    "        '''initialize the pycro-manager bridge'''\n",
    "        bridge = Bridge()\n",
    "        self._core = bridge.get_core()\n",
    "    \n",
    "    def snap_image(self):\n",
    "        '''snape an image with the pycro-manager bridge and return it as a numpy array'''\n",
    "        self._core.snap_image()\n",
    "        image_array = self._core.get_image().reshape(512, 512)\n",
    "        # for display, we can scale the image into the range of 0~255\n",
    "        image_array = (image_array/image_array.max()*255).astype('uint8')\n",
    "        return image_array\n",
    "\n",
    "    async def run(self, ctx):\n",
    "        '''acquire 100 images and show them with itk-vtk-viewer'''\n",
    "        viewer = await api.showDialog(src=\"https://oeway.github.io/itk-vtk-viewer/\")\n",
    "        api.showMessage('Acquiring 100 images')\n",
    "        for i in range(100):\n",
    "            img = self.snap_image()\n",
    "            await viewer.imshow(img)\n",
    "        api.showMessage('Done.')\n",
    "        \n",
    "api.export(MyMicroscope())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build a custom UI plugin for microscope control\n",
    "\n",
    "Since the above image viewer doesn't give much flexiblity, we can build a custom UI plugin to add some custom UI for controlling the microscope.\n",
    "\n",
    "The following example uses [Vue.js](https://vuejs.org/) and [Buefy](https://buefy.org/) to add buttons and controls. In addition, we introduce the [itk-vtk-viewer](https://kitware.github.io/itk-vtk-viewer/docs/embeddedViewer.html) in this UI plugin for displaying the numpy array image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "## ImJoy Plugin\n",
    "from IPython.display import HTML\n",
    "my_plugin_source = HTML('''\n",
    "<docs lang=\"markdown\">\n",
    "[TODO: write documentation for this plugin.]\n",
    "</docs>\n",
    "\n",
    "<config lang=\"json\">\n",
    "{\n",
    "  \"name\": \"PycroCam\",\n",
    "  \"type\": \"window\",\n",
    "  \"tags\": [],\n",
    "  \"ui\": \"\",\n",
    "  \"version\": \"0.1.2\",\n",
    "  \"api_version\": \"0.1.8\",\n",
    "  \"description\": \"A microscopy control plugin built on top of PycroManager\",\n",
    "  \"icon\": \"extension\",\n",
    "  \"inputs\": null,\n",
    "  \"outputs\": null,\n",
    "  \"env\": \"\",\n",
    "  \"requirements\": [\n",
    "      \"https://cdnjs.cloudflare.com/ajax/libs/vue/2.5.22/vue.min.js\",\n",
    "      \"https://unpkg.com/buefy/dist/buefy.min.css\",\n",
    "      \"https://unpkg.com/buefy/dist/buefy.min.js\",\n",
    "      \"https://cdn.materialdesignicons.com/5.3.45/css/materialdesignicons.min.css\",\n",
    "      \"https://oeway.github.io/itk-vtk-viewer/itkVtkViewerCDN.js\"\n",
    "  ],\n",
    "  \"dependencies\": [],\n",
    "  \"defaults\": {\"w\": 20, \"h\": 10},\n",
    "  \"runnable\": true\n",
    "}\n",
    "</config>\n",
    "\n",
    "<script lang=\"javascript\">\n",
    "\n",
    "api.registerCodec({\n",
    "  name: 'itkimage',\n",
    "  decoder: itkVtkViewer.utils.convertToItkImage\n",
    "})\n",
    "\n",
    "api.registerCodec({\n",
    "  name: 'ndarray',\n",
    "  decoder: itkVtkViewer.utils.ndarrayToItkImage\n",
    "})\n",
    "\n",
    "const app = new Vue({\n",
    "  el: '#app',\n",
    "  data: {\n",
    "    mmcore: null,\n",
    "    viewer: null,\n",
    "    liveRunning: false,\n",
    "    binning: 1,\n",
    "    exposure: 100,\n",
    "    cameraDevice: null,\n",
    "  },\n",
    "  methods: {\n",
    "    async init(mmcore){\n",
    "        this.mmcore = mmcore;\n",
    "        this.cameraDevice = await this.mmcore.getCameraDevice()\n",
    "        this.exposure = await this.mmcore.getExposure()\n",
    "        this.binning = await this.mmcore.getProperty(this.cameraDevice, 'Binning')\n",
    "        this.updateImage()\n",
    "    },\n",
    "    async createViewer(imageData, el) {\n",
    "        imageData = itkVtkViewer.utils.vtkITKHelper.convertItkToVtkImage(imageData)\n",
    "        // clear container\n",
    "        el.innerHTML = \"\";\n",
    "        const toolbar = document.createElement('div')\n",
    "        el.appendChild(toolbar)\n",
    "        const view = document.createElement('div')\n",
    "        el.appendChild(view)\n",
    "        const dims = imageData.getDimensions()\n",
    "        const is2D = dims.length === 2 || (dims.length === 3 && dims[2] === 1)\n",
    "        const viewer = itkVtkViewer.createViewer(view, {\n",
    "          image: imageData,\n",
    "          pointSets: null,\n",
    "          geometries: null,\n",
    "          use2D: is2D,\n",
    "          rotate: false,\n",
    "          uiContainer: toolbar\n",
    "        })\n",
    "        return viewer\n",
    "    },\n",
    "    async updateImage(){\n",
    "      if(!this.mmcore){\n",
    "          api.alert('No mmcore api provided.')\n",
    "          return\n",
    "      }\n",
    "      let img;\n",
    "      if(this.liveRunning){\n",
    "          img = await this.mmcore.getImage()\n",
    "      }\n",
    "      else{\n",
    "          img = await this.mmcore.snapImage()\n",
    "      }\n",
    "    \n",
    "      if(!this.viewer){\n",
    "        const elm = document.getElementById('viewer')\n",
    "        this.viewer = await this.createViewer(img, elm)\n",
    "      }\n",
    "      else{\n",
    "        this.viewer.setImage(itkVtkViewer.utils.vtkITKHelper.convertItkToVtkImage(img))\n",
    "        this.viewer.renderLater()\n",
    "          console.log('===>')\n",
    "      }\n",
    "      if(this.liveRunning){\n",
    "          this.updateImage()\n",
    "      }\n",
    "    },\n",
    "    async toggleLive(){\n",
    "      this.liveRunning = !this.liveRunning\n",
    "      if(this.liveRunning){\n",
    "          this.mmcore.startContinuousSequenceAcquisition(0)\n",
    "          this.updateImage()\n",
    "      }\n",
    "      else{\n",
    "          this.mmcore.stopSequenceAcquisition()\n",
    "      }\n",
    "    },\n",
    "    async showDevicePropertyBrowser(){\n",
    "        const browser = await api.showDialog({src: \"https://gist.github.com/oeway/d07f14fca4d9ab1febbde200d0369bf2\", data:{'mmcore': this.mmcore}})\n",
    "        browser.on('property_changed', (item)=>{\n",
    "            if(item.device === this.cameraDevice && item.prop === 'Exposure'){\n",
    "                this.exposure = item.value\n",
    "            }\n",
    "            else if(item.device === this.cameraDevice && item.prop === 'Binning'){\n",
    "                this.binning = item.value\n",
    "            }\n",
    "        })\n",
    "    }\n",
    "  }\n",
    "})\n",
    "\n",
    "class ImJoyPlugin {\n",
    "  async setup() {\n",
    "\n",
    "  }\n",
    "\n",
    "  async run(ctx) {\n",
    "    await app.init(ctx.data.mmcore)\n",
    "  }\n",
    "}\n",
    "\n",
    "api.export(new ImJoyPlugin())\n",
    "</script>\n",
    "\n",
    "<window lang=\"html\">\n",
    "  <div id=\"app\">\n",
    "        <section>\n",
    "            <div class=\"buttons\">\n",
    "                <b-button type=\"is-primary\" @click=\"updateImage()\" :loading=\"liveRunning\">Snap</b-button>\n",
    "                <b-button :type=\"liveRunning?'is-danger': 'is-primary is-light'\"\n",
    "                          @click=\"toggleLive()\">{{liveRunning?'Stop Live': 'Start Live'}}</b-button>\n",
    "                &nbsp;Exposure: <b-field>\n",
    "                    <b-input placeholder=\"Exposure\"\n",
    "                        size=\"is-small\"\n",
    "                        @input=\"mmcore.setExposure(parseFloat(exposure))\"\n",
    "                        v-model=\"exposure\"\n",
    "                        type=\"number\"\n",
    "                        min=\"0\"\n",
    "                        max=\"10000\">\n",
    "                    </b-input>\n",
    "                </b-field>\n",
    "                &nbsp;Binning: <b-field>\n",
    "                    <b-select placeholder=\"Binning\"\n",
    "                              size=\"is-small\"\n",
    "                              @input=\"mmcore.setProperty(cameraDevice, 'Binning', binning)\"\n",
    "                              v-model=\"binning\">\n",
    "                        <option :value=\"1\">1</option>\n",
    "                        <option :value=\"2\">2</option>\n",
    "                        <option :value=\"4\">4</option>\n",
    "                        <option :value=\"8\">8</option>\n",
    "                        <option :value=\"16\">16</option>\n",
    "                    </b-select>\n",
    "                </b-field>\n",
    "                &nbsp;&nbsp;<b-button size=\"is-small\" @click=\"showDevicePropertyBrowser()\">Device Properties</b-button>\n",
    "            </div>\n",
    "            \n",
    "\n",
    "        </section>\n",
    "        <section id=\"viewer-section\">\n",
    "            <div id=\"viewer\" class=\"viewer-container\">\n",
    "\n",
    "            </div>\n",
    "      </section>\n",
    "  </div>\n",
    "</window>\n",
    "\n",
    "<style lang=\"css\">\n",
    "#app{\n",
    "    margin-top: 2px;\n",
    "}\n",
    "#viewer-section{\n",
    "  display: table;\n",
    "  margin-top: 5px;\n",
    "  max-width: calc( 100vh -  36px );\n",
    "}\n",
    ".viewer-container{\n",
    "  display: table-cell;\n",
    "  position: relative;\n",
    "}\n",
    "    \n",
    ".field {\n",
    "    margin-bottom: .2rem!important;\n",
    "}\n",
    "</style>\n",
    "''')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Use the UI plugin with PycroManager"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "from imjoy import api\n",
    "import numpy as np\n",
    "from pycromanager import Bridge\n",
    "\n",
    "\n",
    "class MyMicroscope():\n",
    "    async def setup(self):\n",
    "        bridge = Bridge()\n",
    "        self._core = bridge.get_core()\n",
    "        exposure = self._core.get_exposure()\n",
    "        api.showMessage('MMcore loaded, exposure: ' + str(exposure))\n",
    "\n",
    "    def snap_image(self):\n",
    "        if self._core.is_sequence_running():\n",
    "            self._core.stop_sequence_acquisition()\n",
    "        self._core.snap_image()\n",
    "        tagged_image = self._core.get_tagged_image()\n",
    "        image_array = np.reshape(tagged_image.pix, newshape=[-1, tagged_image.tags['Height'], tagged_image.tags['Width']])\n",
    "        image_array = (image_array/image_array.max()*255).astype('uint8')\n",
    "        return image_array\n",
    "    \n",
    "    def get_image(self):\n",
    "        # we can also check remaining with getRemainingImageCount()\n",
    "        tagged_image = self._core.get_tagged_image()\n",
    "        image_array = np.reshape(tagged_image.pix, newshape=[-1, tagged_image.tags['Height'], tagged_image.tags['Width']])\n",
    "        image_array = (image_array/image_array.max()*255).astype('uint8')\n",
    "        return image_array\n",
    "\n",
    "    def get_device_properties(self):\n",
    "        core = self._core\n",
    "        devices = core.get_loaded_devices()\n",
    "        devices = [devices.get(i) for i in range(devices.size())]\n",
    "        device_items = []\n",
    "        for device in devices:\n",
    "            names = core.get_device_property_names(device)\n",
    "            props = [names.get(i) for i in range(names.size())]\n",
    "            property_items = []\n",
    "            for prop in props:\n",
    "                value = core.get_property(device, prop)\n",
    "                is_read_only = core.is_property_read_only(device, prop)\n",
    "                if core.has_property_limits(device, prop):\n",
    "                    lower = core.get_property_lower_limit(device, prop)\n",
    "                    upper = core.get_property_upper_limit(device, prop)\n",
    "                    allowed = {\"type\": \"range\", \"min\": lower, \"max\": upper, \"readOnly\": is_read_only}\n",
    "                else:\n",
    "                    allowed = core.get_allowed_property_values(device, prop)\n",
    "                    allowed = {\"type\": \"enum\", \"options\": [allowed.get(i) for i in range(allowed.size())], \"readOnly\": is_read_only}\n",
    "                property_items.append({\"device\": device, \"name\": prop, \"value\": value, \"allowed\": allowed})\n",
    "                # print('===>', device, prop, value, allowed)\n",
    "            if len(property_items) > 0:\n",
    "                device_items.append({\"name\": device, \"value\": \"{} properties\".format(len(props)), \"items\": property_items})\n",
    "        return device_items\n",
    "\n",
    "    async def run(self, ctx):\n",
    "        mmcore_api = {\n",
    "            \"_rintf\": True,\n",
    "            \"snapImage\": self.snap_image,\n",
    "            \"getImage\": self.get_image,\n",
    "            \"getDeviceProperties\": self.get_device_properties,\n",
    "            \"getCameraDevice\": self._core.get_camera_device,\n",
    "            \"setCameraDevice\": self._core.set_camera_device,\n",
    "            \"startContinuousSequenceAcquisition\": self._core.start_continuous_sequence_acquisition,\n",
    "            \"stopSequenceAcquisition\": self._core.stop_sequence_acquisition,\n",
    "            \"setExposure\": self._core.set_exposure,\n",
    "            \"getExposure\": self._core.get_exposure,\n",
    "            \"setProperty\": self._core.set_property,\n",
    "            \"getProperty\": self._core.get_property\n",
    "        }\n",
    "        viewer = await api.createWindow(src=my_plugin_source, data={'mmcore': mmcore_api})\n",
    "\n",
    "api.export(MyMicroscope())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}