Micro-Manager-pymmcore-ImJoy-Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "Before start this notebook, you need to:\n",
    "1. Install ImJoy Jupyter Extension by run `pip install imjoy-jupyter-extension`, restart Jupyter notebook, and make sure you see an ImJoy icon in the toolbar in opened notebooks.\n",
    "1. Download the lastest version of [micro-manager 2.0](https://micro-manager.org/wiki/Micro-Manager_Nightly_Builds)\n",
    "1. Install pymmcore using `pip install pymmcore`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "!pip install -U pymmcore imjoy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set the path to your micromanager installation and config file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "MM_DIR = \"/Applications/Micro-Manager-2.0.0-gamma1-20200824/\"\n",
    "MM_CONFIG_FILE = os.path.join(MM_DIR, \"MMConfig_demo.cfg\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Acquire images with pymmcore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from imjoy import api\n",
    "import pymmcore\n",
    "import os.path\n",
    "\n",
    "mmc = pymmcore.CMMCore()\n",
    "mmc.setDeviceAdapterSearchPaths([MM_DIR])\n",
    "mmc.loadSystemConfiguration(MM_CONFIG_FILE)\n",
    "mmc.snapImage()\n",
    "mmc.getImage()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Display image with an ImJoy window plugin\n",
    "\n",
    "In the following example, we will wrap the `MyMicroscope` class as an ImJoy plugin (by adding a `run` function). In the `run` function, we create a viewer by using the `itk-vtk-viewer` hosted on https://oeway.github.io/itk-vtk-viewer/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from imjoy import api\n",
    "import pymmcore\n",
    "import os.path\n",
    "\n",
    "mmc = pymmcore.CMMCore()\n",
    "mmc.setDeviceAdapterSearchPaths([MM_DIR])\n",
    "mmc.loadSystemConfiguration(MM_CONFIG_FILE)\n",
    "mmcore = mmc\n",
    "\n",
    "class MyMicroscope():\n",
    "    def setup(self):\n",
    "        self._core = mmcore\n",
    "        self._core.setExposure(float(10))\n",
    "        exposure = self._core.getExposure()\n",
    "        api.showMessage('MMcore loaded, exposure: ' + str(exposure))\n",
    "    \n",
    "    def snapImage(self):\n",
    "        self._core.snapImage()\n",
    "        image_array = self._core.getImage()\n",
    "        image_array = (image_array/image_array.max()*255).astype('uint8')\n",
    "        return image_array\n",
    "\n",
    "    async def run(self, ctx):\n",
    "        viewer = await api.showDialog(type=\"itk-vtk-viewer\",\n",
    "                                      src=\"https://oeway.github.io/itk-vtk-viewer/\")\n",
    "        api.showMessage('Acquiring 100 images')\n",
    "        for i in range(100):\n",
    "            await viewer.imshow(self.snapImage())\n",
    "        api.showMessage('Done.')\n",
    "        \n",
    "api.export(MyMicroscope())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build a custom UI plugin for microscope control\n",
    "\n",
    "Since the above image viewer doesn't give much flexiblity, we can build a custom UI plugin to add some custom UI for controlling the microscope.\n",
    "\n",
    "The following example uses [Vue.js](https://vuejs.org/) and [Buefy](https://buefy.org/) to add buttons and controls. In addition, we introduce the [itk-vtk-viewer](https://kitware.github.io/itk-vtk-viewer/docs/embeddedViewer.html) in this UI plugin for displaying the numpy array image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "## ImJoy Plugin\n",
    "from IPython.display import HTML\n",
    "my_plugin_source = HTML('''\n",
    "<docs lang=\"markdown\">\n",
    "[TODO: write documentation for this plugin.]\n",
    "</docs>\n",
    "\n",
    "<config lang=\"json\">\n",
    "{\n",
    "  \"name\": \"PycroCam\",\n",
    "  \"type\": \"window\",\n",
    "  \"tags\": [],\n",
    "  \"ui\": \"\",\n",
    "  \"version\": \"0.1.2\",\n",
    "  \"api_version\": \"0.1.8\",\n",
    "  \"description\": \"A microscopy control plugin built on top of PycroManager\",\n",
    "  \"icon\": \"extension\",\n",
    "  \"inputs\": null,\n",
    "  \"outputs\": null,\n",
    "  \"env\": \"\",\n",
    "  \"requirements\": [\n",
    "      \"https://cdnjs.cloudflare.com/ajax/libs/vue/2.5.22/vue.min.js\",\n",
    "      \"https://unpkg.com/buefy/dist/buefy.min.css\",\n",
    "      \"https://unpkg.com/buefy/dist/buefy.min.js\",\n",
    "      \"https://cdn.materialdesignicons.com/5.3.45/css/materialdesignicons.min.css\",\n",
    "      \"https://oeway.github.io/itk-vtk-viewer/itkVtkViewerCDN.js\"\n",
    "  ],\n",
    "  \"dependencies\": [],\n",
    "  \"defaults\": {\"w\": 20, \"h\": 10},\n",
    "  \"runnable\": true\n",
    "}\n",
    "</config>\n",
    "\n",
    "<script lang=\"javascript\">\n",
    "\n",
    "api.registerCodec({\n",
    "  name: 'itkimage',\n",
    "  decoder: itkVtkViewer.utils.convertToItkImage\n",
    "})\n",
    "\n",
    "api.registerCodec({\n",
    "  name: 'ndarray',\n",
    "  decoder: itkVtkViewer.utils.ndarrayToItkImage\n",
    "})\n",
    "\n",
    "const app = new Vue({\n",
    "  el: '#app',\n",
    "  data: {\n",
    "    mmcore: null,\n",
    "    viewer: null,\n",
    "    liveRunning: false,\n",
    "    binning: 1,\n",
    "    exposure: 100,\n",
    "    cameraDevice: null,\n",
    "  },\n",
    "  methods: {\n",
    "    async init(mmcore){\n",
    "        this.mmcore = mmcore;\n",
    "        this.cameraDevice = await this.mmcore.getCameraDevice()\n",
    "        this.exposure = await this.mmcore.getExposure()\n",
    "        this.binning = await this.mmcore.getProperty(this.cameraDevice, 'Binning')\n",
    "        this.updateImage()\n",
    "    },\n",
    "    async createViewer(imageData, el) {\n",
    "        imageData = itkVtkViewer.utils.vtkITKHelper.convertItkToVtkImage(imageData)\n",
    "        // clear container\n",
    "        el.innerHTML = \"\";\n",
    "        const toolbar = document.createElement('div')\n",
    "        el.appendChild(toolbar)\n",
    "        const view = document.createElement('div')\n",
    "        el.appendChild(view)\n",
    "        const dims = imageData.getDimensions()\n",
    "        const is2D = dims.length === 2 || (dims.length === 3 && dims[2] === 1)\n",
    "        const viewer = itkVtkViewer.createViewer(view, {\n",
    "          image: imageData,\n",
    "          pointSets: null,\n",
    "          geometries: null,\n",
    "          use2D: is2D,\n",
    "          rotate: false,\n",
    "          uiContainer: toolbar\n",
    "        })\n",
    "        return viewer\n",
    "    },\n",
    "    async updateImage(){\n",
    "      if(!this.mmcore){\n",
    "          api.alert('No mmcore api provided.')\n",
    "          return\n",
    "      }\n",
    "      let img;\n",
    "      if(this.liveRunning){\n",
    "          img = await this.mmcore.getImage()\n",
    "      }\n",
    "      else{\n",
    "          img = await this.mmcore.snapImage()\n",
    "      }\n",
    "    \n",
    "      if(!this.viewer){\n",
    "        const elm = document.getElementById('viewer')\n",
    "        this.viewer = await this.createViewer(img, elm)\n",
    "      }\n",
    "      else{\n",
    "        this.viewer.setImage(itkVtkViewer.utils.vtkITKHelper.convertItkToVtkImage(img))\n",
    "      }\n",
    "      if(this.liveRunning){\n",
    "          this.updateImage()\n",
    "      }\n",
    "    },\n",
    "    async toggleLive(){\n",
    "      this.liveRunning = !this.liveRunning\n",
    "      if(this.liveRunning){\n",
    "          this.mmcore.startContinuousSequenceAcquisition(0)\n",
    "          this.updateImage()\n",
    "      }\n",
    "      else{\n",
    "          this.mmcore.stopSequenceAcquisition()\n",
    "      }\n",
    "    },\n",
    "    async showDevicePropertyBrowser(){\n",
    "        const browser = await api.showDialog({src: \"https://gist.github.com/oeway/d07f14fca4d9ab1febbde200d0369bf2\", data:{'mmcore': this.mmcore}})\n",
    "        browser.on('property_changed', (item)=>{\n",
    "            if(item.device === this.cameraDevice && item.prop === 'Exposure'){\n",
    "                this.exposure = item.value\n",
    "            }\n",
    "            else if(item.device === this.cameraDevice && item.prop === 'Binning'){\n",
    "                this.binning = item.value\n",
    "            }\n",
    "        })\n",
    "    }\n",
    "  }\n",
    "})\n",
    "\n",
    "class ImJoyPlugin {\n",
    "  async setup() {\n",
    "\n",
    "  }\n",
    "\n",
    "  async run(ctx) {\n",
    "    await app.init(ctx.data.mmcore)\n",
    "  }\n",
    "}\n",
    "\n",
    "api.export(new ImJoyPlugin())\n",
    "</script>\n",
    "\n",
    "<window lang=\"html\">\n",
    "  <div id=\"app\">\n",
    "        <section>\n",
    "            <div class=\"buttons\">\n",
    "                <b-button type=\"is-primary\" @click=\"updateImage()\" :loading=\"liveRunning\">Snap</b-button>\n",
    "                <b-button :type=\"liveRunning?'is-danger': 'is-primary is-light'\"\n",
    "                          @click=\"toggleLive()\">{{liveRunning?'Stop Live': 'Start Live'}}</b-button>\n",
    "                &nbsp;Exposure: <b-field>\n",
    "                    <b-input placeholder=\"Exposure\"\n",
    "                        size=\"is-small\"\n",
    "                        @input=\"mmcore.setExposure(parseFloat(exposure))\"\n",
    "                        v-model=\"exposure\"\n",
    "                        type=\"number\"\n",
    "                        min=\"0\"\n",
    "                        max=\"10000\">\n",
    "                    </b-input>\n",
    "                </b-field>\n",
    "                &nbsp;Binning: <b-field>\n",
    "                    <b-select placeholder=\"Binning\"\n",
    "                              size=\"is-small\"\n",
    "                              @input=\"mmcore.setProperty(cameraDevice, 'Binning', binning)\"\n",
    "                              v-model=\"binning\">\n",
    "                        <option :value=\"1\">1</option>\n",
    "                        <option :value=\"2\">2</option>\n",
    "                        <option :value=\"4\">4</option>\n",
    "                        <option :value=\"8\">8</option>\n",
    "                        <option :value=\"16\">16</option>\n",
    "                    </b-select>\n",
    "                </b-field>\n",
    "                &nbsp;&nbsp;<b-button size=\"is-small\" @click=\"showDevicePropertyBrowser()\">Device Properties</b-button>\n",
    "            </div>\n",
    "            \n",
    "\n",
    "        </section>\n",
    "        <section id=\"viewer-section\">\n",
    "            <div id=\"viewer\" class=\"viewer-container\">\n",
    "\n",
    "            </div>\n",
    "      </section>\n",
    "  </div>\n",
    "</window>\n",
    "\n",
    "<style lang=\"css\">\n",
    "#app{\n",
    "    margin-top: 2px;\n",
    "}\n",
    "#viewer-section{\n",
    "  display: table;\n",
    "  margin-top: 5px;\n",
    "  max-width: calc( 100vh -  36px );\n",
    "}\n",
    ".viewer-container{\n",
    "  display: table-cell;\n",
    "  position: relative;\n",
    "}\n",
    "    \n",
    ".field {\n",
    "    margin-bottom: .2rem!important;\n",
    "}\n",
    "</style>\n",
    "''')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Use the UI plugin with PycroManager"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "class DotDict(dict):\n",
    "    __getattr__ = dict.__getitem__\n",
    "    __setattr__ = dict.__setitem__\n",
    "    __delattr__ = dict.__delitem__\n",
    "\n",
    "def to_camel_case(snake_str):\n",
    "    components = snake_str.split('_')\n",
    "    # We capitalize the first letter of each component except the first one\n",
    "    # with the 'title' method and join them together.\n",
    "    return components[0] + ''.join(x.title() for x in components[1:])\n",
    "\n",
    "\n",
    "class MMCoreWrapper():\n",
    "    '''\n",
    "    make a wrapper to mmcore so the function calls are more pythonic\n",
    "    meaning, all the camel case are converted into snake case\n",
    "    for example: mmc.setExposure are converted into mmc.set_exposure\n",
    "    '''\n",
    "    def __init__(self, obj):\n",
    "        self._wrapped_obj = obj\n",
    "\n",
    "    def get_tagged_image(self):\n",
    "        img = self.get_image()\n",
    "        return DotDict({\"pix\": img, \"tags\": {\"Height\": img.shape[0], \"Width\": img.shape[1]}})\n",
    "\n",
    "    def __getattr__(self, attr):\n",
    "        if attr in self.__dict__:\n",
    "            return getattr(self, attr)\n",
    "        attr = to_camel_case(attr)\n",
    "        return getattr(self._wrapped_obj, attr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "from imjoy import api\n",
    "import numpy as np\n",
    "import pymmcore\n",
    "import os.path\n",
    "\n",
    "mmc = pymmcore.CMMCore()\n",
    "mmc.setDeviceAdapterSearchPaths([MM_DIR])\n",
    "mmc.loadSystemConfiguration(MM_CONFIG_FILE)\n",
    "mmcore = MMCoreWrapper(mmc)\n",
    "\n",
    "\n",
    "class MyMicroscope():\n",
    "    async def setup(self):\n",
    "        self._core = mmcore\n",
    "        exposure = self._core.get_exposure()\n",
    "        api.showMessage('MMcore loaded, exposure: ' + str(exposure))\n",
    "\n",
    "    def snap_image(self):\n",
    "        if self._core.is_sequence_running():\n",
    "            self._core.stop_sequence_acquisition()\n",
    "        self._core.snap_image()\n",
    "        tagged_image = self._core.get_tagged_image()\n",
    "        image_array = np.reshape(tagged_image.pix, newshape=[-1, tagged_image.tags['Height'], tagged_image.tags['Width']])\n",
    "        image_array = (image_array/image_array.max()*255).astype('uint8')\n",
    "        return image_array\n",
    "    \n",
    "    def get_image(self):\n",
    "        # we can also check remaining with getRemainingImageCount()\n",
    "        tagged_image = self._core.get_tagged_image()\n",
    "        image_array = np.reshape(tagged_image.pix, newshape=[-1, tagged_image.tags['Height'], tagged_image.tags['Width']])\n",
    "        image_array = (image_array/image_array.max()*255).astype('uint8')\n",
    "        return image_array\n",
    "\n",
    "    def get_device_properties(self):\n",
    "        devices = core.get_loaded_devices()\n",
    "        if not isinstance(devices, tuple):\n",
    "            devices = [devices.get(i) for i in range(devices.size())]\n",
    "\n",
    "        device_items = []\n",
    "        for device in devices:\n",
    "            props = core.get_device_property_names(device)\n",
    "            if not isinstance(props, tuple):\n",
    "                props = [props.get(i) for i in range(props.size())]\n",
    "            property_items = []\n",
    "            for prop in props:\n",
    "                value = core.get_property(device, prop)\n",
    "                is_read_only = core.is_property_read_only(device, prop)\n",
    "                if core.has_property_limits(device, prop):\n",
    "                    lower = core.get_property_lower_limit(device, prop)\n",
    "                    upper = core.get_property_upper_limit(device, prop)\n",
    "                    allowed = {\"type\": \"range\", \"min\": lower, \"max\": upper, \"readOnly\": is_read_only}\n",
    "                else:\n",
    "                    allowed = core.get_allowed_property_values(device, prop)\n",
    "                    if not isinstance(allowed, tuple):\n",
    "                        allowed = [allowed.get(i) for i in range(allowed.size())]\n",
    "                    allowed = {\"type\": \"enum\", \"options\": allowed, \"readOnly\": is_read_only}\n",
    "                property_items.append({\"device\": device, \"name\": prop, \"value\": value, \"allowed\": allowed})\n",
    "                # print('===>', device, prop, value, allowed)\n",
    "            if len(property_items) > 0:\n",
    "                device_items.append({\"name\": device, \"value\": \"{} properties\".format(len(props)), \"items\": property_items})\n",
    "        return device_items\n",
    "\n",
    "    async def run(self, ctx):\n",
    "        mmcore_api = {\n",
    "            \"_rintf\": True,\n",
    "            \"snapImage\": self.snap_image,\n",
    "            \"getImage\": self.get_image,\n",
    "            \"getDeviceProperties\": self.get_device_properties,\n",
    "            \"getCameraDevice\": self._core.get_camera_device,\n",
    "            \"setCameraDevice\": self._core.set_camera_device,\n",
    "            \"startContinuousSequenceAcquisition\": self._core.start_continuous_sequence_acquisition,\n",
    "            \"stopSequenceAcquisition\": self._core.stop_sequence_acquisition,\n",
    "            \"setExposure\": self._core.set_exposure,\n",
    "            \"getExposure\": self._core.get_exposure,\n",
    "            \"setProperty\": self._core.set_property,\n",
    "            \"getProperty\": self._core.get_property\n",
    "        }\n",
    "        viewer = await api.createWindow(src=my_plugin_source, data={'mmcore': mmcore_api})\n",
    "\n",
    "api.export(MyMicroscope())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}