Benchmarks of JuliaImages compared to OpenCV and PIL (Pillow) on Python

JuliaImages.ipynb

LICENSE

MIT License

Copyright (c) 2019 - 2020 Soumitra Shewale (soumitradev)

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

OpenCV.ipynb

PILPython.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python PIL (Pillow) benchmark\n",
    "Since the Julia Benchmark package [BenchmarkTools.jl](https://github.com/JuliaCI/BenchmarkTools.jl) automatically selects the number of samples, the Python benchmarks (computed using [timeit](https://github.com/python/cpython/blob/3.8/Lib/timeit.py)) are configured according to the number of samples BenchmarkTools takes.\n",
    "\n",
    "Our tests consist of the following benchmarks:\n",
    "1. Loading images\n",
    "2. Saving images\n",
    "3. Resizing (Scaling) images\n",
    "4. Rotating Images\n",
    "5. Getting the histogram (Grayscale Histogram)\n",
    "6. Equalizing the Grayscale Histogram\n",
    "7. Converting image to grayscale\n",
    "8. Changing colorspace to HSV\n",
    "9. Applying Gaussian Blur"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import PIL\n",
    "import timeit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Loading images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.758 ms\n"
     ]
    }
   ],
   "source": [
    "s = \"\"\"\\\n",
    "img = Image.open(\"images/lighthouse.png\")\n",
    "\"\"\"\n",
    "\n",
    "setup = \"from PIL import Image\"\n",
    "\n",
    "print(str(round((timeit.timeit(stmt = s, setup = setup, number = 44)/44)*1000, 3)) + \" ms\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Saving images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "309.918 ms\n"
     ]
    }
   ],
   "source": [
    "s = \"\"\"\\\n",
    "img.save(\"imagecopy.png\")\n",
    "\"\"\"\n",
    "\n",
    "setup = \"\"\"\\\n",
    "from PIL import Image\n",
    "img = Image.open(\"images/lighthouse.png\")\n",
    "\"\"\"\n",
    "\n",
    "print(str(round((timeit.timeit(stmt = s, setup = setup, number = 50)/50)*1000, 3)) + \" ms\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Resizing (Scaling) images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "19.11 ms\n"
     ]
    }
   ],
   "source": [
    "s = \"\"\"\\\n",
    "xsize, ysize = img.size\n",
    "img.resize((xsize*3, ysize*3))\n",
    "\"\"\"\n",
    "\n",
    "setup = \"\"\"\\\n",
    "from PIL import Image\n",
    "img = Image.open(\"images/lighthouse.png\")\n",
    "\"\"\"\n",
    "\n",
    "print(str(round((timeit.timeit(stmt = s, setup = setup, number = 22)/22)*1000, 3)) + \" ms\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Rotating Images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.649 ms\n"
     ]
    }
   ],
   "source": [
    "s = \"\"\"\\\n",
    "img.rotate(90)\n",
    "\"\"\"\n",
    "\n",
    "setup = \"\"\"\\\n",
    "from PIL import Image\n",
    "img = Image.open(\"images/lighthouse.png\")\n",
    "\"\"\"\n",
    "\n",
    "print(str(round((timeit.timeit(stmt = s, setup = setup, number = 184)/184)*1000, 3)) + \" ms\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Getting the histogram (Grayscale Histogram)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.143 ms\n"
     ]
    }
   ],
   "source": [
    "s = \"\"\"\\\n",
    "img.histogram()\n",
    "\"\"\"\n",
    "\n",
    "setup = \"\"\"\\\n",
    "from PIL import Image\n",
    "img = Image.open(\"images/lighthouse.png\")\n",
    "\"\"\"\n",
    "\n",
    "print(str(round((timeit.timeit(stmt = s, setup = setup, number = 125)/125)*1000, 3)) + \" ms\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Equalizing the Grayscale Histogram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5.322 ms\n"
     ]
    }
   ],
   "source": [
    "s = \"\"\"\\\n",
    "PIL.ImageOps.equalize(img)\n",
    "\"\"\"\n",
    "\n",
    "setup = \"\"\"\\\n",
    "import PIL\n",
    "img = PIL.Image.open(\"images/lighthouse.png\")\n",
    "from PIL import ImageOps\n",
    "\"\"\"\n",
    "\n",
    "print(str(round((timeit.timeit(stmt = s, setup = setup, number = 63)/63)*1000, 3)) + \" ms\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Converting image to grayscale"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.831 ms\n"
     ]
    }
   ],
   "source": [
    "s = \"\"\"\\\n",
    "PIL.ImageOps.grayscale(img)\n",
    "\"\"\"\n",
    "\n",
    "setup = \"\"\"\\\n",
    "import PIL\n",
    "img = PIL.Image.open(\"images/lighthouse.png\")\n",
    "from PIL import ImageOps\n",
    "\"\"\"\n",
    "\n",
    "print(str(round((timeit.timeit(stmt = s, setup = setup, number = 1270)/1270)*1000, 3)) + \" ms\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. Changing colorspace to HSV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "26.352 ms\n"
     ]
    }
   ],
   "source": [
    "s = \"\"\"\\\n",
    "img.convert('HSV')\n",
    "\"\"\"\n",
    "\n",
    "setup = \"\"\"\\\n",
    "import PIL\n",
    "img = PIL.Image.open(\"images/lighthouse.png\")\n",
    "from PIL import ImageOps\n",
    "\"\"\"\n",
    "\n",
    "print(str(round((timeit.timeit(stmt = s, setup = setup, number = 362)/362)*1000, 3)) + \" ms\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. Applying Gaussian Blur"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "33.943 ms\n"
     ]
    }
   ],
   "source": [
    "\n",
    "s = \"\"\"\\\n",
    "gauss = img.filter(PIL.ImageFilter.GaussianBlur(radius=3))\n",
    "\"\"\"\n",
    "\n",
    "setup = \"\"\"\\\n",
    "import PIL\n",
    "img = PIL.Image.open(\"images/lighthouse.png\")\n",
    "from PIL import ImageFilter\n",
    "\"\"\"\n",
    "\n",
    "print(str(round((timeit.timeit(stmt = s, setup = setup, number = 75)/75)*1000, 3)) + \" ms\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary:\n",
    "\n",
    "Since `Images.jl` gives us the mean time while `timeit` gives us the total time of the benchmark, we have taken the mean time in `timeit` so that the benchmarks are comparable.\n",
    "\n",
    "### Table for comparison\n",
    "\n",
    "Here is the table of the Benchmarks:\n",
    "\n",
    "| Testcase/Framework               | Images.jl  | Pillow (PIL) | OpenCV    |\n",
    "|----------------------------------|------------|--------------|-----------|\n",
    "| Loading images                   | 114.129 ms | 3.758 ms     | 17.943 ms |\n",
    "| Saving images                    | 100.002 ms | 309.918 ms   | 37.732 ms |\n",
    "| Resizing (Scaling) images        | 230.759 ms | 19.11 ms     | 5.656 ms  |\n",
    "| Rotating images                  | 27.162 ms  | 3.649 ms     | 2.88 ms   |\n",
    "| Calculating Greyscale Histogram  | 39.977 ms  | 2.143 ms     | 0.418 ms  |\n",
    "| Greyscale Histogram equalization | 80.208 ms  | 5.322 ms     | 0.008 ms  |\n",
    "| Converting to Greyscale          | 3.931 ms   | 0.831 ms     | 0.212 ms  |\n",
    "| Changing colorspace to HSV       | 13.802 ms  | 26.352 ms    | 1.277 ms  |\n",
    "| Applying Gaussian Blur           | 67.063 ms  | 33.943 ms    | 1.626 ms  |\n",
    "\n",
    "We find that Pillow (PIL) consistently gives sub great timings for the operations with the one outlier of saving images, while OpenCV varies wildly and takes comparatively (to itself) more time for some basic tasks related to inputting and outputting files.\n",
    "\n",
    "Comparing Images.jl to the other 2 frameworks, Images.jl is never the fastest, which is a point where significant improvement can be made. However, Images.jl compares well with the two other frameworks when the operation includes changing the colorspace and saving images. Another point of concern is the fact that Images.jl is the slowest of the 3 frameworks in 7/9 of the tests. This might be due to OpenCV's powerful C and C++ bindings, or because of the powerful and fast `numpy` backend that Python has.\n",
    "\n",
    "Maybe the algorithms for Images.jl are not the most efficient? Maybe these algorithms need updating?"
   ]
  }
 ],
 "metadata": {
  "kernel_info": {
   "name": "python3"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "nteract": {
   "version": "0.15.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}