Skip to content

Instantly share code, notes, and snippets.

@homm

homm/response to Lilliput.md

Created October 15, 2018 10:47
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save homm/61f3a274c489babdd690c67f1fdca9e2 to your computer and use it in GitHub Desktop.
Save homm/61f3a274c489babdd690c67f1fdca9e2 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
response to Lilliput.md

First of all, never trust a benchmark which you can't reproduce. You can't reproduce lilliput-bench for the obvious reason: there are no test images. And since this is the benchmark which tests graphics codecs, results heavily depend on exact files modes and even content. So the first, what you need, is create some test data.

Now, with the installation instruction, we can start. This is my results on i5-4430 CPU (the same Haswell architecture as in the original test) and Ubuntu 18.04.1 LTS, running on bare metal. Pillow-SIMD version 5.2.0.post0 compiled with AVX2 running on Python 2.7.

JPEG 1920x1080 header read:  1920x1080,  avg: 0.036190 ms  min: 0.033855 ms  max: 7.170200 ms
PNG 1920x1080 header read:   1920x1080,  avg: 0.040357 ms  min: 0.038862 ms  max: 0.266075 ms
WEBP 1920x1080 header read:  1920x1080,  avg: 0.678090 ms  min: 0.149965 ms  max: 14.508009 ms
GIF 1920x1080 header read:   1920x1080,  avg: 0.029355 ms  min: 0.025988 ms  max: 0.365019 ms
JPEG 256x256 => 32x32:  1132 Bytes,     avg: 0.87 ms    min: 0.81 ms    max: 1.06 ms
PNG 256x256 => 32x32:   3022 Bytes,     avg: 3.41 ms    min: 3.40 ms    max: 3.54 ms
WEBP 256x256 => 32x32:  678 Bytes,      avg: 2.25 ms    min: 2.24 ms    max: 2.49 ms
GIF 256x256 => 32x32:   1897 Bytes,     avg: 3.55 ms    min: 3.53 ms    max: 3.91 ms
JPEG 1920x1080 => 800x600:      112307 Bytes,   avg: 25.98 ms   min: 25.38 ms   max: 26.21 ms
PNG 1920x1080 => 800x600:       929497 Bytes,   avg: 331.29 ms  min: 330.92 ms  max: 331.71 ms
WEBP 1920x1080 => 800x600:      92466 Bytes,    avg: 121.59 ms  min: 121.45 ms  max: 126.35 ms
GIF 1920x1080 => 800x600:       411470 Bytes,   avg: 85.91 ms   min: 85.58 ms   max: 89.52 ms
PNG 256x256 => WEBP 256x256:    14668 Bytes,    avg: 12.43 ms   min: 12.41 ms   max: 12.65 ms
JPEG 256x256 => PNG 256x256:    109987 Bytes,   avg: 23.17 ms   min: 23.05 ms   max: 23.42 ms
GIF 256x256 => PNG 256x256:     43766 Bytes,    avg: 3.40 ms    min: 3.37 ms    max: 3.54 ms

Lilliput master (commit 3960219) running on go 1.10.1.

JPEG 1920x1080 header read:  1920x1080,  avg: 0.005600 ms,  min: 0.004915 ms,  max: 0.172405 ms
PNG 1920x1080 header read:   1920x1080,  avg: 0.003186 ms,  min: 0.002956 ms,  max: 0.051915 ms
WEBP 1920x1080 header read:  1920x1080,  avg: 0.001392 ms,  min: 0.001290 ms,  max: 0.017828 ms
GIF 1920x1080 header read:   1920x1080,  avg: 0.004994 ms,  min: 0.004805 ms,  max: 0.029046 ms
JPEG 256x256 => 32x32:  1117 Bytes,     avg: 0.67 ms,   min: 0.67 ms,   max: 0.97 ms
PNG 256x256 => 32x32:   2623 Bytes,     avg: 8.75 ms,   min: 8.72 ms,   max: 8.94 ms
WEBP 256x256 => 32x32:  650 Bytes,      avg: 8.43 ms,   min: 8.42 ms,   max: 9.05 ms
GIF 256x256 => 32x32:   1954 Bytes,     avg: 5.35 ms,   min: 5.29 ms,   max: 5.77 ms
JPEG 1920x1080 => 800x600:      104917 Bytes,   avg: 31.27 ms,  min: 31.22 ms,  max: 33.52 ms
PNG 1920x1080 => 800x600:       748236 Bytes,   avg: 406.31 ms, min: 405.98 ms, max: 407.65 ms
WEBP 1920x1080 => 800x600:      82102 Bytes,    avg: 683.10 ms, min: 682.86 ms, max: 689.18 ms
GIF 1920x1080 => 800x600:       314587 Bytes,   avg: 222.03 ms, min: 221.75 ms, max: 222.30 ms
PNG 256x256 => WEBP 256x256:    14668 Bytes,    avg: 76.34 ms,  min: 76.27 ms,  max: 76.54 ms
JPEG 256x256 => PNG 256x256:    108668 Bytes,   avg: 27.84 ms,  min: 27.79 ms,  max: 28.03 ms
GIF 256x256 => PNG 256x256:     102390 Bytes,   avg: 35.29 ms,  min: 35.09 ms,  max: 35.41 ms

Some conclusions can already be drawn.

  1. Pillow-SIMD totally lost the "header" test. Indeed, most file parsing in Pillow done in Python itself without any speedups. On the other hand, the absolute numbers are often less than 0.1 ms which is acceptable for most cases.
  2. There is a huge difference between 35 ms, the minimal time for querying the WEBP image in the original results, and 0.15 ms in my results. This is due to the problem in earlier versions of Pillow, when whole WEBP image is loaded during the opening. This was fixed eventually.
  3. This is absolutely meaningless to compare PNG and GIF compression ratio and speed. Both formats are lossless, so if you have identical source images, the more time you spend to compression, the better compression you get. The problem that source images are not identical (I'll explain this below) and default preferences are different.
  4. Something is broken with WEBP compression on Linux in the latest Lilliput. It slow as hell. WEBP is a format with lossy compression (at least in this test) and its speed shouldn't highly depend on compression ratio. This is confirmed by this issue.

The most important test (which reflects the performance of the library itself) is a test which is not demanding on compression ratio (i.e. lossy format), is not broken like WEBP and operates large enough images. So the are only one test meet the requirements, this is the JPEG 1920x1080 => 800x600 test. In the original benchmark it faster on Lilliput, in my benchmark it faster on Pillow-SIMD. There is no information which version of Pillow-SIMD was used for the original test or was it compiled with AVX2 support. But from the results, I assume a pre-4.3 version was used without significant optimizations for most general cases (which was already available at the time of the test). But this is only the beginning. In the original test, several more errors were made that distort the results. Let's look at the bencmark code.

Resizing filter

Lilliput uses INTER_AREA interpolation for OpenCV's resize function. Pillow uses a very flexible and high-quality resizing based on convolutions. It allows choosing exact quality for resizing and also affects performance. LANCZOS, which is used in the benchmark, produces very sharp images which looks much better than INTER_AREA interpolation. This affects not only the resize speed itself, but also the speed of following compression and the result size of coded images. BICUBIC filter a bit cheaper and its result is much closer to INTER_AREA, while still slightly better and shaper.

Added alpha-channel to PNG and WEBP images

As @kkopachev mentioned before, there is a very interesting line which converts JPEG images to RGB (while they are already) and adds alpha-channel to other formats. For JPEGs this is only one extra copying, but for other formats, this adds an extra size of compressed images and adds extra time on compression.

Preload codecs

Pillow is written in Python, dynamic scripting language. Each imported python file contributes to the loading time, that is why Pillow loads codecs lazily. The cons are that time is spent during test. You may see that JPEG and WEBP the maximum header read time is 100x higher than average and min.

What we get if we fix all the problems in Pillow-SIMD version:

JPEG 1920x1080 header read:     1920x1080,      avg: 0.068893 ms        min: 0.066996 ms        max: 0.257015 ms
PNG 1920x1080 header read:      1920x1080,      avg: 0.049018 ms        min: 0.043869 ms        max: 0.116825 ms
WEBP 1920x1080 header read:     1920x1080,      avg: 0.559106 ms        min: 0.155926 ms        max: 5.062103 ms
GIF 1920x1080 header read:      1920x1080,      avg: 0.035386 ms        min: 0.032902 ms        max: 0.418901 ms
JPEG 256x256 => 32x32:  1109 Bytes,     avg: 0.70 ms    min: 0.69 ms    max: 0.91 ms
PNG 256x256 => 32x32:   2603 Bytes,     avg: 3.05 ms    min: 3.04 ms    max: 3.15 ms
WEBP 256x256 => 32x32:  652 Bytes,      avg: 1.85 ms    min: 1.84 ms    max: 3.40 ms
GIF 256x256 => 32x32:   1897 Bytes,     avg: 3.42 ms    min: 3.41 ms    max: 3.58 ms
JPEG 1920x1080 => 800x600:      107480 Bytes,   avg: 23.43 ms   min: 22.57 ms   max: 23.53 ms
PNG 1920x1080 => 800x600:       810489 Bytes,   avg: 243.70 ms  min: 243.24 ms  max: 244.43 ms
WEBP 1920x1080 => 800x600:      84690 Bytes,    avg: 112.28 ms  min: 112.14 ms  max: 115.48 ms
GIF 1920x1080 => 800x600:       411470 Bytes,   avg: 84.42 ms   min: 84.26 ms   max: 87.32 ms
PNG 256x256 => WEBP 256x256:    14668 Bytes,    avg: 12.43 ms   min: 12.42 ms   max: 12.62 ms
JPEG 256x256 => PNG 256x256:    109987 Bytes,   avg: 23.18 ms   min: 23.11 ms   max: 23.28 ms
GIF 256x256 => PNG 256x256:     43766 Bytes,    avg: 3.42 ms    min: 3.39 ms    max: 3.50 ms

Fair result:

  • JPEG 256x256 => 32x32 — speed is equal, mainly because of the slow parsing of images in Python. Size is lower for Pillow-SIMD.
  • PNG and WEBP 256x256 => 32x32 — much faster in Pillow with the same size.
  • GIF 256x256 => 32x32 — faster in Pillow with smaller size.
  • JPEG 1920x1080 => 800x600 — faster in Pillow-SIMD (still the fastest software for resizing images on CPU) with the slightly bigger size due to sharper result.
  • PNG and GIF 1920x1080 => 800x600 — time better, size worse. As I said before, the more time you contribute to compression, the smaller output image you get. The difference only in settings.
  • WEBP 1920x1080 => 800x600 — slightly bigger file size due to a sharper result and dramatic slowdown for Lilliput.
  • PNG 256x256 => WEBP 256x256 — dramatic slowdown for WEBP in Lilliput.
  • JPEG 256x256 => PNG 256x256 — a bit faster, a bit larger due to PNG compression.
  • GIF 256x256 => PNG 256x256 — the dramatic slowdown for Lilliput due to unnecessary palette​ to RGB conversion.

So there is no tests except header read when Lilliput truly wins. In turn, I want to say that in Pillow we need to look at the image detection and parsing speed.

In total, I've made five pull requests in lilliput-bench repository:

@brian-armstrong-discord, could you look at them?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment