Create a gist now

Instantly share code, notes, and snippets.

Embed
Lilliput vs. Pillow-simd
Setup: Intel Haswell, Debian. Both tests ran against the same image libraries (libjpeg-turbo etc).
Both ran on a single thread only. Save qualities were 85 for JPEG and WEBP, compress level 7 for PNG.
Both tests ran many iterations and then averaged results.
Benchmarking code can be found at https://github.com/discordapp/lilliput-bench
Test types:
- Header reading: We don't actually know what's in a blob of image bytes when we get it. Reading the header allows us
to decide if we want to resize the image.
- Resize, 256x256 => 32x32: We have lots of icon-sized assets that we need resized into various smaller formats.
These make up a pretty sizable percentage of our resizes.
- Resize, 1920x1080 => 800x600: This test is a somewhat typical case for some arbitrary remote image that we've
fetched and have to crop and resize down to a size the client can use. This gives us a test case for large images.
- Transcode: A few various test cases that are useful for us. Just image decode/encode only, no resizing.
Pillow-simd
======================
JPEG 1920x1080 header read: 1920x1080, avg: 0.037861 ms min: 0.033855 ms max: 13.420820 ms
PNG 1920x1080 header read: 1920x1080, avg: 0.041355 ms min: 0.038862 ms max: 0.269175 ms
WEBP 1920x1080 header read: 1920x1080, avg: 37.851110 ms min: 35.255909 ms max: 65.973997 ms
GIF 1920x1080 header read: 1920x1080, avg: 0.034195 ms min: 0.031948 ms max: 0.703096 ms
JPEG 256x256 => 32x32: 1083 Bytes, avg: 0.97 ms min: 0.89 ms max: 1.75 ms
PNG 256x256 => 32x32: 2090 Bytes, avg: 1.39 ms min: 1.31 ms max: 2.42 ms
WEBP 256x256 => 32x32: 1366 Bytes, avg: 2.88 ms min: 2.65 ms max: 4.70 ms
GIF 256x256 => 32x32: 61644 Bytes, avg: 71.76 ms min: 68.54 ms max: 96.00 ms
JPEG 1920x1080 => 800x600: 123522 Bytes, avg: 39.03 ms min: 34.09 ms max: 42.21 ms
PNG 1920x1080 => 800x600: 856122 Bytes, avg: 278.56 ms min: 272.84 ms max: 298.88 ms
WEBP 1920x1080 => 800x600: 93564 Bytes, avg: 147.06 ms min: 142.81 ms max: 171.15 ms
GIF 1920x1080 => 800x600: 4017933 Bytes, avg: 1819.24 ms min: 1734.70 ms max: 1915.06 ms
PNG 256x256 => WEBP 256x256: 9790 Bytes, avg: 20.67 ms min: 20.12 ms max: 31.40 ms
JPEG 256x256 => PNG 256x256: 38724 Bytes, avg: 7.13 ms min: 6.76 ms max: 9.91 ms
GIF 256x256 => PNG 256x256: 8511 Bytes, avg: 1.50 ms min: 1.44 ms max: 1.92 ms
Lilliput
======================
JPEG 1920x1080 header read: 1920x1080, avg: 0.005083 ms, min: 0.004398 ms, max: 0.663941 ms
PNG 1920x1080 header read: 1920x1080, avg: 0.003636 ms, min: 0.003362 ms, max: 0.134671 ms
WEBP 1920x1080 header read: 1920x1080, avg: 0.002145 ms, min: 0.001795 ms, max: 1.949846 ms
GIF 1920x1080 header read: 1920x1080, avg: 0.003795 ms, min: 0.003521 ms, max: 0.140039 ms
JPEG 256x256 => 32x32: 1078 Bytes, avg: 0.62 ms, min: 0.58 ms, max: 1.07 ms
PNG 256x256 => 32x32: 1411 Bytes, avg: 0.92 ms, min: 0.88 ms, max: 1.43 ms
WEBP 256x256 => 32x32: 946 Bytes, avg: 3.19 ms, min: 2.87 ms, max: 4.57 ms
GIF 256x256 => 32x32: 18310 Bytes, avg: 27.96 ms, min: 27.31 ms, max: 39.77 ms
JPEG 1920x1080 => 800x600: 117991 Bytes, avg: 37.30 ms, min: 36.06 ms, max: 50.85 ms
PNG 1920x1080 => 800x600: 722169 Bytes, avg: 178.61 ms, min: 176.41 ms, max: 203.03 ms
WEBP 1920x1080 => 800x600: 88172 Bytes, avg: 129.14 ms, min: 124.67 ms, max: 171.35 ms
GIF 1920x1080 => 800x600: 2372725 Bytes, avg: 3255.42 ms, min: 3220.95 ms, max: 3301.01 ms
PNG 256x256 => WEBP 256x256: 9790 Bytes, avg: 22.52 ms, min: 21.53 ms, max: 31.95 ms
JPEG 256x256 => PNG 256x256: 40134 Bytes, avg: 6.38 ms, min: 6.20 ms, max: 9.17 ms
GIF 256x256 => PNG 256x256: 18053 Bytes, avg: 5.57 ms, min: 5.44 ms, max: 7.88 ms
Conclusion:
Lilliput seems to be a suitable replacement for pillow simd for our use cases.
@fenollp

This comment has been minimized.

Show comment
Hide comment
@fenollp

fenollp Nov 14, 2017

Kinda weird on GIF 1920x1080 => 800x600:

GIF 1920x1080 => 800x600: 4017933 Bytes, avg: 1819.24 ms min: 1734.70 ms max: 1915.06 ms
lilliput:
GIF 1920x1080 => 800x600: 2372725 Bytes, avg: 3255.42 ms, min: 3220.95 ms, max: 3301.01 ms

Might be a good idea to watch closely clients uploading large GIFs (who does that anyway?)

fenollp commented Nov 14, 2017

Kinda weird on GIF 1920x1080 => 800x600:

GIF 1920x1080 => 800x600: 4017933 Bytes, avg: 1819.24 ms min: 1734.70 ms max: 1915.06 ms
lilliput:
GIF 1920x1080 => 800x600: 2372725 Bytes, avg: 3255.42 ms, min: 3220.95 ms, max: 3301.01 ms

Might be a good idea to watch closely clients uploading large GIFs (who does that anyway?)

@kkopachev

This comment has been minimized.

Show comment
Hide comment
@kkopachev

kkopachev Feb 27, 2018

Benchmark seems weird a bit.
Was pillow-simd compiled with AVX2 optimizations?
Why in pillow case image is converted to RGB/RGBA mode?
Looks like straight comparison of resizes in pillow/opencv is incorrect, since they produce different results. https://www.reddit.com/r/Python/comments/4j5mla/pillowsimd_is_25_times_faster_than_pillow_and_10/d340k8z/
Noticed resized jpegs have different output sizes. Since both libs use libjpeg-turbo with the same version and same settings, given same RGB bytes they should output same encoded jpeg. But output jpeg size is different, which means resize produced different results.
I also noticed that transcode in case of JPEG=>PNG produced different results, which is not expected. PNG=>WEBP produced same output (in terms of filesize).

Not sure if it's significant or not, but bench_transcode had 100 iterations in pillow case, but 1000 in go.

kkopachev commented Feb 27, 2018

Benchmark seems weird a bit.
Was pillow-simd compiled with AVX2 optimizations?
Why in pillow case image is converted to RGB/RGBA mode?
Looks like straight comparison of resizes in pillow/opencv is incorrect, since they produce different results. https://www.reddit.com/r/Python/comments/4j5mla/pillowsimd_is_25_times_faster_than_pillow_and_10/d340k8z/
Noticed resized jpegs have different output sizes. Since both libs use libjpeg-turbo with the same version and same settings, given same RGB bytes they should output same encoded jpeg. But output jpeg size is different, which means resize produced different results.
I also noticed that transcode in case of JPEG=>PNG produced different results, which is not expected. PNG=>WEBP produced same output (in terms of filesize).

Not sure if it's significant or not, but bench_transcode had 100 iterations in pillow case, but 1000 in go.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment