Visual diffing algorithms comparison

comparison.md

Thanks to Awesome Visual Regression Testing and Screenster.io for compiling lists of the things in the space. The research into what each uses was done by me through digging their code or their READMEs.

• OSnap: ReasonML. Uses ODiff, the main contender I am looking at.
• basset: Python. Uses a handrolled algorithm, looks fairly naive, only 63 lines of code
• AyeSpy: JavaScript. Uses looks-same
• Wraith: Ruby. Uses ImageMagick (shells out to it). Made by BBC lol
• BackstopJS: JavaScript. Uses hash comparison then @mirzazeyrek/node-resemble-js a fork of Resemble.js.
• Galen: Java. Uses a handrolled algorithm named rainbow4j. Think they might be looking for a new name now.
• Creevey: JavaScript. Uses pixelmatch.
• CSSCritic: JavaScript. Uses js-imagediff.
• Spectre: Ruby. Uses ImageMagick (shells out). Ruby and ImageMagick, more iconic pair?
• Shoov: PHP. Uses use your eyes! algorithm. Doesn't actually do diffing just displaying images side by side.
• qd_screenshottests: PHP probably. Broken link for me.
• Look-alike: JavaScript. Uses Resemble.js.
• Hardy: JavaScript. Uses js-imagediff.
• TestCafe: JavaScript. Doesn't do visual diffs????? Why is this here.
• Needle: Python. Uses ImageMagick or perceptualdiff or PIL and handrolled simple (https://github.com/python-needle/needle/blob/master/needle/engines/pil_engine.py).
• gatling: Ruby. ImageMagick via RMagick (Ruby bindings for ImageMagick)
• grunt-photobox: JavaScript. Use ImageMagick or a handrolled algorithm that runs in the browser via Canvas API. (https://github.com/stefanjudis/grunt-photobox/blob/9d91025d554a997953deb655de808b40399bc805/tasks/assets/scripts/worker.js)
• vrtest: JavaScript. Uses image-diff.
• Happo: JavaScript. Uses lcs-image-diff.
• Nightmare & Puppeteer & Playwright: Doesn't do visual diffs.
• reg-cli & reg-suit: JavaScript. Uses img-diff-js and x-img-diff-js. (VERY COOL.)
• Chimp: Doesn't do visual diffs.
• Differencify: JavaScript. Uses pixelmatch via jimp.
• Resemble.js: JavaScript. see below
• Muppeteer: JavaScript. Uses pixelmatch
• ember-visual-test: JavaScript. Uses pixelmatch
• AET: Java. Who knows? Who knows really?
• Wendigo: Doesn't do visual diffs.
• Loki: JavaScript. Uses pixelmatch or looks-same or GraphicsMagick. (Special thanks for making it really easy to know what the project uses without digging deep and deep)
• Zombie.js & CodeceptJS & FuncUnit & Nightwatch & Protractor: Doesn't do visual diffs.
• jest-puppeteer-react & jest-image-snapshot: JavaScript. Uses pixelmatch.
• test-crawler: JavaScript. Uses pixdiff-zone.
• wdio-visual-regression: JavaScript. Uses Resemble.js.
• Selenide & Karma: Doesn't do visual diffs.
• Hermione: Uses looks-same.
• Huxley: Uses handrolled algorithm via PIL. https://github.com/facebookarchive/huxley/blob/master/huxley/images.py

Core diff libraries

• ODiff: is based on [1] https://github.com/dmtrKovalenko/odiff
• pixelmatch: implements ideas from [1] and [2]. Is only 150 lines of code. https://www.npmjs.com/package/pixelmatch
• looks-same: ????? https://www.npmjs.com/package/looks-same
• js-imagediff: ????? https://github.com/HumbleSoftware/js-imagediff
• Resemble.js: ?????? https://github.com/rsmbl/Resemble.js
• ImageMagick: ???????? You know what it is. https://imagemagick.org/index.php
• perceptualdiff: ???????? Some ancient C++ code https://github.com/myint/perceptualdiff
• image-diff: ??????? Some Uber deprecated library (recommends looks-same or pixelmatch as replacements) https://www.npmjs.com/package/image-diff
• lcs-image-diff: Uses [3] https://www.npmjs.com/package/lcs-image-diff
• img-diff-js: ??????????? https://github.com/reg-viz/img-diff-js
• x-img-diff-js: ??????????? https://github.com/reg-viz/x-img-diff-js (VERY COOL.)
• GraphicsMagick: ?????????????? http://www.graphicsmagick.org
• pixdiff-zone: Fork of pixelmatch with minor added functionality https://github.com/apiel/test-crawler/tree/master/packages/pixdiff-zone
• blink-diff: ???????????????? https://github.com/yahoo/blink-diff

[1] - Measuring perceived color difference using YIQ NTSC transmission color space in mobile applications
[2] - Anti-aliased pixel and intensity slope detector
[3] - Longest common subsequence algorithm