Create a gist now

Instantly share code, notes, and snippets.

What would you like to do?
Survey of screenshot-based CSS testing tools

Currently considering https://github.com/webdriverio/webdrivercss


Core Goals:

  • Can test in up-to-date versions of all major browsers
  • Can test on up-to-date versions of all major OSes
  • Can test in IE9 (because Bootstrap v4 will support IE9+)
  • Don't want to have to setup/maintain our own cluster of VMs running all the necessary OSes (and all the versions of Windows)
  • Workflow for management of reference/baseline/norm screenshots

Nice-to-haves:

  • Any necessary third-party services being low-cost or free, since this is for an open-source project (namely, Bootstrap)
  • Ability to adjust viewport/screen size for testing of "responsive" features
  • GitHub integration

Non-requirements:

  • Testing interactions with JavaScript widgets
    • A nice-to-have, but not worth it if it overcomplicates things or requires compromises on the main Requirements

Prior art (and boy, is there a lot of it!)

  • Mogotest
    • Went out of business
  • SiteEffect
    • No public source code is yet available; GitHub repo is a stub
  • Bugsnag's unnamed Selenium-based tool
    • Not currently publicly available at all
  • Huxley
    • Has been abandoned by Facebook; is no longer maintained
    • Was geared towards testing interactions with JavaScript widgets, as opposed to testing of static webpages
  • Hardy
    • Offers "CSS property X of element Y has value Z" testing, rather than screenshot comparison
    • Writing/updating this type of test seems prohibitively tedious
  • DalekJS
    • Seems geared more towards functional testing
    • Doesn't seem to have any integrated workflow for management of reference/baseline/norm screenshots
  • CSS Critic
    • Supports Firefox and PhantomJS
    • Has a review workflow
  • Needle
    • Somewhat promising. Offers unit tests with screenshot equality assertions.
    • Supports remote Selenium.
    • Supports setting viewport size.
    • Has some tooling around setting reference/baseline/norm screenshots
      • Not clear whether it supports updating a single reference screenshot at a time, rather than everything at once.
      • Reference/baseline/norm screenshot update workflow doesn't seem quite suited to Bootstrap's particular situation
    • Visual diffs require Perceptual Image Diff tool whose maintenance status is unclear
  • Browsershots
    • Uses virtual cluster of volunteers' computers to take screenshots in various browsers/OSes
    • Throughput/capacity level is unclear, and we'd be asking for a *lot* of screenshots
    • No workflow or higher-level tooling, but it exposes an API which might allow building such workflow tooling
  • CrossBrowserTesting.com
    • Hosted solution with integrated review workflow
    • Pay per minute of test
  • Depicted / dpxdt
    • PhantomJS-based, but also offers "Diff My Images" option to import images taken from other tools
    • Offers web-based test result viewer and reference/baseline/norm management workflow
    • No explicit cross-browser support in its data model
  • Applitools Eyes
    • Smart visual diff algorithm
    • You provide it the images yourself
  • WebdriverCSS
    • Supports Selenium
    • Supports setting viewport dimensions
    • WebdriverCSS Adminpanel
      • Web UI to view test results and manage reference/baseline/norm screenshots
  • Gemini
    • Supports Selenium
    • Offers experimental CSS code coverage feature
    • Gemini GUI
      • Web UI to view test results (not sure if it allows updating of norms)
  • Wraith-Selenium
    • A Wraith fork that supports Selenium instead of CasperJS
  • Screenbeacon
    • Paid service
    • Has GitHub integration
    • Supports responsive testing
    • No cross-browser support. Doesn't even say which browser they're using. Probably Phantom or Chrome.
  • Percy
    • Paid service
    • Currently Firefox-only
    • Currently no responsive testing support
    • They plan to implement these in the future
  • Gatling
    • Supports Selenium
    • Supports cross-browser testing
  • Shoov
    • Supports Selenium
    • Supports cross-browser testing
    • Offers a web UI (in PHP...)
    • Not sure whether this supports responsive testing
  • PhantomJS-related tools
    • Included for completeness, but these fail our cross-browser + cross-OS requirements
    • CasperJS
      • Wrapper for PhantomJS & SlimerJS
      • No higher level tooling around screenshots, just a "take a screenshot" API
    • SlimerJS
      • Firefox/Gecko equivalent of PhantomJS
    • TrifleJS
      • Internet Explorer equivalent of PhantomJS
    • PhantomCSS
      • CasperJS-based CSS regression testing
    • Wraith
      • CasperJS-based CSS regression testing
    • BackstopJS
      • CasperJS-based CSS regression testing of "responsive" webpages
      • Includes web app to review test results (and manage reference/baseline/norm screenshots?)
    • grunt-photobox
      • PhantomJS-based screenshot-taker and web-based visual diff review tool
arikon commented Aug 3, 2016

Web UI to view test results (not sure if it allows updating of norms)

It does.

Currently considering https://github.com/webdriverio/webdrivercss

Why?

Gemini is a way more better and robust. It has slim API for writing test. It implemented CIEDE2000 color comparison algorithm. It manages caret (input fields) difference for you and many other things. It is open for contributions also.

Gemini is already used in production at Yandex by the team of over 70 front-end developers to test UI components library, yandex.ru/search, yandex.ru/images, yandex.ru/video and many other web services for more that 2 years.

Gemini is not a prior art, it appeared nearly at the same time as Webdriver.io and WebdriverCSS.

PS: I'm leading the team who are developing Gemini and Gemini GUI, so I'm biased ;)

mtomara commented Oct 24, 2016

Hi, if the question is still relevant, I would suggest trying Screenster, a visual testing tool. It is able to perform CSS testing. Does not require coding to record tests. You can check it out at free Demo server and decide if it is good enough for your specific case.

elgalu commented Nov 10, 2016

What do you guys think about lineup

Howdy @cvrebert - Percy supports responsive testing now, at any widths you specify. It also has CI and GitHub integration, and discounts for open source projects.

Happo is a screenshot testing tool focused on unit testing (e.g. individual React components). Mostly meant for use in a CI, posts to your code review tool when it detects a diff.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment