Skip to content

Instantly share code, notes, and snippets.

@gtkatakura
Forked from cvrebert/css_regression_testing.md
Created September 26, 2021 07:29
Show Gist options
  • Save gtkatakura/1366cb288272d707f5064b1a00c4183b to your computer and use it in GitHub Desktop.
Save gtkatakura/1366cb288272d707f5064b1a00c4183b to your computer and use it in GitHub Desktop.
Survey of screenshot-based CSS testing tools

Currently considering https://github.com/webdriverio/webdrivercss


Core Goals:

  • Can test in up-to-date versions of all major browsers
  • Can test on up-to-date versions of all major OSes
  • Can test in IE9 (because Bootstrap v4 will support IE9+)
  • Don't want to have to setup/maintain our own cluster of VMs running all the necessary OSes (and all the versions of Windows)
  • Workflow for management of reference/baseline/norm screenshots

Nice-to-haves:

  • Any necessary third-party services being low-cost or free, since this is for an open-source project (namely, Bootstrap)
  • Ability to adjust viewport/screen size for testing of "responsive" features
  • GitHub integration

Non-requirements:

  • Testing interactions with JavaScript widgets
    • A nice-to-have, but not worth it if it overcomplicates things or requires compromises on the main Requirements

Prior art (and boy, is there a lot of it!)

  • Mogotest
    • Went out of business
  • SiteEffect
    • No public source code is yet available; GitHub repo is a stub
  • Bugsnag's unnamed Selenium-based tool
    • Not currently publicly available at all
  • Huxley
    • Has been abandoned by Facebook; is no longer maintained
    • Was geared towards testing interactions with JavaScript widgets, as opposed to testing of static webpages
  • Hardy
    • Offers "CSS property X of element Y has value Z" testing, rather than screenshot comparison
    • Writing/updating this type of test seems prohibitively tedious
  • DalekJS
    • Seems geared more towards functional testing
    • Doesn't seem to have any integrated workflow for management of reference/baseline/norm screenshots
  • CSS Critic
    • Supports Firefox and PhantomJS
    • Has a review workflow
  • Needle
    • Somewhat promising. Offers unit tests with screenshot equality assertions.
    • Supports remote Selenium.
    • Supports setting viewport size.
    • Has some tooling around setting reference/baseline/norm screenshots
      • Not clear whether it supports updating a single reference screenshot at a time, rather than everything at once.
      • Reference/baseline/norm screenshot update workflow doesn't seem quite suited to Bootstrap's particular situation
    • Visual diffs require Perceptual Image Diff tool whose maintenance status is unclear
  • Browsershots
    • Uses virtual cluster of volunteers' computers to take screenshots in various browsers/OSes
    • Throughput/capacity level is unclear, and we'd be asking for a *lot* of screenshots
    • No workflow or higher-level tooling, but it exposes an API which might allow building such workflow tooling
  • CrossBrowserTesting.com
    • Hosted solution with integrated review workflow
    • Pay per minute of test
  • Depicted / dpxdt
    • PhantomJS-based, but also offers "Diff My Images" option to import images taken from other tools
    • Offers web-based test result viewer and reference/baseline/norm management workflow
    • No explicit cross-browser support in its data model
  • Applitools Eyes
    • Smart visual diff algorithm
    • You provide it the images yourself
  • WebdriverCSS
    • Supports Selenium
    • Supports setting viewport dimensions
    • WebdriverCSS Adminpanel
      • Web UI to view test results and manage reference/baseline/norm screenshots
  • Gemini
    • Supports Selenium
    • Offers experimental CSS code coverage feature
    • Gemini GUI
      • Web UI to view test results (not sure if it allows updating of norms)
  • Wraith-Selenium
    • A Wraith fork that supports Selenium instead of CasperJS
  • Screenbeacon
    • Paid service
    • Has GitHub integration
    • Supports responsive testing
    • No cross-browser support. Doesn't even say which browser they're using. Probably Phantom or Chrome.
  • Percy
    • Paid service
    • Currently Firefox-only
    • Currently no responsive testing support
    • They plan to implement these in the future
  • Gatling
    • Supports Selenium
    • Supports cross-browser testing
  • Shoov
    • Supports Selenium
    • Supports cross-browser testing
    • Offers a web UI (in PHP...)
    • Not sure whether this supports responsive testing
  • PhantomJS-related tools
    • Included for completeness, but these fail our cross-browser + cross-OS requirements
    • CasperJS
      • Wrapper for PhantomJS & SlimerJS
      • No higher level tooling around screenshots, just a "take a screenshot" API
    • SlimerJS
      • Firefox/Gecko equivalent of PhantomJS
    • TrifleJS
      • Internet Explorer equivalent of PhantomJS
    • PhantomCSS
      • CasperJS-based CSS regression testing
    • Wraith
      • CasperJS-based CSS regression testing
    • BackstopJS
      • CasperJS-based CSS regression testing of "responsive" webpages
      • Includes web app to review test results (and manage reference/baseline/norm screenshots?)
    • grunt-photobox
      • PhantomJS-based screenshot-taker and web-based visual diff review tool
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment