Skip to content

Instantly share code, notes, and snippets.

@cvrebert

cvrebert/css_regression_testing.md

Last active May 28, 2024 17:42
Show Gist options
  • Save cvrebert/adf91e429906a4d746cd to your computer and use it in GitHub Desktop.
Save cvrebert/adf91e429906a4d746cd to your computer and use it in GitHub Desktop.
Download ZIP
Survey of screenshot-based CSS testing tools
Raw
css_regression_testing.md

Currently considering https://github.com/webdriverio/webdrivercss

Core Goals:

  • Can test in up-to-date versions of all major browsers
  • Can test on up-to-date versions of all major OSes
  • Can test in IE9 (because Bootstrap v4 will support IE9+)
  • Don't want to have to setup/maintain our own cluster of VMs running all the necessary OSes (and all the versions of Windows)
  • Workflow for management of reference/baseline/norm screenshots

Nice-to-haves:

  • Any necessary third-party services being low-cost or free, since this is for an open-source project (namely, Bootstrap)
  • Ability to adjust viewport/screen size for testing of "responsive" features
  • GitHub integration

Non-requirements:

  • Testing interactions with JavaScript widgets
    • A nice-to-have, but not worth it if it overcomplicates things or requires compromises on the main Requirements

Prior art (and boy, is there a lot of it!)

  • Mogotest
    • Went out of business
  • SiteEffect
    • No public source code is yet available; GitHub repo is a stub
  • Bugsnag's unnamed Selenium-based tool
    • Not currently publicly available at all
  • Huxley
    • Has been abandoned by Facebook; is no longer maintained
    • Was geared towards testing interactions with JavaScript widgets, as opposed to testing of static webpages
  • Hardy
    • Offers "CSS property X of element Y has value Z" testing, rather than screenshot comparison
    • Writing/updating this type of test seems prohibitively tedious
  • DalekJS
    • Seems geared more towards functional testing
    • Doesn't seem to have any integrated workflow for management of reference/baseline/norm screenshots
  • CSS Critic
    • Supports Firefox and PhantomJS
    • Has a review workflow
  • Needle
    • Somewhat promising. Offers unit tests with screenshot equality assertions.
    • Supports remote Selenium.
    • Supports setting viewport size.
    • Has some tooling around setting reference/baseline/norm screenshots
      • Not clear whether it supports updating a single reference screenshot at a time, rather than everything at once.
      • Reference/baseline/norm screenshot update workflow doesn't seem quite suited to Bootstrap's particular situation
    • Visual diffs require Perceptual Image Diff tool whose maintenance status is unclear
  • Browsershots
    • Uses virtual cluster of volunteers' computers to take screenshots in various browsers/OSes
    • Throughput/capacity level is unclear, and we'd be asking for a *lot* of screenshots
    • No workflow or higher-level tooling, but it exposes an API which might allow building such workflow tooling
  • CrossBrowserTesting.com
    • Hosted solution with integrated review workflow
    • Pay per minute of test
  • Depicted / dpxdt
    • PhantomJS-based, but also offers "Diff My Images" option to import images taken from other tools
    • Offers web-based test result viewer and reference/baseline/norm management workflow
    • No explicit cross-browser support in its data model
  • Applitools Eyes
    • Smart visual diff algorithm
    • You provide it the images yourself
  • WebdriverCSS
    • Supports Selenium
    • Supports setting viewport dimensions
    • WebdriverCSS Adminpanel
      • Web UI to view test results and manage reference/baseline/norm screenshots
  • Gemini
    • Supports Selenium
    • Offers experimental CSS code coverage feature
    • Gemini GUI
      • Web UI to view test results (not sure if it allows updating of norms)
  • Wraith-Selenium
    • A Wraith fork that supports Selenium instead of CasperJS
  • Screenbeacon
    • Paid service
    • Has GitHub integration
    • Supports responsive testing
    • No cross-browser support. Doesn't even say which browser they're using. Probably Phantom or Chrome.
  • Percy
    • Paid service
    • Currently Firefox-only
    • Currently no responsive testing support
    • They plan to implement these in the future
  • Gatling
    • Supports Selenium
    • Supports cross-browser testing
  • Shoov
    • Supports Selenium
    • Supports cross-browser testing
    • Offers a web UI (in PHP...)
    • Not sure whether this supports responsive testing
  • PhantomJS-related tools
    • Included for completeness, but these fail our cross-browser + cross-OS requirements
    • CasperJS
      • Wrapper for PhantomJS & SlimerJS
      • No higher level tooling around screenshots, just a "take a screenshot" API
    • SlimerJS
      • Firefox/Gecko equivalent of PhantomJS
    • TrifleJS
      • Internet Explorer equivalent of PhantomJS
    • PhantomCSS
      • CasperJS-based CSS regression testing
    • Wraith
      • CasperJS-based CSS regression testing
    • BackstopJS
      • CasperJS-based CSS regression testing of "responsive" webpages
      • Includes web app to review test results (and manage reference/baseline/norm screenshots?)
    • grunt-photobox
      • PhantomJS-based screenshot-taker and web-based visual diff review tool
@zofrex
Copy link

zofrex commented Feb 10, 2018

Really useful list, thank you for putting this together :)

@shmargum
Copy link

shmargum commented Mar 2, 2018

Just want to note here that Wraith now supports using headless Chrome

@kaelig
Copy link

kaelig commented Jun 29, 2018

Percy now supports Chrome as well.

@tobiaslins
Copy link

Take a look at Axcept

@drublic
Copy link

drublic commented Jan 28, 2020

Hello,
unfortunately siteeffect.io is no longer available.

Together with a colleague I am working on Warhol, which analyses Pattern Libraries and uses the data to compare it against the implementation.
Warhol does not use Screenshots but runs in a real browser which makes its error reporting faster and much more reliable than a lot of other tools which exist.

@ygerasimov
Copy link

Take a look at https://diffy.website. We use puppeteer on AWS Lambda (very performant) and also have custom images comparison algorithm that produce way less false positives than standard imagemagick.

@Christopher-Hayes
Copy link

I'm looking at lightweight libraries that would interface with Selenium (not come with Puppeteer or Selenium or anything), just accept screenshots and provide a dashboard. https://github.com/reg-viz/reg-cli and https://github.com/reg-viz/reg-suit look good for accomplishing that.

@tdrhq
Copy link

tdrhq commented Dec 24, 2020 •
edited
Loading

@Christopher-Hayes

I'm looking at lightweight libraries that would interface with Selenium (not come with Puppeteer or Selenium or anything), just accept screenshots and provide a dashboard. https://github.com/reg-viz/reg-cli and https://github.com/reg-viz/reg-suit look good for accomplishing that.

https://screenshotbot.io is just that. We don't try to solve the problem of generating screenshots, or running them. There are many fantastic libraries and CI systems to do that just fine. We just have a simple dashboard that integrates with your Jira/Asana/Github Issues, and will work with Android, iOS, Web, or for whichever platform where there's already a screenshot testing library. I built the screenshot testing service at Facebook, so I have pretty good experience running something like this at scale, and understand what it takes to keep developers happy (or annoyed).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment