Currently considering https://github.com/webdriverio/webdrivercss
Core Goals:
- Can test in up-to-date versions of all major browsers
- Can test on up-to-date versions of all major OSes
- Can test in IE9 (because Bootstrap v4 will support IE9+)
- Don't want to have to setup/maintain our own cluster of VMs running all the necessary OSes (and all the versions of Windows)
- Workflow for management of reference/baseline/norm screenshots
Nice-to-haves:
- Any necessary third-party services being low-cost or free, since this is for an open-source project (namely, Bootstrap)
- Ability to adjust viewport/screen size for testing of "responsive" features
- GitHub integration
Non-requirements:
- Testing interactions with JavaScript widgets
- A nice-to-have, but not worth it if it overcomplicates things or requires compromises on the main Requirements
Prior art (and boy, is there a lot of it!)
- Mogotest
- Went out of business
- SiteEffect
- No public source code is yet available; GitHub repo is a stub
- Bugsnag's unnamed Selenium-based tool
- Not currently publicly available at all
- Huxley
- Has been abandoned by Facebook; is no longer maintained
- Was geared towards testing interactions with JavaScript widgets, as opposed to testing of static webpages
- Hardy
- Offers "CSS property X of element Y has value Z" testing, rather than screenshot comparison
- Writing/updating this type of test seems prohibitively tedious
- DalekJS
- Seems geared more towards functional testing
- Doesn't seem to have any integrated workflow for management of reference/baseline/norm screenshots
- CSS Critic
- Supports Firefox and PhantomJS
- Has a review workflow
- Needle
- Somewhat promising. Offers unit tests with screenshot equality assertions.
- Supports remote Selenium.
- Supports setting viewport size.
- Has some tooling around setting reference/baseline/norm screenshots
- Not clear whether it supports updating a single reference screenshot at a time, rather than everything at once.
- Reference/baseline/norm screenshot update workflow doesn't seem quite suited to Bootstrap's particular situation
- Visual diffs require Perceptual Image Diff tool whose maintenance status is unclear
- Browsershots
- Uses virtual cluster of volunteers' computers to take screenshots in various browsers/OSes
- Throughput/capacity level is unclear, and we'd be asking for a *lot* of screenshots
- No workflow or higher-level tooling, but it exposes an API which might allow building such workflow tooling
- CrossBrowserTesting.com
- Hosted solution with integrated review workflow
- Pay per minute of test
- Depicted / dpxdt
- PhantomJS-based, but also offers "Diff My Images" option to import images taken from other tools
- Offers web-based test result viewer and reference/baseline/norm management workflow
- No explicit cross-browser support in its data model
- Applitools Eyes
- Smart visual diff algorithm
- You provide it the images yourself
- WebdriverCSS
- Supports Selenium
- Supports setting viewport dimensions
- WebdriverCSS Adminpanel
- Web UI to view test results and manage reference/baseline/norm screenshots
- Gemini
- Supports Selenium
- Offers experimental CSS code coverage feature
- Gemini GUI
- Web UI to view test results (not sure if it allows updating of norms)
- Wraith-Selenium
- A Wraith fork that supports Selenium instead of CasperJS
- Screenbeacon
- Paid service
- Has GitHub integration
- Supports responsive testing
- No cross-browser support. Doesn't even say which browser they're using. Probably Phantom or Chrome.
- Percy
- Paid service
- Currently Firefox-only
- Currently no responsive testing support
- They plan to implement these in the future
- Gatling
- Supports Selenium
- Supports cross-browser testing
- Shoov
- Supports Selenium
- Supports cross-browser testing
- Offers a web UI (in PHP...)
- Not sure whether this supports responsive testing
- PhantomJS-related tools
- Included for completeness, but these fail our cross-browser + cross-OS requirements
- CasperJS
- Wrapper for PhantomJS & SlimerJS
- No higher level tooling around screenshots, just a "take a screenshot" API
- SlimerJS
- Firefox/Gecko equivalent of PhantomJS
- TrifleJS
- Internet Explorer equivalent of PhantomJS
- PhantomCSS
- CasperJS-based CSS regression testing
- Wraith
- CasperJS-based CSS regression testing
- BackstopJS
- CasperJS-based CSS regression testing of "responsive" webpages
- Includes web app to review test results (and manage reference/baseline/norm screenshots?)
- grunt-photobox
- PhantomJS-based screenshot-taker and web-based visual diff review tool