Skip to content

Instantly share code, notes, and snippets.

@ModulesUnraveled
Last active June 1, 2018 00:10
Show Gist options
  • Save ModulesUnraveled/3d89d760ae208811bd5a0dde763aa684 to your computer and use it in GitHub Desktop.
Save ModulesUnraveled/3d89d760ae208811bd5a0dde763aa684 to your computer and use it in GitHub Desktop.
Setup Visual Regression Testing on a New Project

Install Java command line tools

Install the latest Java JDK (not JRE)

Require all the things

npm install --save-dev webdriverio chai node-notifier wdio-mocha-framework wdio-browserstack-service wdio-visual-regression-service

Add the test scripts to package.json

{
  "scripts": {
    "test": "wdio",
    "test:quick": "wdio wdio.conf.quick.js"
  },
}

Update the .gitignore

The only screenshots we want to commit to the repo are the baselines, so we need to ignore the others.

# Visual Regression Testing Images
*/**/screenshots/diff
*/**/screenshots/latest
*/**/errorShots

Update the config file

  • Update the URLs, and the Capabilities section of the wdio.conf.js file.

Defining new browsers/OSs

There's a configuration tool at https://www.browserstack.com/automate/capabilities

Select the OS and browser of choice and enter the information into the "capabilities" array of the appropriate wdio.conf file.

Note: When entering browser info, if you omit the browser version, the latest stable release will be used. Typically, this is the best option, unless you need to support previous browser versions. IE9 or IE10, for example.

Create a file at /tests/config/globalHides.js to be used for global site elements that should not appear in visual regression screenshots.

Hide

This section visually hides elements something like this. It will not be visible in the screenshots, but it will still take up the same amount of space in the image.

Good candidates is variable images, or ad blocks.

Remove

This section removes elements from the dom. This is equivalent to display: none.

Good candidates are sticky headers/footers, or pop-over help/chat windows.

Example

module.exports = {
  hide: [
    'img'
  ],
  remove: [
    '.contact-cta',
    '#sticky-footer'
  ]
};

Create a file at tests/config/viewports.js. These are the viewports that all tests will be run against, ulness a test-specific viewport set is defined*.

You can name the viewports whatever you want. mobile, tablet, and desktop below are just examples. The most important part is the width. The height is just to define the browser window, but has little effect on the actual screenshots.

If you're checking an element and it is less than the browser width and/or height, the screenshot will only be as wide and tall as the actual element itself.

If you're checking an entire document (full page), the screenshot will be as wide as you define here, but will be as tall as necessary to capture the entire page. the heights you define here are not relevant.

Example

const viewPorts = {
  mobile: {width: 320, height: 568},
  tablet: {width: 768, height: 1024},
  desktop: {width: 1280, height: 800},
};

module.exports = viewPorts;

Create a file at /wdio.conf.js. This is the main configuration file for visual regression tests on a project. Anything can be changed in here, but two sections must be configured per-project.

  1. Update the baseUrl to the site's local URL. If you are running tests against production (or PR instances like multidevs) update the appropriate baseUrl sections as well.
  2. Update the capabilities array to reflect the OSs and Browsers your tests should be run against by default*.

Everything else is configured to sensible defaults that shouldn't need to be updated unless specifically required.

Optionally update the misMatchTolerance

You may run into an issue where browsers rending things slightly different. e.g. Chrome on a Mac renders fonts slightly different than Chrome on Windows. If you're testing locally (with something like selenium instead of against browserstack) you might want to increase the misMatchTolerance from the default (0.01) to reduce false failures.

Example

var notifier = require('node-notifier');
var VisualRegressionCompare = require('wdio-visual-regression-service/compare');
var path = require('path');
var fs = require('fs-extra');

// Import the defined viewports
var viewPorts = require('./tests/config/viewports.js');

// Configure the baseUrl for local and production test runs
var baseUrl = 'http://local.YOURWEBSITE.com';

if (process.env.SERVER === "prod") {
  baseUrl = 'https://www.YOURWEBSITE.com';
}

// Set timeout based on environment variable
// To debug and get around the 60 second default timeout
// Set a local variable `DEBUG` to `true`
var timeout = process.env.DEBUG ? 99999999 : 60000;

// Configure the screenshot filenames
function getScreenshotName(folder, context){
  var type = context.type;
  var testParent = context.test.parent;
  var testName = context.test.title;
  var browserName = context.browser.name;
  var browserViewport = context.meta.viewport;
  var browserWidth = browserViewport.width;
  var browserHeight = browserViewport.height;

  // Create the screenshot file name
  return path.join(process.cwd(), folder, `${testParent} - ${testName}/${type}_${browserName}_${browserWidth}x${browserHeight}.png`);
}

exports.config = {
    user: process.env.BROWSERSTACK_USER,
    key: process.env.BROWSERSTACK_KEY,
    browserstackLocal: true,

    /////////////////////
    // Specify Test Files
    /////////////////////
    specs: [
        './tests/**/*.test.js'
    ],
    // Patterns to exclude.
    // exclude: [
        // 'path/to/excluded/files'
    // ],
    //

    ///////////////
    // Capabilities
    ///////////////
    maxInstances: 3,
    capabilities: [{
        project: 'YOUR PROJECT NAME',
        os: 'Windows',
        os_version: '10',
        browserName: 'chrome',
        'browserstack.local': true
    }, {
        project: 'YOUR PROJECT NAME',
        os: 'Windows',
        os_version: '10',
        browserName: 'IE',
        browser_version: '11.0',
        'browserstack.local': true
    }, {
        project: 'YOUR PROJECT NAME',
        os: 'Windows',
        os_version: '10',
        browserName: 'Firefox',
        'browserstack.local': true
    }],

    //////////////////////
    // Test Configurations
    //////////////////////
    sync: true,
    // Level of logging verbosity: silent | verbose | command | data | result | error
    logLevel: 'silent',
    // Enables colors for log output.
    coloredLogs: true,
    // Warns when a deprecated command is used
    deprecationWarnings: true,
    // If you only want to run your tests until a specific amount of tests have failed use
    // bail (default is 0 - don't bail, run all tests).
    bail: 0,
    // Saves a screenshot to a given path if a command fails.
    screenshotPath: './tests/errorShots/',
    // Base URL (Defined at the top of this file)
    baseUrl: baseUrl,
    // Default timeout for all waitFor* commands.
    waitforTimeout: timeout,
    // Default timeout in milliseconds for request if Selenium Grid doesn't send response
    connectionRetryTimeout: 90000,
    // Default request retries count
    connectionRetryCount: 3,
    // Test runner services
    services: ['browserstack', 'visual-regression'],
    // Framework you want to run your specs with.
    // The following are supported: Mocha, Jasmine, and Cucumber
    framework: 'mocha',
    // Options to be passed to Mocha.
    // See the full list at http://mochajs.org/
    mochaOpts: {
        ui: 'bdd',
        timeout: timeout
    },
    visualRegression: {
      compare: new VisualRegressionCompare.LocalCompare({
        referenceName: getScreenshotName.bind(null, 'tests/screenshots/baseline'),
        screenshotName: getScreenshotName.bind(null, 'tests/screenshots/latest'),
        diffName: getScreenshotName.bind(null, 'tests/screenshots/diff'),
        // misMatchTolerance: .4,
      }),
      viewports: Object.keys(viewPorts).map((key) => viewPorts[key]),
    },
    // =====
    // Hooks
    // =====
    /**
     * Gets executed once before all workers get launched.
     * @param {Object} config wdio configuration object
     * @param {Array.<Object>} capabilities list of capabilities details
     */
    onPrepare: function (config, capabilities) {
      // Clear the diffs directory so that old diffs aren't still around after furture tests
      return fs.emptyDir('./tests/screenshots/diff');
    },
    /**
     * Gets executed just before initialising the webdriver session and test framework. It allows you
     * to manipulate configurations depending on the capability or spec.
     * @param {Object} config wdio configuration object
     * @param {Array.<Object>} capabilities list of capabilities details
     * @param {Array.<String>} specs List of spec file paths that are to be run
     */
    // beforeSession: function (config, capabilities, specs) {
    // },
    /**
     * Gets executed before test execution begins. At this point you can access to all global
     * variables like `browser`. It is the perfect place to define custom commands.
     * @param {Array.<Object>} capabilities list of capabilities details
     * @param {Array.<String>} specs List of spec file paths that are to be run
     */
    before: function (capabilities, specs) {
      expect = require('chai').expect;
    },

    /**
     * Runs before a WebdriverIO command gets executed.
     * @param {String} commandName hook command name
     * @param {Array} args arguments that command would receive
     */
    // beforeCommand: function (commandName, args) {
    // },

    /**
     * Hook that gets executed before the suite starts
     * @param {Object} suite suite details
     */
    // beforeSuite: function (suite) {
    // },
    /**
     * Function to be executed before a test (in Mocha/Jasmine) or a step (in Cucumber) starts.
     * @param {Object} test test details
     */
    // beforeTest: function (test) {
    // },
    /**
     * Hook that gets executed _before_ a hook within the suite starts (e.g. runs before calling
     * beforeEach in Mocha)
     */
    // beforeHook: function () {
    // },
    /**
     * Hook that gets executed _after_ a hook within the suite ends (e.g. runs after calling
     * afterEach in Mocha)
     */
    // afterHook: function () {
    // },
    /**
     * Function to be executed after a test (in Mocha/Jasmine) or a step (in Cucumber) ends.
     * @param {Object} test test details
     */
    afterTest: function (test) {
      if (!test.passed) {
        notifier.notify({
          title: 'Test failure!',
          message: test.parent + ' ' + test.title
        })
      }
    },
    /**
     * Hook that gets executed after the suite has ended
     * @param {Object} suite suite details
     */
    // afterSuite: function (suite) {
    // },

    /**
     * Runs after a WebdriverIO command gets executed
     * @param {String} commandName hook command name
     * @param {Array} args arguments that command would receive
     * @param {Number} result 0 - command success, 1 - command error
     * @param {Object} error error object if any
     */
    // afterCommand: function (commandName, args, result, error) {
    // },
    /**
     * Gets executed after all tests are done. You still have access to all global variables from
     * the test.
     * @param {Number} result 0 - test pass, 1 - test fail
     * @param {Array.<Object>} capabilities list of capabilities details
     * @param {Array.<String>} specs List of spec file paths that ran
     */
    // after: function (result, capabilities, specs) {
    // },
    /**
     * Gets executed right after terminating the webdriver session.
     * @param {Object} config wdio configuration object
     * @param {Array.<Object>} capabilities list of capabilities details
     * @param {Array.<String>} specs List of spec file paths that ran
     */
    // afterSession: function (config, capabilities, specs) {
    // },
    /**
     * Gets executed after all workers got shut down and the process is about to exit.
     * @param {Object} exitCode 0 - success, 1 - fail
     * @param {Object} config wdio configuration object
     * @param {Array.<Object>} capabilities list of capabilities details
     */
    // onComplete: function(exitCode, config, capabilities) {
    // }
}

* If you want to run against a smaller set of browsers (or even just one) use something like the wdio.conf.quick.js file.

Create a file at /wdio.conf.quick.js to be used for quick testing. Often useful while actively developing/updating a component.

This file first includes all of the config from the default wdio.conf.js file, then overrides aspects. By default, it just sets a single OS/browser to run tests against. Thus speeding up the test run.

(Remember to update the project name)

Example

// Require prod configuration
var prodConfig = require('./wdio.conf.js').config;

// Clone prod config and add new properties/overrides
var localConfig = Object.assign(prodConfig, {
    capabilities: [{
        project: 'YOUR PROJECT NAME',
        os: 'Windows',
        os_version: '10',
        browserName: 'chrome',
        'browserstack.local': true
    }],
});

exports.config = localConfig;

Install Java command line tools

Install the latest Java JDK (not JRE)

Require all the things

npm install

Save browserstack credentials to environment variables

Do this one time per machine. It'll carry over to future projects

Get access credentials from https://www.browserstack.com/automate (Expand the "Username and Access Keys" section)

Bash

  • export BROWSERSTACK_USER="myusername"
  • export BROWSERSTACK_KEY="mysecretkey"

To have this load in future bash sessions, add each export... command to your preference of .profile, .bash_profile, .bashrc etc

Fish

  • set -Ux BROWSERSTACK_USER myusername
  • set -Ux BROWSERSTACK_KEY mysecretkey

This will be Universal, and persistent with no additional steps.

Windows

  • setx BROWSERSTACK_USER myusername
  • setx BROWSERSTACK_KEY mysecretkey

You will need to reload any open shells, but after that, this will be Global, and persistent in all future shells.

Setup Local Testing (BrowserStack)

Do this one time per machine. It'll carry over to future projects

TLDR:

Running Tests

  • Open the browserstack start page (you don't have to select anything, just have it open) https://www.browserstack.com/start
  • In your command line, run the entire test suite with npm test
  • Optional: If you want to run your tests in only one browser (Chrome by default), you can run npm run test:quick
  • Optional: If you just want to run one specific test use the --spec flag. For example:
    • One test in all browsers: npm test --spec path/to/one.test.js
    • One test using the "quick" config (e.g. one browser): npm run test:quick --spec path/to/one.test.js

Defining new browsers/OSs

There's a configuration tool at https://www.browserstack.com/automate/capabilities

Select the OS and browser of choice and enter the information into the "capabilities" array of the appropriate wdio.conf file.

Note: When entering browser info, if you omit the browser version, the latest stable release will be used. Typically, this is the best option, unless you need to support previous browser versions. IE9 or IE10, for example.

Here are a couple example test files. The first is a whole page test, that usees .checkDocument. The second checks the navigation element using .checkElement.

Save them to the visreg tests folder, named something like:

  • /tests/visreg/homepage.test.js
  • /tests/visreg/navigation.test.js

Whole page test

This test takes a screenshot of the entire homepage. It also demonstrates how to increase the timeout length on a specific test.

const visreg = require('../config/globalHides.js');

describe('Home Page', function () {
  it('should look good', function () {
    this.timeout(120000);
    browser
      .url('./')
      .checkDocument({hide: visreg.hide, remove: visreg.remove})
      .forEach((item) => {
        expect(item.isWithinMisMatchTolerance).to.be.true;
      });
  });
});

Single element test

This test suite takes screenshots of the navigation element. The first it section takes a screenshot when the page initially loads. The second it section expands the nav (clicks the hamburger icon), then expands a sub-nav item, then takes a screenshot of the nav element again, capturing all of the expanded sections.

This example demonstrates how to use beforeEach to capture commonalities, in this case, the URL to navigate to, and to set the viewport to mobile before anything else.

This also demonstrates how to use the .waitForVisible command. Since the navigation slides open, we want to wait for it to be expanded before we take the screenshot. Not half-way open.

Lastly, this demonstrates how to take screenshots at specific viewports only. The #toggle-menu element isn't even there on desktop, so the second test would fail if it tried to run at the desktop size simply because it wouldn't be able to complete the test, not because of any regressions.

const visreg = require('../config/globalHides.js');
const {mobile, tablet} = require('../config/viewports.js');

describe('Nav top', function () {
  beforeEach(function () {
    browser.url('./some/specific/url')
      .setViewportSize(mobile);
  });

  it('should look good when initially loaded', () => {
    browser
      .checkElement('.nav-top-wrapper', {hide: visreg.hide, remove: visreg.remove})
      .forEach((item) => {
        expect(item.isWithinMisMatchTolerance).to.be.true;
      });
  });

  it('should look good when subnav is expanded', () => {
    browser
      .click('#toggle-menu')
      .click('.nav-primary-wrapper > ul > li:nth-child(2) > span');
    browser
      .waitForVisible('.nav-primary-wrapper > ul > li:nth-child(2) > ul > li:nth-child(1) > a', 3000);
    browser
      .checkElement('.nav-top-wrapper',
      	{hide: visreg.hide, remove: visreg.remove},
        {viewports: [mobile, tablet]})
      .forEach((item) => {
        expect(item.isWithinMisMatchTolerance).to.be.true;
      });
  });
});
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment