browser e2e testing idea using teenytest and puppeteer

README.md

My main goal is to have tests fail as soon as things go wrong rather than the usual thing where failures in browser testing of single page web apps tend to manifest as timeouts.

Second goal is to have really useful errors so you don't have to try and infer what went wrong quite so often.

In terms of style I want to be able to use async/await, my favourite assertion library and a normal test runner. Basically not nightwatch, not anything selenium/chromedriver related. Also a knowable, sane system.

It is a pity that puppeteer would basically lock you into chrome, but I've already seen a Firefox headless wrapper that mimicks the API. Maybe Microsoft will catch on and do the same? I feel strongly that Selenium/Webdriver is a dead end.

--

I think the first killer feature here is that each page gets a Promise that will reject the moment there's any error in the page. Then I wrap every puppeteer page method with Promise.race() so it will reject if there's a page error or resolve if the operation succeeds. Whichever comes first. No more timing out waiting for something while there's actually an error in the browser's console that you can't see. It actually rethrows the browser error right inside your test.

Then waitForAny wraps puppeteer's waitFor. You can pass it good CSS selectors and bad selectors and it waits for both. It throws if a bad selector was found. You can use this to wait for either the successful state to appear OR an error. Like a confirmation dialog but also an error banner or dialog. Whichever comes first. So unlike some other browser testing frameworks you don't have to wait for it to time out because the success element never appears.

That's it for now. I have lots more ideas, but these are the ones that are already implemented and working.

helper.js

'use strict';

// This assumes you're using teenytest-promise.

const Code = require('code');
const Puppeteer = require('puppeteer');

// these will be available in all the tests
global.Config = require('./config');
global.expect = Code.expect;

global.pause = (ms) => {

    return new Promise((resolve) => {

        setTimeout(resolve, ms);
    })
};

const internals = {
    // Only wrap (and expose) some of puppeteer's page methods so that there is still a chance to port it to Firefox or IE.
    methods: ['goto', 'evaluate', '$', 'click', 'type', 'screenshot']
};

internals.waitForAny = async function (page, good, bad, options) {

    if (!Array.isArray(good)) {
        good = [good];
    }

    if (!Array.isArray(bad)) {
        bad = [bad];
    }

    const allSelectors = [...good, ...bad].join(', ');
    await page.waitFor(allSelectors, options);

    if (good.length > 1) {
        for (const selector of good) {
            const element = await page.$(selector);
            console.log(selector, !!element);
        }
    }

    for (const selector of bad) {
        const element = await page.$(selector);
        expect(element).to.not.exist();
    }
};

internals.wrapPage = function (page) {

    // will reject as soon as an error occurs in the page
    const pageErrorPromise = new Promise((resolve, reject) => {

        page.once('pageerror', (error) => {

            reject(error);
        });
    });

    const wrapped = {};
    for (const method of internals.methods) {
        // The command will fail if a page error occurrs before it completes or if there's already a page error.
        wrapped[method] = (...args) => Promise.race([pageErrorPromise, page[method](...args)]);
    }

    wrapped.waitForAny = (good, bad, options) => {

        return Promise.race([pageErrorPromise, internals.waitForAny(page, good, bad, options)]);
    };

    return wrapped;
};


let browser;

exports.beforeAll = async function() {

    browser = await Puppeteer.launch({ headless: false });

    global.newPage = async () => {

        const page = await browser.newPage();
        page.setViewport({
            width: 1280,
            height: 720,
            deviceScaleFactor: 1
        });

        return internals.wrapPage(page);
    };
};

exports.afterEach = async function() {

    const pages = await browser.pages();
    // Close all the pages except the first empty one so we don't close the browser window after every test which can be annoying if you're trying to follow along.
    for (const page of pages.slice(1)) {
        await page.close();
    }
}

exports.afterAll = async function() {

    await browser.close();
};

This is great. Really like the ability to fail fast with JS errors and the presence of elements which signify error.

I would personally prefer to log in via API, without filling in those fields for every test (rather have a dedicated test for that), but that's an aside to this.

Also really like the ability to use async/await, can't really do that with Cypress (at least without unexpected behaviour), and a familiar assertion library (Cypress uses chai).

Any examples of tests which consume the stuff in this file? At this point it seems the main things that Cypress has going for it over Puppeteer are a pretty nice/terse chained API which is a bit higher level (especially for interacting with elements), the UI/runner/selector playground thing, which is quite pleasant to use, and the support for stubbing requests with fixtures.