Skip to content

Instantly share code, notes, and snippets.

@stedman

stedman/npm-systems-testing-2021.md

Last active January 6, 2021 21:10
Show Gist options
  • Save stedman/317df9d22c486f50174e678cab4b3c34 to your computer and use it in GitHub Desktop.
Save stedman/317df9d22c486f50174e678cab4b3c34 to your computer and use it in GitHub Desktop.
Download ZIP
Systems (e2e) Testing in 2021
Raw
npm-systems-testing-2021.md

Systems (e2e) Testing

This is an evaluation of the choices available for systems, or end-to-end (e2e), testing in 2021.

Requirements

  • JavaScript npm package - consistent with frontend tools
  • runs on local dev environment
  • runs on build system (Jenkins)
  • uses major browser engines: Blink, Gecko, Webkit

Contenders

Head-to-head

The following tests were performed on a common ReactJS todo list application.

  1. verify expected <h1> title on page
  2. add new todo task, verify list update
  3. delete todo task, verify list update
  4. click on list "active" filter, verify list update
  5. click on list "completed" filter, verify list update (tests alternative approach to click verification)
  6. click on print button, verify new window opens with print page (tests new tab/window capability)

Code

Overview

  • Cypress is like jQuery when it comes to selectors and syntax. Get something and then do something.

    It comes with a GUI component that makes test writing more interactive and more enjoyable.

    On the downside, Cypress doesn't work with Webkit and it doesn't handle multiple tabs/windows.

  • CodeceptJS fills an interesting space somewhere between Cypress and Jest-Playwright. It uses a BDD style syntax that seems odd at first but is very terse and smooths over some of the complexities of Playwright.

    On the downside, CodeceptJS is managed by one person in Ukraine.

  • Jest-Playwright is the newest player, having just started in January 2020 by a couple of Puppeteer expats. It provides a mode async/await approach that will be more comfortable with experience JavaScripters.

    On the downside, Jest-Playwright is still rather new and debugging can be a royal pain.

Install and configure

  • Cypress

    npm install --save-dev cypress

    Config:

    // cypress.json

{
  "baseUrl": "http://localhost:3000",
  "integrationFolder": "cypress/integration/e2e",
  "video": false
}

  • CodeceptJS

    npm install --save-dev codeceptjs playwright

    Config:

    // codecept.conf.js

const { setHeadlessWhen } = require('@codeceptjs/configure');

// turn on headless mode when running with HEADLESS=true environment variable
// export HEADLESS=true && npx codeceptjs run
setHeadlessWhen(process.env.HEADLESS);

exports.config = {
  tests: './tests/e2e/todo.spec.js',
  output: './output',
  helpers: {
    Playwright: {
      url: 'http://localhost',
      show: true,
      browser: 'chromium'
    }
  },
  include: {
    I: './steps_file.js'
  },
  bootstrap: null,
  mocha: {},
  name: 'todo-codeceptjs',
  plugins: {
    pauseOnFail: {},
    retryFailedStep: {
      enabled: true
    },
    tryTo: {
      enabled: true
    },
    screenshotOnFail: {
      enabled: true
    }
  }
};
    // steps_file.js

// in this file you can append custom step methods to 'I' object
module.exports = function steps() {
  return actor({
    // Define custom steps here, use 'this' to access default methods of I.
    // It is recommended to place a general 'login' function here.
  });
};

  • Jest-Playwright

    npm install --save-dev jest jest-playwright-preset

    Config:

  // jest-playwright.config.js

  module.exports = {
    // launchOptions: {
    //   headless: true,
    // },
    serverOptions: {
      // auto start server: https://github.com/playwright-community/jest-playwright#start-a-server
      command: 'npm run start',
      debug: true,
      launchTimeout: 30000,
      port: 3000,
      // options: {
      //   env: {
      //     BROWSER: 'none',
      //   },
      // },
    },
  };
// jest.config.e2e.js

module.exports = {
  preset: 'jest-playwright-preset',
  testRegex: '(e2e/.*\\.(test|spec))\\.[jt]sx?$',
};

Setup

  • Cypress

    beforeEach(() => {
  cy.visit('/');
});

  • CodeceptJS

     Before(({ I }) => {
   I.amOnPage('http://localhost:3000/');
 });

  • Jest-Playwright

     beforeAll(async () => {
   browser = await chromium.launch();
 });
 beforeEach(async () => {
   page = await browser.newPage();
   await page.goto(url);
 });

Teardown

  • Cypress

    // NONE

  • CodeceptJS

     // NONE

  • Jest-Playwright

     afterEach(async () => {
   await page.close();
 });
 afterAll(async () => {
   await browser.close();
 });

Test 1 - verify <h1> title

  • Cypress

    it('has the expected title', () => {
  // Get the <h1> content
  cy.get('h1')
    // Verify the content
    .should('have.text', 'React Todo');
});

  • CodeceptJS

     Scenario('has the expected title', ({ I }) => {
   // Verify the content from the <h1>
   I.see('React Todo', 'h1');
 });

  • Jest-Playwright

     it('has the expected title', async () => {
   // Get the <h1> content
   const heading = await page.textContent('h1');
   // Verify the content
   expect(heading).toBe('React Todo');
 });

Test 2 - verify new task

  • Cypress

    it('adds a new todo', () => {
  const text = 'New todo item';

  // Get the "Add" input field
  cy.get('#new-todo-input')
    // Add new task text
    .type(text);

  // Get the "Add" button
  cy.get('.btn__lg')
    // Click "Add" button
    .click();

  // Get the content from the new (last) task
  cy.get('.todo:last-child .todo-label')
    // Verify the content
    .should('have.text', text);
});

  • CodeceptJS

     Scenario('adds new todo', ({ I }) => {
   const text = 'New todo item';
 
   // Add new task text to the "Add" input field
   I.fillField('#new-todo-input', text);
   
   // Click "Add" button
   I.click('Add');
 
   // Verify the content from the new (last) task
   I.see(text, '.todo:last-child .todo-label');
 });

  • Jest-Playwright

     it('adds a new todo', async () => {
   const text = 'New todo item';
 
   // Add new task text to the "Add" input field
   await page.fill('#new-todo-input', text);
 
   // Click "Add" button
   await page.click('.btn__lg');
 
   // Get the content from the new (last) task
   const lastTodo = await page.textContent('.todo:last-child .todo-label');
   // Verify the content
   expect(lastTodo).toBe(text);
 });

Test 6 - click print button

  • Cypress

    // NOT POSSIBLE

  • CodeceptJS

     Scenario('opens a new window (for printing)', async ({ I }) => {
   const titleText = 'print test';
 
   // Click print button
   I.click('print');
   // Go to print popup
   I.switchToNextTab(1);
 
   // Verify print popup details
   I.seeTitleEquals(titleText);
   I.seeTextEquals(titleText.toUpperCase(), 'h1');
 
   // Close new tab (and session)
   I.closeCurrentTab();
 });

  • Jest-Playwright

     it('opens a new window (for printing)', async () => {
   const titleText = 'print test';
 
   // Capture print popup
   const [printPopup] = await Promise.all([
     // Listen for event
     page.waitForEvent('popup'),
     // Click print button
     page.click('.btn__print'),
   ]);
 
   // Verify print popup details
   const title = await printPopup.title();
   expect(title).toEqual(titleText);
 
   const h1 = await printPopup.$eval('h1', (content) => content.textContent);
   expect(h1).toEqual(titleText.toUpperCase());
 });

Run scripts

  • Cypress

    npm run start 		# start http://localhost:3000 server to test
npm run cy:open 	# start Cypress GUI to run tests

  • CodeceptJS

    npm run start 		# start http://localhost:3000 server to test
npm run test:e2e	# start CodeceptJS tests

  • Jest-Playwright

     npm run test:e2e	# start Jest-Playwright tests (auto starts localhost server)

Output

  • Cypress

     ===============================================================================================
   (Run Starting)
 ┌─────────────────────────────────────────────────────────────────────────────────────────────┐
 │ Cypress:    6.1.0                                                                           │
 │ Browser:    Electron 87 (headless)                                                          │
 │ Specs:      1 found (todo.spec.js)                                                          │
 └─────────────────────────────────────────────────────────────────────────────────────────────┘
 ──────────────────────────────────────────────────────────────────────────────────────────────
                                                                                           
 Running:  todo.spec.js                                                                (1 of 1)
 
   Todo app
     ✓ has the expected title (314ms)
     ✓ adds a new todo (508ms)
     ✓ deletes last todo (182ms)
     ✓ shows active tasks (170ms)
     ✓ shows complete tasks (166ms)
 
   5 passing (1s)
 
   (Results)
 ┌─────────────────────────────────────────────────────────────────────────────────────────────┐
 │ Tests:        5                                                                             │
 │ Passing:      5                                                                             │
 │ Failing:      0                                                                             │
 │ Pending:      0                                                                             │
 │ Skipped:      0                                                                             │
 │ Screenshots:  0                                                                             │
 │ Video:        false                                                                         │
 │ Duration:     1 second                                                                      │
 │ Spec Ran:     todo.spec.js                                                                  │
 └─────────────────────────────────────────────────────────────────────────────────────────────┘
 
 ===============================================================================================
 (Run Finished)
 
   Spec                                             Tests  Passing  Failing  Pending  Skipped  
 ┌────────────────────────────────────────────────────────────────────────────────────────────┐
 │ ✔  todo.spec.js                         00:01        5        5        -        -        - │
 └────────────────────────────────────────────────────────────────────────────────────────────┘
   ✔  All specs passed!                    00:01        5        5        -        -        -

  • CodeceptJS

    CodeceptJS v3.0.4
Using test root "/Users/sstedman/Sites/bitbucket/sstedman/todo-codeceptjs"

todo --
  ✔ has the expected title in 11ms
  ✔ adds new todo in 467ms
  ✔ deletes last todo in 143ms
  ✔ shows active tasks in 148ms
  ✔ shows completed tasks in 160ms
  ✔ opens a new window (for printing) in 552ms

  OK  | 6 passed   // 4s

  • Jest-Playwright

    PASS   browser: chromium  tests/e2e/todo.spec.js (8.584 s)
  Playwright todo app
    ✓ has the expected title (4644 ms)
    ✓ adds a new todo (349 ms)
    ✓ deletes last todo (286 ms)
    ✓ shows active tasks (282 ms)
    ✓ shows completed tasks (277 ms)
    ✓ opens a new window (for printing) (329 ms)

Test Suites: 1 passed, 1 total
Tests:       6 passed, 6 total
Snapshots:   0 total
Time:        12.677 s
Ran all test suites.

Performance

The test sample size is rather small for comparing the run times of the three contenders. They all averaged about 10 seconds.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment