RFC Foreman: Integration tests & E2E testing

readme.md

RFC - Integration Tests & End to End tests

Present architecture

E2E - Pipeline bats, but not client edge - server edge running on a browser

integration: foreman uses rails minitest with capybara (phantomJs driver)

While this integration test architecture has some benefits:

Pros

1. uses ruby code
2. Mocking data with fixtures and factoyBot
3. Direct access to models/DB
4. Works naturally with rails

it defiantly has some cons that affect us:

Cons:

1. Too slow
2. Debugging is difficult and frustrating
3. Random test failures
4. Part of the integrations tests are testing some behavior from unit perspective (due to abscent of tools like jest)
5. The current architecture seems to be incompatible with react, some tests are failing due to timeout errors, bad handling with promises, etc.
6. tests are written in ruby, but communicate mostly with client events

Alternatives:

1. Gradually moving to another js driver, which may solves some of the issues above
2. Replacing capybara for react pages with a more dedicated tool
3. Gradually migrate capybara to a different tool
4. Suggest any ? :)

On my searching for a modern testing tool, a tool named cypress took my attention the most

an open source project for full browser E2E tests

Pros:

1. integrates with CI such as jenkins and travis
2. recording tests - automatically record failing test with a video and a snapshot
3. Optional dashbaord - Cypress could be used as a sass with a dashboard, free for open source
4. great for debugging tests
5. adds a local tool for testing and debugging
6. tests are written in javascript - communicate better with the client
7. running tests concurrently

Cons:

1. No "out of the box" server side communication
2. Different envieromet, javascript based
3. ATM only chrome and electron are supported (supprting firefox and IE are developed)

Demonstration

For the demonstration I've created a travis configuration with foreman test environment and cypress

branch with these changes

Test example - 'layout.js'

'layout' test recording, recorded on travis everioment

The new react based layout integration test

import { FOREMAN_ADDR } from './config';

describe('Layout', () => {
  beforeEach(() => {
    cy.visit(FOREMAN_ADDR);
  });

  it('hover all menu items', () => {
    cy.get('.fa-tachometer').trigger('mouseover');
    cy.contains('Dashboard').should('be.visible');

    cy.get('.fa-server').trigger('mouseover');
    cy.contains('All Hosts').should('be.visible');

    cy.get('.fa-wrench').trigger('mouseover');
    cy.contains('Host Group').should('be.visible');

    cy.get('.pficon-network').trigger('mouseover');
    cy.contains('Smart Proxies').should('be.visible');

    cy.get('.fa-cog').trigger('mouseover');
    cy.contains('Users').should('be.visible');
  });

  it('taxonomies dropdown', () => {
    cy.contains('Any Organization').click();
    cy.contains('Manage Organizations').should('be.visible');

    cy.contains('Any Location').click();
    cy.contains('Manage Locations').should('be.visible');
  });

  it('notfication drawer', () => {
    cy.get('#notifications_container').click();
    cy.contains('No Notifications Available').should('be.visible');
  });

  it('user dropdwon', () => {
    cy.get('#account_menu').click();
    cy.contains('My Account').should('be.visible');
  });

  it('mobile view', () => {
    cy.viewport(550, 750);
    cy.contains('Toggle navigation').click();
    cy.get('.visible-xs-block .fa-user').click();
    cy.contains('My Account').should('be.visible');
  });
});

Cypress-dashboard

E2E

Migrate integration tests, even gradually, might be an overkill, even with these great advantages. however we can take these, and create an E2E architecture for foreman (within production environment). With it we can gain great benefits, for example, discovering js runtime errors, which happen after webpack compilation. after all, this will increase foreman builds stability by far.