RFC Foreman: Integration tests & E2E testing

readme.md

RFC - Integration Tests & End to End tests

Present architecture

E2E

Pipeline bats, but not client edge - server edge running on a browser

Integration

foreman uses rails minitest with capybara (phantomJs driver)

While this integration test architecture has some benefits:

1. uses ruby code
2. Mocking data with fixtures and factoyBot
3. Direct access to models/DB
4. Works naturally with rails

It defiantly has some cons that affect us:

1. Too slow
2. Debugging is difficult and frustrating
3. Random test failures
4. Abusing tests - some tests are written from a unit prespecitve
5. The current architecture seems to be incompatible with react, some tests are failing due to timeout errors, bad handling with promises, etc.

Alternatives:

1. Gradually moving to another js driver - (we have couple of PRs for headless chrome i.e #21592),
2. Replacing capybara for react pages with a different tool
3. Gradually migrate capybara to a different tool
4. Suggest any ? :)

On my searching for a modern testing tool, a tool named cypress took my attention the most

an open source project for full browser E2E testing

Pros:

1. integrates with CI such as jenkins and travis
2. recording tests - automatically record failing test with a video and a snapshot
3. Optional dashbaord - Cypress could be used as a sass with a dashboard, free for open source
4. great for debugging tests
5. adds a local tool for testing and debugging
6. tests are written in javascript - communicate better with the client
7. running tests concurrently
8. allows to execute processes - there are few gems that communicates with rails (cypress-on-rails)

Cons:

1. No "out of the box" server side communication
2. Different envieromet, javascript based
3. ATM only chrome and electron are supported (supprting firefox and IE are developed)

Demonstration

For the demonstration I've created a travis configuration with foreman test environment and cypress

branch with these changes

Test example

test recording, recorded on travis everioment

Test Code

import { FOREMAN_ADDR } from './config';

describe('Layout', () => {
  beforeEach(() => {
    cy.visit(FOREMAN_ADDR);
  });

  it('hover all menu items', () => {
    cy.get('.fa-tachometer').trigger('mouseover');
    cy.contains('Dashboard').should('be.visible');

    cy.get('.fa-server').trigger('mouseover');
    cy.contains('All Hosts').should('be.visible');

    cy.get('.fa-wrench').trigger('mouseover');
    cy.contains('Host Group').should('be.visible');

    cy.get('.pficon-network').trigger('mouseover');
    cy.contains('Smart Proxies').should('be.visible');

    cy.get('.fa-cog').trigger('mouseover');
    cy.contains('Users').should('be.visible');
  });

  it('taxonomies dropdown', () => {
    cy.contains('Any Organization').click();
    cy.contains('Manage Organizations').should('be.visible');

    cy.contains('Any Location').click();
    cy.contains('Manage Locations').should('be.visible');
  });

  it('notfication drawer', () => {
    cy.get('#notifications_container').click();
    cy.contains('No Notifications Available').should('be.visible');
  });

  it('user dropdwon', () => {
    cy.get('#account_menu').click();
    cy.contains('My Account').should('be.visible');
  });

  it('mobile view', () => {
    cy.viewport(550, 750);
    cy.contains('Toggle navigation').click();
    cy.get('.visible-xs-block .fa-user').click();
    cy.contains('My Account').should('be.visible');
  });
});

Cypress-dashboard

E2E

Migrate integration tests, even gradually, might be an overkill, even with these great advantages. however we can take these, and create an E2E architecture for foreman (within production environment). With it we can gain great benefits, for example, discovering js runtime errors, which happen after webpack compilation. after all, this will increase foreman builds stability by far.

Looks good to me for starting up this conversation. A few typos I noticed:

It defiantly has some cons that affect us:

Should be "definitely"

Different envieromet, javascript based

Should be "environment"