proposal.md

QUnit - Standard reporter interface

The goal of the project can be accurately described as follows:

More seriously, the goal of the project is to specify and implement a standard for the communication between testing tools like QUnit and integration tools. As of now, the different testing tools don't share a common interface. Thus, each integration tool needs to develop an individual plugin for every testing tool:

This of course is not optimal (N*M doesn't scale) What we want is a scenario that looks as follows:

Detailed Description

In general, there are three major steps:

1. Working out a standard for data formats and APIs
2. Making the test frameworks compatible with the standard
3. Creating reporters that integrate the standard into CI tools, etc.

Data Format

Looking at the discussions in the js-reporters repo there seems to be a general consensus on the event names, which are supported by all major frameworks.
One point that has not been considered yet is the role of the global errors (see this comment).

More challenging is to figure out what minimal set of properties should be chosen. For this, I'd like to create an overview of existing formats as it has been done for the event names here. One thing that needs to be taken into account is the infinite nesting of test suites in Mocha and Jasmine. It needs to be discussed whether that should be reflected in the data format.
For example, an (abbreviated) nested format for the done event may look as follows:

var mainSuite = {
    name: "suite name",
    childSuites: [childSuite1, childSuite2],
    tests: [test1, test2, test3],
    summary: {
        failed: 0,
        passed: 42,
        ...
    }
}

On the other hand, a simple format without nesting may suffice:

var test = {
    name: "suite name > subsuite name > test name",
    status: "passed",
    ...
}

var result = {
    tests: [test1, test2, childTest3],
    summary: {
        failed: 0,
        passed: 42,
        ...
    }
}

Last but not least, the test status needs to be standardized (see the corresponding issue).

Adapter

In an ideal world, we would just call standardReporter.register(StandardTestFramework); e.g. standardReporter.register(QUnit);. However, in practice, each test framework already implements its own methods and returns different data structures. To be independent from the active participation from other test frameworks, we want to write adapters that wrap the existing APIs for the first implementation. This way, we can quickly adapt existing frameworks and spot potential pitfalls.

Integration

Finally, once we have at least two test frameworks which support the standard, we want to write a basic reporter that integrates the standard into karma or browserstack (... or outputs TAP ;) ).

Timeline

Week	Description
1 (25.05\. - 31.05.)	research existing data formats (matrix as for event names)
2 (01.06\. - 07.06.)	propose initial draft for the standard data format
3 (08.06\. - 14.06.)	start QUnit adapter (meanwhile: incorporate changes of the draft)
4 (15.06\. - 21.06.)	complete QUnit adapter
5 (22.06\. - 28.06.)	write one other adapter (Mocha or Jasmine)
6 (29.06\. - 05.07.)	buffer for delayed work (mid- term evaluation on 3rd of July)
7 (06.07\. - 12.07.)	try to finalize specification (given consensus)
8 (13.07\. - 19.07.)	write reporter for one interested integration tool
9 (20.07\. - 26.07.)	create basic website that summarizes the standard
10 (27.07\. - 02.08.)	write a test reporter which can be used to test the adapters
11 (03.08\. - 09.08.)	polish website, promote standard, possibly add another adapter
12 (10.08\. - 16.08.)	buffer for delayed work
13 (17.08\. - 21.08.)	scrub code, improve documentation, etc.

Risks

The major risk for this project is that standards are getting established by adoption, not by writing the standard. The project's success (partially) depends on the support by other frameworks. To combat this, we want to use adapters in our first implementation. This way, we don't depend on other frameworks merging our code within the GSoC schedule.