Test-Everything Spec

Test Everything Spec.md

Test-Everything Spec

This spec aims to define a format for recording the results of tests. It aims to be completely minimal, and can be extended as needed. Reporters should only assume the presense of these features, but may be progressively enhanced when more information is available.

Static Format

Test results consist of a JSON document containing a single "Section"

Section

A "Section" is a JSON object with the following properties:

• name must be a string and is considered to be the name of the section.
• children must be an array and can contain both "Section"s and "Test"s. It can be a mixture of both.

If the name is ommitted, then the children are considered to be part of the parent section. If this is the root section, and name is ommitted then the tests are considered to have an anonymous root section.

Test

A "Test" is a JSON object with two (or more) properties:

• name must be a non-empty string and is considered to be the name of the test.
• passed must be either true or false. It should be false for all cases except when the test has passed in full. Notably it should be false if the test was skipped or is pending.

Streaming Format

Test results can be streamed by representing each node with a start and end node, each of which must be a valid JSON document. This results in 4 types of nodes:

• Section Start
• Section End
• Test Start
• Test End

Test results must begin with a Section Start with a name of "root" and end with a Section End with a name of "root". In between there must be properly nested Section Start/Section End pairs and Test Start/Test End pairs. A test can never contain a section.

Section Start

A Section Start must have a corresponding Section End node with the same name some time after it. Section Start must be an object with the following two properties:

• type must be a string equal to "section-start"
• name must be a string and is considered to be the name of the section.

It may optionally conatin an additional property:

• children is the expected number of direct descendants (sections or tests) which are expected to be run. It may be used to validate that all test results have been successfully reported.

Section End

A Section End must have a corresponding Section Start node with the same name some time before it. Section End must be an object with the following two properties:

• type must be a string equal to "section-end"
• name must be a string equal to the name property of the corresponding Section Start node.

It may optionally conatin an additional property:

• children is the expected number of direct descendants (sections or tests) which are expected to have been run. It may be used to validate that all test results have been successfully reported.

Test Start

A Test Start must be immediately followed by a Test End node. Test Start must be an object with at least the following two properties:

• type must be a string equal to "test-start"
• name must be a non-empty string and is considered to be the name of the test.

Test End

A Test End must be immediately preceded by a Test Start node. Test End must be an object with at least the following three properties:

• type must be a string equal to "test-end"
• name must be a non-empty string equal to the name property of the corresponding Test Start node.
• passed must be either true or false. It should be false for all cases except when the test has passed in full. Notably it should be false if the test was skipped.