Plans for the rebol-test repository.

rebol_test_plans.txt

==Plans for the rebol-test test framework==

===The current state===

====Test format====

=====Carl's requirements=====

The format of the test file has been designed by Carl. His requirements were:

* keep the format as simple as possible
* individual tests shall be Rebol blocks
* use issues as test flags
* every test should be evaluated and should succeed yielding TRUE or fail in any other case
* it should be easy to detect testing regressions

The reasons were:

* a simple format shall alow everybody to write new tests
* a simple format was also meant for the test framework code to be as simple as possible (Carl wanted the test framework to consist of one page of Rebol code)

=====Ladislav's adjustments=====

Respecting Carl's requirements I made two adjustments:

* I put all the R2 and R3 tests into one file at present
* Individual tests are not required to be loadable as Rebol blocks; loadability became one of the properties that are subject to testing; tests are just required to be correctly parenthesized with respect to square brackets and round parentheses and any Rebol strings they contain must be correctly delimited by paired "" or {}.

Reasons:

* the usage of just one file lead to simplifications of the code implementing the crash recovery feature
* the relaxation of the loadability requirement means that even if some test is not loadable in a specific interpreter version, the test file is still not considered malformed if it is correctly parenthesized as specified above; this may happen (and already is happening) between different versions of Rebol interpreter

====Tests====

* Currently, we have 4527 tests in the core-tests.r file.
* According to the latest findings by Andreas the test suite runs 40% of the lines in the r3 source code written in C.
* The total size of the test framework and test files exceeds the total size of the R3 interpreter source code.
* There are sources of new tests that weren't fully exploited yet. One of them is the source code of the interpreter containing some useful tests.
* Another source of new tests is the CureCode database, which looks like growing faster than the current group of test framework maintainers is able to add the related test cases to the test suite.

====Framework properties====

* there are two similar functions: DO-TESTS and DO-RECOVER
* the %run.r script runs the tests depending on the #...crash flags
* the %run-safe.r script runs the tests depending on the #...crash flags but logs the complete information necessary to recover from crash
* the dependency on the #...crahs flags means that the existing flags need to be updated manually whenever the crash information changes
* the %run-recover.r script ignores the #...crash flags being able to automatically resume testing after a crash
* as mentioned above, the %run-recover.r file currently assumes all tests are in one test file

===Plans===

====Tests====

* we would like to recruit people wanting to help with collecting the remaining tests from the R3 source code and insert them into the test suite
* we would like to recruit people wanting to help with defining tests related to CureCode issues not covered by tests yet

====Framework====

This list contains the tasks I want to do in the next days:

* define a separate load-tests function to simplify the framework code a bit
* merge DO-TESTS and DO-RECOVER
* add new statistics like the number of regressions and progressions relative to an existing log file
* get rid of the #...crash flags and the necessity to update the set of crash flags whenever a crash is resoved; the usage of the flags can be replaced by acquiring the necessary informations from an existing log file