First let me state the changes are console only, I think the GUI behavior is intuitive.
existing return codes:
public static readonly int OK = 0;
public static readonly int INVALID_ARG = -1;
public static readonly int FILE_NOT_FOUND = -2;
public static readonly int FIXTURE_NOT_FOUND = -3;
public static readonly int UNEXPECTED_ERROR = -100;
new return codes:
public static readonly int FIXTURE_ERROR = -4;
public static readonly int ADDIN_NOT_FOUND = -5;
FIXTURE_ERROR takes precedence over reporting EXTENSION_NOT_FOUND. All existing errorcodes take precedence over FIXTURE_ERROR and EXTENSION_NOT_FOUND
There are two things I've ran into... I wonder if there are others I should be concerned about? Maybe someone would benefit from another error code indicating if no tests ran because of PlatformAttribute, but I'm speculating on whether thats helpful.
The most valuable thing is to have the callstacks for the non-test errors. It seems to me, these non-test exceptions would be caught by trapping calls to TestSuite.DoOneTimeSetUp and TestSuite.DoOneTimeTearDown.
Besides the callstack, its helpful to know when this happened relative to other tests. From the summary, hopefully it is clear which test ran before and after.
Last, some note indicates if a required extension is missing and there are no tests.
Considering a test DLL that has two test fixtures. The first has a single passing test, but then throws an exception in fixture teardown. The second fixture has a single passing test, but that fixture throws an exception from its constructor. Currently, the output would be:
NUnit version 2.5.9.10348
Copyright (C) 2002-2009 Charlie Poole.
Copyright (C) 2002-2004 James W. Newkirk, Michael C. Two, Alexei A. Vorontsov.
Copyright (C) 2000-2002 Philip Craig.
All Rights Reserved.
Runtime Environment -
OS Version: Microsoft Windows NT 6.1.7600.0
CLR Version: 2.0.50727.4952 ( Net 2.0 )
ProcessModel: Default DomainUsage: Single
Execution Runtime: Default
..F.running test
throwing exception from cleanup
Tests run: 3, Errors: 0, Failures: 1, Inconclusive: 0, Time: 0.0630036 seconds
Not run: 0, Invalid: 0, Ignored: 0, Skipped: 0
Errors and Failures:
1) SetUp Error : TestLibrary.BreaksInFixtureConstructor
SetUp : System.NotFiniteNumberException : fail in consturctor
at TestLibrary.BreaksInFixtureConstructor..ctor() in C:\TestLibrary\TestLibrary\BreaksInFixtureConstructor.cs:line 13
2) Parent Failure : TestLibrary.BreaksInFixtureConstructor.Test1
TestFixtureSetUp failed in BreaksInFixtureConstructor
The fixture exceptions should show in the list of exceptions. The summary remains the same. What changes is that a new error entry describes the teardown error. A similar change would be used in the case of a FixtureSetup exception.
Errors and Failures:
1) FixtureTearDown Error: TestLibrary.AlwaysCrashesAfterFixture
FixtureTearDown : <exception type> : <exception message>
<callstack of exception in teardown>
2) SetUp Error : TestLibrary.BreaksInFixtureConstructor
SetUp : System.NotFiniteNumberException : fail in consturctor
at TestLibrary.BreaksInFixtureConstructor..ctor() in C:\TestLibrary\TestLibrary\BreaksInFixtureConstructor.cs:line 13
3) Parent Failure : TestLibrary.BreaksInFixtureConstructor.Test1
TestFixtureSetUp failed in BreaksInFixtureConstructor
In the case of a RequiredAddinAttribute that is not satisfied, an error message is always printed in the console output last after the summary, error info, and tests not run info. The message should indicate what extension was required. It might be:
RequiredAddinAttribute failure: Could not find required addin "<addinName>".
Current, "Not runnable" messages are shown for every test with a RequiredAddinAttribute is not matched. These messages are excessive when there are many non-tests, and insufficient when the only tests that exist need to be discovered by the addin. Preferably these per-test messages could be excluded from the console output.