jml/runtest.rst

## runtest.rst

      
    Raw
  

              runtest.rst
            
          
    Some notes on a different approach to xUnit style tests. I don't necessarily want to implement it—I don't quite have the motive or opportunity—but I think it interesting enough to share.
Instead of telling users to subclass testtools.TestCase, we would tell them to decorate. Tests would look like:
@test_cases()
class MyTests(object):
    def setUp(self):
        # no upcall
        whatever()
    def test_foo(self):
        assert 1 == 1
    def tearDown(self):
        destroy_things()
@test_cases() would adapt the suite to an object that implements only the bare bones of TestCase: run(), id(), countTestCases(), etc¹.
The actual test classes (MyTests) would have absolutely _nothing inserted into their namespace: no assertions; no fixtures; no details; no methods to upcall. Maybe we'd have an explicit interface that they would have to conform to, perhaps implemented by a purely abstract base class.
@test_cases() would only be responsible for the strategy that translates user code into a runnable TestCase. That is, it would know about setUp and tearDown, and it would know how to translate raised exceptions into various calls on a TestResult².
We could do this, we have the technology.
We might even want to go further, and start with something that adapts functions to runnable test cases and then build the richer strategy in terms of that. That could be quite fun.
If we wanted to provide users with the benefits of details and assertions, we could build something on top of the basic @test_cases that passes a fixture and a detailed into each method, e.g.:
@enriched_test_cases(fixtures=True, details=True)
class MyTests(object):
    def setUp(self, fixture, details):
        fixture.useFixture(TempDir())
    def test_something(self, fixture, details):
        # ...
I think this part needs a lot of experimentation, but I think those experiments should be done building on top of the minimal one. My hunch is that it will lead to simpler, more composable code.
We wouldn't need to provide assertions. assert_that and expect_that just need to take a detailed parameter in order to have full functionality.
Under this regime, RunTest as interface would melt away. The awkward constructor interface would be replaced by whatever parameters @test_cases provides and the run() interface would no longer be needed. Much of the code of RunTest would be moved into the adapter that @test_cases uses, but maybe the hierarchy of method calls could be flattened, since the adapter wouldn't be designed to be inherited from.
Importantly, the relationship would be unidirectional: @test_cases calls to the actual test case; the test case doesn't know about @test_cases at all.
DeferredRunTest and friends would likewise become @deferred_test_cases or something. They wouldn't be obligated to share code with the synchronous version, but we'd be free to factor out common code.
Good acceptance tests for this would be whether something like @flaky decorator support <https://github.com/ClusterHQ/flocker/blob/master/flocker/testtools/_flaky.py#L129> could be written using only public methods but still work with distinct underlying runners, whether Hypothesis works, whether we could build on this approach to implement something like testscenarios, and whether we could move all of the code in testtools.TestCase out into multiple smaller components.
Other quick thoughts:


We'd have to do some dancing around discovery. I looked briefly at this and it looks possible.
I have no idea how this would work with backwards compatibility. That's partly why I haven't tried seriously to write this as code.
I think this would be easier to do with explicit interfaces
Benefits for this plan are mainly implementation-side. End-user benefit is that they have less magic in their tests and a perhaps a vague feeling of being more lightweight. Possibly that a compositional approach makes it easier to build cool stuff for their tests.


Just putting it out there. Thoughts & questions welcome, but please try to be gentle.


https://github.com/testing-cabal/testtools/pull/178/files#diff-4372c1c31a1c37a51a147d74f52e157fR30↩
In my ideal world, it would translate them to a single value (perhaps an algebraic data type) that then gets translated into calls on TestResult↩