Some notes on a different approach to xUnit style tests. I don't necessarily want to implement it—I don't quite have the motive or opportunity—but I think it interesting enough to share.
Instead of telling users to subclass testtools.TestCase
, we would tell them to decorate. Tests would look like:
@test_cases() class MyTests(object): def setUp(self): # no upcall whatever() def test_foo(self): assert 1 == 1 def tearDown(self): destroy_things()
@test_cases()
would adapt the suite to an object that implements only the bare bones of TestCase
: run()
, id()
, countTestCases()
, etc [1].
The actual test classes (MyTests
) would have absolutely _nothing_ inserted into their namespace: no assertions; no fixtures; no details; no methods to upcall. Maybe we'd have an explicit interface that they would have to conform to, perhaps implemented by a purely abstract base class.
@test_cases()
would only be responsible for the strategy that translates user code into a runnable TestCase
. That is, it would know about setUp
and tearDown
, and it would know how to translate raised exceptions into various calls on a TestResult
[2].
We could do this, we have the technology.
We might even want to go further, and start with something that adapts functions to runnable test cases and then build the richer strategy in terms of that. That could be quite fun.
If we wanted to provide users with the benefits of details and assertions, we could build something on top of the basic @test_cases
that passes a fixture
and a detailed
into each method, e.g.:
@enriched_test_cases(fixtures=True, details=True) class MyTests(object): def setUp(self, fixture, details): fixture.useFixture(TempDir()) def test_something(self, fixture, details): # ...
I think this part needs a lot of experimentation, but I think those experiments should be done building on top of the minimal one. My hunch is that it will lead to simpler, more composable code.
We wouldn't need to provide assertions. assert_that
and expect_that
just need to take a detailed
parameter in order to have full functionality.
Under this regime, RunTest
as interface would melt away. The awkward constructor interface would be replaced by whatever parameters @test_cases
provides and the run()
interface would no longer be needed. Much of the code of RunTest
would be moved into the adapter that @test_cases
uses, but maybe the hierarchy of method calls could be flattened, since the adapter wouldn't be designed to be inherited from.
Importantly, the relationship would be unidirectional: @test_cases
calls to the actual test case; the test case doesn't know about @test_cases
at all.
DeferredRunTest
and friends would likewise become @deferred_test_cases
or something. They wouldn't be obligated to share code with the synchronous version, but we'd be free to factor out common code.
Good acceptance tests for this would be whether something like @flaky decorator support <https://github.com/ClusterHQ/flocker/blob/master/flocker/testtools/_flaky.py#L129> could be written using only public methods but still work with distinct underlying runners, whether Hypothesis_ works, whether we could build on this approach to implement something like testscenarios, and whether we could move all of the code in testtools.TestCase
out into multiple smaller components.
Other quick thoughts:
- We'd have to do some dancing around discovery. I looked briefly at this and it looks possible.
- I have no idea how this would work with backwards compatibility. That's partly why I haven't tried seriously to write this as code.
- I think this would be easier to do with explicit interfaces
- Benefits for this plan are mainly implementation-side. End-user benefit is that they have less magic in their tests and a perhaps a vague feeling of being more lightweight. Possibly that a compositional approach makes it easier to build cool stuff for their tests.
Just putting it out there. Thoughts & questions welcome, but please try to be gentle.
[1] | https://github.com/testing-cabal/testtools/pull/178/files#diff-4372c1c31a1c37a51a147d74f52e157fR30 |
[2] | In my ideal world, it would translate them to a single value (perhaps an algebraic data type) that then gets translated into calls on TestResult |
ITestCaseStrategy documents what
RunTest
currently expects of aTestCase
implementation. The relationship is mostly unidirectional, but it has some non-obvious interactions:TestCase
constructsRunTest
TestCase.run
callsRunTest.run
RunTest.run
calls methods onTestCase
TestCase
callTestCase.expectThat
expectThat
setsforce_failure
onTestCase
RunTest
checks forforce_failure
at the end of a test run