testtools's matchers are one of its best parts. They are so good that people have suggested moving them to their own package.
However, they've also received criticism. I recall Martin Pool saying that he liked them, but that they felt too cumbersome, as if one had to expend too much effort in order to define them.
I want to step back from what we have today and do some thinking about what matchers could be and where we should take them.
Here's the current interface, expressed using Haskellish type notation:
match :: a -> Maybe Mismatch
describe :: Mismatch -> Text
details :: Mismatch -> [Detail]This interface is probably OK for callers. I can't really imagine anything simpler, unless we insisted on strict evaluation of the Mismatch description, or dropped support for details.
Another way of saying this is that a matcher, once constructed, is essentially a predicate and an error message.
The "once constructed" is important, because it hides another key factor. Matchers are also constructors, which means they have names (e.g. Equals) and are a means of combining and abstracting a large variety of data.
Assertion: Making new matchers is cumbersome.
You pretty much have to make a new class just to define the match method. If you want to do it right, you also need to define __str__.
For reasons that elude me now, many of our matchers have custom Mismatch implementations. This doesn't seem to be necessary.
Assertion: Many of our existing matchers have awkward names.
Examples:
MatchesAllrather thanAllorAndMatchesSetwiseAfterPreprocessingMatchesStructure
The poor name MatchesAll seems like a surface-level problem that's easy to fix, but I wonder whether names like AfterPreprocessing are clues that we are getting the abstraction wrong.
Assertion: Few matchers have details.
I don't want to remove support for details. I think it's a fantastic innovation. However, I wonder whether we can make the simple case simpler.
Assertion: Matchers support composition, and this is a great feature
The not, and, and or equivalents are great.
- TODO: read up on & paste in radix's alternative approach
- Nice errors are important
- In a sense, this is what distinguishes matchers from predicates
- What if we had a
mismatchmethod onMatcherthat produced mismatches. Would that make anything nice? - Symmetry problem:
assertThat([foo, bar], Contains(foo))vsassertThat(foo, In([foo, bar]))- You have to define both, which sucks
- Almost all matchers fall into broad categories
- Binary comparison
f(x) == f(y)f(x) == g(y)relates(f(x), g(y))- e.g. less than
- Binary comparison
radix's otter.test.utils has two things of relevance:
- https://github.com/rackerlabs/otter/blob/master/otter/test/utils.py#L50
- lifts matchers to equality
- e.g.
Equals(42) == 42,Contains(foo) == [foo, bar]
- https://github.com/rackerlabs/otter/blob/master/otter/test/utils.py#L831
- implements equality after transformation
- e.g.
transform_eq(lambda x: x * 2, 42) == 42 - keeps a log of RHS
http://pyrsistent.readthedocs.org/en/latest/intro.html#invariants
Simple unary predicate function coupled with text message that's shown when predicate doesn't hold. Essentially (a -> Bool, Text), e.g. (lambda x: x % 2 == 0, 'x odd')
https://attrs.readthedocs.org/en/stable/examples.html#validators
https://pypi.python.org/pypi/schema
https://pypi.python.org/pypi/voluptuous
https://pypi.python.org/pypi/validino
Here's a list of all the matchers that exist in testtools. Maybe having them all here can help us identify better ways of expressing these ideas.
ContainsAllMatchesListwiseMatchesSetwiseMatchesStructureContainsEndsWithStartsWithEqualsGreaterThanHasLengthIsIsInstanceLessThanMatchesRegexNotEqualsKeysEqualMatchesDictContainsDictContainedByDictMatchesAllDictDocTestMatchesMatchesExceptionRaisesMatchesAnyMatchesAllNotAnnotateAllMatchAfterPreprocessingAnyMatchMatchesPredicateMatchesPredicateWithParamsPathExistsDirExistsFileExistsDirContainsFileContainsHasPermissionsSamePathTarballContains