Instantly share code, notes, and snippets.

@bionikspoon /README.md Secret
Last active Jul 23, 2016

Embed
What would you like to do?
Getting meaningful results from unit tests

Getting meaningful results from unit tests

Scenario:

We count on unittests failure messages to give us reasonable feedback on how to proceed with a failing test. When you're working with builtin data structures and objects, unit test feedback is usually pretty good. It helps you identify and solve the problem. In contrast, the default results you get from custom objects can be practically useless.

Case

Let me demonstrate what I mean:

Example 1, builtin data structures:

  • Here's an equality comparison of a massively nested tuple:

    def test_tuples_are_equal():
        a = (
            (('a',), (2,), ('c',),),
            (
                (('d',), ('e',),),
                ((('f',),),),
            )
        )
        b = (
            (('a',), (2,), ('c',),),
            (
                ((('d',), ('e',),),),
                ((('f',),),),
            )
        )
        assert a == b
  • py.test (and UnitTest) results show exactly where the problem is:

    >       assert a == b
    E       assert ((('a',), (2,...((('f',),),))) == ((('a',), (2,)...((('f',),),)))
    E         At index 1 diff: ((('d',), ('e',)), ((('f',),),)) != (((('d',), ('e',)),), ((('f',),),))
    E         Full diff:
    E         - ((('a',), (2,), ('c',)), ((('d',), ('e',)), ((('f',),),)))
    E         + ((('a',), (2,), ('c',)), (((('d',), ('e',)),), ((('f',),),)))
    E         ?                          +                  ++
    
    test_data_structure.py:54: AssertionError

Example 2, custom data structures:

In contrast, let's look at a node tree data structure.

  • an equality test (the same test as Example 1):

    def test_nodes_are_equal():
        a = Doc(
            Sequence(String('a'), Integer(2), String('c')),
            Sequence(
                Sequence(String('d'), String('e'), ),
                Sequence(
                    Sequence(String('f'))
                )
            )
        )
        b = Doc(
            Sequence(String('a'), Integer(2), String('c')),
            Sequence(
                Sequence(
                    Sequence(String('d'), String('e'))
                ),
                Sequence(
                    Sequence(String('f'))
                )
            )
        )
    
        assert a == b
  • And the results:

    >       assert a == b
    E       assert <Doc:(<Sequence:(<String:('a',)>, <Integer:(2,)>, <String:('c',)>)>, <Sequence:(<Sequence:(<String:('d',)>, <String:('e',)>)>, <Sequence:(<Sequence:(<String:('f',)>,)>,)>)>)> == <Doc:(<Sequence:(<String:('a',)>, <Integer:(2,)>, <String:('c',)>)>, <Sequence:(<Sequence:(<Sequence:(<String:('d',)>, <String:('e',)>)>,)>, <Sequence:(<Sequence:(<String:('f',)>,)>,)>)>)>
    
    test_data_structure.py:79: AssertionError

    Can anyone spot the problem?

Action:

For my talk, I want to spend a few minutes to show how to get results from Example 2 to look more like Example 1. In both PyTest and UnitTest there are builtin solutions. In PyTest, you can take over the assertion matcher and use python's diff module to output a helpful FAIL message. The documentation for this is sort of hidden in the corners of their site, but it does exist. UnitTest has a conceptually similar solution that can be show in parallel (research required, i know it exists).

Basically the goal is to demonstrate how to give your test runner specific instructions on how to report test failures with your custom objects. In this case, we would want to reuse the same diff logic you can find in pytest/unittest's source code to get beautiful, actionable results for your own objects.

Results

Useful unit test results!!!!! Win-win-win!

edit:
Proof of concept here (conftest.py) and here (utils/node_diff.py). For the talk I would want to reuse pytests diff source code instead of rewriting logic. I'll have to figure that out.

About the talk

Presentation style

This talk would be 100% code probably with some friendly 'copypasta' to keep it short.

Potential follow ups

If this talk goes well, a potential follow up could be a talk about automating some of your debug workflow on test fails. For example setting up pytest to automatically rerun failed tests using some obnoxious debug setting -vvvvvvvv or a helper function that gives better information.

import pytest
class Node(object):
def __init__(self, *value):
self.value = value
def __eq__(self, other):
try:
return self.value == other.value and type(self) == type(other)
except AttributeError:
return False
def __repr__(self):
NodeClass = self.__class__
return '<%s:%s>' % (NodeClass.__name__, self.value)
class Doc(Node):
pass
class Sequence(Node):
pass
class Scalar(Node):
pass
class String(Scalar):
pass
class Integer(Scalar):
pass
# TESTS
# =============================================================================
def test_tuples_are_equal():
a = (
(('a',), (2,), ('c',),),
(
(('d',), ('e',),),
((('f',),),),
)
)
b = (
(('a',), (2,), ('c',),),
(
((('d',), ('e',),),),
((('f',),),),
)
)
assert a == b
def test_nodes_are_equal():
a = Doc(
Sequence(String('a'), Integer(2), String('c')),
Sequence(
Sequence(String('d'), String('e'), ),
Sequence(
Sequence(String('f'))
)
)
)
b = Doc(
Sequence(String('a'), Integer(2), String('c')),
Sequence(
Sequence(
Sequence(String('d'), String('e'))
),
Sequence(
Sequence(String('f'))
)
)
)
assert a == b
if __name__ == '__main__':
pytest.main()
manu@msp ~/Code/chipy_unittest $ python test_data_structure.py -v ve chipy
========================================== test session starts ===========================================
platform linux -- Python 3.5.1+, pytest-2.9.2, py-1.4.31, pluggy-0.3.1 -- /home/manu/.ve/chipy/bin/python
cachedir: .cache
rootdir: /home/manu/Code/chipy_unittest, inifile:
collected 2 items
test_data_structure.py::test_tuples_are_equal FAILED
test_data_structure.py::test_nodes_are_equal FAILED
================================================ FAILURES ================================================
_________________________________________ test_tuples_are_equal __________________________________________
def test_tuples_are_equal():
a = (
(('a',), (2,), ('c',),),
(
(('d',), ('e',),),
((('f',),),),
)
)
b = (
(('a',), (2,), ('c',),),
(
((('d',), ('e',),),),
((('f',),),),
)
)
> assert a == b
E assert ((('a',), (2,...((('f',),),))) == ((('a',), (2,)...((('f',),),)))
E At index 1 diff: ((('d',), ('e',)), ((('f',),),)) != (((('d',), ('e',)),), ((('f',),),))
E Full diff:
E - ((('a',), (2,), ('c',)), ((('d',), ('e',)), ((('f',),),)))
E + ((('a',), (2,), ('c',)), (((('d',), ('e',)),), ((('f',),),)))
E ? + ++
test_data_structure.py:54: AssertionError
__________________________________________ test_nodes_are_equal __________________________________________
def test_nodes_are_equal():
a = Doc(
Sequence(String('a'), Integer(2), String('c')),
Sequence(
Sequence(String('d'), String('e'), ),
Sequence(
Sequence(String('f'))
)
)
)
b = Doc(
Sequence(String('a'), Integer(2), String('c')),
Sequence(
Sequence(
Sequence(String('d'), String('e'))
),
Sequence(
Sequence(String('f'))
)
)
)
> assert a == b
E assert <Doc:(<Sequence:(<String:('a',)>, <Integer:(2,)>, <String:('c',)>)>, <Sequence:(<Sequence:(<String:('d',)>, <String:('e',)>)>, <Sequence:(<Sequence:(<String:('f',)>,)>,)>)>)> == <Doc:(<Sequence:(<String:('a',)>, <Integer:(2,)>, <String:('c',)>)>, <Sequence:(<Sequence:(<Sequence:(<String:('d',)>, <String:('e',)>)>,)>, <Sequence:(<Sequence:(<String:('f',)>,)>,)>)>)>
test_data_structure.py:79: AssertionError
======================================== 2 failed in 0.02 seconds ========================================
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment