Skip to content

Instantly share code, notes, and snippets.

@cefn

cefn/test_example.py

Last active Feb 22, 2019
Embed
What would you like to do?
import pytest
def checkHelper(value):
assert list(value) == list('foo')
@pytest.mark.parametrize("value", (
'foo',
'far'
))
def testInline(value):
assert list(value) == list('foo')
@pytest.mark.parametrize("value", (
'foo',
'far'
))
def testHelper(value):
checkHelper(value)
@cefn

This comment has been minimized.

Copy link
Owner Author

@cefn cefn commented Feb 22, 2019

This compares reports from two cases of a simple test failure, using pytest 4.3.0. I am trying to work out how to get the same quality of failure reports, even in the case that I use a helper function.

In testHelper() the failing assertion is in checkHelper in the same file. Pytest reports the failure poorly.
In testInline() the identical failing assertion is inline in a test function. Pytest reports the failure well.

I have found the low quality reports to be a blocker in real world cases, since the details of the failure are lost and the originating line cannot be navigated to.

Below is the report for testHelper() which relies on assertions taking place in checkHelper(). The report has serious problems:

  • It indicates with a > the source of the error as being in testHelper(), where checkHelper() was called, not the line in checkHelper where the AssertionError was raised.
  • It reports a failure at line 12, which is a blank line as you can see in the source above
  • The assertion has much less detail than the failure report for the identical inline assert failure
test_example.py:12 (testHelper[far])
['f', 'a', 'r'] != ['f', 'o', 'o']

Expected :['f', 'o', 'o']
Actual   :['f', 'a', 'r']
 <Click to see difference>

value = 'far'

    @pytest.mark.parametrize("value", (
        'foo',
        'far'
    ))
    def testHelper(value):
>       checkHelper(value)

test_example.py:18: 

By contrast, the case of testInline() gives me a much better report (except test_example.py:5 doesn't correspond with anything)

test_example.py:5 (testInline[far])
['f', 'a', 'r'] != ['f', 'o', 'o']

Expected :['f', 'o', 'o']
Actual   :['f', 'a', 'r']
 <Click to see difference>

value = 'far'

    @pytest.mark.parametrize("value", (
        'foo',
        'far'
    ))
    def testInline(value):
>       assert list(value) == list('foo')
E       AssertionError: assert ['f', 'a', 'r'] == ['f', 'o', 'o']
E         At index 1 diff: 'a' != 'o'
E         Full diff:
E         - ['f', 'a', 'r']
E         ?        ^    ^
E         + ['f', 'o', 'o']
E         ?        ^    ^

test_example.py:11: AssertionError

Does anyone know how to get checkHelper to have its assert statements rewritten properly by pytest import hooks so it gets proper pytest failure reporting?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.