Skip to content

Instantly share code, notes, and snippets.

@cefn
Last active February 22, 2019 14:03
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save cefn/c5436361a647c8c38e3422943466eca4 to your computer and use it in GitHub Desktop.
import pytest
def checkHelper(value):
assert list(value) == list('foo')
@pytest.mark.parametrize("value", (
'foo',
'far'
))
def testInline(value):
assert list(value) == list('foo')
@pytest.mark.parametrize("value", (
'foo',
'far'
))
def testHelper(value):
checkHelper(value)
@cefn
Copy link
Author

cefn commented Feb 22, 2019

This compares reports from two cases of a simple test failure, using pytest 4.3.0. I am trying to work out how to get the same quality of failure reports, even in the case that I use a helper function.

In testHelper() the failing assertion is in checkHelper in the same file. Pytest reports the failure poorly.
In testInline() the identical failing assertion is inline in a test function. Pytest reports the failure well.

I have found the low quality reports to be a blocker in real world cases, since the details of the failure are lost and the originating line cannot be navigated to.

Below is the report for testHelper() which relies on assertions taking place in checkHelper(). The report has serious problems:

  • It indicates with a > the source of the error as being in testHelper(), where checkHelper() was called, not the line in checkHelper where the AssertionError was raised.
  • It reports a failure at line 12, which is a blank line as you can see in the source above
  • The assertion has much less detail than the failure report for the identical inline assert failure
test_example.py:12 (testHelper[far])
['f', 'a', 'r'] != ['f', 'o', 'o']

Expected :['f', 'o', 'o']
Actual   :['f', 'a', 'r']
 <Click to see difference>

value = 'far'

    @pytest.mark.parametrize("value", (
        'foo',
        'far'
    ))
    def testHelper(value):
>       checkHelper(value)

test_example.py:18: 

By contrast, the case of testInline() gives me a much better report (except test_example.py:5 doesn't correspond with anything)

test_example.py:5 (testInline[far])
['f', 'a', 'r'] != ['f', 'o', 'o']

Expected :['f', 'o', 'o']
Actual   :['f', 'a', 'r']
 <Click to see difference>

value = 'far'

    @pytest.mark.parametrize("value", (
        'foo',
        'far'
    ))
    def testInline(value):
>       assert list(value) == list('foo')
E       AssertionError: assert ['f', 'a', 'r'] == ['f', 'o', 'o']
E         At index 1 diff: 'a' != 'o'
E         Full diff:
E         - ['f', 'a', 'r']
E         ?        ^    ^
E         + ['f', 'o', 'o']
E         ?        ^    ^

test_example.py:11: AssertionError

Does anyone know how to get checkHelper to have its assert statements rewritten properly by pytest import hooks so it gets proper pytest failure reporting?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment