Pytest вывод ошибки

I am relatively new to pytest hooks and plugins and I am unable to figure out how to get my pytest code to give me test execution summary with reason of failure.

Consider the code:

class Foo:
    def __init__(self, val):
        self.val = val

    def test_compare12():
        f1 = Foo(1)
        f2 = Foo(2)
        assert f1 == f2, "F2 does not match F1"

    def test_compare34():
        f3 = Foo(3)
        f4 = Foo(4)
        assert f3 == f4, "F4 does not match F3"

When I run the pytest script with -v option, it gives me the following result on the console:

========================= test session starts=================================
platform darwin -- Python 2.7.5 -- py-1.4.26 -- pytest-2.7.0 --    /Users/nehau/src/QA/bin/python
rootdir: /Users/nehau/src/QA/test, inifile: 
plugins: capturelog
collected 2 items 

test_foocompare.py::test_compare12 FAILED
test_foocompare.py::test_compare34 FAILED

================================ FAILURES ===============================
_______________________________ test_compare12 _________________________

def test_compare12():
    f1 = Foo(1)
    f2 = Foo(2)
>       assert f1 == f2, "F2 does not match F1"
E       AssertionError: F2 does not match F1
E       assert <test.test_foocompare.Foo instance at 0x107640368> == <test.test_foocompare.Foo instance at 0x107640488>

test_foocompare.py:11: AssertionError
_____________________________ test_compare34______________________________

def test_compare34():
    f3 = Foo(3)
    f4 = Foo(4)
>       assert f3 == f4, "F4 does not match F3"
E       AssertionError: F4 does not match F3
E       assert <test.test_foocompare.Foo instance at 0x107640248> == <test.test_foocompare.Foo instance at 0x10761fe60>

test_foocompare.py:16: AssertionError

=============================== 2 failed in 0.01 seconds ==========================

I am running close to 2000 test cases, so it would be really helpful if I could have pytest display output in the following format:

::
test_foocompare.py::test_compare12 FAILED AssertionError:F2 does not match F1
test_foocompare.py::test_compare34 FAILED AssertionError:F2 does not match F1
::

I have looked at pytest_runtest_makereport plugin but can’t seem to get it working. Anyone has any other ideas?

Thanks

I am relatively new to pytest hooks and plugins and I am unable to figure out how to get my pytest code to give me test execution summary with reason of failure.

Consider the code:

class Foo:
    def __init__(self, val):
        self.val = val

    def test_compare12():
        f1 = Foo(1)
        f2 = Foo(2)
        assert f1 == f2, "F2 does not match F1"

    def test_compare34():
        f3 = Foo(3)
        f4 = Foo(4)
        assert f3 == f4, "F4 does not match F3"

When I run the pytest script with -v option, it gives me the following result on the console:

========================= test session starts=================================
platform darwin -- Python 2.7.5 -- py-1.4.26 -- pytest-2.7.0 --    /Users/nehau/src/QA/bin/python
rootdir: /Users/nehau/src/QA/test, inifile: 
plugins: capturelog
collected 2 items 

test_foocompare.py::test_compare12 FAILED
test_foocompare.py::test_compare34 FAILED

================================ FAILURES ===============================
_______________________________ test_compare12 _________________________

def test_compare12():
    f1 = Foo(1)
    f2 = Foo(2)
>       assert f1 == f2, "F2 does not match F1"
E       AssertionError: F2 does not match F1
E       assert <test.test_foocompare.Foo instance at 0x107640368> == <test.test_foocompare.Foo instance at 0x107640488>

test_foocompare.py:11: AssertionError
_____________________________ test_compare34______________________________

def test_compare34():
    f3 = Foo(3)
    f4 = Foo(4)
>       assert f3 == f4, "F4 does not match F3"
E       AssertionError: F4 does not match F3
E       assert <test.test_foocompare.Foo instance at 0x107640248> == <test.test_foocompare.Foo instance at 0x10761fe60>

test_foocompare.py:16: AssertionError

=============================== 2 failed in 0.01 seconds ==========================

I am running close to 2000 test cases, so it would be really helpful if I could have pytest display output in the following format:

::
test_foocompare.py::test_compare12 FAILED AssertionError:F2 does not match F1
test_foocompare.py::test_compare34 FAILED AssertionError:F2 does not match F1
::

I have looked at pytest_runtest_makereport plugin but can’t seem to get it working. Anyone has any other ideas?

Thanks

In these examples I’ll keep both the code under test and the testing function in the same file. You’ve already seen how it would look normally if we imported the functions under test from another module. If not check out the getting started with pytest article.

Also, in order to make the results clear, I’ve removed the summary of the test runs and kept only the actual
error reporting.

Comparing numbers for equality in Pytest

Probably the most basic thing to test is whether a function given some input returns an expected number.

examples/python/pt3/test_number_equal.py

def double(n):
    #return 2*n
    return 2+n

def test_string_equal():
    assert double(2) == 4
    assert double(21) == 42

In the above function double someone has mistakenly used + instead of *. The result of the test looks like this

$ pytest test_number_equal.py

    def test_string_equal():
        assert double(2) == 4
>       assert double(21) == 42
E       assert 23 == 42
E        +  where 23 = double(21)

The line starting with the > sign indicates the assert line that failed. The lines starting with E are the details.

Compare numbers relatively

In certain cases we cannot test for equality. For example if we would like to test if some process finishes within a given time, or whether a timeout is triggered at the right time. In such cases we need to compare if a number is less-than or greater-than some other number.

examples/python/pt3/test_number_less_than.py

def get_number():
    return 23

def test_string_equal():
    assert get_number() < 0 

Running the test will provide the following error message:

$ pytest test_number_less_than.py

    def test_string_equal():
>       assert get_number() < 0
E       assert 23 < 0
E        +  where 23 = get_number()

The error-report looks quite similar to what we had above, but in this case too it is clear what was the comparision operation that failed.

Comparing strings

Similar to numbers we might want to know if a string received from some function is the same as we expect it to be.

examples/python/pt3/test_string_equal.py

def get_string():
    return "abc"

def test_string_equal():
    assert get_string() == "abd"

The result looks familiar:

$ pytest test_string_equal.py

    def test_string_equal():
>       assert get_string() == "abd"
E       AssertionError: assert 'abc' == 'abd'
E         - abc
E         + abd

For such short strings seeing both the expected string and the actual string is ok.
We can look at the strings and compare them character by character to see what was the actual difference.

Compare long strings

If the strings are much longer however, it would be really hard for us to pinpoint the specific location of the character (or characters) that differ. Luckily the authors of Pytest have thought about this problem as well:

examples/python/pt3/test_long_strings.py

import string

def get_string(s):
    return string.printable + s + string.printable

def test_long_strings():
    assert get_string('a') == get_string('b')

string.printable is a string containing all the printable ASCII characters:
0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!»#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~ \t\n\r\x0b\x0c

Our brilliant get_string function will return it twice with an additional character between them.
We use this to create two nasty and long strings that differ by a single character.

The output looks like this:

$ pytest test_long_strings.py

    def test_long_strings():
>       assert get_string('a') == get_string('b')
E       AssertionError: assert '0123456789ab...t\n\r\x0b\x0c' == '0123456789abc...t\n\r\x0b\x0c'
E         Skipping 90 identical leading characters in diff, use -v to show
E         Skipping 91 identical trailing characters in diff, use -v to show
E           {|}~
E
E         - a012345678
E         ? ^
E         + b012345678
E         ? ^

I think this explains quite nicely where have the two strings differ and if you really, really want to see the
whole string you can use the -v flag.

Is string in longer string

If you need to check whether a string is part of a larger string we can use the regular in operator.

examples/python/pt3/test_substring.py

import string

def get_string():
    return string.printable * 30

def test_long_strings():
    assert 'hello' in get_string()

In case of failure the result will include only the beginning and the end of the «long string».
This can be very usefule if you need to test whether a certain string appears or not in an HTML page.

examples/python/pt3/test_substring.txt

    def test_long_strings():
>       assert 'hello' in get_string()
E       assert 'hello' in '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~ \t\n\r\x0b\x0c012345...x0b\x0c0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~ \t\n\r\x0b\x0c'
E        +  where '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~ \t\n\r\x0b\x0c012345...x0b\x0c0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~ \t\n\r\x0b\x0c' = get_string()

Testing any expression

Instead of calling a function we might have an expression on one side of the equation. (Actually I am not sure how often this would happen in the real world.
Maybe we only see these in examples on how pytest works.)

examples/python/pt3/test_expression_equal.py

def test_expression_equal():
    a = 3
    assert a % 2 == 0

The test result:

$ pytest test_expression_equal.py

    def test_expression_equal():
        a = 3
>       assert a % 2 == 0
E       assert (3 % 2) == 0

Is element in a list?

Besides comparing individual values we might also want to compare more complex data. First, let’s see what happens if our test must ensure that a value can be found in a list?

examples/python/pt3/test_in_list.py

def get_list():
    return ["monkey", "cat"]

def test_in_list():
    assert "dog" in get_list()

We can use the in operator of Python. The result will look like this:

$ pytest test_in_list.py

    def test_in_list():
>       assert "dog" in get_list()
E       AssertionError: assert 'dog' in ['monkey', 'cat']
E        +  where ['monkey', 'cat'] = get_list()

Pytest will conveniently show us the list that did not contain the expected value.

Compare lists in Pytest

A more interesting case might be testing if the returned list is the same as the expected list. Using the == operator can tell us if the two lists are equal or not, but if we need to understand what went wrong, we’d better know where do the lists differ.
Or at least where do they start to differ.

examples/python/pt3/test_lists.py

import string
import re

def get_list(s):
    return list(string.printable + s + string.printable)

def test_long_lists():
    assert get_list('a') == get_list('b')

The result:

$ pytest test_lists.py

    def test_long_lists():
>       assert get_list('a') == get_list('b')
E       AssertionError: assert ['0', '1', '2...'4', '5', ...] == ['0', '1', '2'...'4', '5', ...]
E         At index 100 diff: 'a' != 'b'
E         Use -v to get the full diff

We could further explore the output for the cases when multiple elements differ and when one list is a sublist of the other.

Compare dictionaries in Pytest

Dictionaries can differ in a number of ways. The keys might be identical, but some values might differ.
Some keys might be missing in the actual result or there might be some extra keys.

In this example we test all of these:

Using the string.printable we create a dictionary where the keys are the printable characters and the values
are their respective ASCII value returned by the ord function. Then we add (or replace) one of key-value pair.

examples/python/pt3/test_dictionaries.py

import string
import re

def get_dictionary(k, v):
    d = dict([x, ord(x)] for x in  string.printable)
    d[k] = v
    return d

def test_big_dictionary_different_value():
    assert get_dictionary('a', 'def') == get_dictionary('a', 'abc')

def test_big_dictionary_differnt_keys():
    assert get_dictionary('abc', 1) == get_dictionary('def', 2)

The result looks like this:

$ pytest test_dictionaries.py

______________ test_big_dictionary_different_value _______________

    def test_big_dictionary_different_value():
>       assert get_dictionary('a', 'def') == get_dictionary('a', 'abc')
E       AssertionError: assert {'\t': 9, '\n...x0c': 12, ...} == {'\t': 9, '\n'...x0c': 12, ...}
E         Omitting 99 identical items, use -v to show
E         Differing items:
E         {'a': 'def'} != {'a': 'abc'}
E         Use -v to get the full diff

_______________ test_big_dictionary_differnt_keys ________________

    def test_big_dictionary_differnt_keys():
>       assert get_dictionary('abc', 1) == get_dictionary('def', 2)
E       AssertionError: assert {'\t': 9, '\n...x0c': 12, ...} == {'\t': 9, '\n'...x0c': 12, ...}
E         Omitting 100 identical items, use -v to show
E         Left contains more items:
E         {'abc': 1}
E         Right contains more items:
E         {'def': 2}
E         Use -v to get the full diff

The first test function got two dictionaries where the value of a single key differed.

The second test function had an extra key in both dictionaries.

Testing for expected exceptions in Pytest

Finally let’s look at exceptions!

A good test suite will test the expected behaviour both when the input is fine and
also when the input triggers some exception. Without testing the exception we cannot
be sure that they will be really raiesed when necessary. An incorrect refactoring
might eliminate the error checking of our code therby letting through invalid data
and either triggering a different exception as in our example, or not generating any exception
just silently doing the wrong thing.

In this brilliant example the divide function checks if the divider is 0 and raises it
own type of exception instead of letting Python rais its own. If this is the defined behavior
someone using our module will probably wrap our code in some try expression and expect
a ValueError error. If someone changes our divide function and removed our
special exception then we basically have broken the exception-handling of our user.

examples/python/pt3/test_exceptions.py

import pytest

def divide(a, b):
    if b == 0:
        raise ValueError('Cannot divide by Zero')
    return a / b

def test_zero_division():
    with pytest.raises(ValueError) as e:
        divide(1, 0)
    assert str(e.value) == 'Cannot divide by Zero' 

The test ha actually two parts. The first part:

    with pytest.raises(ValueError) as e:
        divide(1, 0)

checks if a ValueError was raised during our call to divide(1, 0)
and will assign the exception object to the arbitrarily named variable e.

The second part is a plain assert that checks if the text of the exception is what
we expect to be.

This is now the expected behaviour. Our test passes:

$ pytest test_exceptions.py

test_exceptions.py .

What if someone changes the error message in our exception from Zero to Null?

examples/python/pt3/test_exceptions_text_changed.py

import pytest

def divide(a, b):
    if b == 0:
        raise ValueError('Cannot divide by Null')
    return a / b

def test_zero_division():
    with pytest.raises(ValueError) as e:
        divide(1, 0)
    assert str(e.value) == 'Cannot divide by Zero' 

The assert in the test will fail indicating the change in the text.
This is actually a plain string comparision.

$ pytest test_exceptions_text_changed.py


    def test_zero_division():
        with pytest.raises(ValueError) as e:
            divide(1, 0)
>       assert str(e.value) == 'Cannot divide by Zero'
E       AssertionError: assert 'Cannot divide by Null' == 'Cannot divide by Zero'
E         - Cannot divide by Null
E         ?                  ^^^^
E         + Cannot divide by Zero
E         ?                  ^^^^

In the second example we show the case when the special exception raising is gone.
Either by mistake or because someone decided that it should not be there.
In this case the first part of our test function will catch the different exception.

examples/python/pt3/test_exceptions_failing.py

import pytest

def divide(a, b):
#    if b == 0:
#        raise ValueError('Cannot divide by Zero')
    return a / b

def test_zero_division():
    with pytest.raises(ValueError) as e:
        divide(1, 0)
    assert str(e.value) == 'Cannot divide by Zero' 

The report will look like this:

$ pytest test_exceptions_failing.py

    def test_zero_division():
        with pytest.raises(ValueError) as e:
>           divide(1, 0)

test_exceptions_failing.py:10:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

a = 1, b = 0

    def divide(a, b):
    #    if b == 0:
    #        raise ValueError('Cannot divide by Zero')
>       return a / b
E       ZeroDivisionError: division by zero

Exception depositing money to the bank

Another case when checking for proper exceptions might be important is when
we want to avoid silently incorrect behavior.

For example in this code we have a function called deposit that expects
a non-negative number. We added our input validation that will raise an exception protecting
the balance of our bank account. (In our example we only indicated the location of the code
that actually changes the balance.)

examples/python/pt3/test_bank.py

import pytest

def deposit(money):
    if money < 0:
        raise ValueError('Cannot deposit negative sum')

    # balance += money

def test_negative_deposit():
    with pytest.raises(ValueError) as e:
        deposit(-1)
    assert str(e.value) == 'Cannot deposit negative sum' 

We have also created a test-case that will ensure that the protection is there,
or at least that the function raises an exception if -1 was passed to it.

Conclusion

Pytest and its automatic error reporting is awesome.