Effective Unit Testing with Hypothesis and pytest

Unit testing is a crucial aspect of software development, ensuring that individual components of a system work as expected. While traditional unit testing involves manually crafting test cases, tools like Hypothesis and pytest can significantly streamline and enhance the testing process.

Introduction to Hypothesis

Hypothesis is a powerful library for property-based testing in Python. Unlike traditional unit testing, where you explicitly define test cases, Hypothesis allows you to specify the properties or invariants that your code should satisfy. It then automatically generates a diverse set of test cases to validate those properties.

One of the key advantages of Hypothesis is its ability to generate edge cases and corner cases that may be overlooked when manually writing test cases. This can greatly improve the robustness and reliability of your code.

Setting up Hypothesis with pytest

As you all know, I'm a FAN (!) of pytest, the popular testing framework for Python that provides a simple and intuitive way to write and run tests. Combining Hypothesis with pytest allows you to leverage the strengths of both tools.

First, you'll need to install both libraries:

pip install pytest hypothesis        

Next, import the necessary modules in your test file:

from hypothesis import given
import hypothesis.strategies as st        

Writing Property-Based Tests with Hypothesis

Let's consider a simple example of testing a function that calculates the sum of two numbers:

def add(x, y):
    return x + y        

With traditional unit testing, you might write test cases like this:

def test_add():
    assert add(2, 3) == 5
    assert add(0, 0) == 0
    assert add(-1, 1) == 0        

While these tests cover some basic scenarios, they may not catch edge cases or handle unexpected inputs.

With Hypothesis, you can define the property that the add function should satisfy: the sum of two integers should equal the expected result. Here's how you can write the property-based test:

@given(st.integers(), st.integers())
def test_add_is_commutative(x, y):
    assert add(x, y) == x + y        

In this example, the @given decorator tells Hypothesis to generate test cases using the provided strategies (st.integers() for both x and y). Hypothesis will automatically generate a wide range of integer values for x and y, ensuring that the add function is thoroughly tested for various inputs.

You can further customize the strategies to explore specific scenarios or edge cases. For example, you can include strategies for negative integers, large integers, or even custom data types.

@given(st.integers(min_value=-1000, max_value=1000), st.integers(min_value=-1000, max_value=1000))
def test_add_with_bounds(x, y):
    assert add(x, y) == x + y        

This test will generate integers between -1000 and 1000 for both x and y, ensuring that the add function works correctly within a specific range.

Running Tests with pytest

With pytest, you can run your tests as you would with any other pytest test suite, assuming you already have pytest, you'll just need to create a module starts with 'test_xyz.py' and it'll automatically fetch all the test cases and hook them into the pytest framework with all the bells and whistles.

from hypothesis import given
import hypothesis.strategies as st


def add(x, y):
    return x + y


@given(st.integers(), st.integers())
def test_add_is_commutative(x, y):
    assert add(x, y) == x + y


@given(st.integers(min_value=-1000, max_value=1000), st.integers(min_value=-1000, max_value=1000))
def test_add_with_bounds(x, y):
    assert add(x, y) == x + y        

When we will run it, the output will be as follows:

Launching pytest with arguments /Users/mordabastany/github/hypotehsis-example/test_example.py --no-header --no-summary -q in /Users/mordabastany/github/hypotehsis-example

============================= test session starts ==============================
collecting ... collected 2 items

a_new_tool.py::test_add_is_commutative PASSED                            [ 50%]
a_new_tool.py::test_add_with_bounds PASSED                               [100%]

============================== 2 passed in 0.38s ===============================

Process finished with exit code 0        

This output indicates that pytest discovered and ran two tests (test_add_is_commutative and test_add_with_bounds), and both tests passed successfully.

However, if one of the tests fails, Hypothesis will provide detailed information about the failing test case. For example, let's modify the add function to introduce a bug:

def add(x, y):
    return x - y  # Incorrect implementation        

Running the tests will produce the following error:

Launching pytest with arguments /Users/mordabastany/github/hypotehsis-example/test_example.py --no-header --no-summary -q in /Users/mordabastany/github/hypotehsis-example

============================= test session starts ==============================
collecting ... collected 2 items

a_new_tool.py::test_add_is_commutative FAILED                            [ 50%]
a_new_tool.py:6 (test_add_is_commutative)
-1 != 1

Expected :1
Actual   :-1
<Click to see difference>

@given(st.integers(), st.integers())
>   def test_add_is_commutative(x, y):

a_new_tool.py:8:

a_new_tool.py::test_add_with_bounds FAILED                               [100%]
a_new_tool.py:10 (test_add_with_bounds)
-1 != 1

Expected :1
Actual   :-1
<Click to see difference>

@given(st.integers(min_value=-1000, max_value=1000), st.integers(min_value=-1000, max_value=1000))
>   def test_add_with_bounds(x, y):

a_new_tool.py:12:        

This output demonstrates how Hypothesis can effectively generates test cases that expose issues in your code, while pytest provides a convenient way to run and report on these tests.

Debugging and looking at the code itself

While it's easy to explain how it happens in theory in reality it's much cooler to look into the bits and bytes.

In the attached example, I've started debugging the code to show how hypothesis generates the tests with the given strategy. it fetches integers and randomly injects them into the test cases skeleton, making it super rubout and reusable. There are however, few ways to use parameterize (the built in pytest generator), but for such cases, I personally would go with Hypothesis.

Conclusion

Combining Hypothesis and pytest offers a powerful approach to unit testing in Python. Hypothesis allows you to define properties and automatically generates test cases, ensuring thorough coverage and catching edge cases that might be missed with traditional unit testing. pytest provides a user-friendly testing framework, making it easy to run and manage your tests.

By leveraging these tools, you can write more robust and reliable code, catch bugs earlier in the development process, and ultimately deliver higher-quality software.