hypothesis-testing

Overview

Hypothesis is the canonical Python property-based testing library (hyp-quickstart (opens in new window)). It generates inputs from strategies and automatically shrinks any failure to the simplest reproducer.

When to use

• A Python function has a non-trivial input domain (numeric ranges, string formats, list shapes) and example-based tests miss edge cases.
• A bug fix needs a property test to prevent regression of a class of inputs (not just the one that caused the bug).
• Refactoring code where the new implementation should be observably equivalent to the old (use Hypothesis to verify property: new(x) == old(x) for all x).
• Testing parsers / serializers / encoders where round-trip properties hold (decode(encode(x)) == x).

If only one or two specific examples need verification, parametrize or fixtures suffice - Hypothesis is overkill.

Step 1 - Install

pip install hypothesis

Pin a version in requirements-dev.txt / pyproject.toml. Hypothesis is well-maintained but adds 1-2 seconds to test runtime per generated case; default 100 examples per @given.

Step 2 - Basic property test

Per hyp-quickstart (opens in new window):

from hypothesis import given, strategies as st

@given(st.integers(0, 100))
def test_example(n):
    assert n < 50

The decorator runs the test 100 times with random n in [0, 100]. On failure, Hypothesis reports the falsifying example (the smallest n that violates the assertion) - typically 50 here.

Step 3 - Strategies

Per hyp-quickstart (opens in new window), a strategy names the shape of the input. The common ones:

st.integers(0, 100)                   # bounded ints
st.floats(allow_nan=False)            # floats, optionally excluding NaN/Inf
st.text(min_size=1)                   # strings
st.lists(st.integers(), max_size=10)  # collections
st.from_regex(r"[a-z]+@[a-z]+\.com", fullmatch=True)  # format-validated
st.sampled_from(Status)               # enum-like

Full catalog (binary, dictionaries, tuples, builds, composite): references/hypothesis-reference.md.

Step 4 - Composite strategies (dependent fields)

Per hyp-quickstart (opens in new window), @st.composite lets later values depend on earlier ones via draw():

from hypothesis import strategies as st

@st.composite
def valid_dates(draw):
    year = draw(st.integers(1900, 2100))
    month = draw(st.integers(1, 12))
    if month in (1, 3, 5, 7, 8, 10, 12):
        day = draw(st.integers(1, 31))
    elif month in (4, 6, 9, 11):
        day = draw(st.integers(1, 30))
    else:
        # Feb: account for leap years
        max_day = 29 if (year % 4 == 0 and year % 100 != 0) or year % 400 == 0 else 28
        day = draw(st.integers(1, max_day))
    return date(year, month, day)

@given(valid_dates())
def test_date_round_trip(d):
    assert date.fromisoformat(d.isoformat()) == d

The draw() call requests a value from a strategy; the composite returns the constructed value.

Step 5 - Filtering and assumptions

Two ways to constrain inputs:

# Filter at strategy level (preferred - Hypothesis can sample efficiently)
@given(st.integers(0, 100).filter(lambda x: x % 2 == 0))
def test_even_squares(n):
    assert (n * n) % 2 == 0

# Filter at test level (fallback when filtering depends on multi-input)
from hypothesis import assume

@given(st.integers(), st.integers())
def test_division(a, b):
    assume(b != 0)   # discard cases where b == 0
    result = a // b
    assert result * b + (a - result * b) == a

Per hyp-quickstart (opens in new window), use .filter() at the strategy level when possible (Hypothesis can sample efficiently). Use assume() inside the test when the precondition involves multiple inputs.

Heavy filtering is a smell - if 90% of generated cases are discarded, redesign the strategy.

Step 6 - Settings and reproducibility

from hypothesis import given, settings, strategies as st

@settings(max_examples=500, deadline=2000)   # 500 cases, 2s deadline per case
@given(st.integers())
def test_expensive(n):
    expensive_function(n)

For CI, set derandomize=True to make failures reproducible across runs (vs random seed = same property test passes locally, fails on CI mysteriously). Full settings table (max_examples, deadline, phases, verbosity): references/hypothesis-reference.md.

Step 7 - Round-trip and metamorphic properties

Two of the most useful property patterns:

Round-trip

import json

@given(st.dictionaries(st.text(), st.integers()))
def test_json_round_trip(d):
    assert json.loads(json.dumps(d)) == d

If decode(encode(x)) == x for all valid x, the encode/decode pair is correct.

Metamorphic (relating two computations on related inputs)

@given(st.lists(st.integers()))
def test_sort_idempotent(xs):
    assert sorted(sorted(xs)) == sorted(xs)

@given(st.lists(st.integers()), st.integers())
def test_sort_commutes_with_offset(xs, offset):
    sorted_xs = sorted(xs)
    sorted_offset = sorted(x + offset for x in xs)
    assert all(s + offset == o for s, o in zip(sorted_xs, sorted_offset))

Metamorphic tests are powerful when the function's "correct output" is hard to specify but its relationship to other inputs is easy.

Step 8 - pytest integration

Per hyp-quickstart (opens in new window): "Hypothesis works seamlessly with pytest fixtures and parametrize decorators."

@pytest.fixture
def db_conn():
    # ... setup ...
    yield conn
    # ... teardown ...

@given(st.integers(0, 1000))
def test_with_fixture(db_conn, n):
    db_conn.insert(id=n)
    assert db_conn.find(id=n) == n

Hypothesis re-runs the test body with new n each time; the fixture is set up once per test (per pytest's normal scope rules, unless it's function-scoped - then once per generated case).

Step 9 - CI integration

- run: pytest --hypothesis-seed=42  # deterministic seed for reproducibility

# OR via @settings(derandomize=True)

When a property test fails, the failure includes the falsifying example - copy that into a regression test:

@given(...)
@example(n=42)   # the falsifying example from the prior run
def test_my_property(n):
    ...

@example cases run before generated ones; locks the regression in permanently.

References

• references/hypothesis-reference.md - full strategies catalog, @settings table, anti-patterns, limitations.
• hyp-quickstart (opens in new window) - Hypothesis quickstart: @given, strategies, shrinking, composite, assume(), settings.
• fast-check-testing, proptest-testing, jqwik-testing, quickcheck-testing - per-language siblings with similar shape.
• schemathesis-fuzzing - applies PBT to API schemas (different layer; same conceptual framework).