PYTHON / TESTING AND TOOLING
Coverage and assertions that mean something
Read a coverage report critically and write assertions that pin exact behaviour, so a passing suite is real evidence the code works.
What you will learn
- Distinguish line from branch coverage and find partial branches with coverage run --branch
- Replace 'is not None' checks with exact expected values worked out independently
- Assert exception type and message using pytest.raises(match=...) or assertRaisesRegex
- Compare float results with math.isclose or pytest.approx instead of ==
Understanding Coverage and assertions that mean something
Coverage tools record which lines of your code executed while the tests ran. They do not know whether anything was checked, so a test that imports a module and calls one function with no assert at all can report 100% coverage. The right mental model is that coverage is a map of what your tests touched, and assertions are the only thing that decides whether what they touched was correct.
Line coverage is the weaker of the two measures coverage.py offers. An `if` with no `else` can have every one of its lines executed while the false outcome was never taken even once, and that untaken outcome is exactly where forgotten defaults and missing validation hide. Running `coverage run --branch -m pytest` followed by `coverage report -m` makes those gaps visible: the Missing column then shows entries like `14->exit`, meaning the condition on line 14 never fell through the way the report indicates.
An assertion is meaningful when it can fail for the reason you care about. That rules out tautologies like `assert result == compute(x)` where `compute` is the function under test, and near-tautologies like `assert result is not None` or `assert len(rows) == 3`, which stay true when every number in the result is wrong. Pin the value you expect from an independent source such as a hand calculation or a fixed fixture, use at least two data points per branch because a single one can pass by coincidence, and compare whole objects when equality is well defined so a new field cannot silently drift.
DISCOUNTS = {"HALF": 0.5, "TENTH": 0.1}
def final_price(price, code):
rate = DISCOUNTS.get(code, 0.0)
return round(price * rate, 2) # bug: should be price * (1 - rate)
def weak_test():
assert final_price(10.0, "HALF") is not None
assert final_price(10.0, "NOPE") is not None
def strong_test():
for code, expected in [("HALF", 5.0), ("TENTH", 9.0), ("NOPE", 10.0)]:
got = final_price(10.0, code)
assert got == expected, f"code={code!r}: expected {expected}, got {got}"
for test in (weak_test, strong_test):
try:
test()
print(f"{test.__name__}: passed")
except AssertionError as exc:
print(f"{test.__name__}: failed -> {exc}")
print("both tests execute every line of final_price")Coverage tells you which code ran; only assertions tell you whether it was right, so the two must be judged together.
Worked examples
Shape checks pass while the money is wrong
A structural assertion cannot fail on a bad total, and exact equality on floats fails for a reason that has nothing to do with the bug.
import math
from dataclasses import dataclass
dataclass
class Order:
items: tuple
total: float
def build_order(prices):
return Order(items=tuple(prices), total=sum(prices))
order = build_order([1.10, 2.20])
print("shape check len(items) == 2:", len(order.items) == 2)
print("exact equality:", order == Order(items=(1.10, 2.20), total=3.30))
print("total:", repr(order.total))
print("tolerant check:", math.isclose(order.total, 3.30, rel_tol=1e-9))Example explained
Line 1len(order.items) == 2 is true for any two-line order, so it can never catch a wrong total.
Line 2The dataclass-generated __eq__ compares every field with ==, so the float noise in total makes the whole comparison False.
Line 3repr shows the real value 3.3000000000000003, which is binary floating point addition, not a bug in build_order.
Line 4math.isclose keeps the assertion about the money while tolerating representation error; pytest.approx does the same inside pytest.
An exception assertion that passes for the wrong reason
Asserting only the exception type accepts a failure from a completely different line, while matching the message pins the code path you meant to test.
import unittest
case = unittest.TestCase()
def parse_port(raw):
port = int(raw)
if not 0 < port < 65536:
raise ValueError(f"port out of range: {port}")
return port
with case.assertRaises(ValueError):
parse_port("not-a-number")
print("broad assertion passed, but the range check never ran")
with case.assertRaisesRegex(ValueError, r"^port out of range: 70000$"):
parse_port("70000")
print("specific assertion passed")
try:
with case.assertRaisesRegex(ValueError, r"^port out of range"):
parse_port("not-a-number")
except AssertionError as exc:
print("caught:", exc)Example explained
Line 1int("not-a-number") already raises ValueError, so assertRaises(ValueError) is satisfied before the range check is reached.
Line 2assertRaisesRegex searches str(exception) against the pattern, so it only passes when the raise you wrote is the one that fired.
Line 3The third block shows the failure message you would see in a real run: the pattern is reported next to the actual exception text.
Line 4unittest.TestCase() can be instantiated on its own here purely to borrow its assertion methods outside a test runner.
Important notes
A covered line means it ran at least once with one set of values; it says nothing about other inputs, so 100% branch coverage is a floor for confidence, not proof.
Use `# pragma: no cover` only for code that genuinely cannot run in tests, and never rely on bare `assert` for production checks since `python -O` removes them.
Common mistakes
Adding tests purely to raise the coverage number, with no assertion or only `assert result is not None`; the report goes green and the wrong arithmetic ships untouched.
Computing the expected value by calling the function under test or repeating its formula in the test; the assertion is a tautology and cannot fail even when the logic is wrong.
Trusting line coverage alone, so an `if` without an `else` counts as fully covered while the fall-through path was never taken and keeps a missing default hidden.
Writing `pytest.raises(ValueError)` or assertRaises with no message check, which passes when an unrelated line raises the same exception type.
Try it yourself
Change, predict, then run
Paste the main example into an editor, fix final_price to return `round(price * (1 - rate), 2)`, and confirm strong_test passes while weak_test stays green either way. Then add the row ("HALF", 0.0) with price 0.0 and one row for a code that is None, and make the function satisfy them.
Open the Python workspaceCheck your understanding
A module reports 100% line coverage and every test passes. What does that guarantee?
- Every line in the module executed at least once while the tests ran
- Every conditional in the module was exercised in both directions
- Every function in the module returns correct results for the inputs tested
- The module contains no unreachable or dead code paths
Show answer
Line coverage only records execution, so the sole guarantee is that each line ran at some point. Option 2 is tempting but wrong: an `if` with no `else` can reach 100% line coverage with the condition only ever true, which is precisely why branch coverage exists. Option 3 depends on the assertions, which coverage never inspects.