Code coverage level with pytest-cov
You say the code coverage on your codebase is 100% - that’s awesome! But is it C0 or higher?
While most developers know about code coverage, not many developers are aware that there are different levels of code coverage: C0, C1, C2 etc. In this blog post, I will show you what they mean, and how to change the coverage level in pytest-cov.
C0 (statement) coverage
C0 coverage measures the coverage of every instruction in the codebase, which simply means every line of the source code. Here is a simple (silly) code example.
foo.py
def is_a_bigger(a, b):
if a > b:
return True
test_foo.py
from foo import is_a_bigger
def teat_is_a_bigger():
a = 2
b = 1
answer = is_a_bigger(a, b)
assert answer
To enable C0 level code coverage, all you have to do is to run pytest with the --cov flag as follows.
pytest test_foo.py --cov foo --cov-report term-missing
=============================================== test session starts ===============================================
platform darwin -- Python 3.9.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /XXX/coverage
plugins: Faker-13.3.4, cov-3.0.0
collected 1 item
test_foo.py . [100%]
---------- coverage: platform darwin, python 3.9.10-final-0 ----------
Name Stmts Miss Cover Missing
--------------------------------------
foo.py 3 0 100%
--------------------------------------
TOTAL 3 0 100%
================================================ 1 passed in 0.04s ================================================
C1 (branch) coverage
C1 coverage measures the coverage of every branch of the control structure, such as an if statement. In the example foo.py above, you notice that there’s no instruction to evaluate False (which I didn’t write on purpose to illustrate the difference between C0 and C1), but C1 will check if you are testing both if a > b and if not a > b. You can enable branch coverage with pytest by adding the --cov-branch flag.
pytest test_foo.py --cov foo --cov-report term-missing --cov-branch foo.py
==================================== test session starts =====================================
platform darwin -- Python 3.9.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /Users/kiyohitokunii/Desktop/coverage
plugins: Faker-13.3.4, cov-3.0.0
collected 1 item
test_foo.py . [100%]
---------- coverage: platform darwin, python 3.9.10-final-0 ----------
Name Stmts Miss Branch BrPart Cover Missing
----------------------------------------------------
foo.py 3 0 2 1 80% 2->exit
----------------------------------------------------
TOTAL 3 0 2 1 80%
===================================== 1 passed in 0.04s ======================================
Now the coverage is 80% because we haven’t tested the case if not a > b, so we need to update the test_foo.py to cover that case.
The updated test_foo.py
from foo import is_a_bigger
def test_is_a_bigger():
a = 2
b = 1
answer = is_a_bigger(a, b)
assert answer
def test_invalid_is_a_bigger():
a = 1
b = 2
answer = is_a_bigger(a, b)
assert not answer
We run the coverage test again with the --cov-branch flag, and now pytest is happy.
pytest test_foo.py --cov foo --cov-report term-missing --cov-branch foo.py
==================================== test session starts =====================================
platform darwin -- Python 3.9.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /Users/kiyohitokunii/Desktop/coverage
plugins: Faker-13.3.4, cov-3.0.0
collected 2 items
test_foo.py .. [100%]
---------- coverage: platform darwin, python 3.9.10-final-0 ----------
Name Stmts Miss Branch BrPart Cover Missing
----------------------------------------------------
foo.py 3 0 2 0 100%
----------------------------------------------------
TOTAL 3 0 2 0 100%
===================================== 2 passed in 0.04s ======================================
C2 (condition) coverage
C2 coverage checks in even more detail! Now I need a different code example.
foo.py
def another_func(a, b):
if a or b:
return True
return False
test_foo.py
from foo import another_func
def test_another_func_true():
a = True
b = True
answer = another_func(a, b)
assert answer
def test_another_func_false():
a = False
b = False
answer = another_func(a, b)
assert not answer
The code coverage in C1 in this case is 100%, but not in C2. To satisfy 100% code coverage in C2, you need to test every expression of True and False. In the above example, you also need to test 2 more cases.
from foo import another_func
def test_another_func_true_true():
a = True
b = True
answer = another_func(a, b)
assert answer
def test_another_func_false_false():
a = False
b = False
answer = another_func(a, b)
assert not answer
def test_another_func_true_false():
a = True
b = False
answer = another_func(a, b)
assert answer
def test_another_func_false_true():
a = False
b = True
answer = another_func(a, b)
assert answer
The tests now cover every combination of the logical condition. At this point, you may be thinking “Oh, this is getting really extreme, I don’t have time to cover this much!” The C2 coverage is not supported by coveragepy (there’s a feature request in their repo), so unfortunately you cannot easily use it in pytest.
It turns out there are a lot more variations in the code coverage, which you can learn more about in the Wikipedia if you are interested. Practically, though, knowing the difference between C0 and C1 is mostly enough, depending on what type of software you are working on.