Subtle mistakes in writing unit tests
While most developers know what a unit test is, I have seen quite a few cases where people claim non-unit tests to be “unit tests”, even among experienced developers. I will highlight two such cases: writing an integration test instead of a unit test, and writing a unit test that is not a “unit”.
1. Writing an integration test instead of a unit test
The most common mistake is mixing up integration tests with unit tests. I have seen “unit tests” that are actually integration tests. These tests often call external requests, such as API requests or database access.
There’s an easy way to detect this. When I see a “unit test” that looks suspicious, I simply turn off the WiFi connection temporarily, and run the test. If the test is properly implemented as a unit test, it should succeed without an internet connection, so you know it is not calling any external requests. If the test fails, it is probably an integration test.
There are two approaches here: (1) realise that what you want to run is actually an integration test rather than a unit test, or (2) fix the test using a mock object library.
For (1), you’ve been wrongly calling it a unit test, but you can simply treat it as an integration test. You should move the test to an integration test directory to keep it separate from your unit tests, so that a CI will run them as separate jobs and you can debug more easily when something goes wrong.
For (2), you need to stop calling those external requests, and replace them with mock tests. In Python, for example, there is the mock library, and it replaces those external requests with something that imitates the request behaviour. You should use the mock library if you just want to test your own code, not the code of external requests libraries or the integration with those external libraries.
2. Writing a unit test that is not a unit
Another mistake I see is unit tests that depend on each other and are not really “unit” tests. Here is a simple example.
from pathlib import Path
import pytest
@pytest.fixture()
def filepath():
return Path("foo.txt")
def save_file(input_path: Path):
with input_path.open("w") as f:
f.write("some text")
def test_save_file(filepath):
assert not filepath.is_file()
save_file(filepath)
assert filepath.is_file()
def test_file_exist(filepath):
assert filepath.is_file()
test_file_exist clearly depends on the execution of test_save_file. If you comment out test_save_file (or change the execution order of these two tests), test_file_exist will fail, which means this test is not an independent unit. While this is an overly simple example, I have seen tests like this where commenting out one unit test will break seemingly unrelated tests elsewhere because of some hidden dependency.
To fix the dependency in the above example, you can add pytest’s temporary directory fixture as follows.
@pytest.fixture()
def filepath(tmp_path):
return Path(tmp_path / "foo.txt")
Now test_file_exist assertion will fail (as it should) because this test is isolated from test_save_file and the file foo.txt doesn’t exist within this test_file_exist.
You could make them dependent on purpose by changing the scope of the fixture. By default, the scope of the pytest fixture is at the function level. You could change the scope to module, class or session by changing the scope argument in the fixture.
@pytest.fixture(scope="module") # or `class` or `session`.
def my_fixture():
xxx
If unit tests are properly isolated, you can even speed them up by running them in parallel using pytest-xdist. pytest-xdist might fail unexpectedly, however, if the tests depend on the execution of other tests.