Suppose I have a test expressed as a simple script with assert
-statements (see background for why), e.g
import foo assert foo(3) == 4
How would I include this script in my pytest test suite — in a nice way?
I have tried two working but less-than-nice approaches:
One approach is to name the script like a test, but this makes the whole pytest discovery fail when the test fails.
My current approach is to import the script from within a test function:
def test_notebooks(): notebook_folder = Path(__file__).parent / 'notebooks' for notebook in notebook_folder.glob('*.py'): import_module(f'{notebook_folder.name}.{notebook.stem}')
This works, but the scripts are not reported individually and test failures have a long and winding stack trace:
__________________________________________________ test_notebooks ___________________________________________________ def test_notebooks(): notebook_folder = Path(__file__).parent / 'notebooks' for notebook in notebook_folder.glob('*.py'): > import_module(f'{notebook_folder.name}.{notebook.stem}') test_notebooks.py:7: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ envsanacondalibimportlib__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) <frozen importlib._bootstrap>:1006: in _gcd_import ... (9 lines removed)... <frozen importlib._bootstrap>:219: in _call_with_frames_removed ??? _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > assert False E AssertionError notebooksnotebook_2.py:1: AssertionError
Background
The reason I have test in script files is that they are really Jupyter notebooks saved as .py
-files with markup by the excellent jupytext plugin.
These notebooks are converted to html for documentation, can be used interactively for learning the system, and serve as cheap functional tests.
Advertisement
Answer
Executing scripts in a test function
There’s nothing wrong with calling the scripts from a test function, so your approach is perfectly fine. However, I would use parametrization instead of running the scripts in a for loop; this way you get the test executed nicely once per script. If you don’t like the long tracebacks, you can cut them in a custom pytest_exception_interact
hookimpl. Example:
# conftest.py def pytest_exception_interact(node, call, report): excinfo = call.excinfo if 'script' in node.funcargs: excinfo.traceback = excinfo.traceback.cut(path=node.funcargs['script']) report.longrepr = node.repr_failure(excinfo)
Parametrized tests:
# test_spam.py import pathlib import runpy import pytest scripts = pathlib.Path(__file__, '..', 'scripts').resolve().glob('*.py') @pytest.mark.parametrize('script', scripts) def test_script_execution(script): runpy.run_path(script)
Test execution yields (for testing, I have created simple scripts with single lines like assert False
or 1 / 0
:
$ pytest -v ======================================= test session starts ======================================== platform linux -- Python 3.6.8, pytest-4.6.3, py-1.8.0, pluggy-0.12.0 -- /home/hoefling/projects/.venvs/stackoverflow/bin/python3.6 cachedir: .pytest_cache rootdir: /home/hoefling/projects/private/stackoverflow/so-56807698 plugins: mock-1.10.4, cov-2.7.1, forked-1.0.2, xdist-1.28.0, django-3.4.8 collected 3 items test_spam.py::test_script_execution[script0] PASSED test_spam.py::test_script_execution[script1] FAILED test_spam.py::test_script_execution[script2] FAILED ============================================= FAILURES ============================================= ____________________________________ test_script_runpy[script1] ____________________________________ > assert False E AssertionError scripts/script_3.py:1: AssertionError ____________________________________ test_script_runpy[script2] ____________________________________ > 1 / 0 E ZeroDivisionError: division by zero scripts/script_2.py:1: ZeroDivisionError ================================ 2 failed, 1 passed in 0.07 seconds ================================
Custom test protocol
If you don’t like the above solution, another thing I can think of is to implement your own test collection & execution protocol. Example:
# conftest.py import pathlib import runpy import pytest def pytest_collect_file(parent, path): p = pathlib.Path(str(path)) if p.suffix == '.py' and p.parent.name == 'scripts': return Script(path, parent) class Script(pytest.File): def collect(self): yield ScriptItem(self.name, self) class ScriptItem(pytest.Item): def runtest(self): runpy.run_path(self.fspath) def repr_failure(self, excinfo): excinfo.traceback = excinfo.traceback.cut(path=self.fspath) return super().repr_failure(excinfo)
This will collect every .py
file in scripts
directory, wrap each script in a test case and invoke runpy
on test execution. The execution log will look pretty much the same, just the tests named differently.