pytest breaks with pip install

I’m working in a repo that has tests and some test helper libraries in a directory named test (https://github.com/covid-projections/covid-data-model/tree/main/test). Mysteriously pip install dash seems to break our tests. I’m using python 3.7.9 in pyenv on a debian laptop. I reproduced the problem with a pretty simple example:

Activate a fresh pyenv virtualenv and install the latest pytest with pip install git+https://github.com/pytest-dev/pytest. Create an empty ./test/__init__.py and the following two files:

$ cat ./test/real_test.py
from test import test_helpers

def test_one():
  assert "foobar" == test_helpers.SOME_CONSTANT
$ cat ./test/test_helpers.py
SOME_CONSTANT = "foo"

Running test/real_test.py does the expected, failing a test. Now pip install dash==1.19.0 and the test fails to run with

_____________________________________________________________________________ ERROR collecting test/real_test.py _____________________________________________________________________________
ImportError while importing test module '/home/thecap/tmp/nomodule/test/real_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../.pyenv/versions/3.7.9/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
E   ModuleNotFoundError: No module named 'test.real_test'

Running pytest --import-mode=importlib test/real_test.py (found in the comments of https://github.com/pytest-dev/pytest/issues/7408) gives me a little hint that it isn’t including the cwd in the path used to search for libraries because it is looking at a test in the pyenv, not in the cwd.

ImportError while importing test module '/home/thecap/tmp/nomodule/test/real_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
test/real_test.py:1: in <module>
    from test import test_helpers
E   ImportError: cannot import name 'test_helpers' from 'test' (/home/thecap/.pyenv/versions/3.7.9/lib/python3.7/test/__init__.py)

Mysteriously to me after running pip uninstall dash the test runs as expected with pytest test/real_test.py but is still broken with pytest --import-mode=importlib test/real_test.py.

I suspect moving our tests from test/ to tests/ will fix this but I’d like to understand why installing dash breaks things and if our current setup is a problem, help other people avoid it.

(I posted this at https://github.com/pytest-dev/pytest/discussions/8270 too and hoping for more response on stack overflow.)

Answer

The issue here is that your local test package shadows the test module from the standard library. In general, name shadowing any stdlib modules is a bad practice and should be avoided; however, since

The test package is meant for internal use by Python only. It is documented for the benefit of the core developers of Python. Any use of this package outside of Python’s standard library is discouraged as code mentioned here can change or be removed without notice between releases of Python.

and

The test package contains all regression tests for Python as well as the modules test.support and test.regrtest. test.support is used to enhance your tests while test.regrtest drives the testing suite.

one would imply that test isn’t used outside of the regression test run. This isn’t true for future, though – it’s importing test.support when installing aliases, causing stdlib’s test to be present in sys.modules before pytest starts test collection. You installed futures along with dash, since it’s a direct dependency. Now when you start a pytest run, it loads the dash plugin which imports future which imports test and afterwards, when trying to collect your local test package, pytest fails on import since another test is already imported!

Solution

I would suggest to rename the local test package into e.g. tests. This will stop the name shadowing of the stdlib module.

If you can’t rename the test package, one can imagine several workarounds to circumvent the import of stdlib’s test:

Clean up the import

Create a file conftest.py in your project root dir (not in the test dir, or it won’t work) with the following contents:

import sys

sys.modules.pop("test", None)

This will “unimport” stdlib’s test, making your local test package importable on test collection.

Alter sys.path to make local test package selectable before stdlib ones

Just run python -m pytest ... or PYTHONPATH=. pytest ... to ensure the root dir of your project is inserted before the site in sys.path, so import test will always end in importing your local package.