test_sequential/demo_package_inline seems flaky when paralelized (pytest -n auto) #2985
Description
Issue
I've observed the following test failures of tox (4.4.11 or main @ ea169d0):
[tox (main)]$ rm tests/demo_pkg_inline/.tox/ -rf && tox -e py311 -- -n=auto --run-integration tests/session/cmd/test_sequential.py
py311 inst-nodeps: .../tox/.tox/.tmp/package/1/tox-4.4.11.tar.gz
py311 installed: aiohttp==3.8.4,aiosignal==1.3.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,async-timeout==4.0.2,attrs==22.2.0,build==0.10.0,cachetools==5.3.0,certifi==2022.12.7,cffi==1.15.1,chardet==5.1.0,charset-normalizer==3.1.0,check-manifest==0.49,colorama==0.4.6,covdefaults==2.3.0,coverage==7.2.3,defusedxml==0.7.1,devpi-client==6.0.3,devpi-common==3.7.0,devpi-process==0.3.0,devpi-server==6.8.0,diff-cover==7.5.0,distlib==0.3.6,editables==0.3,execnet==1.9.0,filelock==3.11.0,flaky==3.7.0,frozenlist==1.3.3,hatch-vcs==0.3.0,hatchling==1.14.0,hupper==1.12,idna==3.4,iniconfig==2.0.0,itsdangerous==2.1.2,Jinja2==3.1.2,lazy==1.5,MarkupSafe==2.1.2,multidict==6.0.4,packaging==23.1,passlib==1.7.4,PasteDeploy==3.0.1,pathspec==0.11.1,pep517==0.13.0,pkginfo==1.9.6,plaster==1.1.2,plaster-pastedeploy==1.0.1,platformdirs==3.2.0,pluggy==1.0.0,psutil==5.9.4,py==1.11.0,pycparser==2.21,Pygments==2.15.0,pyproject_api==1.5.1,pyproject_hooks==1.0.0,pyramid==2.0.1,pytest==7.3.0,pytest-cov==4.0.0,pytest-mock==3.10.0,pytest-xdist==3.2.1,python-dateutil==2.8.2,re-assert==1.1.0,regex==2023.3.23,repoze.lru==0.7,requests==2.28.2,ruamel.yaml==0.17.21,setuptools-scm==7.1.0,six==1.16.0,strictyaml==1.7.3,time-machine==2.9.0,tox @ file://.../tox/.tox/.tmp/package/1/tox-4.4.11.tar.gz,translationstring==1.4,trove-classifiers==2023.3.9,typing_extensions==4.5.0,urllib3==1.26.15,venusian==3.0.0,virtualenv==20.21.0,waitress==2.1.2,WebOb==1.8.7,yarl==1.8.2,zope.deprecation==5.0,zope.interface==6.0
py311 run-test-pre: PYTHONHASHSEED='2384514533'
py311 run-test: commands[0] | pytest -n=auto --run-integration tests/session/cmd/test_sequential.py
============================================== test session starts ==============================================
platform linux -- Python 3.11.3, pytest-7.3.0, pluggy-1.0.0
cachedir: .tox/py311/.pytest_cache
rootdir: .../tox
configfile: pyproject.toml
plugins: time-machine-2.9.0, flaky-3.7.0, xdist-3.2.1, mock-3.10.0, cov-4.0.0, devpi-server-6.8.0
gw0 [33] / gw1 [33] / gw2 [33] / gw3 [33]
.............................FF.. [100%]
=================================================== FAILURES ====================================================
____________________________________ test_sequential_clears_pkg_at_most_once ____________________________________
[gw0] linux -- Python 3.11.3 .../tox/.tox/py311/bin/python
tox_project = <function init_fixture.<locals>._init at 0x7f02ff3439c0>
demo_pkg_inline = PosixPath('.../tox/tests/demo_pkg_inline')
def test_sequential_clears_pkg_at_most_once(tox_project: ToxProjectCreator, demo_pkg_inline: Path) -> None:
project = tox_project({"tox.ini": ""})
result = project.run("r", "--root", str(demo_pkg_inline), "-e", "a,b", "-r")
> result.assert_success()
demo_pkg_inline = PosixPath('.../tox/tests/demo_pkg_inline')
project = ToxProject(path=/tmp/pytest-of-churchyard/pytest-50/popen-gw0/test_sequential_clears_pkg_at_0/p) at 139650797147088
result = code: -1
cmd: .../tox/.tox/py311/bin/python -m tox r --root /home/churchyard/Dokumenty/R...pi/_backend.py True build
a: OK (0.72 seconds)
b: FAIL code 2 (0.02 seconds)
evaluation failed :( (0.75 seconds)
tox_project = <function init_fixture.<locals>._init at 0x7f02ff3439c0>
tests/session/cmd/test_sequential.py:423:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = code: -1
cmd: .../tox/.tox/py311/bin/python -m tox r --root /home/churchyard/Dokumenty/R...pi/_backend.py True build
a: OK (0.72 seconds)
b: FAIL code 2 (0.02 seconds)
evaluation failed :( (0.75 seconds)
def assert_success(self) -> None:
> assert self.success, repr(self)
E AssertionError: code: -1
E cmd: .../tox/.tox/py311/bin/python -m tox r --root .../tox/tests/demo_pkg_inline -e a,b -r
E cwd: /tmp/pytest-of-churchyard/pytest-50/popen-gw0/test_sequential_clears_pkg_at_0/p
E standard output
E .pkg: remove tox env folder .../tox/tests/demo_pkg_inline/.tox/.pkg
E .pkg: _optional_hooks .../tox/tests/demo_pkg_inline> python .../tox/.tox/py311/lib/python3.11/site-packages/pyproject_api/_backend.py True build
E .pkg: get_requires_for_build_sdist .../tox/tests/demo_pkg_inline> python .../tox/.tox/py311/lib/python3.11/site-packages/pyproject_api/_backend.py True build
E .pkg: build_wheel .../tox/tests/demo_pkg_inline> python .../tox/.tox/py311/lib/python3.11/site-packages/pyproject_api/_backend.py True build
E .pkg: build_sdist .../tox/tests/demo_pkg_inline> python .../tox/.tox/py311/lib/python3.11/site-packages/pyproject_api/_backend.py True build
E a: install_package .../tox/tests/demo_pkg_inline> python -I -m pip install --force-reinstall --no-deps .../tox/tests/demo_pkg_inline/.tox/.tmp/package/3/demo_pkg_inline-1.0.0.tar.gz
E a: OK ✔ in 0.72 seconds
E b: internal error
E Traceback (most recent call last):
E File ".../tox/.tox/py311/lib/python3.11/site-packages/tox/session/cmd/run/single.py", line 45, in _evaluate
E tox_env.setup()
E File ".../tox/.tox/py311/lib/python3.11/site-packages/tox/tox_env/api.py", line 250, in setup
E self._setup_with_env()
E File ".../tox/.tox/py311/lib/python3.11/site-packages/tox/tox_env/runner.py", line 143, in _setup_with_env
E self._setup_pkg()
E File ".../tox/.tox/py311/lib/python3.11/site-packages/tox/tox_env/runner.py", line 166, in _setup_pkg
E self._packages = self._build_packages()
E ^^^^^^^^^^^^^^^^^^^^^^
E File ".../tox/.tox/py311/lib/python3.11/site-packages/tox/tox_env/python/runner.py", line 118, in _build_packages
E packages = package_env.perform_packaging(self.conf)
E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E File ".../tox/.tox/py311/lib/python3.11/site-packages/tox/tox_env/package.py", line 47, in _func
E return meth(*args, **kwargs)
E ^^^^^^^^^^^^^^^^^^^^^
E File ".../tox/.tox/py311/lib/python3.11/site-packages/tox/tox_env/python/virtual_env/package/pyproject.py", line 207, in perform_packaging
E sdist = create_session_view(sdist, self._package_temp_path)
E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E File ".../tox/.tox/py311/lib/python3.11/site-packages/tox/util/file_view.py", line 32, in create_session_view
E shutil.copyfile(package, session_package)
E File "/usr/lib64/python3.11/shutil.py", line 256, in copyfile
E with open(src, 'rb') as fsrc:
E ^^^^^^^^^^^^^^^
E FileNotFoundError: [Errno 2] No such file or directory: '.../tox/tests/demo_pkg_inline/.tox/.pkg/dist/demo_pkg_inline-1.0.0.tar.gz'
E .pkg: _exit .../tox/tests/demo_pkg_inline> python .../tox/.tox/py311/lib/python3.11/site-packages/pyproject_api/_backend.py True build
E a: OK (0.72 seconds)
E b: FAIL code 2 (0.02 seconds)
E evaluation failed :( (0.75 seconds)
E
E assert False
E + where False = code: -1\ncmd: .../tox/.tox/py311/bin/python -m tox r --root /home/churchyard/Dokumenty/R...pi/_backend.py True build\n a: OK (0.72 seconds)\n b: FAIL code 2 (0.02 seconds)\n evaluation failed :( (0.75 seconds)\n.success
self = code: -1
cmd: .../tox/.tox/py311/bin/python -m tox r --root /home/churchyard/Dokumenty/R...pi/_backend.py True build
a: OK (0.72 seconds)
b: FAIL code 2 (0.02 seconds)
evaluation failed :( (0.75 seconds)
.tox/py311/lib/python3.11/site-packages/tox/pytest.py:353: AssertionError
__________________________________________ test_rerun_sequential_sdist __________________________________________
[gw0] linux -- Python 3.11.3 .../tox/.tox/py311/bin/python
tox_project = <function init_fixture.<locals>._init at 0x7f02ff343920>
demo_pkg_inline = PosixPath('.../tox/tests/demo_pkg_inline')
@pytest.mark.integration()
def test_rerun_sequential_sdist(tox_project: ToxProjectCreator, demo_pkg_inline: Path) -> None:
proj = tox_project(
{"tox.ini": "[testenv]\npackage=sdist\ncommands=python -c 'from demo_pkg_inline import do; do()'"},
)
result_first = proj.run("--root", str(demo_pkg_inline))
> result_first.assert_success()
demo_pkg_inline = PosixPath('.../tox/tests/demo_pkg_inline')
proj = ToxProject(path=/tmp/pytest-of-churchyard/pytest-50/popen-gw0/test_rerun_sequential_sdist0/p) at 139650848823696
result_first = code: 1
cmd: .../tox/.tox/py311/bin/python -m tox --root ...s/pip/_internal/build_env.py", line 21, in <module>
ModuleNotFoundError: No module named 'pip._internal.metadata'
tox_project = <function init_fixture.<locals>._init at 0x7f02ff343920>
tests/session/cmd/test_sequential.py:154:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = code: 1
cmd: .../tox/.tox/py311/bin/python -m tox --root ...s/pip/_internal/build_env.py", line 21, in <module>
ModuleNotFoundError: No module named 'pip._internal.metadata'
def assert_success(self) -> None:
> assert self.success, repr(self)
E AssertionError: code: 1
E cmd: .../tox/.tox/py311/bin/python -m tox --root .../tox/tests/demo_pkg_inline
E cwd: /tmp/pytest-of-churchyard/pytest-50/popen-gw0/test_rerun_sequential_sdist0/p
E standard output
E .pkg: _optional_hooks .../tox/tests/demo_pkg_inline> python .../tox/.tox/py311/lib/python3.11/site-packages/pyproject_api/_backend.py True build
E .pkg: get_requires_for_build_sdist .../tox/tests/demo_pkg_inline> python .../tox/.tox/py311/lib/python3.11/site-packages/pyproject_api/_backend.py True build
E .pkg: build_wheel .../tox/tests/demo_pkg_inline> python .../tox/.tox/py311/lib/python3.11/site-packages/pyproject_api/_backend.py True build
E .pkg: build_sdist .../tox/tests/demo_pkg_inline> python .../tox/.tox/py311/lib/python3.11/site-packages/pyproject_api/_backend.py True build
E py: install_package .../tox/tests/demo_pkg_inline> python -I -m pip install --force-reinstall --no-deps .../tox/tests/demo_pkg_inline/.tox/.tmp/package/6/demo_pkg_inline-1.0.0.tar.gz
E py: exit 1 (0.27 seconds) .../tox/tests/demo_pkg_inline> python -I -m pip install --force-reinstall --no-deps .../tox/tests/demo_pkg_inline/.tox/.tmp/package/6/demo_pkg_inline-1.0.0.tar.gz pid=2323852
E .pkg: _exit .../tox/tests/demo_pkg_inline> python .../tox/.tox/py311/lib/python3.11/site-packages/pyproject_api/_backend.py True build
E py: FAIL code 1 (0.39 seconds)
E evaluation failed :( (0.42 seconds)
E
E standard error
E Traceback (most recent call last):
E File "<frozen runpy>", line 198, in _run_module_as_main
E File "<frozen runpy>", line 88, in _run_code
E File ".../tox/tests/demo_pkg_inline/.tox/py/lib/python3.11/site-packages/pip/__main__.py", line 29, in <module>
E File ".../tox/tests/demo_pkg_inline/.tox/py/lib/python3.11/site-packages/pip/_internal/cli/main.py", line 9, in <module>
E File ".../tox/tests/demo_pkg_inline/.tox/py/lib/python3.11/site-packages/pip/_internal/cli/autocompletion.py", line 10, in <module>
E File ".../tox/tests/demo_pkg_inline/.tox/py/lib/python3.11/site-packages/pip/_internal/cli/main_parser.py", line 9, in <module>
E File ".../tox/tests/demo_pkg_inline/.tox/py/lib/python3.11/site-packages/pip/_internal/build_env.py", line 21, in <module>
E ModuleNotFoundError: No module named 'pip._internal.metadata'
E
E assert False
E + where False = code: 1\ncmd: .../tox/.tox/py311/bin/python -m tox --root ...s/pip/_internal/build_env.py", line 21, in <module>\nModuleNotFoundError: No module named 'pip._internal.metadata'\n.success
self = code: 1
cmd: .../tox/.tox/py311/bin/python -m tox --root ...s/pip/_internal/build_env.py", line 21, in <module>
ModuleNotFoundError: No module named 'pip._internal.metadata'
.tox/py311/lib/python3.11/site-packages/tox/pytest.py:353: AssertionError
============================================ short test summary info ============================================
FAILED tests/session/cmd/test_sequential.py::test_sequential_clears_pkg_at_most_once - AssertionError: code: -1
FAILED tests/session/cmd/test_sequential.py::test_rerun_sequential_sdist - AssertionError: code: 1
========================================= 2 failed, 31 passed in 4.84s ==========================================
ERROR: InvocationError for command .../tox/.tox/py311/bin/pytest -n=auto --run-integration tests/session/cmd/test_sequential.py (exited with code 1)
____________________________________________________ summary ____________________________________________________
ERROR: py311: commands failed
This only happens sometimes and depends on how xdist schedules the tests. Sometimes 2 tests fails, sometimes 1, sometimes they pass.
$ while true; do rm tests/demo_pkg_inline/.tox/ -rf && tox -e py311 -- -n=auto --run-integration tests/session/cmd/test_sequential.py | grep -F 'FAILED tests/session/cmd/test_sequential.py' || echo OK; echo; done
FAILED tests/session/cmd/test_sequential.py::test_sequential_clears_pkg_at_most_once
FAILED tests/session/cmd/test_sequential.py::test_rerun_sequential_sdist - As...
FAILED tests/session/cmd/test_sequential.py::test_sequential_clears_pkg_at_most_once
FAILED tests/session/cmd/test_sequential.py::test_rerun_sequential_sdist - As...
FAILED tests/session/cmd/test_sequential.py::test_sequential_clears_pkg_at_most_once
FAILED tests/session/cmd/test_sequential.py::test_rerun_sequential_sdist - As...
FAILED tests/session/cmd/test_sequential.py::test_sequential_clears_pkg_at_most_once
FAILED tests/session/cmd/test_sequential.py::test_rerun_sequential_sdist - As...
FAILED tests/session/cmd/test_sequential.py::test_rerun_sequential_sdist - As...
FAILED tests/session/cmd/test_sequential.py::test_sequential_clears_pkg_at_most_once
FAILED tests/session/cmd/test_sequential.py::test_sequential_clears_pkg_at_most_once
FAILED tests/session/cmd/test_sequential.py::test_rerun_sequential_sdist - As...
FAILED tests/session/cmd/test_sequential.py::test_sequential_clears_pkg_at_most_once
FAILED tests/session/cmd/test_sequential.py::test_rerun_sequential_sdist - As...
FAILED tests/session/cmd/test_sequential.py::test_sequential_clears_pkg_at_most_once
FAILED tests/session/cmd/test_sequential.py::test_rerun_sequential_sdist - As...
FAILED tests/session/cmd/test_sequential.py::test_sequential_clears_pkg_at_most_once
FAILED tests/session/cmd/test_sequential.py::test_sequential_clears_pkg_at_most_once
FAILED tests/session/cmd/test_sequential.py::test_rerun_sequential_sdist - As...
FAILED tests/session/cmd/test_sequential.py::test_sequential_clears_pkg_at_most_once
FAILED tests/session/cmd/test_sequential.py::test_rerun_sequential_sdist - As...
FAILED tests/session/cmd/test_sequential.py::test_rerun_sequential_sdist - Ke...
OK
FAILED tests/session/cmd/test_sequential.py::test_sequential_clears_pkg_at_most_once
FAILED tests/session/cmd/test_sequential.py::test_rerun_sequential_sdist - As...
...
When the full test suite runs, other tests using the same demo project, such as test_pyproject_deps_from_static[deps_with_circular_recursive_extra] from tests/tox_env/python/virtual_env/package/test_package_pyproject.py, sometimes failed for me as well, with similar errors.
Depending on the -n value, this happens more or less often. Auto means 4 when I run this. It does not (seem to) happen when the tests run without -n=auto or when I set -n=1.
My assumption is that when multiple tox instances run in the demo_package_inline directory in parallel, they create race conditions. I wonder if the demo_package_inline fixture should prevent parallel usage or if it should copy itself into tmp_path each time.
Environment
Provide at least:
- OS: Fedora Linux 37 or 39
pip listof the host Python wheretoxis installed:
Package Version
-------------------- ---------
aiohttp 3.8.4
aiosignal 1.3.1
argon2-cffi 21.3.0
argon2-cffi-bindings 21.2.0
async-timeout 4.0.2
attrs 22.2.0
build 0.10.0
cachetools 5.3.0
certifi 2022.12.7
cffi 1.15.1
chardet 5.1.0
charset-normalizer 3.1.0
check-manifest 0.49
colorama 0.4.6
covdefaults 2.3.0
coverage 7.2.3
defusedxml 0.7.1
devpi-client 6.0.3
devpi-common 3.7.0
devpi-process 0.3.0
devpi-server 6.8.0
diff-cover 7.5.0
distlib 0.3.6
editables 0.3
execnet 1.9.0
filelock 3.11.0
flaky 3.7.0
frozenlist 1.3.3
hatch-vcs 0.3.0
hatchling 1.14.0
hupper 1.12
idna 3.4
iniconfig 2.0.0
itsdangerous 2.1.2
Jinja2 3.1.2
lazy 1.5
MarkupSafe 2.1.2
multidict 6.0.4
packaging 23.1
passlib 1.7.4
PasteDeploy 3.0.1
pathspec 0.11.1
pep517 0.13.0
pip 23.0.1
pkginfo 1.9.6
plaster 1.1.2
plaster-pastedeploy 1.0.1
platformdirs 3.2.0
pluggy 1.0.0
psutil 5.9.4
py 1.11.0
pycparser 2.21
Pygments 2.15.0
pyproject_api 1.5.1
pyproject_hooks 1.0.0
pyramid 2.0.1
pytest 7.3.0
pytest-cov 4.0.0
pytest-mock 3.10.0
pytest-xdist 3.2.1
python-dateutil 2.8.2
re-assert 1.1.0
regex 2023.3.23
repoze.lru 0.7
requests 2.28.2
ruamel.yaml 0.17.21
setuptools 65.6.3
setuptools-scm 7.1.0
six 1.16.0
strictyaml 1.7.3
time-machine 2.9.0
tox 4.4.11
translationstring 1.4
trove-classifiers 2023.3.9
typing_extensions 4.5.0
urllib3 1.26.15
venusian 3.0.0
virtualenv 20.21.0
waitress 2.1.2
WebOb 1.8.7
wheel 0.40.0
yarl 1.8.2
zope.deprecation 5.0
zope.interface 6.0