Repository navigation
Closed
Labels
plugin: xdistrelated to the xdist external pluginrelated to the xdist external plugintype: bugproblem that needs to be addressedproblem that needs to be addressed
Description
Issues that may be related: #437, #594, #596, #669.
pytest-xdist seems to fail sometimes because it orders parametrized fixtures differently -- e.g., when a fixture depends on two or more parametrized fixtures. Using the latest pytest and pytest-xdist on Python 3.4 with this test script:
import pytest
@pytest.fixture(params=['foo', 'bar'])
def fixture1(request):
return request.param
@pytest.fixture(params=['baz', 'foo'])
def fixture2(request):
return request.param
def test_1(fixture1, fixture2):
passSometimes it works:
$ py.test -v -n 2
============================ test session starts ============================
platform linux -- Python 3.4.3 -- py-1.4.30 -- pytest-2.7.2 -- /home/test/env/bin/python3
plugins: xdist
[gw0] linux Python 3.4.3 cwd: /home/test/foo
[gw1] linux Python 3.4.3 cwd: /home/test/foo
[gw1] Python 3.4.3 [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
[gw0] Python 3.4.3 [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
gw0 [4] / gw1 [4]
scheduling tests via LoadScheduling
foo/test_1.py::test_1[2-3]
foo/test_1.py::test_1[1-3]
[gw0] PASSED foo/test_1.py::test_1[2-3]
[gw1] PASSED foo/test_1.py::test_1[1-3]
foo/test_1.py::test_1[1-4]
foo/test_1.py::test_1[2-4]
[gw0] PASSED foo/test_1.py::test_1[2-4]
[gw1] PASSED foo/test_1.py::test_1[1-4]
========================= 4 passed in 0.60 seconds ==========================And sometimes it doesn't (because it chose to sort the fixtures by the second parameter first for some reason):
$ py.test -v -n 2
============================ test session starts ============================
platform linux -- Python 3.4.3 -- py-1.4.30 -- pytest-2.7.2 -- /home/test/env/bin/python3
plugins: xdist
[gw0] linux Python 3.4.3 cwd: /home/test/foo
[gw1] linux Python 3.4.3 cwd: /home/test/foo
[gw1] Python 3.4.3 [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
[gw0] Python 3.4.3 [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
gw0 [4] / gw1 [4]
scheduling tests via LoadScheduling
collecting 0 items / 1 errors
================================== ERRORS ===================================
___________________________ ERROR collecting gw0 ____________________________
Different tests were collected between gw1 and gw0. The difference is:
--- gw1
+++ gw0
@@ -1,4 +1,4 @@
foo/test_1.py::test_1[foo-foo]
+foo/test_1.py::test_1[foo-bar]
+foo/test_1.py::test_1[bar-bar]
foo/test_1.py::test_1[bar-foo]
-foo/test_1.py::test_1[bar-bar]
-foo/test_1.py::test_1[foo-bar]
========================== 1 error in 0.65 seconds ==========================Metadata
Metadata
Assignees
Labels
plugin: xdistrelated to the xdist external pluginrelated to the xdist external plugintype: bugproblem that needs to be addressedproblem that needs to be addressed
Activity
[-]Non-deterministic test collection order for determinstic fixtures makes xdist fail[/-][+]Non-deterministic test collection order for deterministic fixtures makes xdist fail[/+]RonnyPfannschmidt commentedon Aug 6, 2015
interesting find, my initial guess is, this is a artifact of the python hash-seed and the fixture management is just sorted different in some mappings
it'll take a while to figure the details, and it'll happen after the next release of xdist (which i am working on)
aldanor commentedon Aug 6, 2015
Yea, the test collection mechanism is a bit flaky – in a much larger example which I won't be able to reproduce here I was able to work around it by changing some fixtures from session-scoped to function-scoped, I have no clue why that helped. Also, this doesn't seem to happen on Python 2 (not that I've seen it fail, at least).
By the way, I wonder why should it care about the order at all if the sets are the same?
RonnyPfannschmidt commentedon Aug 7, 2015
That goes a few years back, I recall its a workaround for duplicate test ids or something like that
jstasiak commentedon May 5, 2016
@aldanor FYI I fail to be able to reproduce the failure using the example code you provided. My software: Python 2.7.9 and 3.5.1, pytest 2.9.1, xdist-1.14.
nicoddemus commentedon Jun 18, 2016
Hmm perhaps we can pass
master'sPYTHON_SEEDvalue to workers? This way dict hashing should be the same across master and workers. This does not actually fix the fact that xdist uses indices to distribute tests, but at least seems sensible and would solve this issue (and others related).hectorv commentedon Jul 1, 2016
I'm seeing this while trying to move away from
nosetestson a legacy test suite:this is my environment:
Edit: to be clear, this works fine:
jenkins: Temporary workaround for pytest-dev/pytest#920
jenkins: Temporary workaround for pytest-dev/pytest#920
jenkins: Temporary workaround for pytest-dev/pytest#920
jenkins: Temporary workaround for pytest-dev/pytest#920
renefritze commentedon Aug 2, 2017
I think we still see this in our project with pytest 3.2 and xdist 1.18.2:
https://travis-ci.org/pymor/pymor/jobs/260181761#L744
RonnyPfannschmidt commentedon Aug 2, 2017
@renemilk the issue you show is a different problem that comes from your own self-made parametrization system
all your tests that differ in order may be in a different place based on pythonhashseed
as such pytest cant know the order itself since the methods share the order
renefritze commentedon Aug 2, 2017
Ah, ok. I misunderstood then. Setting
PYTHONHASHSEED=0in env indeed fixes the ordering issue for me locally. Thanks.Unparallelize pytest for Python 3.5 everywhere
liiight commentedon Jan 21, 2019
I think I may have the same issue. I dynamically generate parameterization based on a txt file from S3:
Running this without
-nworks:With
-nit does not:I tried setting PYTHONSEEDHASH to no avail.
Please advise, thanks in advance
EDIT: Realised I wasn't on latest xdist and pytest, upgrade both to latest, still same issue.
RonnyPfannschmidt commentedon Jan 21, 2019
@liiight based on what you show its a completely different issue please open new issues instead of recycling related looking but actually fixed issues
liiight commentedon Jan 22, 2019
maratumba commentedon Dec 1, 2021
I just encountered this issue while collecting tests with parametrized fixtures (
pytest==6.2.5) in Django. The order of tests changes in each collection.pytest.ini