Repository navigation
Revisit xpassed tests and remove xfail from them #6665
Description
For example, there are 3 xpassed tests in https://travis-ci.org/astropy/astropy/jobs/283303458 and https://ci.appveyor.com/project/Astropy/astropy/build/1.0.8822 , respectively (probably the same ones). We should remove xfail from those because they are no longer failing.
astropy\table\tests\test_mixin.py .....................................sss.s.s...................................................X.............
astropy\table\tests\test_operations.py .......x...x.x.x...........x.....x.......................x...xXX.x.........................x.............
Hint: Using this setting turns any xpass into a failure (ref: https://docs.pytest.org/en/latest/skipping.html).
[tool:pytest]
xfail_strict=true
Activity
bsipocz commentedon Oct 5, 2017
and are these xpasses on all platforms?
pllim commentedon Oct 6, 2017
I see them on Travis and Appveyor, so theoretically yes.
MSeifert04 commentedon Oct 6, 2017
Is it possible to enable that setting and let it run through CI without these 3 xfails? That would show if there are more tests that xpass accidentally
bsipocz commentedon Oct 12, 2017
So the 2 xpassing one is due to the heisenbug in operations: #6291
It's great that it's passing, but we need to make sure it's indeed gone for good.
In there, it's suggested (#6291 (comment)) that running the test as
pytest astropy/table/tests/test_operations.py --pdb -sv -k test_col_meta_merge_inneralways makes it fail, but it's still xpasses for me now on master, but also on 2.0...pllim commentedon Oct 12, 2017
Heisenbug... Maybe they only fail when you run them on your second cellphone, I mean, laptop.
bsipocz commentedon Oct 12, 2017
They even pass not using my own laptop now (though for some weird reason I cannot run the tests with
-t, nor the command above, but both run perfectly on my own laptop).Anyway, we can try to remove that xfail and monitor the situation, maybe the fail is indeed gone.
mhvk commentedon Oct 13, 2017
Closing, since the passing tests are now all enabled! (And crossing fingers that they continue to pass...)