Repository navigation
TST: xfail_strict setting not always respected? #8146
Description
We have set xfail_strict = true in setup.cfg, which in theory, would turn xpass into failures. However, in testing for 3.1RC, my Windows 10 reported 2 xpassed but did not change them into failures (as shown in pytest-dev/pytest#1386 ).
Maybe this would naturally be solved if we move away from using setup.py test anyway? 🤷♀️
Activity
pllim commentedon Nov 17, 2018
So, I suspected maybe because I was lazy and used
pip install astropy==3.1rc1(the first thing I saw on the wiki) for testing instead of checking out the tag and testing on source dir withsetup.py. So I did the latter and thexpasseddisappeared. For comparison...Testing from
pipinstall from non-source dir (11762 total):Testing from
setup.pyfrom source dir (12107 total; 345 extra from above):I think I'll bury this as mysterious thingies that happen when you don't test from source dir (with access to
setup.cfg). One more reason to usetox(again), @drdavella ? 😅bsipocz commentedon Nov 17, 2018
Actually testing an installed version from pip in preferred than testing the tag as the former is what users will see. But, well, I agree this is those mysterious ones that doesn't deserve too much time to spend on. ¯_(ツ)_/¯
pllim commentedon Nov 17, 2018
@bsipocz , indeed... though it is obvious that
astropy.test()does not read in preferences set insetup.cfg(e.g., showing tests have 3k warnings instead of ignoring them). Is this somethingtoxwill handle too or do we no longer needastropy.test()when we usetox? 🤔bsipocz commentedon Nov 17, 2018
I don't think
toxis being the answer for everything...