You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
astropy.test() should be able to test docstrings #8153
My conclusion from #8146 is that astropy.test() (testing installed code) is not running the exact same set of tests under the same settings as python setup.py test (testing full source code). Do we care? If so, what can we do about it (probably would involve refactoring the test runner)?
@bsipocz that's almost certainly true, but I wouldn't be entirely surprised to see other differences as well.
@pllim it's fine to keep astropy.test() (although if we were designing things from the ground up I might reconsider), but it could definitely call either pytest.main or tox.main directly.
I'm against in any way of removing astropy.test(). That is a super useful and easy way to test user installations, no tox or else can make it cleaner and easier.
I'm with @bsipocz that astropy.test() has been really useful in the past, otherwise there is no way to test user installations (for example from conda or other package managers)
There's also no way for astropy.test to run the rst docs tests, and I think that's fine - astropy.test() should just be assumed to run a subset of the tests.
In addition of not running doctest, it also is not respecting settings in setup.cfg. Not sure if it is a feature or a bug. (I was going to give examples but I have to get around #8148 first. 🌀
@pllim - No wonder, setup.cfg is not available in the installed version. If we need the settings from it, it has to be a hack that propagates them to somewhere else, e.g. astropy.cfg etc.
I have much bigger differences locally, than your numbers (~1000). Of those are ~390 are doctests.
Yet a fraction of those doctests should be tested as they are docstrings in the source code that is also available in an install.
I'm not yet sure what the other ~600 tests are.
Todo item here: make it possible for astropy.test() to pick up docstrings for testing.
changed the title [-]TST: astropy.test() not testing the same stuff as setup.py test[/-][+]astropy.test() should be able to test docstrings[/+]on Mar 19, 2019
There was a separate conversation on Slack (@Cadair, @astrofrog, and @drdavella) saying something about moving tests outside of distributed package. If that is going to happen, then astropy.test() would disappear and this would become a non-issue...
yes, I know @drdavella and @Cadair would like that happen, but the rest of us keep saying that astropy.test() is way to valuable to see user issues that it would be a waste to get rid of it.
Activity
drdavella commentedon Nov 18, 2018
This is not at all surprising. I'm not sure if we care or not. If possible can you put a diff here?
Friends, I beg of you, let's please consider axing the test runner entirely.
pllim commentedon Nov 18, 2018
@drdavella , are you saying getting rid of
astropy.test()entirely? Or canastropy.test()machinery be done in a different way (tox?)?bsipocz commentedon Nov 18, 2018
I suppose the difference is coming mainly from the fact that the narrative docs is not available in the installed version?
drdavella commentedon Nov 18, 2018
@bsipocz that's almost certainly true, but I wouldn't be entirely surprised to see other differences as well.
@pllim it's fine to keep
astropy.test()(although if we were designing things from the ground up I might reconsider), but it could definitely call eitherpytest.mainortox.maindirectly.bsipocz commentedon Nov 18, 2018
I'm against in any way of removing
astropy.test(). That is a super useful and easy way to test user installations, notoxor else can make it cleaner and easier.astrofrog commentedon Nov 19, 2018
I'm with @bsipocz that
astropy.test()has been really useful in the past, otherwise there is no way to test user installations (for example from conda or other package managers)There's also no way for
astropy.testto run the rst docs tests, and I think that's fine - astropy.test() should just be assumed to run a subset of the tests.pllim commentedon Nov 19, 2018
In addition of not running doctest, it also is not respecting settings in
setup.cfg. Not sure if it is a feature or a bug. (I was going to give examples but I have to get around #8148 first. 🌀bsipocz commentedon Nov 19, 2018
@pllim - No wonder,
setup.cfgis not available in the installed version. If we need the settings from it, it has to be a hack that propagates them to somewhere else, e.g.astropy.cfgetc.pllim commentedon Nov 19, 2018
@drdavella , as requested...
Log from
python setup.py test --remote-data:zlog_source.txt
11889 passed, 107 skipped, 111 xfailed in 337.78 secondsLog from
astropy.test(remote_data=True):zlog_install.txt
11548 passed, 103 skipped, 109 xfailed, 2 xpassed, 3136 warnings in 285.30 secondsIMHO the most perplexing is the 2
xpassedtests; i.e, they failed when tested from source but passed when tested from installation.bsipocz commentedon Mar 19, 2019
I have much bigger differences locally, than your numbers (~1000). Of those are ~390 are doctests.
Yet a fraction of those doctests should be tested as they are docstrings in the source code that is also available in an install.
I'm not yet sure what the other ~600 tests are.
Todo item here: make it possible for astropy.test() to pick up docstrings for testing.
[-]TST: astropy.test() not testing the same stuff as setup.py test[/-][+]astropy.test() should be able to test docstrings[/+]bsipocz commentedon Mar 19, 2019
I suspect some of the pytest parametrization messes up
astropy.test(). E.g. running all tests but then focusing only on the constants one:python setup.py test -V --remote-datain ipython with
astropy.test(remote_data='all', verbose=True)bsipocz commentedon Mar 19, 2019
However, starting a new ipython, and running just the constants tests, twice, I don't get the same number. And this is the point where I give up...:
pllim commentedon Mar 19, 2019
There was a separate conversation on Slack (@Cadair, @astrofrog, and @drdavella) saying something about moving tests outside of distributed package. If that is going to happen, then
astropy.test()would disappear and this would become a non-issue...bsipocz commentedon Mar 19, 2019
yes, I know @drdavella and @Cadair would like that happen, but the rest of us keep saying that
astropy.test()is way to valuable to see user issues that it would be a waste to get rid of it.