Repository navigation
TST: testing fails with minimal test requirements #17681
Description
For some reason the bump to pytest==8.0.0 (#17675) appears to impact test discovery.
This is visible when running with truly pinned oldest test dependencies (#16963), and a periodic reminder that this PR serves a purpose and should be merged sooner rather than later to avoid discovering these issues after the fact (this should have been caught in #17655
example logs
What seems to be happening is that doctests from astropy/extern/configobj/validate.py are not compatible with pytest (they seem to be written with only python -m doctest in mind), which is normally not an issue since this module is somehow excluded from test discovery, but in that specific env (exactly our minimal dependencies), this isn't true, which leads to failing the run.
Further bumping pytest to a bugfix release might remedy this, but it's also possible that something else in our minimal set of test dependencies affects discovery (tox, coverage ?) is just not compatible with pytest 8.0 and might need a bump too. I'll run some experiments on my fork to see how best to address this.
Activity
[-]TST: testing fail with minimal test requirements[/-][+]TST: testing fails with minimal test requirements[/+]neutrinoceros commentedon Jan 28, 2025
Preliminary results seem to indicate that it is sufficient to bump
pytest-doctestplustoo. Version 1.2.1 appears to be sufficient, but I'll try to restrict it as far back as possible.pllim commentedon Jan 28, 2025
Why is pytest not respecting this setting?
astropy/pyproject.toml
Line 207 in 85c72cc
neutrinoceros commentedon Jan 28, 2025
best guess so far: because pytest-doctestplus takes part in test collection and used to be incompatible with this setting
xref scientific-python/pytest-doctestplus#201
Your PR was first ship with pytest-doctestplus 0.13, but unfortunately this doesn't seem to play nice with pytest 8.0.0 (or to put it differently, it is broken by pytest 8.0.0), hence my conclusion that we need version 1.0.0 or newer.
pllim commentedon Jan 28, 2025
So apparently Past-Self fixed it, then it came back, and then fixed again in 1.0? 🧐
That is weird because doctestplus tests against a whole slew of sphinx versions. I added a test in the PR you linked. And I don't ever remember that test failing lately.
neutrinoceros commentedon Jan 28, 2025
I don't know what other pytest plugin might interfere with test collection, but it's seems plausible that our current set of minimal requirements doesn't correspond to anything exact that got tested in the past (as some packages are pinned to older versions than others), so maybe something managed to slip through the net despite everyone's best efforts.