Deprecate astropy test runner in core and downstream

This is what some people call "an epic" in JIRA-land.

A long, long time ago, in the same galaxy, astropy existed when Python was youngish, pytest was younger (remember when we bundled pytest as a single file?) , CI on GitHub was almost non-existent (ask Tom R about Jenkins). There was a need to be able to test installed versions of astropy. And thus, the test runner was created. For many years, it served us well.

astropy/astropy/tests/runner.py

Lines 290 to 293 in 52d2b36

    
           class TestRunner(TestRunnerBase): 
        
               """ 
        
               A test runner for astropy tests. 
        
               """

astropy/astropy/__init__.py

Line 137 in 52d2b36

test = TestRunner.make_test_runner_in(__path__[0])

We even advertised it in package-template (https://github.com/astropy/package-template/blob/36ed48db74c9a0bfcf214e14a1a3fb1fdaae0c6c/%7B%7B%20cookiecutter.package_name%20%7D%7D/%7B%7B%20cookiecutter.module_name%20%7D%7D/_%7B%7B%20cookiecutter._parent_project%20%7D%7D_init.py#L13), so downstream packages may have similar runners too (whether they realized it is there or not is another issue).

Now in 2024, we have multi-architecture CI and wheels. The need for such a test runner has diminished greatly. In fact, having it might even hinder progress like these:

Additionally, it is not aging well now that pytest is very complicated and there are plenty of pytest plugins available:

Running tests from astropy.test() fails when asdf-astropy is installed #16165

All the while, it was never feature complete:

The plea to get rid of the test runner went as far back as 2018 (#8153 (comment)), "Friends, I beg of you, let's please consider axing the test runner entirely."

Proposal

✔ Open issues in affected downstream packages to warn them to check whether they inherited this test runner from package template (or added it intentionally) and tell them to move away from using it. Alternative is to invoke pytest directly (or tox, whatever).
Stop using test runner internally within core lib and docs (including RC testing instructions). -- RFC: Introduce pending deprecation warning of test runner #17874
Step 4 but pending=True -- RFC: Introduce pending deprecation warning of test runner #17874
Deprecate astropy.test() function and astropy.tests.runner module. Look at the diff from MAINT: Remove astropy.test(), should use pytest directly #16208 for an idea on what files to touch. -- Remove deprecated tests/command.py module and really deprecate test runner for v8.0 #17883
After a deprecation period, remove the deprecated stuff from astropy. If downstream packages have not caught up, they need to pin to an older release of astropy or have a broken test runner.

Impact: High!
Benefit: Less technical debt to carry forward.
Cost: Some crying maintainers.

Counter proposal

We keep everything as-is, but we have to start testing astropy.test() in our CI.

I did a search for "astropy.test()" in Python files across GitHub and got 76 results. A similar search for "astropy.tests.runner" in Python files across GitHub got 132 results. (Checking on this was waaaay more fun than timing Sphinx builds.)

@pllim

Thanks for the write up @pllim! I wonder who is the audience for the deprecation? Can we remove astropy.test() directly and only deprecate TestRunner? Assuming the hypothesis that an end user doesn't really need to run astropy.test() today.

There are a bunch of packages which use the TestRunner code downstream but from a quick glance it seems like there isn't much magic happening in the downstream packages, it just directly imports TestRunner and then exposes a test function in the package namespace, so the change hopefully wouldn't be that painful!

I did a search for "astropy.test()" in Python files across GitHub and got 76 results.

If we remove all the forks on github, the only active place astropy.test() seemed to be used is in https://github.com/spack/spack/blob/31de670bd26beca979ebd75dcb0ce90c535a78c4/var/spack/repos/builtin/packages/py-astropy/package.py#L126 and they do support running tests directly with pytest.

Thanks for the searches, y'all!

We advertise astropy.test() in every RC testing (e.g., https://github.com/astropy/astropy/wiki/v6.0-RC-Testing), so I think deprecation is safer bet than straight removal. But whether astropy.test() is considered "public API" or not perhaps could be debated, especially since the public API APE is still WIP:

Public API astropy-APEs#85

Here is an example of active downstream usage of the test runner:

https://github.com/spacetelescope/synphot_refactor/blob/c60d653f311c552b6256099dfc057d3f0a44d176/.github/workflows/publish-to-pypi.yml#L46

I agree we can stop using the test runner (and deprecate the TestRunner class). The main thing really would seem to have good instructions for how people can test astropy without it. The nice thing about our current test instructions at https://github.com/astropy/astropy/wiki/v6.0-RC-Testing is that we do not have to worry at all about what virtual environment people have, etc. But I think one can replace the second line with just a tox incantation, no?

The nice thing about our current test instructions at https://github.com/astropy/astropy/wiki/v6.0-RC-Testing is that we do not have to worry at all about what virtual environment people have, etc. But I think one can replace the second line with just a tox incantation, no?

As long as tests are installed, one can use pytest --pyargs astropy.
Should be quite similar to astropy.test() without a local clone.

A search for make_test_runner_in without forks gives 84 results, seems to give a good list of packages importing the TestRunner (i.e. using the template).

As long as tests are installed

But how, if the wheels no longer come with tests?

Presumably pip install astropy[test] would pull in astropy-tests too!

Oh, we talking about moving the tests out of this repo too? 😱

SunPy 🔥'ed the test runner a long time ago, it's very easy to just call pytest.main and get the same effect: https://github.com/sunpy/sunpy/blob/main/sunpy/tests/self_test.py#L73

We do actually use astropy.test() in astropy's conda recipe 🤷‍♀️

Update: astropy conda recipe no longer uses astropy.test() and I have #17732 to stop using it in our "double run" CI.

I also updated my proposal section above so it makes more sense (at least to me).

As far as my search-fu allows, 194 repos at least import TestRunner. Whether it is actually used or not is another story; but the impact is definitely non-zero. I can warn them with batch issues.

(If I missed any, I apologize in advance. The REST API becomes unreliable when there are too many matches.)

@pllim

What percent of the crying maintainers are me? 😢 Looks like about 5%.

Anyway, thanks for this @pllim. Can you remind me where to find instructions on how best to move away from using astropy for testing? I'm pretty sure this all comes from inheriting from the pre-cookiecutter package template.

Only 1.5% for me... But, yes, from the same template and will now have to update my user-facing instructions about how to test.... But it is good to get rid of old cruft...

I don't have much to do in my package fortunately. I already use pytest/tox; the test runner is just leftover inheritance from package-template, but a pointer to recent documentation of replacements would be appreciated, specifically running tests on a package that is already installed.

😅 Not sure how much this helps but this kinda shows a little example of how to replace specific test() code with pytest directly:

My own packages are also affected. When I get around to them, I can link my PRs here as well.

Examples so far... they vary in amount of diff for various reasons.

As time goes on, there will be more stuff linked to this issue. Or existing linked issues will be marked as resolved. You can refer to those too.

Update: I put in #17874 to kick off a pending deprecation period. 😬 🔥

OK actual deprecation is happening for v8.0. Please review #17883 . Thanks!

Navigation Menu

Repository navigation

Deprecate astropy test runner in core and downstream #16177

Proposal

Counter proposal

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	class TestRunner(TestRunnerBase):
	"""
	A test runner for astropy tests.
	"""

Deprecate astropy test runner in core and downstream #16177

Description

Proposal

Counter proposal

Activity

namurphy commented on Mar 7, 2024

MridulS commented on Mar 7, 2024

pllim commented on Mar 7, 2024

pllim commented on Mar 7, 2024

mhvk commented on Mar 7, 2024

saimn commented on Mar 8, 2024

saimn commented on Mar 8, 2024

pllim commented on Mar 8, 2024

mhvk commented on Mar 8, 2024

pllim commented on Mar 8, 2024

Cadair commented on Mar 8, 2024

bsipocz commented on Nov 26, 2024

pllim commented on Feb 6, 2025

pllim commented on Feb 6, 2025

pllim commented on Feb 14, 2025

keflavich commented on Feb 14, 2025

mhvk commented on Feb 14, 2025

weaverba137 commented on Feb 14, 2025

pllim commented on Feb 15, 2025

pllim commented on Feb 19, 2025

pllim commented on Mar 11, 2025

pllim commented on Nov 4, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions