DEP: redefine dev[_all] extras as PEP 735 Dependency Groups - #18342

Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

• Do the proposed changes actually accomplish desired goals?
• Do the proposed changes follow the Astropy coding guidelines?
• Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
• Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
• Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
• Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
• Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
• Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
• At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

👋 Thank you for your draft pull request! Do you know that you can use [ci skip] or [skip ci] in your commit messages to skip running continuous integration tests until you are ready?

Could you explain what the benefits are of using dependency groups here compared to extras?

providing a clearer separation between

• optional dependencies for end-users and downstream packages (this is the purpose of extras)
• dev-only dependencies that were never really meant to be consumed downstream (but were exposed anyway because extras was all we had in the standard)

Ah I see, so would you plan to deprecate astropy[test]?

This one I don't think so, but astropy[dev], yes

Ok cool, if we don't plan to deprecate test, maybe we should avoid redefining it since anyway for now it requires repeating everything?

I'll touch base with uv folks first. It shouldn't be necessary to duplicate metadata

Actually, I tried to reproduce the problem with self-referential groups + --resolution=lowest-direct in uv and got no problem, as long as I include the following bit in pyproject.toml (and which uv helpfully writes automatically if you use uv add to declare dependencies)

[tool.uv.sources]
astropy = { workspace = true }

Now I'm thinking the problem might be with tox or tox-uv loosing track of pyproject.toml at some point, so I'll need to report there.

Any extras you would not oppose to removing ?

In hindsight I agree there's not much point to have both a test extra and a test group if we're not going to deprecate/remove the extra later. I would still like to move some dependency away from user-facing extras if possible, the question is, are there any extras that are not controversial to deprecate or remove ?

regarding the amount of typing required, and as a purely informative note: if you don't mind pulling in all dependencies, there's uv sync --all-groups (also uv sync --all-extras)

are there any extras that are not controversial to deprecate or remove ?

There were all added for a reason, so probably not, at least not without a lot of opinions involved...

For dev-only extras, I'm assuming the reason was that no equally convenient way existed to define them without exposing them. In other words, I think they were probably exposed as a side effect rather than intentionally. Maybe I'm wrong about it. And even if I'm not, it could be argued that it's too late to put the genie back in the bottle, but I'm not there yet.

dev-only extras

dev and dev_all were added by @nstarman in #16340 . Maybe he remembers why. I never use those.

For dev-only extras, I'm assuming the reason was that no equally convenient way existed to define them without exposing them

Yes.
IMO fine to move.

Ok, reduced the scope of this PR to dev and dev_all extras, which hopefully is much less controversial. This should now be ready for review. Further discussions on whether more extras should be moved or not should happen on #17623

Please rebase. Thanks!

I already did this very morning. Is it still needed ?

Says outdated again and one job failing, so might as well?

Let's try re-running first.

@taldcroft , do you want to re-review?

This was on today's agenda but I think we somehow forgot to go over it...

Overall I don't want to stand in the way of progress. But I'll note one more thing, which is that pip seems to treat a missing extras dependency as a warning not an error. So when people continue to run pip install -e ".[dev-all]" after this is merged, the command appears to run to completion but there is a warning embedded:

WARNING: astropy 8.0.0.dev233+g8ec2d1840 does not provide the extra 'dev-all'

This could be easy to miss, but even if it is spotted this will cause confusion for anyone not following this PR.

ACTION: when this is merged, we need to send notice to the astropy-dev email group and astropy slack (probably #general) with information about the change. We need to do this right away since this applies immediately, not with a release.

@neutrinoceros , can you please address https://github.com/astropy/astropy/pull/18342/changes#r2678530716 ? Thanks.

Thanks for reminding me, I lost track of this PR's state so it seems. I think I've addressed any lingering comments now !

@taldcroft are you able to do a final review to ensure your remaining concern is truly addressed to your liking?

Thanks, all!

@pllim @taldcroft I just rebased this to resolve merge conflicts. The diff is now mostly from the indirect changes in the resolved lowest tree, but this might bitrot more quickly as a result.

failures are unrelated, and also seen on main. I'll inquire there (probably not earlier than tomorrow though), but this can be undrafted in the mean time.

@astrofrog this has one approval already, and would be much better as part of a major release. Can we merge this for 8.0.0 ?

changelog added, auto-merge enabled, and email sent. Thanks !