DEP: migrate dev-only extra dependencies to PEP 735 dependency groups

@nstarman

This was initially proposed by @nstarman in #16963 (comment)

PEP 735, recently accepted, defines standard syntax to specify dev-only dependencies in pyproject.toml in a way that is not visible by users, contrary to extra dependencies defined in the [project.optional-dependencies] table.
This is a natural fit for some of our dependencies, currently defined as extras for convenience, but not intended for users to consume, e.g.

test_all
typing
docs
dev
dev_all

There are however 2 issues to be addressed before we can hope to migrate: tooling, and backward compatibility.

Tooling

Dependency groups are already supported by tox, cibuildwheel, pixi, uv and presumably other modern installers, but not by pip yet. pixi/uv can replace pip (and do much more too), though I would assume most of us don't want to (or can't) move away from pip at the moment and even if it was only a minority, we still don't want to leave them behind, so that's a blocker in my book. Here's the issue to keep track of

Feature Proposal: PEP 735 "Dependency Groups" Support pypa/pip#12963

Backward compatibility

Dependencies like docs are not intended for user consumption, however they are exposed at the moment (it is possible to pip install astropy[docs] from PyPI), so hiding them might be considered a breaking change, and should probably only be allowed in a major release.

If we worry about potential disruption from breaking these suddenly, there is a way to deprecate them first:

create and publish a marker (empty) package astropy-docs-extra
add it to the astropy[docs] extra
at runtime, make import astropy... raise a deprecation warning if astropy-docs-extra is installed

Credit to urllib3 for this clever deprecation strategy and to Anthony Sottile for advertising it on youtube.
In my opinion it's largelly overkill in our case; this hack was designed for a library with massively more users than astropy, and was used to deprecate an extra that was intended for user consumption, but I wanted to show that it existed for completeness.

@neutrinoceros

Thanks for looking into this @neutrinoceros!
I'm looking forward to v8 w/ pip support.

The pip issue was just closed as completed, so if all goes well pip 25.1 should support dependency groups !

pip 25.1 is out (release notes: https://pip.pypa.io/en/stable/news/#features)

It is now possible to install dependency groups as, e.g.

python -m pip install --group typing

see https://pip.pypa.io/en/stable/user_guide/#dependency-groups

Amazing! What's our support policy on versions of pip? When can we bump to this version ?

I'm not aware of a policy regarding support for dev tooling. I'd assume we can make the move even now as long as we also make sure team members get the chance to voice objections and request delays on merging.

And I suppose we can have both the dependency group and optional dependency section during any transition. Just have to be careful to keep them in sync / deprecate the optional dependency.
The things that we want to transition are the dev-related dependencies.

Edit: is this possible? To have an optional dependency collection and dependency group with the same name?

support policy on versions of pip?

pip is also used for end-user installation, so unless this is somehow super urgent, I would give it time. Thanks!

The pip that we use for dev doesn't impact the pip version for users. Users cannot be impacted by this discussion unless they are trying to do pip install astropy[dev] (hence #17623 (comment)).
Move fast and don't break things. 🤣

Is this change backward compatible? I don't remember.

Adding groups is by essence invisible to anyone downstream, so it's perfectly backwards compatible. Removing extras on the other hand may be breaking if anyone relies on them being public. I'd assume we could preserve a selection of them that we know are used downstream, and start by migrating clear dev-only ones to groups.

Another note: at the moment, open astronomy's wheel factory workflow doesn't support test dependencies expressed as groups, so I think we need to start there.

here's the actual first step: OpenAstronomy/build-python-dist#21

new release just dropped: https://github.com/OpenAstronomy/github-actions-workflows/releases/tag/v2.1.0

This part of conf.py could be harder to re-implement in PEP 735 groups (maybe I should just use the dependency-groups library)

astropy/docs/conf.py

Lines 57 to 81 in 470bbc1

    
           missing_requirements = {} 
        
           for line in metadata.requires("astropy"): 
        
               if 'extra == "docs"' in line: 
        
                   req = Requirement(line.split(";")[0]) 
        
                   req_package = req.name.lower() 
        
                   req_specifier = str(req.specifier) 
        
                   try: 
        
                       version = metadata.version(req_package) 
        
                   except metadata.PackageNotFoundError: 
        
                       missing_requirements[req_package] = req_specifier 
        
                   if version not in SpecifierSet(req_specifier, prereleases=True): 
        
                       missing_requirements[req_package] = req_specifier 
        
           if missing_requirements: 
        
               msg = ( 
        
                   "The following packages could not be found and are required to " 
        
                   "build the documentation:\n" 
        
                   "%s" 
        
                   '\nPlease install the "docs" requirements.', 
        
                   "\n".join([f"    * {key} {val}" for key, val in missing_requirements.items()]), 
        
               ) 
        
               logger.error(msg) 
        
               sys.exit(1)

@eteq

If we worry about potential disruption from breaking these suddenly, there is a way to deprecate them first... In my opinion it's largelly overkill in our case

I largely agree. I think [docs] is very likely. [tests] probably. Ping @eteq given recent discussion about test runner.

Navigation Menu

Repository navigation

DEP: migrate dev-only extra dependencies to PEP 735 dependency groups #17623

Tooling

Backward compatibility

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	missing_requirements = {}
	for line in metadata.requires("astropy"):
	if 'extra == "docs"' in line:
	req = Requirement(line.split(";")[0])
	req_package = req.name.lower()
	req_specifier = str(req.specifier)

	try:
	version = metadata.version(req_package)
	except metadata.PackageNotFoundError:
	missing_requirements[req_package] = req_specifier

	if version not in SpecifierSet(req_specifier, prereleases=True):
	missing_requirements[req_package] = req_specifier

	if missing_requirements:
	msg = (
	"The following packages could not be found and are required to "
	"build the documentation:\n"
	"%s"
	'\nPlease install the "docs" requirements.',
	"\n".join([f" * {key} {val}" for key, val in missing_requirements.items()]),
	)
	logger.error(msg)
	sys.exit(1)

DEP: migrate dev-only extra dependencies to PEP 735 dependency groups #17623

Description

Tooling

Backward compatibility

Activity

nstarman commented on Jan 10, 2025

neutrinoceros commented on Feb 22, 2025

neutrinoceros commented on Apr 28, 2025

nstarman commented on Apr 28, 2025

neutrinoceros commented on Apr 28, 2025

nstarman commented on Apr 28, 2025

pllim commented on Apr 28, 2025

nstarman commented on Apr 28, 2025

pllim commented on Apr 28, 2025

neutrinoceros commented on Jun 22, 2025

neutrinoceros commented on Jun 22, 2025

neutrinoceros commented on Jun 24, 2025

neutrinoceros commented on Jun 25, 2025

nstarman commented on Jun 25, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions