Repository navigation
Description
What's the problem this feature will solve?
"Dependency Groups" are proposed in PEP 735 with a variety of target use cases.
In particular, the --only-deps use cases are explicitly a target for this PEP, but additional cases are listed and relevant.
Describe the solution you'd like
As the PEP author, I would like to start discussing what form, in terms of interface and implementation, pip support would take.
I would like to agree on a potential implementation path, implement it in a draft PR, and leave it pending until the PEP is accepted.
My proposal is:
- a variant of the DependencyGroupResolver I wrote moves in-tree here, along with appropriate unit tests
- a new CLI option is added to
pip install,--dependency-groups, which takes a comma-delimited list of group names and attempts to resolve and install all of them frompyproject.tomlin the current working directory
For example,
# in pyproject.toml
[dependency-groups]
test = ["pytest"]
typing = ["pyright"]
# on the CLI
pip install --dependency-groups test,typing
# equivalent to
pip install pytest pyright
Would the pip maintainers be amenable to this addition? Is there an alternative, similar strategy which I could implement?
Alternative Solutions
If pip does not gain support for Dependency Groups, it is possible to workaround the gap.
Primarily, external tooling like dependency-groups can be used to resolve groups and pass them to pip.
For example, in a *nix environment, the following usage covers most cases:
groups=$(python -m dependency_groups pytest typing | tr '\n' ' ')
pip install $groupsHowever, such usages will always be workarounds. pip support provides a superior developer experience.
Additional context
PEP 735 is in Draft, and will be submitted soon. At time of writing, it is not final or accepted, but no further changes to the contents are planned.
Code of Conduct
- I agree to follow the PSF Code of Conduct.
Activity
pfmoore commentedon Sep 11, 2024
Some questions around the UI.
How would something like
pip install requests --dependency-groups xxxwork? Is the flag intended to only be valid for installing from source? What aboutpip install requests --no-binary requests --dependency-groups xxxin that case?How will the flag work with multiple requirements? If I do
pip install --dependency-groups foo ./a ./b ./cis the dependency group looked up for all of a, b, and c? Or is this invalid usage?Basically, I don't think we have an existing flag that works like I'd expect this one to, which might be a usability problem.
RonnyPfannschmidt commentedon Sep 12, 2024
I'd consider pip the wrong place for managing this in a normal way
IMHO it's different enough from normal install that it warrants a distinct command to install dependency groups of a specific source folder
Tools like hatch, rye and poetry should be the place of closer integration
sirosen commentedon Sep 12, 2024
I am a bit concerned about meeting the target use cases for the PEP without pip support. It would be possible, but the user experience is worse in several cases.
That may be, but I see the feature as similar to
-r. The main difference being that the path lookup is implicit in the UX I proposed -- it needn't be. We can discuss usages in which the path to pyproject.toml is given explicitly. (More on this below)I think the answer here is "and" not "or". Dependency Groups will have the greatest positive impact if they are supported by installers, environment managers, and build backends.
I've started with pip and setuptools. Perhaps this is a strategy mistake and I should begin from younger tools like flit.
Before I dive into Paul's specific scenarios, I want to start from the high level concern, make sure I understand it, and consider whether or not the UX I proposed is correct.
Is the issue that this is expected to expand to a set of requirements, whereas
-rexpands to a set of arguments? Or is it the implicit lookup of./pyproject.toml? Both of these? Something else?I'm trying to understand if the issue would be mitigated by making the toml file explicit, e.g.,
pip install --dependency-group pyproject.toml:foo --dependency-group ../otherproj/pyproject.toml:bar.RonnyPfannschmidt commentedon Sep 12, 2024
I believe the proposed installation ux clearly shows my 3rd point
pfmoore commentedon Sep 12, 2024
I think the problem for me was that it wasn't clear that you're proposing that
--dependency-groupsworks the same way as--requirement. In particular, in thepip installhelp, we currently haveIf I understand, you're proposing to add a new form here:
This would look in a
pyproject.tomlin the current directory and extract the list of requirements to install from the specified dependency groups in there. Looking at it that way seems a lot more plausible, and eliminates all of my objections above. We do allow-rand "normal" requirements to be combined, even though the syntax above doesn't show it. I'd assume--dependency-groupswould work the same.That was incorrect, we do, it's
-r. If we mirror the behaviour of-r, that should be OK. Sorry for the confusion here.There's some background information available in #8049. The conclusion there was that we rejected the feature request, but dependency groups are much more specific, and will be standards-based, so I think it's within scope for pip to support them.
Also, there may be some edge cases that won't work, because requirements files are much richer than dependency groups. For example,
--dependency-groupswon't work well with hash checking mode, as (unlike requirement files) dependency groups don't offer a way to specify a hash. I think that's fine, but it might be worth explicitly documenting any limitations we are aware of somewhere, so we don't get user reports saying "hash checking fails with dependency groups" and expecting us to fix this.pfmoore commentedon Sep 12, 2024
Strong -1 on me for allowing anything but "read dependency groups from
pyproject.tomlin the current directory". Finding thepyproject.tomlfor some idea of the "current project" should be left to project management tools, and explicitly stating the path topyproject.tomlis unnecessary complexity that (as far as I'm aware) there's no valid use case for.notatallshaw commentedon Sep 17, 2024
So one of the repos I work with is a messy monorepo, in that repo we would never put a
pyproject.tomlin there because it might stomp all over the settings of one team or another.This has never been a problem because front end and utility tools, like pip, pre-commit, linters, formatters, etc. allow you to specify where you want the configuration / requirements to come from. And backend tools that do not allow this are generally not suitable for the repo anyway i.e. we never want to build a package in the root of the repo.
Were pip to use this as a "requirements-like install file but with structured options" it would be a departure from pip's current approach to allow the user to specify the path of the requirements file or constraints file, and other front end tools in general.
If you don't consider "messy monorepo" as a valid use case, fair enough, but we're not trying to build a package from it, so it would be annoying for frontend tools, that are not project management tools, to say how we should structure our project.
That said, this isn’t a hill I want to die on, just want to give an alternative view point.
sirosen commentedon Oct 2, 2024
I think "messy monorepo" is a valid use-case to consider for
pip(or, really, any ecosystem tool), but considering it isn't the same as letting it drive the UX/DX.If
pipsupport for dependency groups implicitly always used./pyproject.toml, I can see that it might hamper some usages, but not that it would block them entirely. You can alwayscd ...(shell),WORKDIR ...(docker), or similar -- I can't think of any tools capable of runningpipwhich are deficient in this regard.I'm not strictly against something like
--dependency-groups-pyproject=...with a default ofpyproject.toml, but I trust the judgement of thepipmaintainers about what is a good fit (or not) for the existing tool.I'm not sure if I should open a draft PR or not at this stage? I can add a
pipwrapper todependency-groups, e.g.pip-dependency-groups install ...?redeboer commentedon Oct 11, 2024
Update: PEP 735 has been accepted 🎉
https://discuss.python.org/t/pep-735-dependency-groups-in-pyproject-toml/39233/312
potiuk commentedon Oct 12, 2024
Fantastic. We have some good use cases for those in Airflow and once
pipand other tools will start supporting those, we will definitely make a very good use of those.henryiii commentedon Oct 25, 2024
FWIW, uv has shipped dependency groups (as has tox). It would potentially be nice to mimic the CLI interface names here unless there's a reason not to.
7 remaining items
sirosen commentedon Oct 25, 2024
Ah, good point. I think that mitigates my concern about supporting it in
pipand users bringing an expectation that it will exactly matchuv.I'll work with
--groupfor now, but I'll make a note at review-time that this is chosen to matchuvand may need more thought.Yes, once the dust settles, I do want to pursue this idea. (Or related ideas which cover this functionality.)
I'm not sure how far into the future that will be though.
notatallshaw commentedon Oct 25, 2024
Ah yes, I see they're not exposed via
uv sync, butuv addalso installs a group, and to my eyesaddlooks closer topip installthansync.Probably a cue that project semantics are not the same as install semantics and pip should choose independently of uv.
zanieb commentedon Oct 26, 2024
We have some additional experience that may help here.
uv already supports
uv pip install --extra. However, this is only allowed if you provide-rwith a project source, e.g.:In this case, we actually don't install the project itself, just the dependencies declared in the file. Generally, this is confusing to people using
pip install, e.g.:uv pip install <PyPI package> --all-extrasastral-sh/uv#7295uv pip install --all-extras .fail to locatepyproject.tomlastral-sh/uv#4762uv pip compiledoes not include local package astral-sh/uv#6317I believe we added this flag for
uv pip compileuse-cases (i.e., you want to export the requirements of your project), then propagated it throughout the interface. We needed a separate flag to power this because there are no semantics for requesting extras from a project file, e.g.,pip install -e example[test]is standard butpip install -r pyproject.toml[test]is not.Adding support for PEP 735 looks like a very similar problem. There is no standard way to request groups in the package specifier. In hindsight, this probably should have been considered in the PEP (I do not recall seeing discussion, if there was any, please point me to it!).
There's a bit more complexity here, in that pip does not support project files via
-rlike uv:So, where do we go from here? In uv, I think we'll need to support
--groupinuv pipfor parity with--extraand the rest of our interface. However, there's not a clear equivalent for that flag here since there's not a way to provide a path to thepyproject.toml. Much of the conversation thus far has been focused on how a user says which project they want the groups to come from. Notably, I think "read dependency groups from pyproject.toml in the current directory" would be incredibly confusing; pip does not do any discovery of projects today, right? Does pip have any reliance on the contents of the working directory today? This seems like a weird and surprising change to the experience. I have a strong expectation that the path to the project would need to be provided.Concretely, I'd recommend adding a
--groupflag to pip, which, enables the group for for any paths in the installation:If there are no paths (e.g.,
pip install anyio --group test) or the group does not exist in any path, I'd expect pip to fail. I don't think groups should be read from source distributions. Groups should only be read from<path>/pyproject.toml, not discover apyproject.tomlelsewhere in the directory. Groups could be read from remote source directories like VCS URLs.I think this is the most guessable solution for users, e.g., you'd just tack on
pip install -e . --group dev. It also makes it clear the project itself is being included in the installation (which would otherwise be ambiguous), and gives control over installing the project in editable or regular mode.This leaves an important use-case unaddressed: installing a group without the project itself. I'm not sure what the best path forward on this is for pip.
uv pip install -r pyproject.toml --group devis the most logical interface in uv, but we've seen significant user confusion around that interface and I can't strongly recommend it given there's not already support for-r pyproject.toml. Regardless I think you end up with some need to specify the project path via a dedicated flag, e.g.,pip install --project . --group dev, or in the group specification (if you want to install groups from multiple projects), e.g.,pip install --group .[dev](similarly, proposed in #12963 (comment)). As a minor note, something like #11440 doesn't address this since the project's other dependencies would still be installed.Similarly, we haven't figured out how to expose an interface for groups without a
[project]table in uv, but the user expectation is thatuv runwill discover apyproject.tomlso something likeuv run --group testshould "just work". As I mentioned before, adding project discovery to pip (even in the working directory) feels like a big change. I think this is also a possible cause of behavior differences between uv and pip, asuv pipwould probably need to follow our typicalpyproject.tomlworkspace discovery rules which goes beyond the working directory.To be clear, I don't have any expectation that pip conforms to uv's interface. I do think it's in the best interest of the users to have a coherent experience across tools though. I'm hoping that our experiences and feedback from users can help inform decisions here (and vice-versa!).
pfmoore commentedon Oct 26, 2024
I believe this was deliberate, and in any case it's correct IMO, as allowing requirement specifiers to include dependency groups would allow projects to depend on dependency groups, which isn't possible because dependency groups aren't stored in project metadata.
Requesting groups via an installer flag is the intended approach.
I don't think that's something we'd want to support, either. The
--requirement(-r) flag is specifically to indicate a requirements file, it's not for specifying thepyproject.tomlfile.This is the root of why dependency groups are a bit of an uncomfortable "fit" for pip. We very deliberately and explicitly don't require any sort of concept of "the current project" in order to use pip. But dependency groups are very much a project-oriented concept (because they are defined in the project metadata file!) But maybe that suggests a way forward here. We could add a new
--projectoption to pip, which points to the "selected project" (a local directory). Note in particular that you specify the project rather than thepyproject.tomlfile. Any pip commands that need the concept of a project would need to specify it via the--projectoption - at the moment, that's just dependency groups, but longer term it might include other features based off thepyproject.tomlfile1. So the way to install a dependency group would then be:Just to be clear, this is not a fully-formed idea at this point, it's just an off-the-cuff suggestion for an approach that fits better with pip's design. It will need some work to flesh it out.
In particular, I don't like this option. As you say, it doesn't offer a way to not install the project, and furthermore, it complicates things when there's multiple paths in the install command (which might not even agree on what group names mean). And it also makes local directory names special, in a way that would immediately prompt the question "why not allow reading
pyproject.tomlfrom sdists as well?" and then why not allow remote sdists (viat http)...Agreed, and at least personally, it's not a direction I want to go in (I haven't asked the other @pypa/pip-committers their views on this). In particular, while I don't want to fall into the trap of making every detail a big design question, I also don't want a significant change in philosophy to "slip in" as part of what should be a fairly straightforward feature addition.
Thanks for that comment. And I do agree that being consistent is better for users. I think that pip and the
uv pipinterface should definitely try to be consistent, but we may have to diverge from the "native"2uvinterface, simply becauseuv(in its native form) is a project focused tool, whereas pip isn't. I don't know how comfortable that would be foruvusers, though (do your users expectuv pipto be more like pip, or more like uv?)Footnotes
The proposed
--only-depsoption might fall into this category... ↩Is there an official term for the
uvinterface that isn'tuv pip? ↩zanieb commentedon Oct 27, 2024
I agree. We adapted this from
pip-toolsthen made it consistent across our interface — I don't think pip needs this.Did you see I recommended this? Basically, yeah this makes sense to me. I especially like that you highlighted that there is future work that will require project awareness in pip. However, I worry that it'll be confusing for users to need a command like:
This is what led me to having
--groupapply to source trees. Is there an easier way to install the project and the group? It seems worth optimizing for that over cases where you want the group separately from the project.There are still confusing cases where it's not clear what
groupapplies to, e.g.:I think the idea here is that we would support this for source trees, but not distributions. I don't think it'd be hard distinction to make, i.e., we won't consume parts of the
pyproject.tomlthat are not part of the distribution metadata. I understand your hesitation, but I feel like the entire focus of dependency groups is local. It seems feasible that users won't know that groups are even available in source distributions, and I think there's a lot of room to guide expectations with error messages. Mostpip installinvocations are probably for a single target package, which makes it less likely to be confusing (unless using a requirements file, in which case I don't think groups really applies). I think there will need to be some careful work to set expectations about what--groupapplies to regardless, since it's a different paradigm.They want both 🙃
In this context, it might be fine if we infer the value of
--projecteven if pip does not since pip would just exit with an error otherwise.I guess I'd say... "uv's interface" or "uv's project interface" vs "uv's pip-compatible interface". The project interface includes all of our top-level commands, though
uv runcan be used in more contexts.chore: Introduce pip support and make uv optional
chore: Introduce pip support and make uv optional
chore: Introduce pip support and make uv optional
notatallshaw commentedon Jan 23, 2025
FYI, it looks like uv got bored waiting to see exactly what pip's interface looks like and they've added
--groupfor theiruv pipinterfaces: astral-sh/uv#10861It looks like the syntax there will be
uv pip install -r pyproject.toml --group foo, which will not be the same as pip's currently proposed syntax, but pretty close, and uv already supporteduv pip install -r pyproject.toml.pfmoore commentedon Jan 23, 2025
Fair enough. I don't think we should change the syntax we've agreed on just to match uv1, and I think there's been long enough for anyone with objections to raise them, so I'm planning on merging the existing PR once the requested documentation additions have been included.
Footnotes
It was uv's choice to call their command
uv pipand then diverge from pip in UI and functionality. I disagree with that choice, but it's their decision, and while it's useful to know how they've diverged, I think we should continue to make our own UI choices. ↩