Add experimental uv resolver for Python lockfiles#22949
Conversation
|
Wow, this is a really creative design 🤩 |
|
Thanks for the contribution. We've just branched for 2.31.x, so merging this pull request now will come out in 2.32.x, please move the release notes updates to docs/notes/2.32.x.md if that's appropriate. |
bc17fc2 to
efc7d06
Compare
|
Hi maintainers 👋 — just rebased this onto the latest Summary of what this PR adds:
Known limitations (documented in the option help):
Would love a review from anyone familiar with the Python backend / lockfile generation — @cburroughs, @stuhood, @benjy, or whoever is currently shepherding the lockfile work. Happy to iterate! |
|
Thanks for this! Excited to review. I will try and take a look in the next day or two. I somehow missed this back when it was first created in, looks like, December. Sorry about that! Meanwhile, if you used LLMs in the making of this PR, can you elaborate on that? We are working on an official AI disclosure policy but haven't yet put one into practice. |
benjyw
left a comment
There was a problem hiding this comment.
Before I review the meat of this PR: a couple of seemingly unrelated changes got pulled in here.
| root_str, | ||
| dep_entries | ||
| .iter() | ||
| .map(|d| format!("\"{}\"", d.entry_str)) |
There was a problem hiding this comment.
Why these changes?
They aren't wrong, but AFAICT they are irrelevant to the purpose of this PR, and they don't seem to have any benefit. These strings are going to be consumed either here or immediately after, when they go out of scope.
| // Licensed under the Apache License, Version 2.0 (see LICENSE). | ||
|
|
||
| use std::convert::{AsRef, Infallible}; | ||
| use std::convert::Infallible; |
|
Hello, @Liam-Deacon @benjyw Can I help you somehow with PR? |
|
One question - why universal style is unsupported? I see no reason not to support him here. |
sureshjoshi
left a comment
There was a problem hiding this comment.
Thanks for the contribution. We've just branched for 2.32.x, so merging this pull request now will come out in 2.33.x, please move the release notes updates to docs/notes/2.33.x.md if that's appropriate.
|
Note that this is superseded by the more comprehensive solution in #23302. Thanks for all the effort on this front, and for the prodding to finally get this done properly. |
uv lockfiles are first-class entities that users check in. When a pex is needed, we use uv to create a venv, which it does very efficiently, and then use pex's `--venv-repository` feature to build a pex from (a subset of) that venv's requirements. Note that we still create VenvPexes to run tools, and in some cases this might be unnecessary, since we already have the uv venv. But utilizing those venvs directly would be an even bigger change (and this one is already massive), and there are caching tradeoffs to figure out . So we leave that for the future (and probably for the new python backend). Note also that the dynamics of lockfile generation remain pants's (you must generate explicitly) and not uv's (lockfiles are automatically updated for you when input requirements change). The latter is desirable, but again this is for the future. The general thrust of the changes is: - Add uv as a third lockfile format (after pex and the deprecated legacy constraints file "locking") - Update the lockfile metadata to include the format (and also the name of the resolve that created the lockfile). - Make lockfile metadata reading much more robust and useful, so we can get the format from it reliably. Unfortunately this logic is tricky since it has to handle pex lockfiles both with and without separate metadata files. For uv we simply require separate metadata files and do not add any metadata inside the lockfile itself, as we foolishly did for pex lockfiles at one point. - Update the lockfile diffing code to support diffing across all combinations of old/new and pex/uv. - Update the existing `uv` subsystem for more general use. - Add src/python/pants/backend/python/util_rules/uv.py, which runs `uv`, first generating ephemeral pyproject.toml and uv.toml config files. We handle environments correctly, which requires some `realpath` shenanigans. - Update some iffy logic in PythonToolBase that parses lockfiles to find useful info for help messages. It was previously relying on pex lockfile internals (which are not guaranteed to be stable) and it now also relies on uv lockfile internals (similarly not stable). However now at least it fails gracefully if the lockfile parsing fails and just prints a less useful error message. - Use all this in src/python/pants/backend/python/util_rules/pex.py, instead of the previous, uv pex building solution. Note that this supersedes the more limited (but much appreciated) change in #23197 that implemented only the uv venv -> pex part. It also supersedes the competing pending changes in #22949 and #23212, which implemented partial support for uv resolution.
|
Closing since #23302 is now merged. We'd love your feedback on that one. Thanks! |
…ick of #23302) (#23320) uv lockfiles are first-class entities that users check in. When a pex is needed, we use uv to create a venv, which it does very efficiently, and then use pex's `--venv-repository` feature to build a pex from (a subset of) that venv's requirements. Note that we still create VenvPexes to run tools, and in some cases this might be unnecessary, since we already have the uv venv. But utilizing those venvs directly would be an even bigger change (and this one is already massive), and there are caching tradeoffs to figure out . So we leave that for the future (and probably for the new python backend). Note also that the dynamics of lockfile generation remain pants's (you must generate explicitly) and not uv's (lockfiles are automatically updated for you when input requirements change). The latter is desirable, but again this is for the future. The general thrust of the changes is: - Add uv as a third lockfile format (after pex and the deprecated legacy constraints file "locking") - Update the lockfile metadata to include the format (and also the name of the resolve that created the lockfile). - Make lockfile metadata reading much more robust and useful, so we can get the format from it reliably. Unfortunately this logic is tricky since it has to handle pex lockfiles both with and without separate metadata files. For uv we simply require separate metadata files and do not add any metadata inside the lockfile itself, as we foolishly did for pex lockfiles at one point. - Update the lockfile diffing code to support diffing across all combinations of old/new and pex/uv. - Update the existing `uv` subsystem for more general use. - Add src/python/pants/backend/python/util_rules/uv.py, which runs `uv`, first generating ephemeral pyproject.toml and uv.toml config files. We handle environments correctly, which requires some `realpath` shenanigans. - Update some iffy logic in PythonToolBase that parses lockfiles to find useful info for help messages. It was previously relying on pex lockfile internals (which are not guaranteed to be stable) and it now also relies on uv lockfile internals (similarly not stable). However now at least it fails gracefully if the lockfile parsing fails and just prints a less useful error message. - Use all this in src/python/pants/backend/python/util_rules/pex.py, instead of the previous, uv pex building solution. Note that this supersedes the more limited (but much appreciated) change in #23197 that implemented only the uv venv -> pex part. It also supersedes the competing pending changes in #22949 and #23212, which implemented partial support for uv resolution.
Summary
This PR adds an experimental
uv-backed resolver option for Python user lockfile generation.Pants still generates and consumes Pex lockfiles, but when opted in it uses
uv pip compileto pre-resolve pins, then usespex lock create --no-transitiveto materialize the Pex lock.pex lock create)[uv].argsMotivation
Issue: #20679 — faster, more ergonomic lockfile generation by leveraging
uv.How it works
Usage
1) Enable resolves and select the uv resolver
2) Pass arbitrary uv flags (passthrough)
Use the standard
argspattern (same as other downloadable tools in Pants):Equivalent CLI:
pants generate-lockfiles --resolve=python-default \ --python-lockfile-resolver=uv \ --uv-args='--index-strategy unsafe-first-match'Current limitations (intentional safeguards)
lock_style = "universal"uv pip compileresolves for a single interpreter environment todaycomplete_platformssources/overrides/excludesnot yet modeled in uv stepBenchmark (real-world example)
Environment:
CPython==3.11.*strictpipvsuv~/repos/python-mono/3rdparty/python/requirements.txt~/repos/python-mono/3rdparty/python/requirements-dev.txtpip.antarcticaam.app/privatereferencesblpapi,StressVaR,module-wrapper,pytest-localstackTiming (
/usr/bin/time -p, Pantsd disabled; repeated to show warm-cache behavior):Method:
/tmp/pants-uv-lockfile-benchmark(isolated), usingpython_requirements(...)pointing at the filteredrequirements.in.PANTS_WORKDIRto avoid Pants's local process cache reusing results.PANTS_GLOBAL_PEX_ROOT=/tmp/pants-uv-lockfile-benchmark/cache/pex_rootPIP_CACHE_DIR=/tmp/pants-uv-lockfile-benchmark/cache/pip_cacheUV_CACHE_DIR=/tmp/pants-uv-lockfile-benchmark/cache/uv_cachepippipuvuvSpeedup (lower is better):
pip/uv = 3.58xpip/uv = 1.16xNotes:
uvpre-solve materially reduces cold-cache solve time in this real-world-ish set.Additional visibility (uv mode only):
uv pip compilepre-solve (previous run-set)In this run-set,
uvreduced end-to-end wall time vspip(especially cold-cache). Warm-cache repeats narrow the gap, which suggests resolver overhead is a larger fraction of first-time generation than subsequent runs.Code changes
src/python/pants/backend/python/subsystems/setup.py[python].lockfile_resolver = {pip,uv}src/python/pants/backend/python/subsystems/uv.pyuvtool +[uv].argspassthroughsrc/python/pants/backend/python/goals/lockfile.pyuv pip compilepre-step (opt-in) thenpex lock create --no-transitivesrc/python/pants/backend/python/goals/lockfile_test.pydocs/docs/python/overview/lockfiles.mdx