Skip to content

Add experimental uv resolver for Python lockfiles#22949

Closed
Liam-Deacon wants to merge 3 commits into
pantsbuild:mainfrom
Liam-Deacon:uv-lockfile-resolver
Closed

Add experimental uv resolver for Python lockfiles#22949
Liam-Deacon wants to merge 3 commits into
pantsbuild:mainfrom
Liam-Deacon:uv-lockfile-resolver

Conversation

@Liam-Deacon

@Liam-Deacon Liam-Deacon commented Dec 16, 2025

Copy link
Copy Markdown
Contributor

Summary

This PR adds an experimental uv-backed resolver option for Python user lockfile generation.

Pants still generates and consumes Pex lockfiles, but when opted in it uses uv pip compile to pre-resolve pins, then uses pex lock create --no-transitive to materialize the Pex lock.

Capability Before After (opt-in)
Lockfile generator Pex (pex lock create) still Pex
Resolver used for the solve step pip uv (PubGrub)
Extra flags pip/pex flags only uv flags via [uv].args

Motivation

Issue: #20679 — faster, more ergonomic lockfile generation by leveraging uv.

How it works

requirements (targets)
        |
        |  (opt-in) uv pip compile
        v
 pinned requirements.txt
        |
        |  pex lock create --no-transitive
        v
     Pex lockfile

Usage

1) Enable resolves and select the uv resolver

[python]
enable_resolves = true
lockfile_resolver = "uv"  # <-- opt in

# uv currently requires a single Python major/minor
interpreter_constraints = ["CPython==3.11.*"]

[python.resolves]
python-default = "3rdparty/python/default.lock"

# uv is currently supported for strict/sources, not universal
[python.resolves_to_lock_style]
python-default = "strict"

2) Pass arbitrary uv flags (passthrough)

Use the standard args pattern (same as other downloadable tools in Pants):

[uv]
# Example requested in the issue: prefer the first matching index.
args = ["--index-strategy", "unsafe-first-match"]

Equivalent CLI:

pants generate-lockfiles --resolve=python-default \
  --python-lockfile-resolver=uv \
  --uv-args='--index-strategy unsafe-first-match'

Current limitations (intentional safeguards)

Limitation Reason
No lock_style = "universal" uv pip compile resolves for a single interpreter environment today
No complete_platforms not wired through in this initial integration
Interpreter constraints must select one Python major/minor ensures uv resolves for exactly one target interpreter
Per-resolve sources / overrides / excludes not yet modeled in uv step avoids silently changing semantics

Benchmark (real-world example)

Environment:

  • Python: CPython==3.11.*
  • Lock style: strict
  • Resolver modes compared: pip vs uv
  • Input: annotated + sorted concat of:
    • ~/repos/python-mono/3rdparty/python/requirements.txt
    • ~/repos/python-mono/3rdparty/python/requirements-dev.txt
  • Excluded from the benchmark input to keep it public/reproducible:
    • any pip.antarcticaam.app/private references
    • blpapi, StressVaR, module-wrapper, pytest-localstack

Timing (/usr/bin/time -p, Pantsd disabled; repeated to show warm-cache behavior):

Method:

  • Buildroot: /tmp/pants-uv-lockfile-benchmark (isolated), using python_requirements(...) pointing at the filtered requirements.in.
  • Each run uses a fresh PANTS_WORKDIR to avoid Pants's local process cache reusing results.
  • Tool caches persist across repeats:
    • PANTS_GLOBAL_PEX_ROOT=/tmp/pants-uv-lockfile-benchmark/cache/pex_root
    • PIP_CACHE_DIR=/tmp/pants-uv-lockfile-benchmark/cache/pip_cache
    • UV_CACHE_DIR=/tmp/pants-uv-lockfile-benchmark/cache/uv_cache
  • “cold” = cache dirs removed before the run; “warm” = immediate repeat with caches retained.
Mode Run real (s) user (s) sys (s)
pip cold 323.58 144.50 40.03
pip warm (repeat) 117.52 136.15 29.27
uv cold 90.49 108.48 29.05
uv warm (repeat) 101.41 121.51 32.02

Speedup (lower is better):

  • Cold: pip/uv = 3.58x
  • Warm: pip/uv = 1.16x

Notes:

  • Results vary with network / platform and the resolver's cache state. The intent is to show that the uv pre-solve materially reduces cold-cache solve time in this real-world-ish set.

Additional visibility (uv mode only):

Phase Observed duration
uv pip compile pre-solve (previous run-set) 25.39s

In this run-set, uv reduced end-to-end wall time vs pip (especially cold-cache). Warm-cache repeats narrow the gap, which suggests resolver overhead is a larger fraction of first-time generation than subsequent runs.

Code changes

Area Change
src/python/pants/backend/python/subsystems/setup.py Adds [python].lockfile_resolver = {pip,uv}
src/python/pants/backend/python/subsystems/uv.py Adds hermetic uv tool + [uv].args passthrough
src/python/pants/backend/python/goals/lockfile.py Runs uv pip compile pre-step (opt-in) then pex lock create --no-transitive
src/python/pants/backend/python/goals/lockfile_test.py Adds unit tests for uv-mode validation behavior
docs/docs/python/overview/lockfiles.mdx Documents uv resolver configuration and flags

@jasonwbarnett

Copy link
Copy Markdown
Contributor

Wow, this is a really creative design 🤩

@cburroughs

Copy link
Copy Markdown
Contributor

Thanks for the contribution. We've just branched for 2.31.x, so merging this pull request now will come out in 2.32.x, please move the release notes updates to docs/notes/2.32.x.md if that's appropriate.

@Liam-Deacon
Liam-Deacon force-pushed the uv-lockfile-resolver branch from bc17fc2 to efc7d06 Compare March 15, 2026 05:08
@Liam-Deacon

Copy link
Copy Markdown
Contributor Author

Hi maintainers 👋 — just rebased this onto the latest main (resolving conflicts with the new sync feature in generate-lockfiles and the updated execute_process_or_raise API).

Summary of what this PR adds:

  • A new [python].lockfile_resolver option (default: pex, opt-in: uv) that uses uv pip compile to pre-resolve fully-pinned requirements before calling pex lock create --no-transitive.
  • Motivated by pants generate-lockfiles to use uv for dependency resolution #20679uv's --unsafe-first-match resolver behaviour lets large repos with pinned transitive deps generate lockfiles that pex's resolver struggles with.
  • The uv subsystem integration (src/python/pants/backend/python/subsystems/uv.py) is already present in Pants, so no new subsystem plumbing is needed.
  • Release note added to docs/notes/2.32.x.md.

Known limitations (documented in the option help):

  • Requires a single Python major/minor interpreter constraint (e.g. CPython==3.11.*)
  • Does not yet support lock_style="universal" or complete_platforms

Would love a review from anyone familiar with the Python backend / lockfile generation — @cburroughs, @stuhood, @benjy, or whoever is currently shepherding the lockfile work. Happy to iterate!

@benjyw

benjyw commented Mar 16, 2026

Copy link
Copy Markdown
Contributor

Thanks for this! Excited to review. I will try and take a look in the next day or two. I somehow missed this back when it was first created in, looks like, December. Sorry about that!

Meanwhile, if you used LLMs in the making of this PR, can you elaborate on that? We are working on an official AI disclosure policy but haven't yet put one into practice.

@benjyw benjyw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before I review the meat of this PR: a couple of seemingly unrelated changes got pulled in here.

root_str,
dep_entries
.iter()
.map(|d| format!("\"{}\"", d.entry_str))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why these changes?

They aren't wrong, but AFAICT they are irrelevant to the purpose of this PR, and they don't seem to have any benefit. These strings are going to be consumed either here or immediately after, when they go out of scope.

// Licensed under the Apache License, Version 2.0 (see LICENSE).

use std::convert::{AsRef, Infallible};
use std::convert::Infallible;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?

@Niccolum

Niccolum commented Apr 1, 2026

Copy link
Copy Markdown

Hello, @Liam-Deacon @benjyw

Can I help you somehow with PR?

@Niccolum

Niccolum commented Apr 1, 2026

Copy link
Copy Markdown

One question - why universal style is unsupported? I see no reason not to support him here.

@sureshjoshi sureshjoshi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution. We've just branched for 2.32.x, so merging this pull request now will come out in 2.33.x, please move the release notes updates to docs/notes/2.33.x.md if that's appropriate.

@benjyw

benjyw commented May 1, 2026

Copy link
Copy Markdown
Contributor

Note that this is superseded by the more comprehensive solution in #23302. Thanks for all the effort on this front, and for the prodding to finally get this done properly.

benjyw added a commit that referenced this pull request May 5, 2026
uv lockfiles are first-class entities that users check in.
When a pex is needed, we use uv to create a venv, which
it does very efficiently, and then use pex's
`--venv-repository` feature to build a pex from
(a subset of) that venv's requirements.

Note that we still create VenvPexes to run tools, and
in some cases this might be unnecessary, since we
already have the uv venv. But utilizing those venvs 
directly would be an even bigger change (and this one
is already massive), and there are caching tradeoffs
to figure out . So we leave that for the future (and
probably for the new python  backend).

Note also that the dynamics of lockfile generation remain
pants's (you must generate explicitly) and not uv's
(lockfiles are automatically updated for you when
input requirements change). The latter is desirable, but
again this is for the future.

The general thrust of the changes is:

- Add uv as a third lockfile format (after pex and the deprecated
  legacy constraints file "locking")
- Update the lockfile metadata to include the format (and
  also the name of the resolve that created the lockfile).
- Make lockfile metadata reading much more robust 
  and useful, so we can get the format from it reliably.
  Unfortunately this logic is tricky since it has to handle
  pex lockfiles both with and without separate metadata files.
  For uv we simply require separate metadata files and do
  not add any metadata inside the lockfile itself, as we
  foolishly did for pex lockfiles at one point.
- Update the lockfile diffing code to support diffing
  across all combinations of old/new and pex/uv.
- Update the existing `uv` subsystem for more general use.
- Add src/python/pants/backend/python/util_rules/uv.py, 
  which runs `uv`, first generating ephemeral pyproject.toml
  and uv.toml config files. We handle environments correctly, 
  which requires some `realpath` shenanigans.
- Update some iffy logic in PythonToolBase that parses lockfiles
  to find useful info for help messages. It was previously relying
  on pex lockfile internals (which are not guaranteed to be stable)
and it now also relies on uv lockfile internals (similarly not stable).
  However now at least it fails gracefully if the lockfile parsing fails
   and just prints a less useful error message.
- Use all this in src/python/pants/backend/python/util_rules/pex.py, 
  instead of the previous, uv pex building solution.

Note that this supersedes the more limited (but much appreciated)
change in #23197 that implemented only the uv venv -> pex part.
It also supersedes the competing pending changes in #22949 
and #23212, which implemented partial support for uv resolution.
@benjyw

benjyw commented May 5, 2026

Copy link
Copy Markdown
Contributor

Closing since #23302 is now merged. We'd love your feedback on that one. Thanks!

@benjyw benjyw closed this May 5, 2026
benjyw added a commit that referenced this pull request May 6, 2026
…ick of #23302) (#23320)

uv lockfiles are first-class entities that users check in.
When a pex is needed, we use uv to create a venv, which
it does very efficiently, and then use pex's
`--venv-repository` feature to build a pex from
(a subset of) that venv's requirements.

Note that we still create VenvPexes to run tools, and
in some cases this might be unnecessary, since we
already have the uv venv. But utilizing those venvs 
directly would be an even bigger change (and this one
is already massive), and there are caching tradeoffs
to figure out . So we leave that for the future (and
probably for the new python  backend).

Note also that the dynamics of lockfile generation remain
pants's (you must generate explicitly) and not uv's
(lockfiles are automatically updated for you when
input requirements change). The latter is desirable, but
again this is for the future.

The general thrust of the changes is:

- Add uv as a third lockfile format (after pex and the deprecated
  legacy constraints file "locking")
- Update the lockfile metadata to include the format (and
  also the name of the resolve that created the lockfile).
- Make lockfile metadata reading much more robust 
  and useful, so we can get the format from it reliably.
  Unfortunately this logic is tricky since it has to handle
  pex lockfiles both with and without separate metadata files.
  For uv we simply require separate metadata files and do
  not add any metadata inside the lockfile itself, as we
  foolishly did for pex lockfiles at one point.
- Update the lockfile diffing code to support diffing
  across all combinations of old/new and pex/uv.
- Update the existing `uv` subsystem for more general use.
- Add src/python/pants/backend/python/util_rules/uv.py, 
  which runs `uv`, first generating ephemeral pyproject.toml
  and uv.toml config files. We handle environments correctly, 
  which requires some `realpath` shenanigans.
- Update some iffy logic in PythonToolBase that parses lockfiles
  to find useful info for help messages. It was previously relying
  on pex lockfile internals (which are not guaranteed to be stable)
and it now also relies on uv lockfile internals (similarly not stable).
  However now at least it fails gracefully if the lockfile parsing fails
   and just prints a less useful error message.
- Use all this in src/python/pants/backend/python/util_rules/pex.py, 
  instead of the previous, uv pex building solution.

Note that this supersedes the more limited (but much appreciated)
change in #23197 that implemented only the uv venv -> pex part.
It also supersedes the competing pending changes in #22949 
and #23212, which implemented partial support for uv resolution.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants