Support for uv lockfiles as an alternative to pex lockfiles by benjyw · Pull Request #23302 · pantsbuild/pants

benjyw · 2026-04-30T18:37:51Z

uv lockfiles are first-class entities that users check in.
When a pex is needed, we use uv to create a venv, which
it does very efficiently, and then use pex's
--venv-repository feature to build a pex from
(a subset of) that venv's requirements.

Note that we still create VenvPexes to run tools, and
in some cases this might be unnecessary, since we
already have the uv venv. But utilizing those venvs
directly would be an even bigger change (and this one
is already massive), and there are caching tradeoffs
to figure out . So we leave that for the future (and
probably for the new python backend).

Note also that the dynamics of lockfile generation remain
pants's (you must generate explicitly) and not uv's
(lockfiles are automatically updated for you when
input requirements change). The latter is desirable, but
again this is for the future.

The general thrust of the changes is:

Add uv as a third lockfile format (after pex and the deprecated
legacy constraints file "locking")
Update the lockfile metadata to include the format (and
also the name of the resolve that created the lockfile).
Make lockfile metadata reading much more robust
and useful, so we can get the format from it reliably.
Unfortunately this logic is tricky since it has to handle
pex lockfiles both with and without separate metadata files.
For uv we simply require separate metadata files and do
not add any metadata inside the lockfile itself, as we
foolishly did for pex lockfiles at one point.
Update the lockfile diffing code to support diffing
across all combinations of old/new and pex/uv.
Update the existing uv subsystem for more general use.
Add src/python/pants/backend/python/util_rules/uv.py,
which runs uv, first generating ephemeral pyproject.toml
and uv.toml config files. We handle environments correctly,
which requires some realpath shenanigans.
Update some iffy logic in PythonToolBase that parses lockfiles
to find useful info for help messages. It was previously relying
on pex lockfile internals (which are not guaranteed to be stable)
and it now also relies on uv lockfile internals (similarly not stable).
However now at least it fails gracefully if the lockfile parsing fails
and just prints a less useful error message.
Use all this in src/python/pants/backend/python/util_rules/pex.py,
instead of the previous, uv pex building solution.

Note that this supersedes the more limited (but much appreciated)
change in #23197 that implemented only the uv venv -> pex part.
It also supersedes the competing pending changes in #22949
and #23212, which implemented partial support for uv resolution.

benjyw · 2026-04-30T18:40:06Z

Reviewers: apologies for the massiveness. Happy to pair on the review if that is helpful. Breaking this up into smaller chunks would have added even more complexity.

The proof of this pudding is the green CI for #23293, which regenerates all our internal and default lockfiles to be uv lockfiles.

benjyw · 2026-04-30T18:41:07Z

question - do we want to publicly deem this "experimental"? I couldn't think of a practical advantage to doing so. We're not going to undo this, only fix whatever issues users find, so it isn't meaningfully experimental, just new.

jsirois · 2026-04-30T20:22:12Z

                extra_args=argv,
                additional_input_digest=merged_digest,
                description=_build_pex_description(request, req_strings, python_setup.resolves),
+                append_only_caches=venv_repo.append_only_caches() if venv_repo else None,


This cannot be correct for multi-platform pex_binary targets - you need 1 venv repo per platform (c.f. uv's --python-platform) . Is a different code path used to generate pex_binary PEXes?

I demoed this here: #20679 (comment)

No doubt in other threads too.

Multi-platform pex_binary are not yet supported. I plan that for a followup. For now we error if complete_platforms are passed:

https://github.com/benjyw/pants/blob/uv_lockfile/src/python/pants/backend/python/util_rules/pex.py#L791

Although this will actually be caught at an earlier validation step:

https://github.com/benjyw/pants/blob/uv_lockfile/src/python/pants/backend/python/util_rules/pex_requirements.py#L530

I will mention this in the release notes.

Aha, OK. I guess that means lambdas too. 1 platform, but, IIRC, uses complete-platform.

Yes, that's correct. Either user-supplied or default ones we bundle for the known lambda platforms. I will call that out in the release notes.

It shouldn't be hard to support (thanks to your demos), but this was already a big change so I decided to push it to a followup.

sureshjoshi · 2026-04-30T23:48:03Z

question - do we want to publicly deem this "experimental"? I couldn't think of a practical advantage to doing so. We're not going to undo this, only fix whatever issues users find, so it isn't meaningfully experimental, just new.

Is this intended to be cherry picked into 2.32? Or stay on 2.33?

benjyw · 2026-04-30T23:56:50Z

question - do we want to publicly deem this "experimental"? I couldn't think of a practical advantage to doing so. We're not going to undo this, only fix whatever issues users find, so it isn't meaningfully experimental, just new.

Is this intended to be cherry picked into 2.32? Or stay on 2.33?

Cherrypicked into 2.32, so that the other, limited uv change doesn't make it into a release.

benjyw · 2026-05-02T04:59:01Z

Addressed comments (and also offline comment on Slack from @sureshjoshi about disambiguating cached venvs by buildroots).

PTAL!

tdyas · 2026-05-02T15:35:16Z


+A new `[python].resolver` option allows using [uv](https://github.com/astral-sh/uv) lockfiles instead of
+pex lockfiles. uv's resolver is substantially faster than pip's, so this should support much better
+`generate-lockfiles` times. PEXes are still created as needed, for internal and external uses, by first letting


Capitalize for?

sureshjoshi · 2026-05-02T18:30:14Z

Blocking merge here for now.

I’ve been doing some tests today on my machine. There is still -something- wrong here, but I’m not quite sure what is causing it yet, or even whether it’s new or not, or if I’m chasing ghosts.

I used the previous PR (before the cache tweak), and saw that there was some hard to explain inconsistency - where I could generate-lockfiles ::, I see the console diff claiming I have a new version, but my local .lock and .lock.metadata weren’t updated. I run my code, and it’s using the old version of my dep.

So, in this case, the runtime is consistent with the lockfile, which is “good”. It’s wrong that re-generating the lockfiles failed to correctly show, but at least lockfile == runtime dep is a safe-ish failure (while it still sucks).

I pulled the latest merge, and now, trying to run a similar test, the console diff is showing the updated dep, the lockfiles are showing the updated dep, but runtime is using the old dep (somehow).

I need to dig into a reproducible workflow and whether this is the case today in 2.30.2 or not. This latter problem is weird, because this feels like something that should be caught in a unit test already, so need to check whether this has to do with the two side-by-side (almost identical) repos I had going at the same time.

Rough example workflow (I had two repos side-by-side with the same deps, so unsure if that’s related yet - but that’s the workflow I wanted to see pass/fail). Will try to clean these notes up into a bash file or Github repo when I can get to a computer again later today (writing from my phone, bear with me on typos).

Setup:

pants 2.30.2
enable_resolves=True
requirements.txt: fastapi==0.100.0
main.py: print(fastapi.__version__)

PANTS_SOURCE=… pants generate_lockfiles ::
# lockfiles updated to 0.100.0
PANTS_SOURCE=… pants run main.py
# see 0.100.0

# Update to fastapi==0.101.0
PANTS_SOURCE=… pants generate_lockfiles ::
# lockfiles updated to 0.101.0
PANTS_SOURCE=… pants run main.py
# see 0.101.0

# Update to fastapi (no value)
PANTS_SOURCE=… pants generate_lockfiles ::
# lockfiles updated to 0.136.0
PANTS_SOURCE=… pants run main.py
# see 0.101.0 <——-

Running Linux x86_64, Fedora Kinoite.

I don’t ever run unconstrained requirements in my real projects, so for all I know, this is somehow known behaviour. But, sure as hell surprising to me.

benjyw · 2026-05-02T18:49:40Z

That would be good to get a repro on. My manual testing didn't show anything similar.

sureshjoshi · 2026-05-02T19:50:12Z

Working on one. It took wiping the entire cache (after selectively wiping pex_root, parts of uv cache, etc).

I hope it’s not a bug with running PANTS_SOURCE as that would be a pain

sureshjoshi · 2026-05-02T20:02:47Z

Alright, annoyingly, I wiped the repos, wiped the caches, killed all daemons, etc… And now my reproduction doesn’t reproduce the error anymore.

It’s possible it was still running off the pre-cache-fix, though I don’t know how, as I pulled it and verified I was on the correct commit.

sureshjoshi · 2026-05-02T20:18:34Z

Alright, I can’t repro.

That’s it for me for now. I have a bunch of other smaller issues/nits/what-if’s (around uv interpreter constraints, locking, and other random things), but those require more testing that I don’t have time for this weekend.

Not sure if Chris has put his feedback in, but I think we’d be better getting this in people’s hands and testing whether their systems break, moreso than more static reviews and small one-off tests.

cburroughs

To make sure I have the fundamentals right, this takes a "coupled" approach where:

pex makes venvs from pex lockfiles
uv makes makes venvs from uv lockfiles (and then hands off to pex)

As opposed to a mix and match approach like #23197 where a pex lockfile could be passed to uv. Do I have that right?

Regarding "experimental"-ness and [python].resolver I'm a unsure what this
would look like if pylock files ever took off as a "native" format and there
was a matrix of resovlers x lockfile-formats people wanted to use. I don't want to over-engineer for a case that might never happen and am not sure if there is anything more future-proof here. (I'm not seeing [python].lockfile_format being clearly better for example.)

Otherwise I think language in the notes over the config name is satisfactory.

benjyw · 2026-05-04T23:32:14Z

To make sure I have the fundamentals right, this takes a "coupled" approach where:

pex makes venvs from pex lockfiles

uv makes makes venvs from uv lockfiles (and then hands off to pex)

Yes, exactly. The pex handoff uses `--venv-repository.

As opposed to a mix and match approach like #23197 where a pex lockfile could be passed to uv. Do I have that right?

Sort of? Depends what you mean by "passed to"...

The mix-and-match approach is in #23197 was "convert a pex lockfile to a fully-pinned set of requirements, use uv to create a venv from those, and then build the pex with --venv-repository pointing to that venv.

Then there is another mix-and-max approach on the front side (as in #22949 and #23212) which is to use uv for lockfile generation, but then immediately turn the uv lockfile into a pex lockfile and discard the former.

Regarding "experimental"-ness and [python].resolver I'm a unsure what this would look like if pylock files ever took off as a "native" format and there was a matrix of resovlers x lockfile-formats people wanted to use. I don't want to over-engineer for a case that might never happen and am not sure if there is anything more future-proof here. (I'm not seeing [python].lockfile_format being clearly better for example.)

Otherwise I think language in the notes over the config name is satisfactory.

cburroughs

I think there are a dozen ways we could do this, but the "coupled" approach has fewer sharp edges or things that won't "magically work". 🚢

benjyw · 2026-05-05T20:40:11Z

I think there are a dozen ways we could do this, but the "coupled" approach has fewer sharp edges or things that won't "magically work". 🚢

Yeah, I think running tools directly from uv venvs instead of creating pexes would be a possible next step.

benjyw · 2026-05-05T21:33:59Z

OK, going to merge now, and iterate as needed. Best to get this in users' hands and get real world feedback.

Thanks for the reviews!

WorkerPants · 2026-05-05T21:35:14Z

I tried to automatically cherry-pick this change back to each relevant milestone, so that it is available in those older releases of Pants.

❌ 2.32.x

I was unable to cherry-pick this PR to 2.32.x, likely due to merge-conflicts.

Steps to Cherry-Pick locally

To resolve:

(Ensure your git working directory is clean)

Run the following script to reproduce the merge-conflicts:

git fetch https://github.com/pantsbuild/pants main \
  && git fetch https://github.com/pantsbuild/pants 2.32.x \
  && git checkout -b cherry-pick-23302-to-2.32.x FETCH_HEAD \
  && git cherry-pick bdfc95dceb2d1d65409bcbae24dab048740a9375

Fix the merge conflicts and commit the changes
Run build-support/cherry_pick/make_pr.sh "23302" "2.32.x"

Please note that I cannot re-run CI if a job fails. Please work with your PR approver(s) to re-run CI if necessary.

When you're done manually cherry-picking, please remove the needs-cherrypick label on this PR.

Thanks again for your contributions!

🤖 Beep Boop here's my run link

…ick of #23302) (#23320) uv lockfiles are first-class entities that users check in. When a pex is needed, we use uv to create a venv, which it does very efficiently, and then use pex's `--venv-repository` feature to build a pex from (a subset of) that venv's requirements. Note that we still create VenvPexes to run tools, and in some cases this might be unnecessary, since we already have the uv venv. But utilizing those venvs directly would be an even bigger change (and this one is already massive), and there are caching tradeoffs to figure out . So we leave that for the future (and probably for the new python backend). Note also that the dynamics of lockfile generation remain pants's (you must generate explicitly) and not uv's (lockfiles are automatically updated for you when input requirements change). The latter is desirable, but again this is for the future. The general thrust of the changes is: - Add uv as a third lockfile format (after pex and the deprecated legacy constraints file "locking") - Update the lockfile metadata to include the format (and also the name of the resolve that created the lockfile). - Make lockfile metadata reading much more robust and useful, so we can get the format from it reliably. Unfortunately this logic is tricky since it has to handle pex lockfiles both with and without separate metadata files. For uv we simply require separate metadata files and do not add any metadata inside the lockfile itself, as we foolishly did for pex lockfiles at one point. - Update the lockfile diffing code to support diffing across all combinations of old/new and pex/uv. - Update the existing `uv` subsystem for more general use. - Add src/python/pants/backend/python/util_rules/uv.py, which runs `uv`, first generating ephemeral pyproject.toml and uv.toml config files. We handle environments correctly, which requires some `realpath` shenanigans. - Update some iffy logic in PythonToolBase that parses lockfiles to find useful info for help messages. It was previously relying on pex lockfile internals (which are not guaranteed to be stable) and it now also relies on uv lockfile internals (similarly not stable). However now at least it fails gracefully if the lockfile parsing fails and just prints a less useful error message. - Use all this in src/python/pants/backend/python/util_rules/pex.py, instead of the previous, uv pex building solution. Note that this supersedes the more limited (but much appreciated) change in #23197 that implemented only the uv venv -> pex part. It also supersedes the competing pending changes in #22949 and #23212, which implemented partial support for uv resolution.

uv lockfile support.

0c38d5d

benjyw requested review from cburroughs and sureshjoshi April 30, 2026 18:41

jsirois reviewed Apr 30, 2026

View reviewed changes

BUILD file fix, and notes update

270ec7c

sureshjoshi reviewed Apr 30, 2026

View reviewed changes

Comment thread src/python/pants/backend/helm/subsystems/BUILD

sureshjoshi reviewed Apr 30, 2026

View reviewed changes

Comment thread src/python/pants/core/util_rules/system_binaries.py

A couple of test fixes, and notes update.

55f4809

sureshjoshi reviewed Apr 30, 2026

View reviewed changes

Comment thread build-support/bin/generate_builtin_lockfiles.py Outdated

This was referenced May 1, 2026

Add experimental uv resolver for Python lockfile generation #23212

Closed

Add experimental uv resolver for Python lockfiles #22949

Closed

Add experimental uv-based PEX builder ([python].pex_builder) #23197

Merged

sureshjoshi reviewed May 1, 2026

View reviewed changes

Comment thread build-support/bin/generate_builtin_lockfiles.py

sureshjoshi reviewed May 1, 2026

View reviewed changes

Comment thread src/python/pants/backend/python/subsystems/uv.py

benjyw added this to the 2.32.x milestone May 1, 2026

benjyw added the needs-cherrypick [CI] label May 1, 2026

sureshjoshi reviewed May 1, 2026

View reviewed changes

Comment thread src/python/pants/backend/python/util_rules/uv.py

tdyas reviewed May 1, 2026

View reviewed changes

benjyw added 2 commits May 1, 2026 21:12

Merge branch 'main' into uv_lockfile

915ca4c

Address code review comments

ddb51b5

benjyw requested review from sureshjoshi and tdyas May 2, 2026 14:02

tdyas approved these changes May 2, 2026

View reviewed changes

sureshjoshi requested changes May 2, 2026

View reviewed changes

sureshjoshi approved these changes May 2, 2026

View reviewed changes

Affanmir mentioned this pull request May 4, 2026

uv: support private-index auth via named indexes, extra_env_vars, and keyring-provider benjyw/pants#24

Open

4 tasks

cburroughs reviewed May 4, 2026

View reviewed changes

benjyw added 2 commits May 4, 2026 16:33

Merge branch 'main' into uv_lockfile

434670d

Code review comments

ef6123e

benjyw requested a review from cburroughs May 5, 2026 00:06

cburroughs approved these changes May 5, 2026

View reviewed changes

Whoops

2769d8b

benjyw merged commit bdfc95d into pantsbuild:main May 5, 2026
48 of 50 checks passed

WorkerPants added the auto-cherry-picking-failed [CI] Auto Cherry-Picking Failed label May 5, 2026

benjyw mentioned this pull request May 5, 2026

pants generate-lockfiles to use uv for dependency resolution #20679

Closed

benjyw removed needs-cherrypick [CI] auto-cherry-picking-failed [CI] Auto Cherry-Picking Failed labels May 5, 2026

Uh oh!

Uh oh!

Conversation

benjyw commented Apr 30, 2026

Uh oh!

benjyw commented Apr 30, 2026

Uh oh!

benjyw commented Apr 30, 2026

Uh oh!

jsirois Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

jsirois Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

benjyw Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

jsirois Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

benjyw Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

sureshjoshi commented Apr 30, 2026

Uh oh!

Uh oh!

Uh oh!

benjyw commented Apr 30, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

benjyw commented May 2, 2026

Uh oh!

tdyas May 2, 2026

Choose a reason for hiding this comment

Uh oh!

sureshjoshi commented May 2, 2026

Uh oh!

benjyw commented May 2, 2026

Uh oh!

sureshjoshi commented May 2, 2026

Uh oh!

sureshjoshi commented May 2, 2026

Uh oh!

sureshjoshi commented May 2, 2026

Uh oh!

cburroughs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

benjyw commented May 4, 2026

Uh oh!

cburroughs left a comment

Choose a reason for hiding this comment

Uh oh!

benjyw commented May 5, 2026

Uh oh!

benjyw commented May 5, 2026

Uh oh!

Uh oh!

WorkerPants commented May 5, 2026

❌ 2.32.x

Uh oh!

Reviewers