Add experimental uv-based PEX builder ([python].pex_builder)#23197
Conversation
jsirois
left a comment
There was a problem hiding this comment.
Thanks for taking this up @seungwoo-ji-03. I hope you have better luck than previous folks attempting to un-stick Pants here. I encourage you to push hard on Pants reviewers to pay attention.
I don't use / maintain Pants, but I maintain Pex and added the --venv-repository feature you're using here exactly for this purpose; so I left a note about how you could take this further if you wish in follow-ups.
|
|
||
| use_uv_builder = python_setup.pex_builder == PexBuilder.uv | ||
| # uv builder only applies to non-internal PEXes with requirements and a | ||
| # local interpreter (not cross-platform builds). |
There was a problem hiding this comment.
FYI: You can do foreign platform builds with uv + Pex. You can also do multi-platform PEX builds (You can create multiple venvs and specify --venv-repository more than once when building the PEX). I demonstrate a single foreign platform PEX build below:
# I'm on x86_64 Linux:
:; uname -om
x86_64 GNU/Linux
# Build a venv for aarch64 linux using uv:
:; rm -rf /tmp/example/
:; uv venv --python 3.13 --no-project /tmp/example
Using CPython 3.13.12 interpreter at: /home/jsirois/.pyenv/versions/3.13.12/bin/python3.13
Creating virtual environment at: /tmp/example
Activate with: source /tmp/example/bin/activate
:; uv pip install --prefix /tmp/example --python 3.13 --python-platform aarch64-unknown-linux-gnu psutil cowsay
Using CPython 3.13.12 interpreter at: /home/jsirois/.pyenv/versions/3.13.12/bin/python3.13
Resolved 2 packages in 1ms
░░░░░░░░░░░░░░░░░░░░ [0/2] Installing wheels... warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance.
If the cache and target directories are on different filesystems, hardlinking may not be supported.
If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning.
Installed 2 packages in 2ms
+ cowsay==6.1
+ psutil==7.2.2
# Use that foreign platform venv to create a foreign platform PEX:
:; pex --venv-repository /tmp/example/ -o example.pex
example.pex
# Prove it works:
:; docker run --rm --platform linux/arm64 -v $PWD/example.pex:/example.pex python:3.13 python /example.pex -c 'import cowsay, psutil; cowsay.tux(str(psutil.Process()))'
_________________________________________________
/ \
| psutil.Process(pid=1, name='python', status='runn |
| ing') |
\ /
=================================================
\
\
\
.--.
|o_o |
|:_/ |
// \ \
(| | )
/'\_ _/`\
\___)=(___/
:; unzip -qc example.pex PEX-INFO | jq .distributions
{
"cowsay-6.1-py3-none-any.whl": "cb195ef66765a3fa4c4a1243417d30d04849faa0d2de8eda533b807d59da0e50",
"psutil-7.2.2-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_28_aarch64.whl": "ac787d25959dbae14884467d31b49fe20915ae22c8cc10793c93fae2243ef470"
}There was a problem hiding this comment.
Thanks for the detailed example @jsirois and for building --venv-repository in the first place, it's exactly what makes this work!
Cross-platform support via uv pip install --python-platform and multiple --venv-repository flags looks great. If Pants reviewers think it fits in this PR's scope, I'm happy to add it here — otherwise it'd be a natural follow-up.
There was a problem hiding this comment.
Let's do this in a follow up.
|
Thanks for this! There is an existing PR for integrating In fact, a future follow up to this PR could support uv lockfiles directly, which would obviate that other PR, but one thing at a time. I will review this one orthogonally to that one though. Meanwhile please note our new AI disclosure policy in case it is relevant to this change. Thanks! |
Thanks @benjyw! I looked through #22949 before starting — the scope is different enough so I proceeded independently. Happy to rebase whichever lands second. LLM disclosure added to the PR description. Thanks for the pointer. |
benjyw
left a comment
There was a problem hiding this comment.
Thanks again for this impactful change. See my comments in line.
|
|
||
| use_uv_builder = python_setup.pex_builder == PexBuilder.uv | ||
| # uv builder only applies to non-internal PEXes with requirements and a | ||
| # local interpreter (not cross-platform builds). |
There was a problem hiding this comment.
Let's do this in a follow up.
| if not req_strings: | ||
| logger.debug( | ||
| "pex_builder=uv: no individual requirement strings for %s " | ||
| "(e.g. using a whole-lockfile resolve or no third-party deps). " |
There was a problem hiding this comment.
We shouldn't need to fall back to PEX/pip in this case. If we're in the WholeLockfile situation, then we can get the entire set of input reqs from the lockfile metadata (in generated_with_requirements). And if there are truly no requirements, then that is a trivial case.
Fine to do this in a followup though.
|
Thanks for the quick and thorough review @benjyw! All feedback has been addressed except the two items marked as follow-ups And thanks for maintaining such a great project! |
benjyw
left a comment
There was a problem hiding this comment.
Looks great! Just one more comment and I think this will be ready to merge.
|
Such amazing news to see this happen! Looking forward to the next pants release to try it out / test |
|
Great! I will merge on green. |
|
Note that this is superseded by the more comprehensive solution in #23302. Thanks for all the effort on this front, and for the prodding to finally get this done properly. |
uv lockfiles are first-class entities that users check in. When a pex is needed, we use uv to create a venv, which it does very efficiently, and then use pex's `--venv-repository` feature to build a pex from (a subset of) that venv's requirements. Note that we still create VenvPexes to run tools, and in some cases this might be unnecessary, since we already have the uv venv. But utilizing those venvs directly would be an even bigger change (and this one is already massive), and there are caching tradeoffs to figure out . So we leave that for the future (and probably for the new python backend). Note also that the dynamics of lockfile generation remain pants's (you must generate explicitly) and not uv's (lockfiles are automatically updated for you when input requirements change). The latter is desirable, but again this is for the future. The general thrust of the changes is: - Add uv as a third lockfile format (after pex and the deprecated legacy constraints file "locking") - Update the lockfile metadata to include the format (and also the name of the resolve that created the lockfile). - Make lockfile metadata reading much more robust and useful, so we can get the format from it reliably. Unfortunately this logic is tricky since it has to handle pex lockfiles both with and without separate metadata files. For uv we simply require separate metadata files and do not add any metadata inside the lockfile itself, as we foolishly did for pex lockfiles at one point. - Update the lockfile diffing code to support diffing across all combinations of old/new and pex/uv. - Update the existing `uv` subsystem for more general use. - Add src/python/pants/backend/python/util_rules/uv.py, which runs `uv`, first generating ephemeral pyproject.toml and uv.toml config files. We handle environments correctly, which requires some `realpath` shenanigans. - Update some iffy logic in PythonToolBase that parses lockfiles to find useful info for help messages. It was previously relying on pex lockfile internals (which are not guaranteed to be stable) and it now also relies on uv lockfile internals (similarly not stable). However now at least it fails gracefully if the lockfile parsing fails and just prints a less useful error message. - Use all this in src/python/pants/backend/python/util_rules/pex.py, instead of the previous, uv pex building solution. Note that this supersedes the more limited (but much appreciated) change in #23197 that implemented only the uv venv -> pex part. It also supersedes the competing pending changes in #22949 and #23212, which implemented partial support for uv resolution.
…ick of #23302) (#23320) uv lockfiles are first-class entities that users check in. When a pex is needed, we use uv to create a venv, which it does very efficiently, and then use pex's `--venv-repository` feature to build a pex from (a subset of) that venv's requirements. Note that we still create VenvPexes to run tools, and in some cases this might be unnecessary, since we already have the uv venv. But utilizing those venvs directly would be an even bigger change (and this one is already massive), and there are caching tradeoffs to figure out . So we leave that for the future (and probably for the new python backend). Note also that the dynamics of lockfile generation remain pants's (you must generate explicitly) and not uv's (lockfiles are automatically updated for you when input requirements change). The latter is desirable, but again this is for the future. The general thrust of the changes is: - Add uv as a third lockfile format (after pex and the deprecated legacy constraints file "locking") - Update the lockfile metadata to include the format (and also the name of the resolve that created the lockfile). - Make lockfile metadata reading much more robust and useful, so we can get the format from it reliably. Unfortunately this logic is tricky since it has to handle pex lockfiles both with and without separate metadata files. For uv we simply require separate metadata files and do not add any metadata inside the lockfile itself, as we foolishly did for pex lockfiles at one point. - Update the lockfile diffing code to support diffing across all combinations of old/new and pex/uv. - Update the existing `uv` subsystem for more general use. - Add src/python/pants/backend/python/util_rules/uv.py, which runs `uv`, first generating ephemeral pyproject.toml and uv.toml config files. We handle environments correctly, which requires some `realpath` shenanigans. - Update some iffy logic in PythonToolBase that parses lockfiles to find useful info for help messages. It was previously relying on pex lockfile internals (which are not guaranteed to be stable) and it now also relies on uv lockfile internals (similarly not stable). However now at least it fails gracefully if the lockfile parsing fails and just prints a less useful error message. - Use all this in src/python/pants/backend/python/util_rules/pex.py, instead of the previous, uv pex building solution. Note that this supersedes the more limited (but much appreciated) change in #23197 that implemented only the uv venv -> pex part. It also supersedes the competing pending changes in #22949 and #23212, which implemented partial support for uv resolution.
Closes #20679
Add an experimental
[python].pex_builderoption that allows usinguv to install dependencies when
building PEX files via
pants package.When set to
"uv", Pants:uvbinary (as anExternalTool).uv venv.uv pip install.--venv-repository.When a PEX-native lockfile is available, uv installs the exact pinned
versions with
--no-deps, preserving reproducibility. Otherwise itfalls back to transitive resolution from requirement strings.
Builds that cannot use uv (internal-only, cross-platform, no local
interpreter) silently fall back to the default pip path.
Benchmark
Raw
pip installvsuv pip installfor 11 packages (requests, boto3,cryptography, aiohttp, sqlalchemy, pillow, etc.) measured with
hyperfine:
Within Pants, the end-to-end improvement is smaller because scheduler
and bootstrap overhead dominates, but the dependency installation step
itself is significantly faster — especially with warm caches on
repeated builds.
LLM Disclosure
Code was written by the author. Claude was used for code review, catching edge cases, and verifying test coverage.