Reuse build (virtual) environments across resolution and installation by charliermarsh · Pull Request #14338 · astral-sh/uv

charliermarsh · 2025-06-27T23:41:42Z

Summary

The basic idea here is that we can (should) reuse a build environment across resolution (prepare_metadata_for_build_wheel) and installation. This also happens to solve the build-PyTorch-from-source problem, since we use a consistent build environment between the invocations.

Since SourceDistributionBuilder is stateless, we instead store the builds on BuildContext, and we key them by various properties: the underlying interpreter, the configuration settings, etc. This just ensures that if we build the same package twice within a process, we don't accidentally reuse an incompatible build (virtual) environment. (Note that still drop build environments at the end of the command, and don't attempt to reuse them across processes.)

Closes #14269.

charliermarsh · 2025-06-27T23:51:24Z

(Converting to draft while I review test failures.)

charliermarsh · 2025-06-28T00:40:45Z

There's a deadlock somewhere related to --no-binary but having a lot of trouble figuring out where.

charliermarsh · 2025-06-28T01:23:07Z

Okay, tests passing.

konstin · 2025-07-01T11:13:23Z

crates/uv-distribution/src/source/mod.rs

+
+            if let Some(builder) = self.build_context.build_arena().remove(&build_key) {
+                debug!("Creating build environment for: {source}");
+                builder.wheel(temp_dir.path()).await.map_err(Error::Build)?


After we removed it, should we re-add in case we need it again? We could have prepare_metadata_for_build_wheel, then get_requires_for_build_wheel and then build_wheel, which would be three steps.

(I want to keep the removing over borrowing/sharing as it safeguards against accidental parallelism)

Yeah this makes sense. (I didn't want to borrow for the same reason.)

konstin · 2025-07-01T14:02:50Z

crates/uv-distribution/src/source/mod.rs

+                let wheel = builder.wheel(temp_dir.path()).await.map_err(Error::Build)?;
+
+                // Store the build context.
+                self.build_context
+                    .build_arena()
+                    .insert(build_key, builder);
+
+                wheel


nit: we can move the duplicated code below the if let.

Which code?

This MR contains the following updates: | Package | Update | Change | |---|---|---| | [astral-sh/uv](https://github.com/astral-sh/uv) | patch | `0.7.16` -> `0.7.19` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>astral-sh/uv (astral-sh/uv)</summary> ### [`v0.7.19`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0719) [Compare Source](astral-sh/uv@0.7.18...0.7.19) The **[uv build backend](https://docs.astral.sh/uv/concepts/build-backend/) is now stable**, and considered ready for production use. The uv build backend is a great choice for pure Python projects. It has reasonable defaults, with the goal of requiring zero configuration for most users, but provides flexible configuration to accommodate most Python project structures. It integrates tightly with uv, to improve messaging and user experience. It validates project metadata and structures, preventing common mistakes. And, finally, it's very fast — `uv sync` on a new project (from `uv init`) is 10-30x faster than with other build backends. To use uv as a build backend in an existing project, add `uv_build` to the `[build-system]` section in your `pyproject.toml`: ```toml [build-system] requires = ["uv_build>=0.7.19,<0.8.0"] build-backend = "uv_build" ``` In a future release, it will replace `hatchling` as the default in `uv init`. As before, uv will remain compatible with all standards-compliant build backends. ##### Python - Add PGO distributions of Python for aarch64 Linux, which are more optimized for better performance See the [python-build-standalone release](https://github.com/astral-sh/python-build-standalone/releases/tag/20250702) for more details. ##### Enhancements - Ignore Python patch version for `--universal` pip compile ([#14405](astral-sh/uv#14405)) - Update the tilde version specifier warning to include more context ([#14335](astral-sh/uv#14335)) - Clarify behavior and hint on tool install when no executables are available ([#14423](astral-sh/uv#14423)) ##### Bug fixes - Make project and interpreter lock acquisition non-fatal ([#14404](astral-sh/uv#14404)) - Includes `sys.prefix` in cached environment keys to avoid `--with` collisions across projects ([#14403](astral-sh/uv#14403)) ##### Documentation - Add a migration guide from pip to uv projects ([#12382](astral-sh/uv#12382)) ### [`v0.7.18`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0718) [Compare Source](astral-sh/uv@0.7.17...0.7.18) ##### Python - Added arm64 Windows Python 3.11, 3.12, 3.13, and 3.14 These are not downloaded by default, since x86-64 Python has broader ecosystem support on Windows. However, they can be requested with `cpython-<version>-windows-aarch64`. See the [python-build-standalone release](https://github.com/astral-sh/python-build-standalone/releases/tag/20250630) for more details. ##### Enhancements - Keep track of retries in `ManagedPythonDownload::fetch_with_retry` ([#14378](astral-sh/uv#14378)) - Reuse build (virtual) environments across resolution and installation ([#14338](astral-sh/uv#14338)) - Improve trace message for cached Python interpreter query ([#14328](astral-sh/uv#14328)) - Use parsed URLs for conflicting URL error message ([#14380](astral-sh/uv#14380)) ##### Preview features - Ignore invalid build backend settings when not building ([#14372](astral-sh/uv#14372)) ##### Bug fixes - Fix equals-star and tilde-equals with `python_version` and `python_full_version` ([#14271](astral-sh/uv#14271)) - Include the canonical path in the interpreter query cache key ([#14331](astral-sh/uv#14331)) - Only drop build directories on program exit ([#14304](astral-sh/uv#14304)) - Error instead of panic on conflict between global and subcommand flags ([#14368](astral-sh/uv#14368)) - Consistently normalize trailing slashes on URLs with no path segments ([#14349](astral-sh/uv#14349)) ##### Documentation - Add instructions for publishing to JFrog's Artifactory ([#14253](astral-sh/uv#14253)) - Edits to the build backend documentation ([#14376](astral-sh/uv#14376)) ### [`v0.7.17`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0717) [Compare Source](astral-sh/uv@0.7.16...0.7.17) ##### Bug fixes - Apply build constraints when resolving `--with` dependencies ([#14340](astral-sh/uv#14340)) - Drop trailing slashes when converting index URL from URL ([#14346](astral-sh/uv#14346)) - Ignore `UV_PYTHON_CACHE_DIR` when empty ([#14336](astral-sh/uv#14336)) - Fix error message ordering for `pyvenv.cfg` version conflict ([#14329](astral-sh/uv#14329)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).

charliermarsh force-pushed the charlie/shared-builds branch from b4c2b4b to de2023c Compare June 27, 2025 23:43

charliermarsh marked this pull request as ready for review June 27, 2025 23:46

charliermarsh temporarily deployed to uv-test-registries June 27, 2025 23:46 — with GitHub Actions Inactive

charliermarsh requested review from konstin and zanieb June 27, 2025 23:50

charliermarsh added the enhancement New feature or improvement to existing functionality label Jun 27, 2025

charliermarsh marked this pull request as draft June 27, 2025 23:51

charliermarsh temporarily deployed to uv-test-registries June 28, 2025 00:16 — with GitHub Actions Inactive

charliermarsh force-pushed the charlie/shared-builds branch from f1b6aaf to 490cebf Compare June 28, 2025 00:22

charliermarsh temporarily deployed to uv-test-registries June 28, 2025 00:25 — with GitHub Actions Inactive

charliermarsh temporarily deployed to uv-test-registries June 28, 2025 00:47 — with GitHub Actions Inactive

charliermarsh force-pushed the charlie/shared-builds branch 3 times, most recently from c0ee25c to 27030b6 Compare June 28, 2025 01:17

charliermarsh temporarily deployed to uv-test-registries June 28, 2025 01:19 — with GitHub Actions Inactive

charliermarsh marked this pull request as ready for review June 28, 2025 01:23

charliermarsh force-pushed the charlie/shared-builds branch from 27030b6 to e4f4e13 Compare June 28, 2025 01:40

charliermarsh temporarily deployed to uv-test-registries June 28, 2025 01:43 — with GitHub Actions Inactive

konstin mentioned this pull request Jul 1, 2025

Only drop build directories on program exit #14304

Merged

konstin reviewed Jul 1, 2025

View reviewed changes

charliermarsh force-pushed the charlie/shared-builds branch from e4f4e13 to 67db2fd Compare July 1, 2025 14:01

konstin approved these changes Jul 1, 2025

View reviewed changes

charliermarsh had a problem deploying to uv-test-registries July 1, 2025 14:04 — with GitHub Actions Error

charliermarsh force-pushed the charlie/shared-builds branch from 67db2fd to 1c1e91e Compare July 1, 2025 14:04

charliermarsh temporarily deployed to uv-test-registries July 1, 2025 14:12 — with GitHub Actions Inactive

charliermarsh added 2 commits July 1, 2025 12:50

Use builds across lock and solve

412374f

Drop lock temporarily

8c5f5ef

charliermarsh force-pushed the charlie/shared-builds branch from 1c1e91e to 8c5f5ef Compare July 1, 2025 16:50

charliermarsh temporarily deployed to uv-test-registries July 1, 2025 16:52 — with GitHub Actions Inactive

charliermarsh merged commit d9f9ed4 into main Jul 1, 2025
136 checks passed

charliermarsh deleted the charlie/shared-builds branch July 1, 2025 17:15

BrewTestBot mentioned this pull request Jul 1, 2025

uv 0.7.18 Homebrew/homebrew-core#228785

Merged

charliermarsh mentioned this pull request Jul 3, 2025

Using uv pip install to install pytorch from its source code fails, while pip install works. #11184

Closed

thalassemia mentioned this pull request Jan 30, 2026

Source redundant environment variables from single script CovertLab/vEcoli#375

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

Reuse build (virtual) environments across resolution and installation#14338

Reuse build (virtual) environments across resolution and installation#14338
charliermarsh merged 2 commits intomainfrom
charlie/shared-builds

charliermarsh commented Jun 27, 2025 •

edited

Loading

Uh oh!

charliermarsh commented Jun 27, 2025

Uh oh!

charliermarsh commented Jun 28, 2025

Uh oh!

charliermarsh commented Jun 28, 2025

Uh oh!

konstin Jul 1, 2025

Uh oh!

charliermarsh Jul 1, 2025

Uh oh!

charliermarsh Jul 1, 2025

Uh oh!

konstin Jul 1, 2025

Uh oh!

charliermarsh Jul 1, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Comments

Conversation

charliermarsh commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

charliermarsh commented Jun 27, 2025

Uh oh!

charliermarsh commented Jun 28, 2025

Uh oh!

charliermarsh commented Jun 28, 2025

Uh oh!

konstin Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

charliermarsh Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

charliermarsh Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

konstin Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

charliermarsh Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

charliermarsh commented Jun 27, 2025 •

edited

Loading