Support Git LFS with opt-in by samypr100 · Pull Request #16143 · astral-sh/uv

samypr100 · 2025-10-06T21:13:10Z

Summary

Follow up to #15563
Closes #13485

This is a first-pass at adding support for conditional support for Git LFS between git sources, initial feedback welcome.

e.g.

[tool.uv.sources]
test-lfs-repo = { git = "https://github.com/zanieb/test-lfs-repo.git", lfs = true }

For context previously a user had to set UV_GIT_LFS to have uv fetch lfs objects on git sources. This env var was all or nothing, meaning you must always have it set to get consistent behavior and it applied to all git sources. If you fetched lfs objects at a revision and then turned off lfs (or vice versa), the git db, corresponding checkout lfs artifacts would not be updated properly. Similarly, when git source distributions were built, there would be no distinction between sources with lfs and without lfs. Hence, it could corrupt the git, sdist, and archive caches.

In order to support some sources being LFS enabled and other not, this PR adds a stateful layer roughly similar to how subdirectory works but for lfs since the git database, the checkouts and the corresponding caching layers needed to be LFS aware (requested vs installed). The caches also had to isolated and treated entirely separate when handling LFS sources.

Summary

Adds lfs = true or lfs = false to git sources in pyproject.toml
Added lfs=true query param / fragments to most relevant url structs (not parsed as user input)
- In the case of uv add / uv tool, --lfs is supported instead
- UV_GIT_LFS environment variable support is still functional for non-project entrypoints (e.g. uv pip)
direct-url.json now has an custom git_lfs entry under VcsInfo (note, this is not in the spec currently -- see caveats).
git database and checkouts have an different cache key as the sources should be treated effectively different for the same rev.
sdists cache also differ in the cache key of a built distribution if it was built using LFS enabled revisions to distinguish between non-LFS same revisions. This ensures the strong assumption for archive-v0 that an unpacked revision "doesn't change sources" stays valid.

Caveats

pylock.toml import support has not been added via git_lfs=true, going through the spec it wasn't clear to me it's something we'd support outside of the env var (for now).
direct-url struct was modified by adding a non-standard git_lfs field under VcsInfo which may be undersirable although the PEP 610 does say Additional fields that would be necessary to support such VCS SHOULD be prefixed with the VCS command name which could be interpret this change as ok.
There will be a slight lockfile and cache churn for users that use UV_GIT_LFS as all git lockfile entries will get a lfs=true fragment. The cache version does not need an update, but LFS sources will get their own namespace under git-v0 and sdist-v9/git hence a cache-miss will occur once but this can be sufficient to label this as breaking for workflows always setting UV_GIT_LFS.

Test Plan

Some initial tests were added. More tests likely to follow as we reach consensus on a final approach.

For IT test, we may want to move to use a repo under astral namespace in order to test lfs functionality.

Manual testing was done for common pathological cases like killing LFS fetch mid-way, uninstalling LFS after installing an sdist with it and reinstalling, fetching LFS artifacts in different commits, etc.

PSA: Please ignore the docker build failures as its related to depot OIDC issues.

codspeed-hq · 2025-10-17T22:11:35Z

CodSpeed Performance Report

Merging #16143 will not alter performance

_{Comparing samypr100:conditional_lfs_support (dc1b6ba) with main (5947fb0)}

Summary

✅ 6 untouched

We're not very consistent in adding spans for git operations, but we should at least for new functions.

Add a script and associated CI check that checks whether a new uv build can use the cache of the previous uv version. This prevents accidental changes to cache keys, e.g. by changing nested data structures. I've tested the script by using a previous change of #16143. The check can be disabled with a PR label for PRs that change the cache layout. What's missing here is that the base is the last release, meaning that once a PR with that label merges, all following PRs will fail this check, as we currently don't have a good way to ask whether there was a change previously or to download the latest build binary from main as baseline.

konstin

Thank you!

Red-Eyed · 2025-12-03T06:17:02Z

Thank you!
Having worked with LFS, and I wonder how to deal with different LFS endpoints and credentials?
For example, at work we have GitHub enterprise and external git LFS server (not github)

samypr100 · 2025-12-03T14:34:53Z

Thank you! Having worked with LFS, and I wonder how to deal with different LFS endpoints and credentials? For example, at work we have GitHub enterprise and external git LFS server (not github)

@Red-Eyed That would still be configured as part of your git configuration. Since LFS objects have the originating url on them, as long as git itself can authenticate, fetch, and materialize the objects you shouldn't have issues with uv.

gsemet · 2025-12-06T10:15:56Z

Great ! Is there a way to use it when doing « uv tool install git+https//… »?

samypr100 · 2025-12-06T12:08:39Z

Great ! Is there a way to use it when doing « uv tool install git+https//… »?

uv tool install --lfs

This MR contains the following updates: | Package | Update | Change | |---|---|---| | [astral-sh/uv](https://github.com/astral-sh/uv) | patch | `0.9.13` -> `0.9.17` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>astral-sh/uv (astral-sh/uv)</summary> ### [`v0.9.17`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0917) [Compare Source](astral-sh/uv@0.9.16...0.9.17) Released on 2025-12-09. ##### Enhancements - Add `torch-tensorrt` and `torchao` to the PyTorch list ([#17053](astral-sh/uv#17053)) - Add hint for misplaced `--verbose` in `uv tool run` ([#17020](astral-sh/uv#17020)) - Add support for relative durations in `exclude-newer` (a.k.a., dependency cooldowns) ([#16814](astral-sh/uv#16814)) - Add support for relocatable nushell activation script ([#17036](astral-sh/uv#17036)) ##### Bug fixes - Respect dropped (but explicit) indexes in dependency groups ([#17012](astral-sh/uv#17012)) ##### Documentation - Improve `source-exclude` reference docs ([#16832](astral-sh/uv#16832)) - Recommend `UV_NO_DEV` in Docker installs ([#17030](astral-sh/uv#17030)) - Update `UV_VERSION` in docs for GitLab CI/CD ([#17040](astral-sh/uv#17040)) ### [`v0.9.16`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0916) [Compare Source](astral-sh/uv@0.9.15...0.9.16) Released on 2025-12-06. ##### Python - Add CPython 3.14.2 - Add CPython 3.13.11 ##### Enhancements - Add a 5m default timeout to acquiring file locks to fail faster on deadlock ([#16342](astral-sh/uv#16342)) - Add a stub `debug` subcommand to `uv pip` announcing its intentional absence ([#16966](astral-sh/uv#16966)) - Add bounds in `uv add --script` ([#16954](astral-sh/uv#16954)) - Add brew specific message for `uv self update` ([#16838](astral-sh/uv#16838)) - Error when built wheel is for the wrong platform ([#16074](astral-sh/uv#16074)) - Filter wheels from PEP 751 files based on `--no-binary` et al in `uv pip compile` ([#16956](astral-sh/uv#16956)) - Support `--target` and `--prefix` in `uv pip list`, `uv pip freeze`, and `uv pip show` ([#16955](astral-sh/uv#16955)) - Tweak language for build backend validation errors ([#16720](astral-sh/uv#16720)) - Use explicit credentials cache instead of global static ([#16768](astral-sh/uv#16768)) - Enable SIMD in HTML parsing ([#17010](astral-sh/uv#17010)) ##### Preview features - Fix missing preview warning in `uv workspace metadata` ([#16988](astral-sh/uv#16988)) - Add a `uv auth helper --protocol bazel` command ([#16886](astral-sh/uv#16886)) ##### Bug fixes - Fix Pyston wheel compatibility tags ([#16972](astral-sh/uv#16972)) - Allow redundant entries in `tool.uv.build-backend.module-name` but emit warnings ([#16928](astral-sh/uv#16928)) - Fix infinite loop in non-attribute re-treats during HTML parsing ([#17010](astral-sh/uv#17010)) ##### Documentation - Clarify `--project` flag help text to indicate project discovery ([#16965](astral-sh/uv#16965)) - Regenerate the crates.io READMEs on release ([#16992](astral-sh/uv#16992)) - Update Docker integration guide to prefer `COPY` over `ADD` for simple cases ([#16883](astral-sh/uv#16883)) - Update PyTorch documentation to include information about supporting CUDA 13.0.x ([#16957](astral-sh/uv#16957)) - Update the versioning policy ([#16710](astral-sh/uv#16710)) - Upgrade PyTorch documentation to latest versions ([#16970](astral-sh/uv#16970)) ### [`v0.9.15`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0915) [Compare Source](astral-sh/uv@0.9.14...0.9.15) Released on 2025-12-02. ##### Python - Add CPython 3.14.1 - Add CPython 3.13.10 ##### Enhancements - Add ROCm 6.4 to `--torch-backend=auto` ([#16919](astral-sh/uv#16919)) - Add a Windows manifest to uv binaries ([#16894](astral-sh/uv#16894)) - Add LFS toggle to Git sources ([#16143](astral-sh/uv#16143)) - Cache source reads during resolution ([#16888](astral-sh/uv#16888)) - Allow reading requirements from scripts without an extension ([#16923](astral-sh/uv#16923)) - Allow reading requirements from scripts with HTTP(S) paths ([#16891](astral-sh/uv#16891)) ##### Configuration - Add `UV_HIDE_BUILD_OUTPUT` to omit build logs ([#16885](astral-sh/uv#16885)) ##### Bug fixes - Fix `uv-trampoline-builder` builds from crates.io by moving bundled executables ([#16922](astral-sh/uv#16922)) - Respect `NO_COLOR` and always show the command as a header when paging `uv help` output ([#16908](astral-sh/uv#16908)) - Use `0o666` permissions for flock files instead of `0o777` ([#16845](astral-sh/uv#16845)) - Revert "Bump `astral-tl` to v0.7.10 ([#16887](astral-sh/uv#16887))" to narrow down a regression causing hangs in metadata retrieval ([#16938](astral-sh/uv#16938)) ##### Documentation - Link to the uv version in crates.io member READMEs ([#16939](astral-sh/uv#16939)) ### [`v0.9.14`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0914) [Compare Source](astral-sh/uv@0.9.13...0.9.14) Released on 2025-12-01. ##### Performance - Bump `astral-tl` to v0.7.10 to enable SIMD for HTML parsing ([#16887](astral-sh/uv#16887)) ##### Bug fixes - Allow earlier post releases with exclusive ordering ([#16881](astral-sh/uv#16881)) - Prefer updating existing `.zshenv` over creating a new one in `tool update-shell` ([#16866](astral-sh/uv#16866)) - Respect `-e` flags in `uv add` ([#16882](astral-sh/uv#16882)) ##### Enhancements - Attach subcommand to User-Agent string ([#16837](astral-sh/uv#16837)) - Prefer `UV_WORKING_DIR` over `UV_WORKING_DIRECTORY` for consistency ([#16884](astral-sh/uv#16884)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).

samypr100 force-pushed the conditional_lfs_support branch 8 times, most recently from b34deec to 52cbd5c Compare October 13, 2025 19:37

samypr100 force-pushed the conditional_lfs_support branch 2 times, most recently from 6b19653 to 17d5d16 Compare October 17, 2025 21:51

samypr100 force-pushed the conditional_lfs_support branch 6 times, most recently from ce1cfb1 to f222e06 Compare October 21, 2025 15:55

samypr100 marked this pull request as ready for review October 21, 2025 15:59

samypr100 force-pushed the conditional_lfs_support branch 11 times, most recently from 8d3c6f0 to 50cf0e5 Compare October 29, 2025 16:20

samypr100 force-pushed the conditional_lfs_support branch from 50cf0e5 to 81d1024 Compare October 29, 2025 22:00

samypr100 force-pushed the conditional_lfs_support branch 2 times, most recently from b88ff97 to b802ba7 Compare November 29, 2025 00:45

zanieb and others added 10 commits December 1, 2025 23:51

Add lfs = true support in Git source declarations

6102f52

chore: initial cleanup

4fa3e77

fix: ensure lfs state is propagated in the git database and checkouts

1a9cb16

fix: support stateful lfs fragments and indicators

070c2e7

fix: proper caching lfs-based built wheels

f34b1ac

tests: initial uv-sync test adjustments

511c8b1

feat: support --lfs in uv add

0ce00eb

feat: support --lfs in uv tool run/install

bd397b2

docs: add documentation for lfs usage and configuration

70bd58b

Add span for LFS git fsck

d90ebc7

We're not very consistent in adding spans for git operations, but we should at least for new functions.

samypr100 force-pushed the conditional_lfs_support branch from b802ba7 to d90ebc7 Compare December 2, 2025 04:53

konstin mentioned this pull request Dec 2, 2025

Add a check for accidental cache modifications #16927

Draft

konstin changed the title ~~Conditional Git LFS Support~~ Support Git LFS with opt-in Dec 2, 2025

Final touch-ups

dc1b6ba

konstin approved these changes Dec 2, 2025

View reviewed changes

konstin enabled auto-merge (squash) December 2, 2025 12:10

konstin merged commit fee7f9d into astral-sh:main Dec 2, 2025
162 checks passed

BrewTestBot mentioned this pull request Dec 3, 2025

uv 0.9.15 Homebrew/homebrew-core#256853

Merged

samypr100 deleted the conditional_lfs_support branch December 3, 2025 14:39

zanieb mentioned this pull request Dec 13, 2025

Add lfs = true support in Git source declarations #15563

Closed

traversaro mentioned this pull request Feb 11, 2026

Smudge errors when installing a package from git prefix-dev/pixi#2000

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

Support Git LFS with opt-in#16143

Support Git LFS with opt-in#16143
konstin merged 11 commits intoastral-sh:mainfrom
samypr100:conditional_lfs_support

samypr100 commented Oct 6, 2025 •

edited

Loading

Uh oh!

codspeed-hq bot commented Oct 17, 2025 •

edited

Loading

Uh oh!

konstin left a comment

Uh oh!

Uh oh!

Red-Eyed commented Dec 3, 2025 •

edited

Loading

Uh oh!

samypr100 commented Dec 3, 2025

Uh oh!

gsemet commented Dec 6, 2025

Uh oh!

samypr100 commented Dec 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Comments

Conversation

samypr100 commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test Plan

Uh oh!

codspeed-hq bot commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CodSpeed Performance Report

Merging #16143 will not alter performance

Summary

Uh oh!

konstin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Red-Eyed commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

samypr100 commented Dec 3, 2025

Uh oh!

gsemet commented Dec 6, 2025

Uh oh!

samypr100 commented Dec 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

samypr100 commented Oct 6, 2025 •

edited

Loading

codspeed-hq bot commented Oct 17, 2025 •

edited

Loading

Red-Eyed commented Dec 3, 2025 •

edited

Loading