Add options for sparse checkout in GitFetcher by psakievich · Pull Request #45473 · spack/spack

psakievich · 2024-07-26T23:16:53Z

Newer versions of git have a beta feature called sparse checkout that allow users to check out a portion of a large repo.

This feature will be ideal for monolithic repo projects that want to model their infrastructure via spack. This PR implements an addition to the GitFetcher that allows users to add a git_sparse_paths attribute to package classes or versions which will then use sparse checkout on those directories/files for the package.

Steps for completion

Add in git_sparse_paths attribute and test it can be called for versions and package attributes
Update git fetch calls to operate with sparse fetch under these conditions
Update unit-test infrastructure to handle case with sparse checkout
Test on a live application
Update documentation

This PR also refactors the GitFetcher.clone method to break it up into somewhat smaller functions. This is to improve readability and compatibility with further refactors with #44319 when we will be able to do the vast majority of git clones per commit instead per branch/tag.

Newer versions of git have a beta feature called sparse checkout that allow users to check out a portion of a large repo. This feature will be ideal for monolithic repo projects that want to model their infrastructure via spack. This PR implements an addition to the GitFetcher that allows users to add a `git_sparse_paths` attribute to package classes or versions which will then use sparse checkout on those directories/files for the package.

psakievich · 2024-08-06T22:26:25Z

@spackbot fix-style

psakievich · 2024-08-06T22:27:47Z

@spackbot fix style

spackbot-app · 2024-08-06T22:27:50Z

Let me see if I can fix that for you!

spackbot-app · 2024-08-06T22:28:54Z

I was able to run spack style --fix for you!

spack style --fix

==> Running style checks on spack
  selected: isort, black, flake8, mypy
==> Modified files
  lib/spack/spack/fetch_strategy.py
  lib/spack/spack/test/conftest.py
  lib/spack/spack/test/git_fetch.py
  lib/spack/spack/test/packages.py
  lib/spack/spack/version/git_ref_lookup.py
  var/spack/repos/builtin.mock/packages/git-monorepo-mock/package.py
==> Running isort checks
  isort checks were clean
==> Running black checks
reformatted lib/spack/spack/test/git_fetch.py
reformatted lib/spack/spack/test/conftest.py
All done! ✨ 🍰 ✨
2 files reformatted, 4 files left unchanged.
  black checks were clean
==> Running flake8 checks
  flake8 checks were clean
==> Running mypy checks
lib/spack/spack/version/version_types.py:145: error: Argument 2 to "StandardVersion" has incompatible type "*Tuple[Tuple[Any, ...], Tuple[Any, ...]]"; expected "Tuple[Tuple[Any, ...], Tuple[Any, ...]]"  [arg-type]
lib/spack/spack/version/version_types.py:452: error: Argument 2 to "StandardVersion" has incompatible type "*Tuple[Tuple[Any, ...], Tuple[Any, ...]]"; expected "Tuple[Tuple[Any, ...], Tuple[Any, ...]]"  [arg-type]
lib/spack/spack/version/version_types.py:481: error: Argument 2 to "StandardVersion" has incompatible type "*Tuple[Tuple[Any, ...], Tuple[Any, ...]]"; expected "Tuple[Tuple[Any, ...], Tuple[Any, ...]]"  [arg-type]
Found 3 errors in 1 file (checked 621 source files)
  mypy found errors

Keep in mind that I cannot fix your flake8 or mypy errors, so if you have any you'll need to fix them and update the pull request. If I was able to push to your branch, if you make further changes you will need to pull from your updated branch before pushing again.

I've updated the branch with style fixes.

mdmosby · 2024-08-08T21:04:21Z

We are modeling the internal products in the SIERRA code suite as spack packages and this sparse checkout feature is reducing the fetch times from O(min) to O(s) for each package and there are O(50) packages and growing (we are actively converting things). This is a huge win for us in monorepo land!

lib/spack/docs/packaging_guide.rst

Co-authored-by: Matthew Mosby <[email protected]>

lib/spack/spack/fetch_strategy.py

…ut' into f/git-sparse-checkout

psakievich · 2024-08-14T18:25:21Z

@spackbot fix style

@spackbot

* Add options for sparse checkout in GitFetcher Newer versions of git have a beta feature called sparse checkout that allow users to check out a portion of a large repo. This feature will be ideal for monolithic repo projects that want to model their infrastructure via spack. This PR implements an addition to the GitFetcher that allows users to add a `git_sparse_paths` attribute to package classes or versions which will then use sparse checkout on those directories/files for the package. * Style * Split git clone into multiple functions * Add sparse-checkout impl * Internalize src clone functions * Docs * Adding sparse clone test * Add test for partial clone * [@spackbot] updating style on behalf of psakievich * Small fixes * Restore default branch status * Fix attributes for package * Update lib/spack/docs/packaging_guide.rst Co-authored-by: Matthew Mosby <[email protected]> * Extend unit test to multiple git versions * style --------- Co-authored-by: psakievich <[email protected]> Co-authored-by: Matthew Mosby <[email protected]>

@spackbot

* Add options for sparse checkout in GitFetcher Newer versions of git have a beta feature called sparse checkout that allow users to check out a portion of a large repo. This feature will be ideal for monolithic repo projects that want to model their infrastructure via spack. This PR implements an addition to the GitFetcher that allows users to add a `git_sparse_paths` attribute to package classes or versions which will then use sparse checkout on those directories/files for the package. * Style * Split git clone into multiple functions * Add sparse-checkout impl * Internalize src clone functions * Docs * Adding sparse clone test * Add test for partial clone * [@spackbot] updating style on behalf of psakievich * Small fixes * Restore default branch status * Fix attributes for package * Update lib/spack/docs/packaging_guide.rst Co-authored-by: Matthew Mosby <[email protected]> * Extend unit test to multiple git versions * style --------- Co-authored-by: psakievich <[email protected]> Co-authored-by: Matthew Mosby <[email protected]>

@spackbot

* Add options for sparse checkout in GitFetcher Newer versions of git have a beta feature called sparse checkout that allow users to check out a portion of a large repo. This feature will be ideal for monolithic repo projects that want to model their infrastructure via spack. This PR implements an addition to the GitFetcher that allows users to add a `git_sparse_paths` attribute to package classes or versions which will then use sparse checkout on those directories/files for the package. * Style * Split git clone into multiple functions * Add sparse-checkout impl * Internalize src clone functions * Docs * Adding sparse clone test * Add test for partial clone * [@spackbot] updating style on behalf of psakievich * Small fixes * Restore default branch status * Fix attributes for package * Update lib/spack/docs/packaging_guide.rst Co-authored-by: Matthew Mosby <[email protected]> * Extend unit test to multiple git versions * style --------- Co-authored-by: psakievich <[email protected]> Co-authored-by: Matthew Mosby <[email protected]>

spackbot-app bot added core PR affects Spack core functionality fetching new-version tests General test capability(ies) labels Jul 26, 2024

psakievich added 2 commits July 29, 2024 06:10

Style

0cc2ee0

Split git clone into multiple functions

cb7c080

spackbot-app bot added the versions label Jul 29, 2024

psakievich added 3 commits July 29, 2024 14:39

Add sparse-checkout impl

cc36ab0

Internalize src clone functions

10f4097

Docs

4143a17

spackbot-app bot added the documentation Improvements or additions to documentation label Aug 1, 2024

psakievich added 2 commits August 5, 2024 23:00

Adding sparse clone test

53fb6b5

Add test for partial clone

b4afb94

[@spackbot] updating style on behalf of psakievich

be4efe0

psakievich added 4 commits August 6, 2024 16:33

Merge branch 'develop' into f/git-sparse-checkout

3d26fd0

Small fixes

f2d90fb

Restore default branch status

4b96858

Fix attributes for package

6c2dcf2

psakievich marked this pull request as ready for review August 8, 2024 03:55

tgamblin self-requested a review August 8, 2024 19:30

mdmosby reviewed Aug 9, 2024

View reviewed changes

lib/spack/docs/packaging_guide.rst Outdated Show resolved Hide resolved

Update lib/spack/docs/packaging_guide.rst

a421a79

Co-authored-by: Matthew Mosby <[email protected]>

becker33 requested changes Aug 12, 2024

View reviewed changes

lib/spack/spack/fetch_strategy.py Outdated Show resolved Hide resolved

lib/spack/spack/fetch_strategy.py Show resolved Hide resolved

Extend unit test to multiple git versions

d8e2a81

Merge remote-tracking branch 'refs/remotes/origin/f/git-sparse-checko…

c736937

…ut' into f/git-sparse-checkout

psakievich added 2 commits August 14, 2024 12:26

style

a729134

Merge branch 'develop' into f/git-sparse-checkout

cb9e5f0

becker33 approved these changes Aug 14, 2024

View reviewed changes

psakievich enabled auto-merge (squash) August 14, 2024 19:39

psakievich merged commit 1b82779 into spack:develop Aug 15, 2024

psakievich deleted the f/git-sparse-checkout branch August 15, 2024 14:29

greenc-FNAL mentioned this pull request Aug 22, 2024

maintenance/merge from upstream 2024 08 22 FNALssi/spack#428

Merged

psakievich mentioned this pull request Sep 22, 2024

Add a custom hook for dev_path changes #46529

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add options for sparse checkout in GitFetcher#45473

Add options for sparse checkout in GitFetcher#45473
psakievich merged 18 commits intospack:developfrom
psakievich:f/git-sparse-checkout

psakievich commented Jul 26, 2024 •

edited

Loading

Uh oh!

psakievich commented Aug 6, 2024

Uh oh!

psakievich commented Aug 6, 2024

Uh oh!

spackbot-app bot commented Aug 6, 2024

Uh oh!

spackbot-app bot commented Aug 6, 2024

Uh oh!

mdmosby commented Aug 8, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

psakievich commented Aug 14, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

psakievich commented Jul 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

psakievich commented Aug 6, 2024

Uh oh!

psakievich commented Aug 6, 2024

Uh oh!

spackbot-app bot commented Aug 6, 2024

Uh oh!

spackbot-app bot commented Aug 6, 2024

Uh oh!

mdmosby commented Aug 8, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

psakievich commented Aug 14, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

psakievich commented Jul 26, 2024 •

edited

Loading