Skip to content

Recommend Git's manyFiles feature#25977

Merged
tgamblin merged 1 commit intospack:developfrom
michaelkuhn:git-clone-manyfiles
Sep 16, 2021
Merged

Recommend Git's manyFiles feature#25977
tgamblin merged 1 commit intospack:developfrom
michaelkuhn:git-clone-manyfiles

Conversation

@michaelkuhn
Copy link
Copy Markdown
Member

Git 2.24 introduced a feature flag for repositories with many files, see:
https://github.blog/2019-11-03-highlights-from-git-2-24/#feature-macros

Since Spack's Git repository contains roughly 8,500 files, it can be
worthwhile to enable this, especially on slow file systems such as NFS:

$ hyperfine --warmup 3 'cd spack-default; git status' 'cd spack-manyfiles; git status'
Benchmark #1: cd spack-default; git status
  Time (mean ± σ):      3.388 s ±  0.095 s    [User: 256.2 ms, System: 625.8 ms]
  Range (min … max):    3.168 s …  3.535 s    10 runs

Benchmark #2: cd spack-manyfiles; git status
  Time (mean ± σ):     168.7 ms ±  10.9 ms    [User: 98.6 ms, System: 126.1 ms]
  Range (min … max):   144.8 ms … 188.0 ms    19 runs

Summary
  'cd spack-manyfiles; git status' ran
   20.09 ± 1.42 times faster than 'cd spack-default; git status'

Git 2.24 introduced a feature flag for repositories with many files, see:
https://github.blog/2019-11-03-highlights-from-git-2-24/#feature-macros

Since Spack's Git repository contains roughly 8,500 files, it can be
worthwhile to enable this, especially on slow file systems such as NFS:
```
$ hyperfine --warmup 3 'cd spack-default; git status' 'cd spack-manyfiles; git status'
Benchmark #1: cd spack-default; git status
  Time (mean ± σ):      3.388 s ±  0.095 s    [User: 256.2 ms, System: 625.8 ms]
  Range (min … max):    3.168 s …  3.535 s    10 runs

Benchmark #2: cd spack-manyfiles; git status
  Time (mean ± σ):     168.7 ms ±  10.9 ms    [User: 98.6 ms, System: 126.1 ms]
  Range (min … max):   144.8 ms … 188.0 ms    19 runs

Summary
  'cd spack-manyfiles; git status' ran
   20.09 ± 1.42 times faster than 'cd spack-default; git status'
```
@spackbot-app spackbot-app bot added the documentation Improvements or additions to documentation label Sep 15, 2021
@sethrj
Copy link
Copy Markdown
Contributor

sethrj commented Sep 15, 2021

Will the config flag be ignored on older git versions that don't support the feature?

@michaelkuhn
Copy link
Copy Markdown
Member Author

Will the config flag be ignored on older git versions that don't support the feature?

I tested it with Git 2.17 and it just doesn't do anything then. git clone will still add it to the config file, so it will start working once Git is updated.

@adamjstewart
Copy link
Copy Markdown
Member

It's not possible to ship a git config file within Spack where this is preset, is it?

@michaelkuhn
Copy link
Copy Markdown
Member Author

michaelkuhn commented Sep 15, 2021

It's not possible to ship a git config file within Spack where this is preset, is it?

I don't think so. It needs to be set in .git/config as far as I know.

Edit: To be precise, we could also put it into a separate config file (see https://stackoverflow.com/questions/18329621/how-to-store-a-git-config-as-part-of-the-repository) but then this file needs to be referenced in .git/config. Moreover, if I read the man page correctly, at least some of the implied options should be set before initializing the repository:

index.version
Specify the version with which new index files should be initialized. This does not affect existing repositories. If feature.manyFiles is enabled, then the default is 4.

Copy link
Copy Markdown
Member

@adamjstewart adamjstewart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As long as older versions of git don't crash on this unsupported config setting, I agree with this feature!

@tgamblin tgamblin enabled auto-merge (squash) September 16, 2021 00:59
@haampie
Copy link
Copy Markdown
Member

haampie commented Sep 16, 2021

@spackbot run pipeline

@spackbot-app
Copy link
Copy Markdown

spackbot-app bot commented Sep 16, 2021

I've started that pipeline for you!

@tgamblin tgamblin merged commit d73fe19 into spack:develop Sep 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants