Reading commits in parallel seems like a way to speed this up a bit, but I'm not sure if this is possible with libgit2.
git-repository from gitoxide seems like a promising alternative.

It looks like major speedups are possible even on a single thread for commit graph traversal, for example I'd expect onefetch to go from ~19s on the v5.16 linux kernel checkout to something like 11s.

If more/all operations are done in parallel, like the syntax analysis, it should become as fast as the slowest of these operations, the commit graph traversal which clocks in at about ~7s.

I wouldn't know how to parallelize the commit graph traversal though - the only way I can imagine this to work is to traverse different branches on multiple threads. This usually comes at the overhead of avoiding them to do duplicate work which requires a parallel hashset (like dashmap) which will limit the amount of threads that are effective there. That also depends on the graph, a linear history with a single trunk can't be sped up at all. Anyway, it's sounds like an interesting task to implement high-performance parallel traversal, maybe gitoxide can provide one once all other options are exhausted here.

The above is my profiling run on the linux kernel v5.16. Most of it is the commit graph traversal, the spike towards the end is tokei, and there is about ~1s of releasing memory (which can and should probably be avoided with process::abort() or by leaking the values with mem::forget()).

All in all, I think switching to gitoxide and running the syntax analysis alone in parallel, along with process::abort() to avoid memory deallocation at the end, one should be able to get roughly 2x the speed on the linux kernel.

Extremely slow with large repositories. #621

Description

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions