gix-ref: skip name validation in packed-refs binary search#2604
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8726942cfc
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "Codex (@codex) address that feedback".
8726942 to
a8f7474
Compare
|
Pushed a small follow-up that elides the redundant lifetime on |
…wins Combines two upstream PRs that are not yet released: - GitoxideLabs/gitoxide#2604 — gix-ref: skip name validation in packed-refs binary search. - GitoxideLabs/gitoxide#2605 — gix: cache packed-refs in a HashMap during fetch update_refs. On `gitaur -Sy` against the AUR mirror (154k branches in packed-refs, warm cache, no incoming updates) the combo brings wall time from ~11.0s on stock gix 0.83 to ~3.4-5.0s, putting gitaur on par with system `git fetch` on the same workload. Profile evidence: docs/PROFILING.md plus the `gitaur-bc.json.gz` samply capture. Revert to a versioned crates.io dependency once both PRs land and a gix release ships with them. Co-Authored-By: Claude Opus 4.7 <[email protected]>
d778df6 to
0cf8980
Compare
…ormance `packed::Buffer::binary_search_by` calls `decode::reference` on every comparison step (~17 per lookup on a 154k-ref store), and each decode runs `gix_validate::reference::name` -> `gix_validate::tag::name_inner` plus full hex validation on the hash. The binary search only needs the name bytes for ordered comparison; the final match in `try_find_full_name` already re-parses the record through `decode::reference` and surfaces any parse error. Introduce `decode::name_at_record_start`, which scans for the `<hash><space><name><newline>` shape and returns just the name slice without validating either piece, and use it from the binary search. Parse-failure detection is preserved via the same flag the previous implementation set. On a wide-refs incremental fetch (gitaur AUR mirror, 154k branches) this is the single largest CPU sink — `gix_validate::tag::name_inner` shows up at ~14% of total fetch CPU in `samply`, all of it from this loop. End-to-end on the same fetch the change brings wall time from ~11.0s to ~7.7s (~30%) and user CPU from ~8.1s to ~5.1s (~37%). Co-authored-by: Claude Opus 4.7 <[email protected]>
9b86d7c to
c3ee599
Compare
There was a problem hiding this comment.
Thanks so much, and sorry for the long, long wait!
I can confirm the speed-boost, excited about its effects in the AUR mirror, and will bring this PR to completion.
Old
Enumerated 150003 refs in 0.015734458s (9533407 refs/s)
Found 10000 refs in 0.026713792s (374338.47 refs/s)
New
Found 10000 refs in 0.007547166s (1325000.6 refs/s)
Enumerated 150003 refs in 0.013016916s (11523697 refs/s)
9e6fa75 to
3ae3f26
Compare
- refactor - [P2] Reject bare CR before accepting a record name — /Users/byron/dev/github.com/GitoxideLabs/gitoxide.skip-validation-in-binary-search/gix-ref/src/store/packed/decode/mod.rs:118-118 When a packed-refs record contains a bare `\r` that is not followed by `\n` and the lookup is for a different name, this fast path now returns `Some(name)` and leaves `encountered_parse_failure` unset. The full parser rejects the same record because `parse::newline` only accepts LF or CRLF, so this can turn a malformed sorted packed-refs file into `Ok(None)` instead of `Error::Parse`. Co-authored-by: Sebastian Thiel <[email protected]>
3ae3f26 to
a8b885d
Compare
Summary
packed::Buffer::binary_search_bycurrently callsdecode::referenceon every comparison step of the binary search (~17 per lookup on a 154k-ref store). Eachdecode::referencerunsgix_validate::reference::name→gix_validate::tag::name_innerand full hex validation on the hash. The binary search only needs the name bytes for ordered comparison; the final match intry_find_full_namealready re-parses the record throughdecode::referenceand surfaces any parse error.This PR introduces
decode::name_at_record_start, which scans for the<hash><space><name><newline>shape and returns just the name slice without validating either piece, and uses it from the binary search. Parse-failure detection is preserved via the same flag the previous implementation set.Measured impact
Profiled on a real-world wide-refs incremental fetch —
gitaur -Syagainst the AUR mirror (154k branches in packed-refs, ~12 MB file) on a warm cache with no incoming ref updates, sampled withsamply:gix_validate::tag::name_innerself timegix_ref::packed::decode::referenceself timegix_validate::tag::name_innerwas the single largest CPU sink, all of it coming from inside this loop.Test coverage
cargo test -p gix-refpasses locally (161/161). The existing tests fortry_find/try_find_full_nameexercise both the hit and miss paths against packed-refs fixtures; behavior on malformed records is preserved by detecting the shape mismatch and setting the sameencountered_parse_failureflag as before, so a corrupt packed-refs file still surfaces asError::Parse.AI disclosure
Per
CONTRIBUTING.md, this change was drafted with the help of Claude Opus 4.7 (see theCo-authored-bytrailer on the commit). The investigation, profiling methodology, and final review were performed by me.