-
Notifications
You must be signed in to change notification settings - Fork 762
Unify CRC implementations to crc-fast #873
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
houseme
merged 5 commits into
copilot/fix-data-recovery-during-disconnection
from
copilot/unify-crc-implementation
Nov 17, 2025
Merged
Unify CRC implementations to crc-fast #873
houseme
merged 5 commits into
copilot/fix-data-recovery-during-disconnection
from
copilot/unify-crc-implementation
Nov 17, 2025
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: houseme <[email protected]>
Copilot
AI
changed the title
[WIP] Unify CRC implementation by replacing with crc-fast
Unify CRC implementations to crc-fast
Nov 17, 2025
…pilot/unify-crc-implementation
2c73e22
into
copilot/fix-data-recovery-during-disconnection
1 check passed
houseme
added a commit
that referenced
this pull request
Nov 17, 2025
#871) * Initial plan * Fix: Trigger self-heal when missing shards detected during read - Added proactive heal detection in get_object_with_fileinfo - When reading an object, now checks if any shards are missing even if read succeeds - Sends low-priority heal request to reconstruct missing shards on rejoined nodes - This fixes the issue where data written during node outage is not healed when node rejoins Co-authored-by: houseme <[email protected]> * fix * Unify CRC implementations to crc-fast (#873) * Initial plan * Replace CRC libraries with unified crc-fast implementation Co-authored-by: houseme <[email protected]> * fix * fix: replace low to Normal --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: houseme <[email protected]> Co-authored-by: houseme <[email protected]> --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: houseme <[email protected]> Co-authored-by: houseme <[email protected]>
houseme
added a commit
that referenced
this pull request
Nov 17, 2025
* Initial plan * Fix large file upload freeze by increasing StreamReader buffer size Co-authored-by: houseme <[email protected]> * Add comprehensive documentation for large file upload freeze fix Co-authored-by: houseme <[email protected]> * upgrade s3s version * Fix compilation error: use BufReader instead of non-existent StreamReader::with_capacity Co-authored-by: houseme <[email protected]> * Update documentation with correct BufReader implementation Co-authored-by: houseme <[email protected]> * add tokio feature `io-util` * Implement adaptive buffer sizing based on file size Co-authored-by: houseme <[email protected]> * Constants are managed uniformly and fmt code * fix * Fix: Trigger self-heal on read when shards missing from rejoined nodes (#871) * Initial plan * Fix: Trigger self-heal when missing shards detected during read - Added proactive heal detection in get_object_with_fileinfo - When reading an object, now checks if any shards are missing even if read succeeds - Sends low-priority heal request to reconstruct missing shards on rejoined nodes - This fixes the issue where data written during node outage is not healed when node rejoins Co-authored-by: houseme <[email protected]> * fix * Unify CRC implementations to crc-fast (#873) * Initial plan * Replace CRC libraries with unified crc-fast implementation Co-authored-by: houseme <[email protected]> * fix * fix: replace low to Normal --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: houseme <[email protected]> Co-authored-by: houseme <[email protected]> --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: houseme <[email protected]> Co-authored-by: houseme <[email protected]> --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: houseme <[email protected]> Co-authored-by: houseme <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Consolidates four different CRC libraries (
crc32c,crc32fast,crc64fast-nvme, andcrc-fast) into a single dependency oncrc-fast.Changes
Dependency cleanup
crc32c,crc32fast,crc64fast-nvmefrom workspace dependenciescrc-fast = "1.6.0"across all cratesAlgorithm mappings
crc32fast::Hasher→crc_fast::Digest::new(Crc32IsoHdlc)crc32c::crc32c_append→crc_fast::Digest::new(Crc32Iscsi)crc64fast_nvme::Digest→crc_fast::Digest::new(Crc64Nvme)Code migrations
crates/rio/src/checksum.rs: Migrated hasher implementations for all three CRC variantscrates/rio/src/{encrypt_reader,compress_reader}.rs: Replaced one-shothash()calls with inline digest usagecrates/filemeta/src/fileinfo.rs: Migrated hash computation for erasure coding index calculationcrates/utils/src/hash.rs: Updated CRC hash function for bucket distributionExample
Before:
After:
Note: Old CRC libraries remain in dependency tree as transitive dependencies from
aws-smithy-eventstream,google-cloud-storage, ands3s.Original prompt
✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.