fix: normalize ETag comparison in multipart upload and replication #627
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Normalize ETags by removing quotes before comparison in complete_multipart_upload - Fix ETag comparison in replication logic to handle quoted ETags from API responses - Fix ETag comparison in transition object logic - Add unit tests for trim_etag function This fixes the ETag mismatch error when uploading large files (5GB+) via multipart upload, which was caused by PR #592 adding quotes to ETag responses while internal storage remains unquoted. Fixes #625
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
Issue #625 reported an error when uploading 5GB files via multipart upload:
This was caused by PR #592, which introduced the
format_etag()function to ensure HTTP responses comply with RFC 7232 (ETags wrapped in quotes). However, this created a mismatch:"2c7ab85a893283e98c931e9511add182"2c7ab85a893283e98c931e9511add182"etag"≠etagSolution
Normalize ETags by removing quotes before comparison using
trim_etag()function in three critical locations:set_disk.rs:6601-6610): Normalize client-submitted and stored ETags before comparisonreplication_resyncer.rs:2315-2329): Normalize internal and external S3 API ETagsset_disk.rs:5671-5678): Normalize transition options and stored ETagsTesting
trim_etag()functionVerification Commands
cargo test --package rustfs-utils --features path -- test_trim_etag cargo check --workspaceFixes #625