Repository navigation
s3: treat UploadPart success without ETag as retryable error - #9823
Merged
Merged
Conversation
A successful UploadPart whose response carries no ETag header made WriteChunk panic dereferencing uout.ETag in a debug log line. The part ETag is required by CompleteMultipartUpload, so an ETag-less 200 is unusable: return a retryable error from inside the pacer callback so the chunk is retried instead of crashing the transfer or completing the upload with a broken part list. Fixes rclone#9822
… unrelated to s3)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this change do?
Fixes a panic in the S3 multipart upload path reported in #9822: when a
backend (or a proxy / load balancer in front of it, e.g. MinIO behind
sidekick) answers anUploadPartrequest with HTTP 200 but noETagheader,
s3ChunkWriter.WriteChunkdereferencesuout.ETagunconditionallyin a debug log line and crashes the whole ~85 GB transfer with a nil pointer
panic — even though nothing load-bearing needed the ETag at that point.
Instead of only guarding the log line, this treats an ETag-less successful
response as what it actually is — an unusable part — and returns a retryable
error from inside the pacer callback:
The pacer then retries that chunk with the usual backoff, which both removes
the panic and gives transient proxy glitches a chance to heal themselves.
This follows the direction suggested in the issue report.
Note on behaviour without this change beyond the crash itself: with only a
log-line guard, the nil ETag would be recorded via
addCompletedPartand thetransfer would proceed all the way to
CompleteMultipartUpload, which wouldthen fail server-side (
InvalidPart) — so retrying at the chunk level is thecorrect recovery point.
Linked issue
Fixes #9822
(The issue contains the reporter's own root-cause analysis against v1.75.0,
which I verified is still accurate on master:
addCompletedPartdoes notderef, the following
fs.Debugfline does.)For new or changed backends
No new backend; small bug fix in the existing s3 backend. Not run:
go run ./fstest/test_all -backends :s3needs live S3 credentials which I donot have. Local verification instead:
go vet ./backend/s3/— cleango test ./backend/s3/ -count=1— okgo build ./...— cleanChecklist
test_allpasses for this backend and if submitting a new backend can provide a test account for the integration tester - see CONTRIBUTING.md.AI-assistance disclosure: this PR was prepared with AI assistance (code
location and fix drafted by an agent, verified locally by me as described
above). I have read rclone's AI-assisted contributions guidance and take full
ownership of this change.
Notes: checklist items for tests/docs/
test_allare intentionally unticked —no new tests added (the failure mode requires an S3-compatible endpoint that
omits
ETagon 200 responses; happy to add one if maintainers point me atthe right harness seam), docs unaffected, and
test_allneeds live S3credentials.