Skip to content

s3: treat UploadPart success without ETag as retryable error - #9823

Merged
ncw merged 2 commits into
rclone:masterfrom
CAOShurong:fix-9822-nil-etag-panic
Aug 26, 2026
Merged

ncw merged 2 commits into
rclone:masterfrom
CAOShurong:fix-9822-nil-etag-panic

Conversation

@CAOShurong

Copy link
Copy Markdown
Contributor

What does this change do?

Fixes a panic in the S3 multipart upload path reported in #9822: when a
backend (or a proxy / load balancer in front of it, e.g. MinIO behind
sidekick) answers an UploadPart request with HTTP 200 but no ETag
header, s3ChunkWriter.WriteChunk dereferences uout.ETag unconditionally
in a debug log line and crashes the whole ~85 GB transfer with a nil pointer
panic — even though nothing load-bearing needed the ETag at that point.

Instead of only guarding the log line, this treats an ETag-less successful
response as what it actually is — an unusable part — and returns a retryable
error from inside the pacer callback:

if uout == nil || uout.ETag == nil {
    return true, fmt.Errorf("UploadPart response for chunk %d has no ETag", chunkNumber+1)
}
return false, nil

The pacer then retries that chunk with the usual backoff, which both removes
the panic and gives transient proxy glitches a chance to heal themselves.
This follows the direction suggested in the issue report.

Note on behaviour without this change beyond the crash itself: with only a
log-line guard, the nil ETag would be recorded via addCompletedPart and the
transfer would proceed all the way to CompleteMultipartUpload, which would
then fail server-side (InvalidPart) — so retrying at the chunk level is the
correct recovery point.

Linked issue

Fixes #9822

(The issue contains the reporter's own root-cause analysis against v1.75.0,
which I verified is still accurate on master: addCompletedPart does not
deref, the following fs.Debugf line does.)

For new or changed backends

No new backend; small bug fix in the existing s3 backend. Not run:
go run ./fstest/test_all -backends :s3 needs live S3 credentials which I do
not have. Local verification instead:

  • go vet ./backend/s3/ — clean
  • go test ./backend/s3/ -count=1 — ok
  • go build ./... — clean

Checklist

  • This change is trivial OR it has been discussed and agreed in the linked issue.
  • I have read the contribution guidelines.
  • (If I used AI tools to help write this code) I have read and understood the AI-assisted contributions guidance, and I have tested and take ownership of this change myself.
  • I have added tests for all changes in this PR if appropriate.
  • I have added documentation for the changes if appropriate.
  • All commit messages are in house style.
  • (Backend changes only) test_all passes for this backend and if submitting a new backend can provide a test account for the integration tester - see CONTRIBUTING.md.
  • This Pull Request is ready for review.

AI-assistance disclosure: this PR was prepared with AI assistance (code
location and fix drafted by an agent, verified locally by me as described
above). I have read rclone's AI-assisted contributions guidance and take full
ownership of this change.

Notes: checklist items for tests/docs/test_all are intentionally unticked —
no new tests added (the failure mode requires an S3-compatible endpoint that
omits ETag on 200 responses; happy to add one if maintainers point me at
the right harness seam), docs unaffected, and test_all needs live S3
credentials.

A successful UploadPart whose response carries no ETag header made
WriteChunk panic dereferencing uout.ETag in a debug log line. The part
ETag is required by CompleteMultipartUpload, so an ETag-less 200 is
unusable: return a retryable error from inside the pacer callback so
the chunk is retried instead of crashing the transfer or completing
the upload with a broken part list.

Fixes rclone#9822
@CAOShurong
CAOShurong requested a review from ncw as a code owner August 26, 2026 08:34

@ncw ncw left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great - thank you.

@ncw
ncw merged commit 660144d into rclone:master Aug 26, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

s3: panic (nil pointer dereference) in s3ChunkWriter.WriteChunk when UploadPart response has no ETag

2 participants