s3: panic (nil pointer dereference) in s3ChunkWriter.WriteChunk when UploadPart response has no ETag #9822

Before you start

• I have searched the forum and the existing issues for this problem.
• I have tested with the latest beta or stable release and can still reproduce the problem.
• This is a reproducible bug, not a usage question (questions belong on the forum).

Associated forum post URL

No response

What is the problem you are having with rclone?

rclone copy panics with a nil pointer dereference partway through an ~85GB
multipart upload to an S3-compatible backend (MinIO, fronted by a MinIO
sidekick load balancer). The panic is not from a network error — it happens
after UploadPart returns successfully (no error from the pacer), while
logging the result.

Root cause (traced against the v1.75.0 tag):

backend/s3/s3.go, in s3ChunkWriter.WriteChunk:

w.addCompletedPart(s3PartNumber, uout.ETag)          // line 4666 — fine, doesn't deref
fs.Debugf(w.o, "multipart upload wrote chunk %d with %v bytes and etag %v",
    chunkNumber+1, currentChunkSize, *uout.ETag)       // line 4668 — unconditional deref

If the backend answers a given UploadPart request with HTTP success but an
empty/absent ETag header, uout.ETag is nil, err is nil (so the pacer
loop doesn't retry), and the very next line panics dereferencing it — purely
for a debug log message, not for anything load-bearing. This is inconsistent
with how the same file handles a possibly-nil ETag elsewhere, e.g.
uploadSinglepartPutObject guards it via stringClone(deref(resp.ETag)).

Whether the backend should ever omit ETag on a 200 is a separate
question (still being investigated on our infra side — MinIO/proxy issue
under load), but rclone crashing the whole transfer on it, from a log
statement, is a bug regardless.

Suggested fix: nil-guard the log line (or better, treat a nil ETag on a
successful response as a retryable/error condition inside the pacer
callback, since silently completing a part with no ETag is itself
questionable).

rclone version

rclone v1.75.0

• os/version: rocky 9.8 (64 bit)
• os/kernel: 5.14.0-687.15.1.el9_8.x86_64 (x86_64)
• os/type: linux
• os/arch: amd64
• go/version: go1.26.5
• go/linking: static
• go/tags: none

Operating system

Linux

Which cloud storage system are you using?

S3-compatible (MinIO, provider=Minio in rclone config), accessed through a local sidekick (MinIO's own client-side load balancer/health-checker) reverse proxy on http://127.0.0.1:<port>.

Output of rclone config redacted

[object_storage]
type = s3
provider = Minio
access_key_id = XXX
secret_access_key = XXX
endpoint = http://127.0.0.1:9000

The command you were trying to run

rclone --config <redacted>.conf copy --progress <source-dir> \
        --retries=10 --retries-sleep=30s --low-level-retries=20 \
        --buffer-size=64M --transfers=4 \
        <remote>:<bucket>/<date>

Source is a single ~85GB tarball.

A log from the command with the -vv flag

panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x20bde91]

    goroutine 634 [running]:
    github.com/rclone/rclone/backend/s3.(*s3ChunkWriter).WriteChunk(0x38b7d512e000, {0x3463148, 0x38b7d4dbc960}, 0x1db, {0x3448e88, 0x38b7d48c5d10})
          github.com/rclone/rclone/backend/s3/s3.go:4668 +0x991
    github.com/rclone/rclone/fs/operations.(*multiThreadCopyState).copyChunk(0x38b7d4dbc9b0, {0x3463148, 0x38b7d4dbc960}, 0x1db, {0x3459360, 0x38b7d512e000}, 0xed8000000, 0xee0000000, 0x8000000, 0x0)
          github.com/rclone/rclone/fs/operations/multithread.go:103 +0x5d3
    github.com/rclone/rclone/fs/operations.multiThreadCopy.func2()
          github.com/rclone/rclone/fs/operations/multithread.go:236 +0x3f
    golang.org/x/sync/errgroup.(*Group).Go.func1()
          golang.org/x/[email protected]/errgroup/errgroup.go:93 +0x50
    created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 77
          golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x95

    (We're working on capturing a -vv --dump=headers log that shows the raw
    UploadPart response with the missing ETag header, and will attach it here
    once we reproduce it again)

Anything else?

No response