Skip to content

Send part checksums when completing checksummed multipart uploads - #1088

Open
haeganm wants to merge 1 commit into
fsspec:mainfrom
haeganm:fix/multipart-part-checksums
Open

haeganm wants to merge 1 commit into
fsspec:mainfrom
haeganm:fix/multipart-part-checksums

Conversation

@haeganm

@haeganm haeganm commented Nov 4, 2026

Copy link
Copy Markdown

Fixes #672

#775 fixed this for S3File with SHA256, but pipe_file and put_file still complete their multipart uploads with only PartNumber and ETag per part. if the upload was created with a ChecksumAlgorithm (e.g. s3_additional_kwargs={"ChecksumAlgorithm": "SHA256"}), AWS refuses to complete it without the part checksums, which is the error in #672. S3File also only forwarded ChecksumSHA256, so CRC32 and friends were dropped there.

the change is a small _completed_part helper in core.py that all three paths use now. it takes the ChecksumAlgorithm s3fs asked for (per-call kwargs over s3_additional_kwargs, same precedence _call_s3 uses) and copies Checksum<ALGORITHM> from the upload_part response into the part, which is the same rule s3transfer's UploadPartTask uses. the algorithm in the create_multipart_upload response is only a fallback, so servers that don't echo it still get the checksums, and the SHA256 behaviour from #775 stays the same there. uploads created without an algorithm send exactly what they sent before. botocore adds a CRC32 to parts by default, but that's left out of the complete call unless the upload asked for it.

test: test_multipart_parts_carry_checksums runs pipe_file, put_file and open(..., "wb") against moto with SHA256 and CRC32, records the UploadPart responses and the CompleteMultipartUpload params through botocore's event hooks, and checks every completed part carries the checksum its upload returned. each case also runs with ChecksumAlgorithm stripped from the CreateMultipartUpload response, like a server that doesn't echo it. moto itself completes these uploads without the checksums, so the test checks what we send rather than relying on the server to reject it.

  • before the fix: 10 of the 12 cases fail (pipe_file and put_file for both algorithms, open with CRC32, with and without the echo). open with SHA256 already passed both ways thanks to Fix for issue #672 #775.
  • after: all 12 pass
  • full suite on this branch: 473 passed, 1 skipped, 4 xfailed
  • full suite on main: 461 passed, 1 skipped, 4 xfailed, so the 12 new cases are the only difference
  • pre-commit hooks (black 25.9.0, flake8 7.3.0, whitespace/eof) pass on the changed files

not touched here: the upload_part_copy paths (merge, managed copies, append mode in S3File). there are open PRs on merge and multipart copies, so I left those alone. also, a ChecksumAlgorithm passed straight to put_file/pipe_file as a call kwarg only reaches create_multipart_upload, not upload_part. that's a separate kwargs-forwarding thing, this PR covers the s3_additional_kwargs route from the issue.

S3 only completes a multipart upload created with a ChecksumAlgorithm
if each part in CompleteMultipartUpload carries its checksum for that
algorithm. pipe_file and put_file built their parts from PartNumber
and ETag only, and S3File only forwarded ChecksumSHA256, so uploads
using s3_additional_kwargs={"ChecksumAlgorithm": ...} failed on AWS
once they went multipart.

Build every completed part through one helper that adds the
Checksum<ALGORITHM> value from the upload_part response. The algorithm
is the one s3fs requested, as s3transfer does, with the
CreateMultipartUpload response as a fallback, so servers that don't
echo it back still get the checksums.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Checksum failing with Multipart Uploads

1 participant