persist: read multipart blobs from stores without x-amz-mp-parts-count - #39115

Issue

S3Blob::get reads a multipart-uploaded object part by part. It learns how many parts to fetch from the x-amz-mp-parts-count header on its first request, and treats a missing header as "this object has one part". A store that does not send that header therefore returns only the object's first 8 MiB, and persist hands the truncated bytes to the parquet decoder, which panics with Invalid Parquet file. Corrupt footer.

That is not hypothetical. rustfs answers GetObject?partNumber=1 with 206 and a correct Content-Range but no parts count, and its HeadObject ignores partNumber entirely. Running persist against it, every cold read of a blob above the 8 MiB multipart threshold, which is every large compaction output, crashed environmentd. The blob cache masks it for freshly written blobs, so it only surfaces on a cold read, for example after a restart or once the cache is under pressure.

minio, garage, seaweedfs and S3 all return the header, so this does not affect any store we run today. I found it while benchmarking blob stores as possible minio replacements.

Solution

get now also takes the object's total size from Content-Range, which every store returns on a request that names a part. When the part count is missing and the first part is short of the total, the remainder is fetched by byte range, in chunks the size of the first part. The reassembled length is then checked against the total, so a store returning a short object fails the read, which persist retries, rather than corrupting data downstream.

Where the header is present, which is every store we currently use, nothing changes: same requests, same code path.

The length check is the part worth keeping regardless of rustfs. It turns silent truncation into a loud failure for any S3-compatible store, and it costs one comparison per get.

Testing

Unit tests for the Content-Range parser and for the byte-range chunking.

Verified end to end against rustfs: the scenario that reproducibly crashed environmentd, with the blob cache disabled so every scan fetches from the store, now completes about 20 GiB of cold reads with no failures, twice. Before the change it panicked within a minute.

Gotchas for the reviewer

• The Content-Range denominator is load-bearing. The completeness check assumes it is the length of the whole object, not of the part that was requested. That is what S3 specifies, and what minio, garage, seaweedfs and rustfs all return. A store that reported the part's length there would fail every multipart get instead of silently truncating. The assumption is now called out in a comment at the parse site.
• The check runs for every store, not only the ones needing the byte-range fallback. That is deliberate: it is what converts a short read into a retryable error rather than a corrupt blob. It is also the only behavioural change visible to stores that already send the header.
• The fallback only engages when the part count is absent and the first part is shorter than the total, so a genuine single-part object still takes exactly one request.
• Byte-range chunks are sized from the first part rather than from persist's own multipart chunk size, so they line up with however the object was actually uploaded rather than with how we would upload it today. A store reporting a zero-length first part leaves no chunk size to reuse, so the remainder is fetched as a single range.
• x-amz-mp-parts-count is part of the S3 GetObject contract and rustfs already defines the constant internally without sending it. Worth an upstream report, which I have not filed yet.

This is split out of #38800, which has the blob store benchmarking work that led to it.

🤖 Generated with Claude Code