persist: fix multipart blob reads and add blob store benchmarks - #38800

Issue

S3Blob::get reads a multipart-uploaded object part by part. It learns how many parts to fetch from the x-amz-mp-parts-count header on its first request, and treats a missing header as "this object has one part". A store that does not send that header therefore returns only the object's first 8 MiB, and persist hands the truncated bytes to the parquet decoder, which panics with Invalid Parquet file. Corrupt footer.

rustfs is such a store. Every cold read of a blob above the multipart threshold, which is every large compaction output, crashed environmentd. The blob cache masks it for freshly written blobs, so it only surfaces on a cold read, for example after a restart or once the cache is under pressure. minio, garage, seaweedfs and S3 all send the header.

I found this while benchmarking blob stores, and could not tell the two stores apart on the existing instruments, which is the other half of this PR.

Solution

The read path. get now also takes the object's total size from Content-Range, which every store returns on a request that names a part. When the part count is missing and the first part is short of the total, the remainder is fetched by byte range, in chunks the size of the first part. The reassembled length is then checked against the total, so a store returning a short object fails the read, which persist retries, rather than corrupting data downstream. Where the header is present, nothing changes.

The length check is the part worth keeping regardless of rustfs: it turns silent truncation into a loud failure for any S3-compatible store.

Measuring a blob store. Two instruments, because the existing ones cannot separate the store from Materialize's own work:

• persistcli bench blob drives one store through persist's own blob client at a given object size and concurrency. It writes, lists, reads at random and deletes, reporting throughput, latency percentiles, and the retries persist's retry loop needed, so a throttling store shows up as latency and a retry count instead of an error. Reads verify object length.
• test/blob-store-benchmark sweeps that across object sizes, concurrency levels and stored volumes for several stores, samples each store container's CPU and memory, and writes a CSV. Manual-only, so it is excluded from the CI reference lint.

More blob stores in the compositions. garage, rustfs and seaweedfs join minio and azurite as external_blob_store targets for Materialized and Testdrive, resolved through a new blob_store module. The parallel benchmark takes --blob-store and --other-blob-store in place of --azurite (nothing in CI passed that flag to this composition), so two stores can be A/B'd directly, and it prints persist's blob operation counts and latencies after each scenario. The new BlobStoreReadsWrites scenario disables the blob cache so reads actually reach the store.

garage ships an image with no shell, so it gets an mzbuild image that adds one for the cluster setup a fresh node needs before it serves S3.

What the benchmarks found

Measured on one laptop under Docker, one store at a time, no resource limits, fsync off everywhere. Treat these as relative and provisional.

A control run matters here: minio's configuration was unchanged between two rounds, yet its per-cell throughput moved by a median of -6% with a 10th-to-90th range of -44% to +43%. A single-run cell in this harness is not interpretable below roughly 50% difference. Only the large, consistent gaps below clear that bar.

Throughput against minio, across all five object sizes at an empty store, from one round with all four stores measured back to back. Above 1.0 is slower than minio.

• garage is roughly 4x behind minio on reads at every size and saturates by 32 operations in flight, but does it on a fraction of the memory. Its defaults were badly wrong for persist's object sizes: with 10 MiB blocks and a raised read-concurrency cap, its 8 MiB reads went from 70 to 470 ops/s. The reasoning lives in test/garage/garage.toml.
• rustfs tracks minio on reads and writes, is 3 to 4x behind on deletes, and refuses concurrent 64 MiB uploads with SlowDown from an internal write admission limit. Its RUSTFS_HEAL_ENABLED and RUSTFS_SCANNER_ENABLED switches appear inert in 1.0.0-rc.5, so it carries background work minio has switched off.
• seaweedfs is the fastest of the four on small objects and on deletes at every size, and the worst on large writes. At 64 MiB with 128 operations in flight it stopped making progress: 0.0 ops/s and a p50 of about 93 minutes per write with 62 retries, against minio's 28 ops/s, while its resident memory climbed to 25.8 GiB. Reads in that same cell still ran, so writes specifically fell over. That is a harsher failure than rustfs's, which at least refuses work promptly.

For persist, which writes batch parts from a few hundred KiB up to a 128 MiB target, seaweedfs is fastest where persist writes least and slowest where it writes most.

blob-store-benchmark-report.md has the full matrices, scaling curves and caveats. Its numbers are specific to one laptop, so say the word if you would rather it not live in the repo and I will drop that commit.

Testing

Unit tests for the Content-Range parser and the byte-range chunking. The behavioral fix was verified end to end: the scenario that reproducibly crashed environmentd on rustfs (blob cache off, so every scan fetches from the store) now completes with about 20 GiB of cold reads and no failures, twice.

Gotchas for the reviewer

• This is really two changes, the persist fix and the benchmark tooling, and I am happy to split them if you would rather review the production change on its own. The persist change is about 130 lines of the diff.
• --azurite is gone from the parallel benchmark in favour of --blob-store azurite. No CI pipeline passed that flag to this composition, so nothing in CI changes.
• CI will build the new test/garage mzbuild image on this PR.
• seaweedfs is given a static S3 key deliberately. With no S3 config it serves every request unauthenticated and skips the signature verification the other stores perform, which would flatter it in a comparison.
• x-amz-mp-parts-count is part of the S3 GetObject contract and rustfs already defines the constant internally without sending it. Worth an upstream report, which I have not filed yet.
• Lint and formatting were not run against the final state of this branch.

🤖 Generated with Claude Code