objectstore-plugin: upload reservations cover a path the upload never takes, which caps snapshot throughput on large nodes #2847

What happened?

Follow-up to #2553 / #2708. On a large node, snapshot uploads at high density spend most of their time waiting for the shared upload budget, not uploading. Raising --upload-memory-budget helps less than it should, because each upload's reservation grows with the budget and with GOMAXPROCS.

Uploader.plan reserves max(streaming, ranged) for every file:

• streaming: uploadBufferMemory (320 MiB on GCS) + workers × parallelWorkerMemory (64 MiB each), with workers = min(GOMAXPROCS, (budget − 320 MiB) / 64 MiB, populated / 8 MiB + 1)
• ranged: parts × sparseUploadMemory (48 MiB each), at most 12 parts

On GCS, a file with two or more parts always takes the ranged path: PutSparseFile only falls back to streaming below two parts. So the streaming term never runs for these files, but it dominates their reservation. On a 192-vCPU node with a 16Gi budget:

Two things make it worse:

• golang.org/x/sync/semaphore is FIFO, so a metadata file or a manifest waits behind multi-GiB reservations.
• The manifest is uploaded in a second UploadSnapshot call after the data files, so under load every snapshot waits in the queue twice.

The measurements below use synchronous suspends on v0.5. The plugin code is unchanged on main, where SuspendActor is asynchronous by default (#2805). There, background uploads go through the same two plugin calls, so the budget also caps how fast they drain. At the default budget, one memory image fills it, so only one of the two default upload workers (#2802) makes progress at a time.

Related work: #2633 reduces what small files actually allocate. #2835 measures a part's encoder at ~26 MiB, inside the 32 MiB serialWorkerMemory that the ranged reservation below relies on. Neither changes how much plan() reserves.

Measurements

Setup: one c3-highmem-192-metal node (v0.5, microVM, GCS backend), snapshot staging on a dedicated Hyperdisk Balanced disk. Three probe actors (512 MiB Glutton) run synchronous SuspendActor and resume every 20–30 s, next to 64 SWE-bench replay actors that suspend and resume back to back. Each run is 12 minutes; p95 is taken after a 3-minute warm-up, one run per configuration. Phase times come from per-request traces and the plugin's Compressed zstd upload logs.

At 16Gi, 82% of a suspend's checkpoint time (p50 over ~2,000 suspends) is waiting for the budget: 4.3 s for the data files and 4.2 s for the manifest. The memory-image upload itself takes 1.2 s and the manifest PUT 0.07 s.

The two patched rows ran an earlier revision that reserved only the 64 MiB lookahead for single-request objects. The fix above also counts the client's chunk buffer, so it reserves 64 MiB more per metadata file and manifest. That shouldn't affect the 16Gi row: modelled reservations peaked at 11.5 GiB, well under the budget. It may trim the 512Mi gain somewhat; that isn't re-measured.

The probes-only suspend p95 is 3.0–4.5 s. Each upload's median speed stayed between 248 and 284 MB/s for the SWE-bench snapshots in every run, and a raw upload test from the same node reached at least 2.5 GB/s to GCS. So the node's throughput is set by how many uploads the budget admits, not by the network, compression or CPU: the plugin used 5.5 cores at 16Gi and 11 cores with the patch. GOMAXPROCS=16 helps only because it caps workers and so shrinks each reservation.

Proposed fix

Have plan() reserve for the path that will run:

• For a file whose backend implements sparseFilePutter and that has ≥ 2 parts, reserve parts × sparseUploadMemory. PutSparseFile will split it and never fall back.
• For a streamed object whose sparse stream is bounded below uploadCompositeMin, reserve the single request's buffers (uploadCompositeMin lookahead + uploadChunkSize client chunk buffer) + workers, not the composite part buffers, since GCS sends it in one request.
• Size Send's reservation by the file when it is a small regular file.

With the patch, the modelled reservations average 2.2 GiB at 16Gi against a measured 2.0 GiB average working set. With today's sizing they average 14.6 GiB for 1.3 GiB. So the budget keeps bounding memory (#2553) without capping throughput.

At the 512Mi default the patch helps about 3×, since two memory images fit instead of one. But with 64 SWE-bench actors, suspend p95 is still ~10× the quiet baseline, so the default itself should scale with the node, as suggested in #2553. With the patch at 512Mi, the plugin's working set peaked at 1.3 GiB (0.5 GiB without it), because more uploads run at once and the budget only counts upload buffers. A total-memory cap needs GOMEMLIMIT or a container limit.

Smaller follow-ups:

• The composed parts are deleted one by one before the upload returns: 0.17 s for 4 parts, 0.48 s for 9.
• With nothing queuing, synchronized suspends can start 50+ uploads at once. If that tail matters, a cap on concurrent uploads is a separate knob from memory.

I'll send a PR with the change.