fix(rust): bound remote insert request size to avoid ingestion timeouts#3630
Conversation
| // lands exactly on the end of input from emitting a trailing empty part. | ||
| let mut first = match input.next().await { | ||
| Some(batch) => batch?, | ||
| None => return Ok(()), |
There was a problem hiding this comment.
Could we keep staging one schema-only part when the whole multipart input is empty, or avoid multipart for known zero-row inputs? This early return means an empty multipart overwrite can create an upload and complete it without any insert request; since mode=overwrite is only attached to the per-part insert request, complete has no staged part carrying the overwrite intent. That seems like a regression from the previous schema-only empty insert request behavior.
There was a problem hiding this comment.
Good point. Let's see if we can initiate the upload only when we have data.
There was a problem hiding this comment.
Fixed in e642f9f: add() now peeks the input regardless of write_parallelism and falls back to the single-request path when it's fully empty, so multipart is never entered for a zero-batch write. Added test_multipart_write_empty_overwrite_uses_single_partition to cover the empty-overwrite-with-explicit-parallelism case.
An explicit write_parallelism forced the multipart path even for fully empty input. Since an empty partition stages no part, an all-empty multipart write left nothing for `complete` to commit, silently dropping `mode=overwrite`. Peek the input regardless of write_parallelism and fall back to the single-request path (which always sends a schema-only request) when it's empty. Addresses review feedback on lancedb#3630.
On the remote write path each write partition was uploaded as a single `/insert?upload_id=...` request that stayed open until the whole partition was streamed and written to object storage. For large bulk ingests a partition can be many GB, so one request could exceed the client read timeout (default 300s), surfacing as `HttpError: operation timed out`. Split each partition into parts of at most `max_bytes_per_request` (Arrow IPC, compressed) bytes, each uploaded as its own request under the shared `upload_id` (a distinct `upload_part_id` per part). The server already stages multiple parts per session and merges them atomically on complete, so no server change is needed. Each part body is still streamed through a bounded channel while the request is in flight, so peak memory stays at a couple of batches per partition regardless of the part size. An empty partition still sends exactly one part; a size cut landing on the end of input does not emit a trailing empty part. Configurable via `ClientConfig::max_bytes_per_request` or the `LANCE_CLIENT_MAX_BYTES_PER_REQUEST` env var; defaults to 1 GiB, `0` disables splitting. Related to ENT-1883 Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
- Take &str/&client in send_multipart_chunked instead of owned String - Early-return on empty partitions instead of staging a schema-only part - Split the long chunking fn into build_part_request / send_part_request / send_one_part helpers Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The client read timeout covers the request-body upload, not just the response, so a slow or throttled upload of a single large part can hit it even while making progress. Splitting only by bytes doesn't help when throughput is low. Add a per-part wall-clock budget (max_request_duration, env LANCE_CLIENT_MAX_REQUEST_DURATION, default read_timeout/2) and cut a part on whichever of bytes/time is reached first. Raise the byte default from 1 GiB to 8 GiB so a part holds at least one full Lance data file (1M rows) and fragments aren't split undersized across parts; the time cut keeps large parts from risking the read timeout. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Add unit tests for the max-bytes / max-duration resolution helpers (config precedence, env parsing, invalid-env errors, zero-disables, default derivation) and for multipart chunking behaviors that were previously untested: empty partitions staging nothing, distinct upload_part_id per part, per-partition chunking, and surfacing the original input error over an induced HTTP error. Also fold the per-request-constant arguments of the multipart part helpers into a PartRequestCtx struct, dropping the too_many_arguments allows, and document the count-0 result, per-part overwrite mode, join!-not-spawn choice, and the Option semantics that invert between the config and resolved-client layers. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Address review feedback: - The chunked multipart path recorded a part's bytes only after the whole part finished uploading, so the progress bar jumped once per part and looked like it was stalling and restarting. Push byte recording down into the per-part producer stream so progress advances as each chunk is produced, matching the non-multipart path. send_one_part now records on the tracker and returns only whether the input ended. - The max-bytes-per-request budget was a usize, so the 8 GiB default overflows on 32-bit targets. Use u64 throughout (config, resolution, client, exec) so the default is representable everywhere and the per-part byte accumulator cannot overflow. Add a test asserting progress is reported multiple times, monotonically, within a single part. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
An explicit write_parallelism forced the multipart path even for fully empty input. Since an empty partition stages no part, an all-empty multipart write left nothing for `complete` to commit, silently dropping `mode=overwrite`. Peek the input regardless of write_parallelism and fall back to the single-request path (which always sends a schema-only request) when it's empty. Addresses review feedback on lancedb#3630.
Rebasing onto upstream/main pulls in a Lance/DataFusion bump where ExecutionPlan no longer declares as_any() as a trait method (downcast goes through the Any supertrait coercion instead, as the rest of the codebase already does). The test-only ErroringExec mock still defined it explicitly, which no longer compiles.
e642f9f to
ae525bf
Compare
|
Docs PR opened: lancedb/docs#314 Added a subsection to the performance guide covering two environment variables that cap remote insert request size and duration. |
Problem
On the remote (LanceDB Cloud) write path, each write partition is uploaded as a single
/insert?upload_id=...request that stays open until the whole partition has been streamed and the server has written it to object storage. For large bulk ingests a partition can be many GB, so a single request can run longer than the client read timeout (default 300s), surfacing as:The server already supports staging multiple parts under one
upload_id(each/insertwrites a separate transaction thatcompletemerges atomically), but the client never used that — it sent one part per partition.Change
Split each partition into multiple parts of at most
max_bytes_per_request(Arrow IPC, LZ4-compressed) bytes, each uploaded as its own/insert?upload_id=...&upload_part_id=...request. This bounds how long any single request stays open, independent of total data size or write parallelism.Key properties:
futures::join!of a producer + the send), so peak memory stays at a couple of batches per partition regardless of the part size. Backpressure from a slow/throttled server still propagates upstream.completehas a transaction to commit; a size cut landing exactly on the end of input does not emit a trailing empty part.Config
New
ClientConfig::max_bytes_per_request: Option<usize>, also settable via theLANCE_CLIENT_MAX_BYTES_PER_REQUESTenvironment variable. Default 1 GiB (Some(0)disables splitting → one request per partition). Python users pick up the default/env automatically through the remote client.Tests
test_multipart_chunked_splits_into_parts: a 1-byte budget puts each batch in its own part → N requests, each carrying the sharedupload_idand a distinctupload_part_id.test_multipart_single_part_when_under_budget: a large budget keeps the partition in a single request.Related to ENT-1883.
🤖 Generated with Claude Code