Skip to content

fix(rust): bound remote insert request size to avoid ingestion timeouts#3630

Merged
wjones127 merged 7 commits into
lancedb:mainfrom
wjones127:will/ent-1883-multipart-request-chunking
Jul 17, 2026
Merged

fix(rust): bound remote insert request size to avoid ingestion timeouts#3630
wjones127 merged 7 commits into
lancedb:mainfrom
wjones127:will/ent-1883-multipart-request-chunking

Conversation

@wjones127

Copy link
Copy Markdown
Contributor

Problem

On the remote (LanceDB Cloud) write path, each write partition is uploaded as a single /insert?upload_id=... request that stays open until the whole partition has been streamed and the server has written it to object storage. For large bulk ingests a partition can be many GB, so a single request can run longer than the client read timeout (default 300s), surfacing as:

lancedb.remote.errors.HttpError: operation timed out

The server already supports staging multiple parts under one upload_id (each /insert writes a separate transaction that complete merges atomically), but the client never used that — it sent one part per partition.

Change

Split each partition into multiple parts of at most max_bytes_per_request (Arrow IPC, LZ4-compressed) bytes, each uploaded as its own /insert?upload_id=...&upload_part_id=... request. This bounds how long any single request stays open, independent of total data size or write parallelism.

Key properties:

  • Still streamed, not buffered. Each part's body is driven through a bounded channel while the request is in flight (futures::join! of a producer + the send), so peak memory stays at a couple of batches per partition regardless of the part size. Backpressure from a slow/throttled server still propagates upstream.
  • Correct part accounting. An empty partition still sends exactly one (schema-only) part so complete has a transaction to commit; a size cut landing exactly on the end of input does not emit a trailing empty part.
  • Multipart only. The single-request (non-multipart) path is unchanged.

Config

New ClientConfig::max_bytes_per_request: Option<usize>, also settable via the LANCE_CLIENT_MAX_BYTES_PER_REQUEST environment variable. Default 1 GiB (Some(0) disables splitting → one request per partition). Python users pick up the default/env automatically through the remote client.

Tests

  • test_multipart_chunked_splits_into_parts: a 1-byte budget puts each batch in its own part → N requests, each carrying the shared upload_id and a distinct upload_part_id.
  • test_multipart_single_part_when_under_budget: a large budget keeps the partition in a single request.
  • Verified end-to-end against a live remote table: a forced-chunked multipart add (many parts) assembles to the correct row count.

Related to ENT-1883.

🤖 Generated with Claude Code

@github-actions github-actions Bot added bug Something isn't working Python Python SDK Rust Rust related issues labels Jul 7, 2026
Comment thread rust/lancedb/src/remote/table/insert.rs Outdated
Comment thread rust/lancedb/src/remote/table/insert.rs Outdated
Comment thread rust/lancedb/src/remote/table/insert.rs Outdated
Comment thread rust/lancedb/src/remote/table/insert.rs Outdated
Comment thread rust/lancedb/src/remote/client.rs Outdated
@wjones127
wjones127 marked this pull request as ready for review July 10, 2026 16:10

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

// lands exactly on the end of input from emitting a trailing empty part.
let mut first = match input.next().await {
Some(batch) => batch?,
None => return Ok(()),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we keep staging one schema-only part when the whole multipart input is empty, or avoid multipart for known zero-row inputs? This early return means an empty multipart overwrite can create an upload and complete it without any insert request; since mode=overwrite is only attached to the per-part insert request, complete has no staged part carrying the overwrite intent. That seems like a regression from the previous schema-only empty insert request behavior.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. Let's see if we can initiate the upload only when we have data.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in e642f9f: add() now peeks the input regardless of write_parallelism and falls back to the single-request path when it's fully empty, so multipart is never entered for a zero-batch write. Added test_multipart_write_empty_overwrite_uses_single_partition to cover the empty-overwrite-with-explicit-parallelism case.

wjones127 added a commit to wjones127/lancedb that referenced this pull request Jul 16, 2026
An explicit write_parallelism forced the multipart path even for
fully empty input. Since an empty partition stages no part, an
all-empty multipart write left nothing for `complete` to commit,
silently dropping `mode=overwrite`. Peek the input regardless of
write_parallelism and fall back to the single-request path (which
always sends a schema-only request) when it's empty.

Addresses review feedback on lancedb#3630.
wjones127 and others added 7 commits July 17, 2026 09:11
On the remote write path each write partition was uploaded as a single
`/insert?upload_id=...` request that stayed open until the whole partition
was streamed and written to object storage. For large bulk ingests a
partition can be many GB, so one request could exceed the client read
timeout (default 300s), surfacing as `HttpError: operation timed out`.

Split each partition into parts of at most `max_bytes_per_request` (Arrow
IPC, compressed) bytes, each uploaded as its own request under the shared
`upload_id` (a distinct `upload_part_id` per part). The server already
stages multiple parts per session and merges them atomically on complete,
so no server change is needed. Each part body is still streamed through a
bounded channel while the request is in flight, so peak memory stays at a
couple of batches per partition regardless of the part size. An empty
partition still sends exactly one part; a size cut landing on the end of
input does not emit a trailing empty part.

Configurable via `ClientConfig::max_bytes_per_request` or the
`LANCE_CLIENT_MAX_BYTES_PER_REQUEST` env var; defaults to 1 GiB, `0`
disables splitting.

Related to ENT-1883

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
- Take &str/&client in send_multipart_chunked instead of owned String
- Early-return on empty partitions instead of staging a schema-only part
- Split the long chunking fn into build_part_request / send_part_request /
  send_one_part helpers

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The client read timeout covers the request-body upload, not just the
response, so a slow or throttled upload of a single large part can hit it
even while making progress. Splitting only by bytes doesn't help when
throughput is low.

Add a per-part wall-clock budget (max_request_duration, env
LANCE_CLIENT_MAX_REQUEST_DURATION, default read_timeout/2) and cut a part
on whichever of bytes/time is reached first. Raise the byte default from
1 GiB to 8 GiB so a part holds at least one full Lance data file
(1M rows) and fragments aren't split undersized across parts; the time
cut keeps large parts from risking the read timeout.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Add unit tests for the max-bytes / max-duration resolution helpers
(config precedence, env parsing, invalid-env errors, zero-disables,
default derivation) and for multipart chunking behaviors that were
previously untested: empty partitions staging nothing, distinct
upload_part_id per part, per-partition chunking, and surfacing the
original input error over an induced HTTP error.

Also fold the per-request-constant arguments of the multipart part
helpers into a PartRequestCtx struct, dropping the
too_many_arguments allows, and document the count-0 result, per-part
overwrite mode, join!-not-spawn choice, and the Option semantics that
invert between the config and resolved-client layers.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Address review feedback:

- The chunked multipart path recorded a part's bytes only after the whole
  part finished uploading, so the progress bar jumped once per part and
  looked like it was stalling and restarting. Push byte recording down
  into the per-part producer stream so progress advances as each chunk is
  produced, matching the non-multipart path. send_one_part now records on
  the tracker and returns only whether the input ended.

- The max-bytes-per-request budget was a usize, so the 8 GiB default
  overflows on 32-bit targets. Use u64 throughout (config, resolution,
  client, exec) so the default is representable everywhere and the
  per-part byte accumulator cannot overflow.

Add a test asserting progress is reported multiple times, monotonically,
within a single part.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
An explicit write_parallelism forced the multipart path even for
fully empty input. Since an empty partition stages no part, an
all-empty multipart write left nothing for `complete` to commit,
silently dropping `mode=overwrite`. Peek the input regardless of
write_parallelism and fall back to the single-request path (which
always sends a schema-only request) when it's empty.

Addresses review feedback on lancedb#3630.
Rebasing onto upstream/main pulls in a Lance/DataFusion bump where
ExecutionPlan no longer declares as_any() as a trait method (downcast
goes through the Any supertrait coercion instead, as the rest of the
codebase already does). The test-only ErroringExec mock still defined
it explicitly, which no longer compiles.
@wjones127
wjones127 force-pushed the will/ent-1883-multipart-request-chunking branch from e642f9f to ae525bf Compare July 17, 2026 16:41
@wjones127
wjones127 merged commit 5d0a1ef into lancedb:main Jul 17, 2026
25 checks passed
@mintlify

mintlify Bot commented Jul 17, 2026

Copy link
Copy Markdown

Docs PR opened: lancedb/docs#314

Added a subsection to the performance guide covering two environment variables that cap remote insert request size and duration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working Python Python SDK Rust Rust related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants