fix(index): give each FTS partition a distinct scheduler base priority#7449
Merged
Conversation
Multi-partition inverted-index reads (FTS prewarm, corpus stats, df) fan out over partitions through one shared ScanScheduler. Every partition opened its files at base_priority 0, so all concurrent reads tied at the same priority. The scheduler's backpressure deadlock-break admits the lowest-priority in-flight request without a byte-budget check, which requires a total order over in-flight priorities to guarantee one request can always advance and drain the byte budget. With every partition at priority 0 there is no unique lowest request, so when the shared 64-IOP scheduler saturates, freed IOP slots go to arbitrary tied tasks and no consumer's batch is guaranteed to complete and refund resources -> the I/O loop can wedge with work pending but none admissible. Stamp each partition's IndexStore with its enumerate index as the base priority at load time, mirroring how a filtered read scan prioritizes each fragment. The stores share the same scheduler (Arc), so global backpressure is unchanged; only the priority differs, restoring the total order across partitions that the deadlock-break relies on. The single, non-concurrent metadata read stays at priority 0 (no tie). Adds IndexStore::with_base_priority (default no-op for backends without a priority concept) and a test asserting partitions load with distinct, dense priorities. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Multi-partition inverted-index reads (FTS prewarm, corpus stats, df) fan out over partitions through one shared ScanScheduler. Every partition opened its files at base_priority 0, so all concurrent reads tied at the same priority. The scheduler's backpressure deadlock-break admits the lowest-priority in-flight request without a byte-budget check, which requires a total order over in-flight priorities to guarantee one request can always advance and drain the byte budget. With every partition at priority 0 there is no unique lowest request, so when the shared 64-IOP scheduler saturates, freed IOP slots go to arbitrary tied tasks and no consumer's batch is guaranteed to complete and refund resources -> the I/O loop can wedge with work pending but none admissible. Stamp each partition's IndexStore with its enumerate index as the base priority at load time, mirroring how a filtered read scan prioritizes each fragment. The stores share the same scheduler (Arc), so global backpressure is unchanged; only the priority differs, restoring the total order across partitions that the deadlock-break relies on. The single, non-concurrent metadata read stays at priority 0 (no tie). Adds IndexStore::with_base_priority (default no-op for backends without a priority concept) and a test asserting partitions load with distinct, dense priorities. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Xuanwo
pushed a commit
that referenced
this pull request
Jun 26, 2026
#7495) ## What The `lance-index` **test build** is currently broken on `main`: `error[E0046]: not all trait items implemented, missing: with_io_priority` at `rust/lance-index/src/scalar/fmindex.rs`. `with_io_priority` was added to the `IndexStore` trait in #7449. The `FailNewFileStore` test fixture was added later in #7422 (which branched before #7449 landed), so it never implemented the new method. Each PR was green on its own; together on `main` the test build fails — a semantic merge conflict. It only surfaces in the `Build tests` step because the fixture is `#[cfg(test)]`. ## Fix Implement `with_io_priority` on `FailNewFileStore` by delegating to the inner store and re-wrapping, mirroring how the fixture already delegates its other read methods while keeping the injected `new_index_file` failure. ## Verification - `cargo check -p lance-index --tests` — compiles (was failing on `main`). - `cargo test -p lance-index --lib fmindex` — 24 passed. - `cargo fmt -p lance-index -- --check` and `cargo clippy -p lance-index --tests` — clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
FTS prewarm fans out over partitions through one shared
ScanScheduler. Every partition opened its files atbase_priority = 0, so all concurrent reads tied at the same priority.The scheduler's deadlock-break admits the lowest-priority in-flight request without a byte-budget check (
task.priority <= min_in_flight), which needs a total order to guarantee one request always advances. With all priorities tied at 0, freed IOP slots go to arbitrary tasks → no consumer drains → the I/O loop wedges (FTS index-cache prewarm hang on large multi-partition indexes).Fix
Stamp each partition's
IndexStorewith its index asbase_priorityat load time (mirrors how a scan prioritizes each fragment). Stores share the same scheduler, so backpressure is unchanged — only the priority differs, restoring a total order across partitions.Validation
Reproduced the hang on a 3-node cluster (600M-row multi-partition FTS index), verified the fix reaches
complete=3/3. Adds a test for distinct, dense partition priorities.🤖 Generated with Claude Code