Skip to content

perf(index): parallelize FMIndex partition builds#7422

Merged
jackye1995 merged 1 commit into
lance-format:mainfrom
jackye1995:jack/fix-fmindex-build-regression
Jun 26, 2026
Merged

perf(index): parallelize FMIndex partition builds#7422
jackye1995 merged 1 commit into
lance-format:mainfrom
jackye1995:jack/fix-fmindex-build-regression

Conversation

@jackye1995

Copy link
Copy Markdown
Contributor

Summary:

  • Build streamed FMIndex partitions with bounded parallel CPU workers.
  • Add row and byte partition controls with deterministic file ordering.
  • Cover byte-based partition splits and worker write failures.

@github-actions github-actions Bot added A-index Vector index, linalg, tokenizer performance labels Jun 23, 2026
@codecov

codecov Bot commented Jun 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.45161% with 42 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
rust/lance-index/src/scalar/fmindex.rs 86.45% 30 Missing and 12 partials ⚠️

📢 Thoughts on this report? Let us know!

@jackye1995
jackye1995 merged commit efc6c13 into lance-format:main Jun 26, 2026
31 checks passed
Xuanwo pushed a commit that referenced this pull request Jun 26, 2026
#7495)

## What

The `lance-index` **test build** is currently broken on `main`:
`error[E0046]: not all trait items implemented, missing:
with_io_priority` at `rust/lance-index/src/scalar/fmindex.rs`.

`with_io_priority` was added to the `IndexStore` trait in #7449. The
`FailNewFileStore` test fixture was added later in #7422 (which branched
before #7449 landed), so it never implemented the new method. Each PR
was green on its own; together on `main` the test build fails — a
semantic merge conflict. It only surfaces in the `Build tests` step
because the fixture is `#[cfg(test)]`.

## Fix

Implement `with_io_priority` on `FailNewFileStore` by delegating to the
inner store and re-wrapping, mirroring how the fixture already delegates
its other read methods while keeping the injected `new_index_file`
failure.

## Verification

- `cargo check -p lance-index --tests` — compiles (was failing on
`main`).
- `cargo test -p lance-index --lib fmindex` — 24 passed.
- `cargo fmt -p lance-index -- --check` and `cargo clippy -p lance-index
--tests` — clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-index Vector index, linalg, tokenizer performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants