feat(fts): read inverted index params without opening the segment by LuQQiu · Pull Request #7816 · lance-format/lance

LuQQiu · 2026-07-15T21:53:47Z

Summary

Reading a segment's tokenizer configuration previously required a full InvertedIndex open — partition construction, token dictionaries loaded into memory, and an index-cache entry — even when the caller only needs the params, e.g. to tokenize query text identically to the index or to inspect how an index was built. Measured on one segment of a large text dataset: 7.7 MB resident and a full open, for a few hundred bytes of config.

The manifest's InvertedIndexDetails is not a substitute: it is a lossy copy of the params that cannot carry custom_stop_words (unbounded user data that doesn't belong in the manifest) nor the json doc type, so a tokenizer rebuilt from it can silently diverge from the index's real tokenizer.

This PR adds a params-only read path:

InvertedIndex::load_params(store) — reads the params JSON from the metadata file's schema metadata; falls back to the legacy tokens-file location, mirroring load_legacy_index. No partitions, no dictionaries, nothing cached.
load_segment_params(dataset, segment) — dataset-level helper via LanceIndexStore::from_dataset_for_existing, re-exported from index::scalar.

The returned params are the complete serialized struct, byte-faithful — custom_stop_words and lance_tokenizer included. The metadata file is read through the session file-metadata cache the store wires up, so repeated calls are memory hits and a later full open of the same segment reuses the read.

Measured (one segment, local FS, cold process):

	latency	resident
`load_segment_params`	0.76 ms	293 B
full index open	17.7 ms (with the file cache already warm)	7.68 MB pinned in the index cache

Test plan

New test test_load_segment_params_full_fidelity: builds an FTS index with custom_stop_words (exactly the field InvertedIndexDetails cannot carry) and asserts load_segment_params equals the fully opened segment's params() field-for-field
cargo check -p lance -p lance-index / fmt clean

🤖 Generated with Claude Code

coderabbitai · 2026-07-15T21:54:07Z

📝 Walkthrough

Walkthrough

Adds metadata-only loading of inverted-index parameters, exposes it through a dataset-level helper, re-exports the helper, and tests that segment-loaded parameters preserve fields such as custom stop words.

Changes

Inverted index parameter loading

Layer / File(s)	Summary
Metadata-only parameter API `rust/lance-index/src/scalar/inverted/index.rs`	`InvertedIndex::load_params` reads current metadata parameters and falls back to legacy tokenizer metadata or defaults.
Segment helper and fidelity validation `rust/lance/src/index/scalar/inverted.rs`, `rust/lance/src/index/scalar.rs`, `rust/lance/src/dataset/tests/dataset_index.rs`	Adds and re-exports `load_segment_params`, then verifies it matches parameters from a fully opened inverted index.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Dataset
  participant load_segment_params
  participant LanceIndexStore
  participant InvertedIndex
  Dataset->>load_segment_params: Provide dataset and segment
  load_segment_params->>LanceIndexStore: Create existing-segment store
  load_segment_params->>InvertedIndex: Load serialized parameters
  InvertedIndex-->>load_segment_params: Return InvertedIndexParams
  load_segment_params-->>Dataset: Return segment parameters

Suggested reviewers: westonpace, xuanwo, bubblecal

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: adding a params-only read path for inverted indexes without opening the segment.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch lu/fts_params_read

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@rust/lance-index/src/scalar/inverted/index.rs`:
- Around line 1209-1215: Add documentation examples and cross-links for both
public APIs: in rust/lance-index/src/scalar/inverted/index.rs (lines 1209-1215),
document InvertedIndex::load_params with a minimal metadata-only loading example
and links to IndexStore and the dataset-level helper; in
rust/lance/src/index/scalar/inverted.rs (lines 223-239), document the
segment-loading API with a minimal example and a link to
InvertedIndex::load_params.
- Around line 1216-1236: Update the metadata probe in the parameter-loading path
to fall back to the legacy TOKENS_FILE format only when
open_index_file(METADATA_FILE) returns the store’s not-found error; propagate
permission, I/O, and metadata errors unchanged. Extract or reuse a shared
metadata-presence probe between this path and load so both use identical
not-found handling.

In `@rust/lance/src/dataset/tests/dataset_index.rs`:
- Around line 4141-4163: Expand the inverted-index parameter test around
create_index and load_segment_params to cover a non-default lance_tokenizer and
a dataset containing multiple fragments, verifying parameters for the relevant
segments. Add a checked-in older-version legacy fixture loaded through
copy_test_data_to_tmp, assert its datagen.py version, and verify the TOKENS_FILE
fallback preserves and reloads the expected parameters.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: eeea5a4a-4c01-471c-bf76-2d518700974e

📥 Commits

Reviewing files that changed from the base of the PR and between 3ece36f and d709f84.

📒 Files selected for processing (4)

rust/lance-index/src/scalar/inverted/index.rs
rust/lance/src/dataset/tests/dataset_index.rs
rust/lance/src/index/scalar.rs
rust/lance/src/index/scalar/inverted.rs

coderabbitai · 2026-07-15T21:57:55Z

+    /// Read only the index's [`InvertedIndexParams`] from `store` — no
+    /// partitions, no token dictionaries, no [`InvertedIndex`]
+    /// construction. This is the complete serialized params (including
+    /// fields the manifest's `InvertedIndexDetails` cannot carry, such as
+    /// `custom_stop_words` and `lance_tokenizer`), so callers can rebuild
+    /// the index's exact tokenizer from one small metadata read.
+    pub async fn load_params(store: &dyn IndexStore) -> Result<InvertedIndexParams> {


📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win

Add examples and cross-links for both new public APIs.

rust/lance-index/src/scalar/inverted/index.rs#L1209-L1215: add a minimal metadata-only loading example and link to IndexStore plus the dataset-level helper.

rust/lance/src/index/scalar/inverted.rs#L223-L239: add a minimal segment-loading example and link to InvertedIndex::load_params.

As per coding guidelines, “Document all public APIs with examples and links to relevant structs and methods.”

📍 Affects 2 files

rust/lance-index/src/scalar/inverted/index.rs#L1209-L1215 (this comment)

rust/lance/src/index/scalar/inverted.rs#L223-L239

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/lance-index/src/scalar/inverted/index.rs` around lines 1209 - 1215, Add documentation examples and cross-links for both public APIs: in rust/lance-index/src/scalar/inverted/index.rs (lines 1209-1215), document InvertedIndex::load_params with a minimal metadata-only loading example and links to IndexStore and the dataset-level helper; in rust/lance/src/index/scalar/inverted.rs (lines 223-239), document the segment-loading API with a minimal example and a link to InvertedIndex::load_params.

Source: Coding guidelines

coderabbitai · 2026-07-15T21:57:55Z

+        match store.open_index_file(METADATA_FILE).await {
+            Ok(reader) => {
+                let params = reader
+                    .schema()
+                    .metadata
+                    .get("params")
+                    .ok_or(Error::index("params not found in metadata".to_owned()))?;
+                Ok(serde_json::from_str::<InvertedIndexParams>(params)?)
+            }
+            Err(_) => {
+                // Legacy single-partition format: params live in the
+                // tokens file, defaulting when absent — mirrors
+                // `load_legacy_index`.
+                let reader = store.open_index_file(TOKENS_FILE).await?;
+                Ok(reader
+                    .schema()
+                    .metadata
+                    .get("tokenizer")
+                    .map(|s| serde_json::from_str::<InvertedIndexParams>(s))
+                    .transpose()?
+                    .unwrap_or_default())


🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash set -euo pipefail # Locate the relevant file and surrounding symbols. FILE="rust/lance-index/src/scalar/inverted/index.rs" echo "== outline ==" ast-grep outline "$FILE" --view expanded || true echo echo "== line numbers around load/load_params ==" grep -n "async fn load\|async fn load_params\|METADATA_FILE\|TOKENS_FILE\|params not found\|tokenizer" -n "$FILE" || true echo echo "== relevant slices ==" sed -n '1140,1265p' "$FILE"

Repository: lance-format/lance

Length of output: 1922

🏁 Script executed:

#!/bin/bash set -euo pipefail FILE="rust/lance-index/src/scalar/inverted/index.rs" grep -n "async fn load\|async fn load_params\|METADATA_FILE\|TOKENS_FILE\|params not found\|tokenizer" "$FILE" || true sed -n '1180,1265p' "$FILE"

Repository: lance-format/lance

Length of output: 1922

Only fall back when the metadata file is missing.
Err(_) here treats permission, I/O, and corrupt-metadata failures as the legacy format, which hides the real error behind a tokens-file read or default params. Match the store’s not-found error only, and share this probe with load so both paths stay aligned.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/lance-index/src/scalar/inverted/index.rs` around lines 1216 - 1236, Update the metadata probe in the parameter-loading path to fall back to the legacy TOKENS_FILE format only when open_index_file(METADATA_FILE) returns the store’s not-found error; propagate permission, I/O, and metadata errors unchanged. Extract or reuse a shared metadata-presence probe between this path and load so both use identical not-found handling.

Source: Coding guidelines

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@rust/lance/src/dataset/tests/dataset_index.rs`:
- Around line 4121-4137: Update the test setup around Dataset::write: move the
shown imports to file scope, replace the manual RecordBatch construction with
arrow_array’s record_batch!() macro, and change the storage URI to plain
"memory://". Preserve the existing test data and schema behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: a2ed6814-2c56-4a51-a402-6f9aee17a5aa

📥 Commits

Reviewing files that changed from the base of the PR and between d709f84 and ad910fd.

📒 Files selected for processing (4)

rust/lance-index/src/scalar/inverted/index.rs
rust/lance/src/dataset/tests/dataset_index.rs
rust/lance/src/index/scalar.rs
rust/lance/src/index/scalar/inverted.rs

coderabbitai · 2026-07-15T22:05:19Z

+    use crate::index::DatasetIndexInternalExt;
+    use lance_index::metrics::NoOpMetricsCollector;
+    use lance_index::scalar::inverted::InvertedIndex;
+
+    let batch = RecordBatch::try_new(
+        arrow_schema::Schema::new(vec![Field::new("text", DataType::Utf8, false)]).into(),
+        vec![Arc::new(StringArray::from(vec![
+            "the quick brown fox",
+            "lazy dogs sleep",
+        ]))],
+    )
+    .unwrap();
+    let schema = batch.schema();
+    let stream = RecordBatchIterator::new(vec![batch].into_iter().map(Ok), schema);
+    let mut dataset = Dataset::write(stream, "memory://test/segment_params", None)
+        .await
+        .unwrap();


📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Use the standard Rust test setup conventions.

Move the use declarations to file scope, construct the batch with record_batch!(), and use plain "memory://" instead of a path-qualified URI.

As per coding guidelines, “Place use imports at the top of the file,” “Use record_batch!() from arrow_array to construct RecordBatch in tests,” and “Use plain "memory://" URIs in tests.”

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/lance/src/dataset/tests/dataset_index.rs` around lines 4121 - 4137, Update the test setup around Dataset::write: move the shown imports to file scope, replace the manual RecordBatch construction with arrow_array’s record_batch!() macro, and change the storage URI to plain "memory://". Preserve the existing test data and schema behavior.

Source: Coding guidelines

Distributed callers need a segment's tokenizer config (to tokenize query text for cross-segment BM25 stats), but the only way to get it was a full index open: partition construction, token dictionaries resident in memory, and an index-cache entry — hundreds of ms and tens-to-hundreds of MB for a few hundred bytes of config. The manifest's InvertedIndexDetails is not a substitute: it is a lossy copy that cannot carry custom_stop_words or the json doc type. Add InvertedIndex::load_params (params JSON from the metadata file's schema metadata, with the legacy tokens-file fallback mirroring load_legacy_index) and a dataset-level load_segment_params helper. One small metadata read, byte-faithful params, nothing cached. Co-Authored-By: Claude Fable 5 <[email protected]>

codecov · 2026-07-15T22:45:08Z

Codecov Report

❌ Patch coverage is 56.00000% with 11 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-index/src/scalar/inverted/index.rs	44.44%	8 Missing and 2 partials ⚠️
rust/lance/src/index/scalar/inverted.rs	85.71%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

) ## Summary Reading a segment's tokenizer configuration previously required a full `InvertedIndex` open — partition construction, token dictionaries loaded into memory, and an index-cache entry — even when the caller only needs the params, e.g. to tokenize query text identically to the index or to inspect how an index was built. Measured on one segment of a large text dataset: **7.7 MB resident and a full open, for a few hundred bytes of config**. The manifest's `InvertedIndexDetails` is not a substitute: it is a lossy copy of the params that cannot carry `custom_stop_words` (unbounded user data that doesn't belong in the manifest) nor the json doc type, so a tokenizer rebuilt from it can silently diverge from the index's real tokenizer. This PR adds a params-only read path: - `InvertedIndex::load_params(store)` — reads the params JSON from the metadata file's schema metadata; falls back to the legacy tokens-file location, mirroring `load_legacy_index`. No partitions, no dictionaries, nothing cached. - `load_segment_params(dataset, segment)` — dataset-level helper via `LanceIndexStore::from_dataset_for_existing`, re-exported from `index::scalar`. The returned params are the complete serialized struct, byte-faithful — `custom_stop_words` and `lance_tokenizer` included. The metadata file is read through the session file-metadata cache the store wires up, so repeated calls are memory hits and a later full open of the same segment reuses the read. Measured (one segment, local FS, cold process): | | latency | resident | |---|---|---| | `load_segment_params` | 0.76 ms | 293 B | | full index open | 17.7 ms (with the file cache already warm) | 7.68 MB pinned in the index cache | ## Test plan - New test `test_load_segment_params_full_fidelity`: builds an FTS index with `custom_stop_words` (exactly the field `InvertedIndexDetails` cannot carry) and asserts `load_segment_params` equals the fully opened segment's `params()` field-for-field - `cargo check -p lance -p lance-index` / fmt clean 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <[email protected]> (cherry picked from commit 2ba078c)

github-actions Bot added A-index Vector index, linalg, tokenizer enhancement New feature or request labels Jul 15, 2026

coderabbitai Bot reviewed Jul 15, 2026

View reviewed changes

LuQQiu force-pushed the lu/fts_params_read branch from d709f84 to ad910fd Compare July 15, 2026 21:59

coderabbitai Bot reviewed Jul 15, 2026

View reviewed changes

LuQQiu force-pushed the lu/fts_params_read branch from ad910fd to e968f2a Compare July 15, 2026 22:07

LuQQiu force-pushed the lu/fts_params_read branch from e968f2a to 74d687d Compare July 15, 2026 22:12

LuQQiu requested a review from jackye1995 July 15, 2026 22:36

jackye1995 approved these changes Jul 15, 2026

View reviewed changes

LuQQiu merged commit 2ba078c into main Jul 15, 2026
39 checks passed

LuQQiu deleted the lu/fts_params_read branch July 15, 2026 22:50

This was referenced Jul 20, 2026

fix: make FTS metadata loading retry-safe #7838

Merged

fix(fts): preserve parameter compatibility #7874

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(fts): read inverted index params without opening the segment#7816

feat(fts): read inverted index params without opening the segment#7816
LuQQiu merged 1 commit into
mainfrom
lu/fts_params_read

LuQQiu commented Jul 15, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jul 15, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

coderabbitai Bot left a comment

Uh oh!

coderabbitai Bot Jul 15, 2026

Uh oh!

coderabbitai Bot Jul 15, 2026

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Uh oh!

coderabbitai Bot Jul 15, 2026

Uh oh!

codecov Bot commented Jul 15, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

LuQQiu commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

coderabbitai Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Jul 15, 2026

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

LuQQiu commented Jul 15, 2026 •

edited

Loading

coderabbitai Bot commented Jul 15, 2026 •

edited

Loading