Skip to content

feat: support segment selection in pylance prewarm#7677

Merged
jackye1995 merged 1 commit into
lance-format:mainfrom
jackye1995:jack/pylance-segment-prewarm
Jul 8, 2026
Merged

feat: support segment selection in pylance prewarm#7677
jackye1995 merged 1 commit into
lance-format:mainfrom
jackye1995:jack/pylance-segment-prewarm

Conversation

@jackye1995

@jackye1995 jackye1995 commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Expose segment-level prewarm through the existing pylance prewarm_index API by adding an optional index_segments argument. Callers can pass UUIDs from describe_indices()[i].segments[j].uuid to prewarm only selected physical segments of the named logical index, while preserving the existing full-index behavior when index_segments is omitted.

The resolved physical index segments are opened and prewarmed concurrently, including both full-index prewarm and filtered segment prewarm. This also supports the existing FTS with_position option for selected segments and adds focused coverage in the index prewarm test.

Adds Rust-level tracing for prewarm investigation: request-level selected/available/requested segment counts, selected on-disk bytes when known, index cache entries/bytes before and after prewarm with deltas, per-segment start/open/finish/failure with UUID, fragment count, index version, dataset version, index type, elapsed time, and FTS partition-level start/posting-list/docset/finish timing. Also exposes ds.session().index_cache_size_bytes() in pylance for direct before/after measurement. Checked with cargo fmt --all and cargo check --manifest-path python/Cargo.toml.

@github-actions github-actions Bot added A-python Python bindings enhancement New feature or request labels Jul 7, 2026
@jackye1995
jackye1995 force-pushed the jack/pylance-segment-prewarm branch from 7df1832 to a7fa763 Compare July 7, 2026 23:53
@jackye1995 jackye1995 changed the title feat: expose segment index prewarm in pylance feat: support segment selection in pylance prewarm Jul 7, 2026
@jackye1995
jackye1995 force-pushed the jack/pylance-segment-prewarm branch 2 times, most recently from bfa442f to 25e1d28 Compare July 8, 2026 00:11
@github-actions github-actions Bot added the A-index Vector index, linalg, tokenizer label Jul 8, 2026
@jackye1995
jackye1995 force-pushed the jack/pylance-segment-prewarm branch from 25e1d28 to b046f47 Compare July 8, 2026 00:16
@codecov

codecov Bot commented Jul 8, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 74.34555% with 49 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
rust/lance/src/index.rs 69.17% 44 Missing and 1 partial ⚠️
rust/lance/src/index/api.rs 0.00% 4 Missing ⚠️

📢 Thoughts on this report? Let us know!

@jackye1995
jackye1995 merged commit c3218b2 into lance-format:main Jul 8, 2026
33 of 34 checks passed
jackye1995 added a commit that referenced this pull request Jul 13, 2026
## Summary

`hamming_clustering_for_ivf_partition` and `get_ivf_partition_info`
resolved the target index with `find(name)` — the first physical segment
only. On a logical IVF_FLAT index made of multiple segments (delta
segments from append-mode `optimize_indices`, or distributed builds
committed via `commit_existing_index_segments`), this silently processed
only the oldest segment: appended rows were excluded and cross-segment
duplicate pairs were never compared, with no error raised.

Both functions now cover **all** segments of the logical index. Delta
segments of one IVF index share centroids, so partition `p` denotes the
same centroid region in every segment; the correct clustering unit is
the union of partition `p`'s rows across all segments in a single
pairwise pass (representative = min row id, so results are
order-independent).

## Changes

- **Rust** (`lance-index::vector::hamming`): all-segments behavior for
both functions. Every selected segment is validated to share
byte-identical global IVF centroids; a mismatch (or missing centroids)
raises a descriptive error naming the diverging segment UUIDs and
partition counts, advising a retrain. New
`hamming_clustering_for_ivf_partition_segments` /
`get_ivf_partition_info_segments` accept explicit segment UUIDs,
following the `prewarm_index_segments` precedent (#7677).
- **Python**: keyword-only `index_segments` argument on
`hamming_clustering_for_ivf_partition` and `get_ivf_partition_info`
(`None` = all segments, the default). Segment-id normalization used in
three places (these wrappers, `prewarm_index`,
`ScannerBuilder.with_index_segments`) is consolidated into
`lance.util._normalize_index_segment_ids`.

## Notes

- Fixes a latent bug: `get_ivf_partition_info` returned `size: 0` for
every partition on v3 index files because the loaded IVF model carries
no partition lengths. Sizes now come from partition storage via
`VectorIndex::partition_size`.
- Behavior tightening: `get_ivf_partition_info` now validates the
indexed column is `FixedSizeList<UInt8, 8>`, a check it previously
skipped. This is consistent with the clustering function, which always
required 8-byte hashes.
- An empty explicit segment selection is rejected rather than silently
returning "no duplicates", to avoid a silent data-quality hazard in
dedup pipelines.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added support for running IVF_FLAT Hamming clustering and partition
analysis across multiple index segments.
* Added optional segment selection to limit clustering and partition
statistics to specific physical segments.
* Segment identifiers now accept UUID strings or UUID objects with
validation for invalid selections.

* **Bug Fixes**
* Improved handling and validation of index segment identifiers across
dataset and scanner operations.
* Added validation to ensure selected segments are compatible before
processing.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-index Vector index, linalg, tokenizer A-python Python bindings enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants