feat: support segment selection in pylance prewarm#7677
Merged
jackye1995 merged 1 commit intoJul 8, 2026
Conversation
jackye1995
force-pushed
the
jack/pylance-segment-prewarm
branch
from
July 7, 2026 23:53
7df1832 to
a7fa763
Compare
jackye1995
force-pushed
the
jack/pylance-segment-prewarm
branch
2 times, most recently
from
July 8, 2026 00:11
bfa442f to
25e1d28
Compare
jackye1995
force-pushed
the
jack/pylance-segment-prewarm
branch
from
July 8, 2026 00:16
25e1d28 to
b046f47
Compare
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
LuQQiu
approved these changes
Jul 8, 2026
BubbleCal
approved these changes
Jul 8, 2026
jackye1995
added a commit
that referenced
this pull request
Jul 13, 2026
## Summary `hamming_clustering_for_ivf_partition` and `get_ivf_partition_info` resolved the target index with `find(name)` — the first physical segment only. On a logical IVF_FLAT index made of multiple segments (delta segments from append-mode `optimize_indices`, or distributed builds committed via `commit_existing_index_segments`), this silently processed only the oldest segment: appended rows were excluded and cross-segment duplicate pairs were never compared, with no error raised. Both functions now cover **all** segments of the logical index. Delta segments of one IVF index share centroids, so partition `p` denotes the same centroid region in every segment; the correct clustering unit is the union of partition `p`'s rows across all segments in a single pairwise pass (representative = min row id, so results are order-independent). ## Changes - **Rust** (`lance-index::vector::hamming`): all-segments behavior for both functions. Every selected segment is validated to share byte-identical global IVF centroids; a mismatch (or missing centroids) raises a descriptive error naming the diverging segment UUIDs and partition counts, advising a retrain. New `hamming_clustering_for_ivf_partition_segments` / `get_ivf_partition_info_segments` accept explicit segment UUIDs, following the `prewarm_index_segments` precedent (#7677). - **Python**: keyword-only `index_segments` argument on `hamming_clustering_for_ivf_partition` and `get_ivf_partition_info` (`None` = all segments, the default). Segment-id normalization used in three places (these wrappers, `prewarm_index`, `ScannerBuilder.with_index_segments`) is consolidated into `lance.util._normalize_index_segment_ids`. ## Notes - Fixes a latent bug: `get_ivf_partition_info` returned `size: 0` for every partition on v3 index files because the loaded IVF model carries no partition lengths. Sizes now come from partition storage via `VectorIndex::partition_size`. - Behavior tightening: `get_ivf_partition_info` now validates the indexed column is `FixedSizeList<UInt8, 8>`, a check it previously skipped. This is consistent with the clustering function, which always required 8-byte hashes. - An empty explicit segment selection is rejected rather than silently returning "no duplicates", to avoid a silent data-quality hazard in dedup pipelines. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added support for running IVF_FLAT Hamming clustering and partition analysis across multiple index segments. * Added optional segment selection to limit clustering and partition statistics to specific physical segments. * Segment identifiers now accept UUID strings or UUID objects with validation for invalid selections. * **Bug Fixes** * Improved handling and validation of index segment identifiers across dataset and scanner operations. * Added validation to ensure selected segments are compatible before processing. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Expose segment-level prewarm through the existing pylance
prewarm_indexAPI by adding an optionalindex_segmentsargument. Callers can pass UUIDs fromdescribe_indices()[i].segments[j].uuidto prewarm only selected physical segments of the named logical index, while preserving the existing full-index behavior whenindex_segmentsis omitted.The resolved physical index segments are opened and prewarmed concurrently, including both full-index prewarm and filtered segment prewarm. This also supports the existing FTS
with_positionoption for selected segments and adds focused coverage in the index prewarm test.Adds Rust-level tracing for prewarm investigation: request-level selected/available/requested segment counts, selected on-disk bytes when known, index cache entries/bytes before and after prewarm with deltas, per-segment start/open/finish/failure with UUID, fragment count, index version, dataset version, index type, elapsed time, and FTS partition-level start/posting-list/docset/finish timing. Also exposes
ds.session().index_cache_size_bytes()in pylance for direct before/after measurement. Checked withcargo fmt --allandcargo check --manifest-path python/Cargo.toml.