Summary

Creating an IVF index whose num_partitions is large enough to trigger hierarchical k-means (num_partitions > 256) can fail badly when the training data cannot be split into that many non-empty clusters (small dataset and/or many near-duplicate vectors). The failure mode is build-dependent:

• Debug / debug-assertions on: hard panic at debug_assert_eq!(heap.len(), target_k) (rust/lance-index/src/vector/kmeans.rs:1159). Surfaces downstream (e.g. via PyO3) as an opaque rust future panicked: unknown error.
• Release (debug-assertions off): no panic — the assert is compiled out and the index is silently built with far fewer than num_partitions non-empty partitions, degrading recall with no error.

Both are bad: a crash in one build, silent bad data in the other.

Reproduction

• 1537 rows, list<float32>[35], IVF_FLAT, distance_type="dot", num_partitions=472.
• Debug build (lance v8.0.0-beta.19): panics at kmeans.rs:1159, assertion left == right failed.
• Released lancedb==0.34.0b6 (lance rev e25b71e, 9.0.0-beta.10), macOS-arm64: no panic across 15/15 runs; index builds with 239 of 472 partitions empty.

Downstream report with a self-contained repro (data + script): lancedb/lancedb#3649

Root cause

num_partitions > 256 routes into KMeans::train_hierarchical_kmeans (rust/lance-index/src/vector/kmeans.rs, added in #4726). It seeds hierarchical_k (=16) clusters, then greedily splits the largest cluster until heap.len() == target_k:

while heap.len() < target_k {
    // ... pop largest cluster, sub-kmeans split, push sub-clusters ...
}
debug_assert_eq!(heap.len(), target_k);   // kmeans.rs:1159

The loop has several early/ineffective exits that leave heap.len() < target_k:

• popped cluster already finalized → push back and break;
• largest cluster has <= 1 point → break;
• a split where every point lands in one sub-cluster (all_same) → the cluster is finalized, no net growth;
• empty sub-clusters are dropped (if new_cluster_indices.is_empty() { continue; }), so a split can add fewer than cluster_k clusters.

For a small dataset with dot metric (which makes degenerate/all_same splits common — see the repeated "clusters are empty" warnings), the loop routinely terminates with fewer than target_k clusters. The only thing guarding this is a debug_assert_eq!, so debug builds panic and release builds silently proceed with a short, half-empty centroid set.

Note the existing num_rows < k guard in train_kmeans does not catch this — 1537 >= 472 passes, yet the data still can't form 472 distinct clusters.

Proposed fix

Replace the debug_assert_eq! with a returned error when the splitter cannot reach target_k, so both debug and release fail cleanly and actionably:

if heap.len() < target_k {
    return Err(ArrowError::InvalidArgumentError(format!(
        "Cannot create {target_k} IVF partitions: k-means could only form {} \
         non-empty clusters from {n} training vectors. The dataset is likely too \
         small or has too many (near-)duplicate vectors for this many partitions. \
         Reduce num_partitions to <= {} or provide more diverse data.",
        heap.len(), heap.len()
    )));
}

This propagates up through train_kmeans → do_train_ivf_model → index creation as a normal Result, replacing the opaque panic / silent degenerate index with a clear message.

Scope note: the splitter can also overshoot target_k by one (the .max(2) on the final split), which would also trip the current == assert. That's a separate, minor concern; this fix targets only the heap.len() < target_k (unreachable) case, which is the reported failure.