Skip to content

perf: hierarchical clustering#4726

Merged
BubbleCal merged 20 commits into
lance-format:mainfrom
BubbleCal:hierarchical-clustering
Sep 17, 2025
Merged

perf: hierarchical clustering#4726
BubbleCal merged 20 commits into
lance-format:mainfrom
BubbleCal:hierarchical-clustering

Conversation

@BubbleCal

@BubbleCal BubbleCal commented Sep 15, 2025

Copy link
Copy Markdown
Contributor

resolve #4663

This boosts the KMeans algo by ~30x faster, and even much more for large k (num_partitions)

With this, it's possible to run KMeans with k=16384 within 25s (which requires over 10mins before this).
And no obvious accuracy loss observed according to my benchmark.

@github-actions github-actions Bot added A-python Python bindings performance labels Sep 15, 2025
Signed-off-by: BubbleCal <[email protected]>

@wjones127 wjones127 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have some minor comments, but once addressed (and CI fixed) this looks good!

Comment thread rust/lance-index/src/vector/kmeans.rs Outdated
Comment thread rust/lance-index/src/vector/kmeans.rs
Comment thread rust/lance-index/src/vector/kmeans.rs Outdated
Comment thread rust/lance-index/src/vector/kmeans.rs Outdated
Signed-off-by: BubbleCal <[email protected]>
Signed-off-by: BubbleCal <[email protected]>
Signed-off-by: BubbleCal <[email protected]>
@codecov-commenter

codecov-commenter commented Sep 16, 2025

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.06818% with 42 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.65%. Comparing base (76a710e) to head (0237406).
⚠️ Report is 88 commits behind head on main.

Files with missing lines Patch % Lines
rust/lance-index/src/vector/kmeans.rs 86.85% 33 Missing and 8 partials ⚠️
rust/lance-index/src/optimize.rs 75.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4726      +/-   ##
==========================================
- Coverage   80.72%   80.65%   -0.07%     
==========================================
  Files         321      321              
  Lines      124043   125015     +972     
  Branches   124043   125015     +972     
==========================================
+ Hits       100131   100830     +699     
- Misses      20340    20591     +251     
- Partials     3572     3594      +22     
Flag Coverage Δ
unittests 80.65% <88.06%> (-0.07%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@BubbleCal
BubbleCal merged commit 1f40a49 into lance-format:main Sep 17, 2025
26 checks passed
jackye1995 pushed a commit to jackye1995/lance that referenced this pull request Jan 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

perf: faster and more balanced clustering

3 participants