Update README.md#1
Merged
Merged
Conversation
changhiskhan
approved these changes
Mar 20, 2023
changhiskhan
left a comment
Contributor
There was a problem hiding this comment.
maybe remove the "billion+" for now until we can publish a benchmark. Just say "Production-scale vector search with no additional servers to manage" ?
gsilvestrin
added a commit
that referenced
this pull request
Jun 17, 2023
raghavdixit99
pushed a commit
to raghavdixit99/lancedb
that referenced
this pull request
Apr 5, 2024
Update README.md
raghavdixit99
pushed a commit
to raghavdixit99/lancedb
that referenced
this pull request
Apr 5, 2024
…ncedb#1… (lancedb#206) …97)" This reverts commit e5c3690.
alexkohler
pushed a commit
to alexkohler/lancedb
that referenced
this pull request
Apr 20, 2024
wjones127
pushed a commit
that referenced
this pull request
Jul 1, 2026
## What
`MRRReranker.rerank_multivector` averages each document's reciprocal
ranks over the wrong denominator. It divides by the number of rankings
the document *happens to appear in*, instead of the total number of
rankings being fused.
```python
# python/python/lancedb/rerankers/mrr.py
for result_id, reciprocal_ranks in mrr_score_map.items():
mean_rr = np.mean(reciprocal_ranks) # divides by len(present systems)
```
`mrr_score_map[doc]` only accumulates a reciprocal rank for the systems
in which the document was returned, so `np.mean` never accounts for the
systems that missed it.
## Why it's wrong
Mean Reciprocal Rank fusion treats a system that didn't return a
document as a reciprocal rank of `0` and averages across **all**
systems. That's the exact mechanism by which it rewards cross-system
consensus. Dividing by the appearance count removes that, so a document
liked by a single ranking can beat one ranked highly by every ranking.
Concretely, fusing 3 vector rankings:
| Doc | Ranks | Current score | Correct score |
|-----|-------|---------------|---------------|
| A | #1 in 1 system only | `mean([1.0]) = 1.000` | `1.0 / 3 = 0.333` |
| B | #1, #1, #2 across all 3 | `mean([1, 1, .5]) = 0.833` | `2.5 / 3 =
0.833` |
The current code ranks **A above B** - a document two of three rankings
ignored outranks one all three ranked at or near the top.
This also makes `rerank_multivector` inconsistent with `rerank_hybrid`
in the same file, which already treats a missing system as `0`
(`vector_rr = 0.0` / `fts_rr = 0.0`), and with the class docstring
("average of reciprocal ranks across different search results").
## Fix
Divide the summed reciprocal ranks by the total number of rankings:
```python
num_systems = len(vector_results)
...
mean_rr = float(np.sum(reciprocal_ranks)) / num_systems
```
## Tests
Adds `test_mrr_multivector_rewards_consensus`, which asserts the exact
MRR scores and that the consensus document ranks first. It fails on
`main` and passes with this change. Existing reranker tests are
unaffected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.