Repository navigation
Exclude unretrieved positives from observed RRF ranking metrics - #4114
Merged
tomaarsen merged 3 commits intoNov 2, 2026
Merged
Conversation
Compute DCG directly and normalize against all known positives, matching the reranking evaluator. Consolidate regression coverage and evaluate the hybrid-search example using only retrieved candidates.
# Conflicts: # examples/sparse_encoder/applications/retrieve_rerank/README.md # tests/sparse_encoder/test_reciprocal_rank_fusion.py
Member
|
Hello! I've pushed a revision that simplifies the metric calculation to match #4054 and consolidates the regression tests. The hybrid-search example now reranks only retrieved documents. I've rerun the evaluation and refreshed the results: RRF MAP drops from 22.64% to 12.81%.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Top-k retrieval can omit some known relevant documents. The evaluator currently appends those positives to the retrieved relevance vector and assigns them ordinary descending scores. That gives unretrieved documents a rank and positive gain, inflating both average precision and nDCG for dense, sparse, and fused rankings.
For example, with positives
[p, missing]and retrieved documents[irrelevant, p], AP is currently 7/12 instead of 1/4. At k=10, nDCG is about 0.6934 instead of 0.3869: the missing document incorrectly contributes DCG at rank 3. This is observable without loading a model.The change evaluates only actual retrieved documents, counts every known positive in the AP denominator and ideal DCG, and keeps existing sklearn score-tie behavior. MRR and fusion ordering remain unchanged. A single retrieved positive is handled directly because sklearn nDCG requires at least two candidates. At a zero cutoff, MRR and nDCG are zero while AP still uses the full ranking.
Validation: 14 focused regression cases plus the 3 existing RRF cases pass against the complete current-main package. The regressions cover omitted positives at several cutoffs, empty and singleton rankings, zero cutoff, score ties, query averaging, prediction output, and complementary retrievers. Before adding the two zero-cutoff cases, pristine main yielded 9 failures and 6 passes.