MOD-7896: Add weight factor to BM25 scoring calculation by Itzikvaknin · Pull Request #5670 · RediSearch/RediSearch

Itzikvaknin · 2025-02-19T10:00:33Z

This PR fixes an issue where the weight attribute was not applied when querying a single term using the BM25 scorer, as demonstrated in the example below:

DB:

JSON.SET key:1 $ '{"tag": "foo", "content": "The cat is on the mat", "description": "The cat is on the mat", "content_vector": [1,1], "description_vector": [1,1]}'
JSON.SET key:2 $ '{"tag": "foo", "content": "My dog and cat are the best animals to ever exist in the universe.  My cat\'s name is Fred", "description": "My dog and cat are the best animals to ever exist in the universe.  My cat\'s name is Fred", "content_vector": [1,2], "description_vector": [1,2]}'
JSON.SET key:3 $ '{"tag": "bar", "content": "There was a solar eclipse yesterday", "description": "There was a solar eclipse yesterday", "content_vector": [1,3], "description_vector": [1,3]}'
FT.CREATE idx ON JSON PREFIX 1 key: SCHEMA $.tag AS tag TAG $.content AS content TEXT $.description AS description TEXT $.content_vector AS content_vector VECTOR FLAT 6 TYPE FLOAT32 DIM 2 DISTANCE_METRIC COSINE $.description_vector AS description_vector VECTOR FLAT 6 TYPE FLOAT32 DIM 2 DISTANCE_METRIC COSINE

Behavior before the fix:

> FT.SEARCH idx "@content:cat" SCORER BM25 WITHSCORES
1) "2"
2) "key:2"
3) "0.33898304194357526"
4) 1) "$"
   2) "{\"tag\":\"foo\",\"content\":\"My dog and cat are the best animals to ever exist in the universe.  My cat's name is Fred\",\"description\":\"My dog and cat are the best animals to ever exist in the universe.  My cat's name is Fred\",\"content_vector\":[1,2],\"description_vector\":[1,2]}"
5) "key:1"
6) "0.20408162619858095"
7) 1) "$"
   2) "{\"tag\":\"foo\",\"content\":\"The cat is on the mat\",\"description\":\"The cat is on the mat\",\"content_vector\":[1,1],\"description_vector\":[1,1]}"


> FT.SEARCH idx "((@content:cat) => {$weight: 0.3;})" SCORER BM25 WITHSCORES
1) "2"
2) "key:2"
3) "0.33898304194357526"
4) 1) "$"
   2) "{\"tag\":\"foo\",\"content\":\"My dog and cat are the best animals to ever exist in the universe.  My cat's name is Fred\",\"description\":\"My dog and cat are the best animals to ever exist in the universe.  My cat's name is Fred\",\"content_vector\":[1,2],\"description_vector\":[1,2]}"
5) "key:1"
6) "0.20408162619858095"
7) 1) "$"
   2) "{\"tag\":\"foo\",\"content\":\"The cat is on the mat\",\"description\":\"The cat is on the mat\",\"content_vector\":[1,1],\"description_vector\":[1,1]}"

Behavior after the fix:

> FT.SEARCH idx "@content:cat" SCORER BM25 WITHSCORES
1) (integer) 2
2) "key:2"
3) "0.33898304194357526"
4) 1) "$"
   2) "{\"tag\":\"foo\",\"content\":\"My dog and cat are the best animals to ever exist in the universe.  My cat's name is Fred\",\"description\":\"My dog and cat are the best animals to ever exist in the universe.  My cat's name is Fred\",\"content_vector\":[1,2],\"description_vector\":[1,2]}"
5) "key:1"
6) "0.20408162619858095"
7) 1) "$"
   2) "{\"tag\":\"foo\",\"content\":\"The cat is on the mat\",\"description\":\"The cat is on the mat\",\"content_vector\":[1,1],\"description_vector\":[1,1]}"


> FT.SEARCH idx "((@content:cat) => {$weight: 0.3;})" SCORER BM25 WITHSCORES
1) (integer) 2
2) "key:2"
3) "0.10169491258307257"
4) 1) "$"
   2) "{\"tag\":\"foo\",\"content\":\"My dog and cat are the best animals to ever exist in the universe.  My cat's name is Fred\",\"description\":\"My dog and cat are the best animals to ever exist in the universe.  My cat's name is Fred\",\"content_vector\":[1,2],\"description_vector\":[1,2]}"
5) "key:1"
6) "0.06122448785957428"
7) 1) "$"
   2) "{\"tag\":\"foo\",\"content\":\"The cat is on the mat\",\"description\":\"The cat is on the mat\",\"content_vector\":[1,1],\"description_vector\":[1,1]}"

EXPLAIN was updated accordingly:

> FT.SEARCH idx '((@content:(cat => {$weight: 0.5;}|dog) => {$weight: 0.7;}) => {$weight: 0.3;})' WITHSCORES EXPLAINSCORE SCORER BM25 NOCONTENT
1) (integer) 2
2) "key:2"
3) 1) "0.17329643201068487"
   2) 1) Final BM25 : words BM25 0.17 * document score 1.00 / slop 1
      2) 1) 1) (Weight 0.30 * children BM25 0.58)
            2) 1) (0.17 = Weight 0.50 * IDF 1.00 * F 4 / (F 4 + k1 1.2 * (1 - b 0.5 + b 0.5 * Average Len 12.00)))
               2) (0.41 = Weight 1.00 * IDF 2.00 * F 2 / (F 2 + k1 1.2 * (1 - b 0.5 + b 0.5 * Average Len 12.00)))
4) "key:1"
5) 1) "0.03061224392978714"
   2) 1) Final BM25 : words BM25 0.03 * document score 1.00 / slop 1
      2) 1) 1) (Weight 0.30 * children BM25 0.10)
            2) 1) (0.10 = Weight 0.50 * IDF 1.00 * F 2 / (F 2 + k1 1.2 * (1 - b 0.5 + b 0.5 * Average Len 12.00)))

…anations

codecov · 2025-02-19T10:26:49Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 88.02%. Comparing base (c2bab6e) to head (22a17df).
Report is 8 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5670      +/-   ##
==========================================
- Coverage   88.04%   88.02%   -0.03%     
==========================================
  Files         197      197              
  Lines       35577    35576       -1     
==========================================
- Hits        31322    31314       -8     
- Misses       4255     4262       +7

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Expand testTFIDFScorerExplanation in test_scorers

to test_scorers

raz-mon · 2025-02-20T12:36:54Z

@Itzikvaknin Update the PR comment please.
Remove the default part you did not use, and add some details as to when the issue happened and what is the new behavior. An example would be great

raz-mon

Nice
See a few comments

src/ext/default.c

tests/pytests/test_scorers.py

Fix BM25STD calculation

src/ext/default.c

tests/pytests/test_scorers.py

raz-mon

😎

tests/pytests/test_scorers.py

…TDScorerExplanation

Itzikvaknin · 2025-02-27T09:37:11Z

/backport

redisearch-backport-pull-request · 2025-02-27T09:37:29Z

Backport failed for 2.10, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin 2.10
git worktree add -d .worktree/backport-5670-to-2.10 origin/2.10
cd .worktree/backport-5670-to-2.10
git switch --create backport-5670-to-2.10
git cherry-pick -x 65a4c9e396228d024d7ca4d34a9e99bccbcced31

redisearch-backport-pull-request · 2025-02-27T09:37:32Z

Successfully created backport PR for 8.0:

[8.0] MOD-7896: Add weight factor to BM25 scoring calculation #5702

* Update BM25 scoring calculations to incorporate weight factor in explanations * Add weight to explainscore Expand testTFIDFScorerExplanation in test_scorers * Add testBM25STDScoreWithWeight and testBM25ScoreWithWeight to test_scorers * Fix testOptionalAndWildcardScoring * Fix testBM25STDScorerExplanation * Add docs to scorer_with_weight_test Fix BM25STD calculation * Remove unnecessary isCluster condition from test_scorers * Add hash seed to force docs distribution * Name change * Fix test_scorers:testBM25ScorerExplanation and test_scorers:testBM25STDScorerExplanation (cherry picked from commit 65a4c9e)

* Update BM25 scoring calculations to incorporate weight factor in explanations * Add weight to explainscore Expand testTFIDFScorerExplanation in test_scorers * Add testBM25STDScoreWithWeight and testBM25ScoreWithWeight to test_scorers * Fix testOptionalAndWildcardScoring * Fix testBM25STDScorerExplanation * Add docs to scorer_with_weight_test Fix BM25STD calculation * Remove unnecessary isCluster condition from test_scorers * Add hash seed to force docs distribution * Name change * Fix test_scorers:testBM25ScorerExplanation and test_scorers:testBM25STDScorerExplanation

MOD-7896: Add weight factor to BM25 scoring calculation (#5670) * Update BM25 scoring calculations to incorporate weight factor in explanations * Add weight to explainscore Expand testTFIDFScorerExplanation in test_scorers * Add testBM25STDScoreWithWeight and testBM25ScoreWithWeight to test_scorers * Fix testOptionalAndWildcardScoring * Fix testBM25STDScorerExplanation * Add docs to scorer_with_weight_test Fix BM25STD calculation * Remove unnecessary isCluster condition from test_scorers * Add hash seed to force docs distribution * Name change * Fix test_scorers:testBM25ScorerExplanation and test_scorers:testBM25STDScorerExplanation (cherry picked from commit 65a4c9e) Co-authored-by: Itzikvaknin <[email protected]>

MOD-7896: Add weight factor to BM25 scoring calculation (#5670) * Update BM25 scoring calculations to incorporate weight factor in explanations * Add weight to explainscore Expand testTFIDFScorerExplanation in test_scorers * Add testBM25STDScoreWithWeight and testBM25ScoreWithWeight to test_scorers * Fix testOptionalAndWildcardScoring * Fix testBM25STDScorerExplanation * Add docs to scorer_with_weight_test Fix BM25STD calculation * Remove unnecessary isCluster condition from test_scorers * Add hash seed to force docs distribution * Name change * Fix test_scorers:testBM25ScorerExplanation and test_scorers:testBM25STDScorerExplanation

Itzikvaknin · 2025-03-02T08:26:04Z

/backport

* Update BM25 scoring calculations to incorporate weight factor in explanations * Add weight to explainscore Expand testTFIDFScorerExplanation in test_scorers * Add testBM25STDScoreWithWeight and testBM25ScoreWithWeight to test_scorers * Fix testOptionalAndWildcardScoring * Fix testBM25STDScorerExplanation * Add docs to scorer_with_weight_test Fix BM25STD calculation * Remove unnecessary isCluster condition from test_scorers * Add hash seed to force docs distribution * Name change * Fix test_scorers:testBM25ScorerExplanation and test_scorers:testBM25STDScorerExplanation

…#5712) * MOD-7896: Add weight factor to BM25 scoring calculation (#5670) * Update BM25 scoring calculations to incorporate weight factor in explanations * Add weight to explainscore Expand testTFIDFScorerExplanation in test_scorers * Add testBM25STDScoreWithWeight and testBM25ScoreWithWeight to test_scorers * Fix testOptionalAndWildcardScoring * Fix testBM25STDScorerExplanation * Add docs to scorer_with_weight_test Fix BM25STD calculation * Remove unnecessary isCluster condition from test_scorers * Add hash seed to force docs distribution * Name change * Fix test_scorers:testBM25ScorerExplanation and test_scorers:testBM25STDScorerExplanation * Add test with scoring

…#5709) * MOD-7896: Add weight factor to BM25 scoring calculation (#5670) * Update BM25 scoring calculations to incorporate weight factor in explanations * Add weight to explainscore Expand testTFIDFScorerExplanation in test_scorers * Add testBM25STDScoreWithWeight and testBM25ScoreWithWeight to test_scorers * Fix testOptionalAndWildcardScoring * Fix testBM25STDScorerExplanation * Add docs to scorer_with_weight_test Fix BM25STD calculation * Remove unnecessary isCluster condition from test_scorers * Add hash seed to force docs distribution * Name change * Fix test_scorers:testBM25ScorerExplanation and test_scorers:testBM25STDScorerExplanation * Add Env import

) * Update BM25 scoring calculations to incorporate weight factor in explanations * Add weight to explainscore Expand testTFIDFScorerExplanation in test_scorers * Add testBM25STDScoreWithWeight and testBM25ScoreWithWeight to test_scorers * Fix testOptionalAndWildcardScoring * Fix testBM25STDScorerExplanation * Add docs to scorer_with_weight_test Fix BM25STD calculation * Remove unnecessary isCluster condition from test_scorers * Add hash seed to force docs distribution * Name change * Fix test_scorers:testBM25ScorerExplanation and test_scorers:testBM25STDScorerExplanation

Update BM25 scoring calculations to incorporate weight factor in expl…

9d194f1

…anations

Itzikvaknin added 4 commits February 19, 2025 14:51

Add weight to explainscore

81a1681

Expand testTFIDFScorerExplanation in test_scorers

Add testBM25STDScoreWithWeight and testBM25ScoreWithWeight

bfe1fa9

to test_scorers

Fix testOptionalAndWildcardScoring

8bf4a87

Fix testBM25STDScorerExplanation

797321d

Itzikvaknin requested review from oshadmi and raz-mon February 19, 2025 16:30

raz-mon reviewed Feb 20, 2025

View reviewed changes

src/ext/default.c Outdated Show resolved Hide resolved

tests/pytests/test_scorers.py Outdated Show resolved Hide resolved

tests/pytests/test_scorers.py Show resolved Hide resolved

Add docs to scorer_with_weight_test

61620de

Fix BM25STD calculation

Itzikvaknin requested a review from raz-mon February 20, 2025 18:27

raz-mon reviewed Feb 22, 2025

View reviewed changes

src/ext/default.c Show resolved Hide resolved

Itzikvaknin requested a review from raz-mon February 23, 2025 10:01

raz-mon reviewed Feb 23, 2025

View reviewed changes

tests/pytests/test_scorers.py Outdated Show resolved Hide resolved

Remove unnecessary isCluster condition from test_scorers

8b7d290

raz-mon previously approved these changes Feb 23, 2025

View reviewed changes

Add hash seed to force docs distribution

014cbee

Itzikvaknin dismissed raz-mon’s stale review via 014cbee February 23, 2025 12:56

Itzikvaknin requested a review from raz-mon February 23, 2025 13:23

Name change

e9c1611

raz-mon previously approved these changes Feb 23, 2025

View reviewed changes

oshadmi reviewed Feb 26, 2025

View reviewed changes

tests/pytests/test_scorers.py Show resolved Hide resolved

oshadmi reviewed Feb 26, 2025

View reviewed changes

tests/pytests/test_scorers.py Show resolved Hide resolved

Fix test_scorers:testBM25ScorerExplanation and test_scorers:testBM25S…

50b2968

…TDScorerExplanation

Itzikvaknin dismissed raz-mon’s stale review via 50b2968 February 26, 2025 12:10

oshadmi approved these changes Feb 26, 2025

View reviewed changes

oshadmi mentioned this pull request Feb 26, 2025

MOD-8923: Add Normalized BM25STD scorer #5686

Merged

2 tasks

Itzikvaknin requested a review from raz-mon February 26, 2025 13:07

Merge branch 'master' into ivaknin-bm25-weights-issue

22a17df

Itzikvaknin deleted the ivaknin-bm25-weights-issue branch February 26, 2025 15:28

Itzikvaknin added backport 2.10 backport 8.0 labels Feb 27, 2025

redisearch-backport-pull-request bot mentioned this pull request Feb 27, 2025

[8.0] MOD-7896: Add weight factor to BM25 scoring calculation #5702

Merged

Itzikvaknin mentioned this pull request Feb 27, 2025

[2.10] MOD-7896: Add weight factor to BM25 scoring calculation #5703

Closed

Itzikvaknin mentioned this pull request Feb 27, 2025

[2.10] MOD-7896: Add weight factor to BM25 scoring calculation #5704

Merged

Itzikvaknin added backport 2.8 backport 2.6 labels Mar 2, 2025

RediSearch deleted a comment from redisearch-backport-pull-request bot Mar 2, 2025

Itzikvaknin mentioned this pull request Mar 2, 2025

[2.8] MOD-7896: Add weight factor to BM25 scoring calculation (#5670) #5709

Merged

Itzikvaknin mentioned this pull request Mar 2, 2025

[2.6] MOD-7896: Add weight factor to BM25 scoring calculation (#5670) #5712

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MOD-7896: Add weight factor to BM25 scoring calculation#5670

MOD-7896: Add weight factor to BM25 scoring calculation#5670
Itzikvaknin merged 11 commits intomasterfrom
ivaknin-bm25-weights-issue

Itzikvaknin commented Feb 19, 2025 •

edited

Loading

Uh oh!

codecov bot commented Feb 19, 2025 •

edited

Loading

Uh oh!

raz-mon commented Feb 20, 2025 •

edited

Loading

Uh oh!

raz-mon left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

raz-mon left a comment

Uh oh!

Uh oh!

Uh oh!

Itzikvaknin commented Feb 27, 2025

Uh oh!

redisearch-backport-pull-request bot commented Feb 27, 2025

Uh oh!

redisearch-backport-pull-request bot commented Feb 27, 2025

Uh oh!

Itzikvaknin commented Mar 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Itzikvaknin commented Feb 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Feb 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

raz-mon commented Feb 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

raz-mon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

raz-mon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Itzikvaknin commented Feb 27, 2025

Uh oh!

redisearch-backport-pull-request bot commented Feb 27, 2025

Uh oh!

redisearch-backport-pull-request bot commented Feb 27, 2025

Uh oh!

Itzikvaknin commented Mar 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Itzikvaknin commented Feb 19, 2025 •

edited

Loading

codecov bot commented Feb 19, 2025 •

edited

Loading

raz-mon commented Feb 20, 2025 •

edited

Loading