You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 4016f20
Browse filesBrowse the repository at this point in the historyBrowse files
[fix](be) Support BM25 scoring on keyword SEARCH indexes (apache#69051)
### What problem does this PR solve?
Issue Number: None
Related PR: apache#68758
Problem Summary: SEARCH can match a keyword inverted index, but
projecting score() can drop matched rows or report an unsupported index.
Keyword indexes omit norms and may omit frequencies; they still need
usable BM25 defaults.
Preserve whole keyword values, including case and whitespace. Use
frequency and encoded norm 1 for docs-only/no-norm postings. Legacy
indexes without total-term-frequency metadata use average length 1; SNII
keeps its physical statistics. Analyzed segments missing required
scoring metadata and numeric BKD scoring retain their existing
restrictions.
This branch incorporates master's unified MATCH/SEARCH engine from
apache#68758. Keyword norm handling lives in the CLucene postings cursor,
including partition switches and WAND bounds; SNII cursors provide
docs-only frequencies and norms. The obsolete query_v2/SNII scoring
implementations are removed with master.
Keyword matching and missing-metadata behavior follow Elasticsearch
while retaining Doris's existing BM25 formula and collection-statistics
conventions.
#### P1 follow-up: keyword filters and internal conjunctions
1. A keyword index explicitly declared with `parser=none` can inherit
`support_phrase=true` from FE defaults. Ordinary MATCH filters then
entered scoring although the statistics collector skipped that field.
Both IndexReaderHelper query-type overloads now share the canonical
analyzer-and-phrase eligibility check used by statistics collection. The
implementation is out of line to keep analyzer dependencies out of the
header. Named-analyzer and normalizer behavior is preserved. Equality
and keyword MATCH filters keep the SEARCH-only baseline score.
2. Scored SNII AND grouped a docs-only keyword term into a path that
counts positions. Such sources now use the existing individual term
scorer with implicit unit frequency; position-bearing sources retain
batched scoring. Internal `SEARCH('body:apple AND tag:x')` continues to
score both fields.
0 commit comments