Skip to content

feat(java): expose global BM25 statistics for FTS queries - #8822

Closed
zhangstar333 wants to merge 6 commits into
lance-format:mainfrom
zhangstar333:fts
Closed

feat(java): expose global BM25 statistics for FTS queries#8822
zhangstar333 wants to merge 6 commits into
lance-format:mainfrom
zhangstar333:fts

Conversation

@zhangstar333

@zhangstar333 zhangstar333 commented Aug 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor
  • Collect query-bound BM25 statistics across all committed segments of a single logical FTS index.
  • Preserve the prepared vocabulary and original token positions for each scoring leaf, including fuzzy expansions.
  • Expose the statistics through the Java/JNI API as an opaque protobuf payload.
  • Reject queries whose scoring leaves span multiple FTS corpora.

This is the implementation for #8937 and depends on #8938.

It supports the distributed FTS use case described in apache/doris#67435.

@github-actions

github-actions Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

  • Start a vote following the Lance community voting process.
    Format specification modifications need 3 binding +1 votes (excluding the
    proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
  • Once the vote passes, link the completed vote in this PR. It should not be
    merged until the vote is linked.

@github-actions github-actions Bot added A-index Vector index, linalg, tokenizer A-java Java bindings + JNI A-format On-disk format: protos and format spec docs enhancement New feature or request labels Aug 27, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

Sign in to view

@lance-gatekeeper lance-gatekeeper Bot added K-changes Latest Gatekeeper recommendation requests changes. and removed K-changes Latest Gatekeeper recommendation requests changes. labels Aug 27, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

Sign in to view

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Aug 27, 2026
@github-actions github-actions Bot added the format-change A change to the format spec, which requires a vote. Remove if minor (e.g. fixing typo). label Sep 1, 2026
@github-actions

github-actions Bot commented Sep 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Important

Format specification vote

This PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer) and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process.

Status: ❌ Blocked — 0 of 3 required approvals
Approvals (this commit) none (0/3)
Vetoes none
Voting period elapsed — ended Fri 2026-09-04 03:56 UTC (Thu 20:56 PDT)

Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the format-waived label to waive the vote for a trivial edit (typo, wording, formatting).

@lance-gatekeeper lance-gatekeeper Bot removed K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Sep 1, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

Sign in to view

@lance-gatekeeper lance-gatekeeper Bot added the K-changes Latest Gatekeeper recommendation requests changes. label Sep 1, 2026
This was referenced Sep 2, 2026
Open
Open
@lance-gatekeeper lance-gatekeeper Bot removed the K-changes Latest Gatekeeper recommendation requests changes. label Sep 2, 2026
lance-gatekeeper[bot]

This comment was marked as outdated.

Sign in to view

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. and removed K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Sep 2, 2026
@lance-gatekeeper lance-gatekeeper Bot removed K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Sep 4, 2026
lance-gatekeeper[bot]
lance-gatekeeper Bot reviewed Sep 4, 2026
View reviewed changes

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Gate recommendation: approve with a non-blocking risk.

The rebase preserves the producer behavior and adds explicit cross-column rejection coverage. V1 now makes its trust boundary clear: a planner must attach the payload only to the query that produced it because consumers cannot independently verify source-query identity.

That leaves a bounded correctness risk if a trusted planner misbinds the payload. The downstream validation of independently checkable snapshot, index, and segment metadata, plus distributed Top-K equivalence tracked by #8937, also remain outside this revision. No further producer change is requested here.

@lance-gatekeeper lance-gatekeeper Bot added K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk. labels Sep 4, 2026
@zhangstar333 zhangstar333 mentioned this pull request Oct 21, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lance-gatekeeper lance-gatekeeper[bot] lance-gatekeeper[bot] left review comments

Assignees

No one assigned

Labels

A-format On-disk format: protos and format spec docs A-index Vector index, linalg, tokenizer A-java Java bindings + JNI enhancement New feature or request format-change A change to the format spec, which requires a vote. Remove if minor (e.g. fixing typo). K-approved Latest Gatekeeper recommendation permits acceptance. K-risk Latest Gatekeeper recommendation includes a non-blocking risk.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@zhangstar333