You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 6a79f95
Browse filesBrowse the repository at this point in the historyBrowse files
[feat](inverted-index) Support configurable ngram size difference (#67917)
### What problem does this PR solve?
Issue Number:
Related PR: None
Problem Summary:
Custom ngram tokenizers hard-code the allowed difference between
max_gram and min_gram to 1. This prevents valid wider ngram ranges while
offering no explicit override.
This PR adds max_ngram_diff, keeps 1 as the backward-compatible default,
requires ASCII integer syntax, and validates the same 0 through 255
range in FE and BE. The upper bound limits per-position token fan-out.
Newly created policies also cap absolute custom ngram sizes at 1024,
while an explicit persisted compatibility marker preserves marker-less
policies accepted before that cap during rolling upgrades. Invalid
replayed policies cannot block valid replacements. Because
max_ngram_diff controls policy admission but does not affect emitted
tokens, valid policies exclude it from analyzer identity so equivalent
analyzers cannot bypass duplicate-index detection.
### Release note
Allow custom ngram tokenizers to configure the maximum difference
between max_gram and min_gram with max_ngram_diff values from 0 through
255.
0 commit comments