Repository navigation
Commit ef72b93
authored
[feat](inverted-index) Add IK tokenizers to custom analyzers (apache#67918)
### What problem does this PR solve?
Custom inverted-index analyzers could not use IK smart or max-word
tokenization with their configured character and token filters. This
adds the `ik_smart` and `ik_max_word` tokenizer factories and policy
validation.
Composed pipelines need source-byte provenance across normalization,
splitting, concatenation, trimming, and reset/reuse. Opt-in offset
tracking propagates through IK, keyword, standard, ICU normalization,
word-delimiter, and pinyin processing. Separate rune end offsets
preserve gaps removed by delimiter concatenation.
Review fixes preserve exact saved policy bindings across replay,
lifecycle operations, MATCH serialization, provider dispatch, physical
reader selection, and scoring-statistics collection. FE matches explicit
built-in IK to its default max-word configuration. BE selection keys
distinguish mode, lowercase configuration, and effective outer character
filters without changing provider names, Thrift, or stored index
formats. Encoded configurations cannot collide with arbitrary policy
names.
For backend-first upgrades, BE now binds an old FE's normalized analyzer
name to the canonical saved policy and accepts a lowercase metadata key
only when it resolves to the same policy. Exact metadata remains
preferred. The compatibility alias cannot select a differently bound
policy or a built-in analyzer with different tokenization.
Named character-filter identities account for transformations absorbed
by the resolved tokenizer while preserving meaningful ordered
transformations. The two new global-policy suites run in `nonConcurrent`
instead of contributing to concurrent P0 policy-quota exhaustion.
Classic writer and query boundaries now convert lazy custom-analyzer
failures to non-OK `Status` results. This includes ARRAY token-stream
creation before `add_document()`, which clears the partial document and
closes the writer on error instead of allowing a Doris exception to
escape. FE identity construction also preserves case-folding absorption
through the default `icu_normalizer` `nfkc_cf` filter, preventing
equivalent lower-then-fold and fold-only pipelines from receiving
different identities.
FE policy validation now checks an exact replayed policy before falling
back to a normalized built-in analyzer or normalizer. This prevents a
legacy wrong-type `IK` or `LOWERCASE` policy from passing FE validation
and failing later in BE. `CREATE ... IF NOT EXISTS` also returns for an
existing replayed `ik_smart` or `ik_max_word` policy before applying the
reserved built-in-name check, while fresh creations with those names
remain rejected.
Nested tokenizer, token-filter, and character-filter validation uses the
same exact-first policy lookup as BE and enforces the expected component
type. Replayed case-colliding policies can no longer bind a different
saved component or the wrong filter family. When ICU token normalization
changes text, its output carries a conservative source map to the
original token span; unchanged tokens continue to delegate their precise
upstream maps, and reset/reuse clears rewritten-map state.
The latest AI-review fixes reject duplicate non-IK analyzer selectors
before full identity comparison, including the implicit default
selector, so outer character-filter differences cannot create two
indexes that the query syntax cannot distinguish. Basic, CharGroup,
Empty, ICU, and NGram tokenizers now publish document-relative source
boundaries for offset-aware Pinyin. ICU resolves monotonically
increasing UTF-16 break boundaries with a constant-size UTF-8/UTF-16
cursor, transformations that change rune counts use a conservative
whole-source span, and NGram byte offsets preserve full `UChar32`
values.
The memory fixes remove full-input dense offset tables from ICU
tokenization and normalization. The tokenizer preserves exact spans with
its incremental cursor. The normalizer retains opt-in provenance, stores
no provenance when unused, and represents a changed token with one
compact conservative source span when enabled; Pinyin consumes that span
without forcing dense maps.
The current review fixes prevent rune-expanding ASCII folding and
lowercase filters from forwarding invalid exact provenance, avoid
allocating Pinyin rune and source arrays in the default ignore-offset
mode, canonicalize FE analyzer identities using effective component
defaults, and convert lazy scoring-analyzer failures back into `Status`
results.
The final CI follow-up aligns the duplicate-analyzer unit test with the
intentional selector-based diagnostic introduced by those validation
fixes.
Current candidate: `31450230c09d6ba10b6f0c2e8de70f3765948543`.
### Release note
Add IK smart and max-word tokenizers to custom inverted-index analyzers.
Correct composed-tokenizer source offsets, preserve custom-analyzer
MATCH bindings during backend-first upgrades, select the matching
physical IK configuration, and fix effective duplicate-index identities.
### Check List (For Author)
- Test:
- Native FE and BE ASAN builds passed in the isolated task worktree. The
BE build included the GLIBC 2.17 compatibility gate and its CDC client
stage.
- The official BE ASAN runner passed all 9 focused writer, reader, and
analyzer failure-path tests, including the ARRAY pre-`add_document()`
exception boundary. The complete `InvertedIndexWriterTest` suite passed
20/20.
- The official FE runner passed all 22 `AnalyzerIdentityBuilderTest`
tests. After policy-validation repairs, it also passed the replay/image
coverage and all 40 `PolicyValidatorTests`; `./build.sh --fe` completed
successfully with zero Checkstyle violations.
- The official BE ASAN runner passed 77/77 tests across
`IndexPolicyMgrTest`, `PinyinFilterTest`, and
`ICUNormalizerFilterFactoryTest`, including component type collisions
and ICU-normalizer-to-pinyin source spans before and after reset.
- A broader official BE ASAN run passed 354 tests in 10 related suites.
A later tokenizer/filter run passed 150/150, and the two new
offset-focused Pinyin tests passed 2/2 on the final source content.
- The memory-fix candidate passed 203/203 related ASAN tokenizer and
filter tests, including deterministic large-input allocation coverage
for the ICU tokenizer and both disabled and enabled ICU-normalizer
provenance paths. The full ASAN BE build then passed.
- An official FE run passed 118 tests in four classes, including real
index selection and MATCH-to-Thrift translation. The selector-ambiguity
coverage passed 58/58 in `InvertedIndexPropertiesTest`.
- The current review-fix candidate passed 147/147 related ASAN BE tests
covering Pinyin, ASCII folding, lowercase, and scoring collection. A
full ASAN BE build passed after the C++ changes.
- The latest review-fix head passed 112/112 targeted BE ASAN tests
across ICU, Keyword, NGram, and Pinyin coverage. It also passed 86/86 FE
tests across `AnalyzerIdentityBuilderTest` and
`InvertedIndexPropertiesTest`; the full 80-module Maven reactor and
Checkstyle completed successfully.
- The final candidate passed all 22 `SchemaChangeHandlerTest` tests and
the complete 80-module Maven reactor after updating its stale
duplicate-analyzer diagnostic expectation.
- Discriminating native RED/GREEN tests cover legacy exact reader
metadata, normalized index metadata, policy-name collisions, a reserved
built-in-name collision, canonical-key precedence, legacy normalizers,
scoring selection, encoded-policy-name collisions, ambiguous non-IK
selectors, document-relative Pinyin offsets for every custom tokenizer,
retained allocations for large ICU inputs, rune-expansion provenance,
default Pinyin allocation behavior, explicit component defaults, and
scoring analyzer failures.
- `test_ik_custom_analyzer`, `test_analyzer_identity_semantics`, and
existing `test_ik_analyzer` passed earlier against the task-owned native
ASAN cluster. The targeted analyzer-identity regression passed 1/1; the
Basic CREATE/ALTER cases added by the latest review fix were not run
locally.
- The current candidate contains 26 PR commits relative to merge base
`bf127c2d9ca1cdbb70c604320120fea1f1e33002`; exact merge-tree preflight
against target snapshot `3750a819a2af157ad539d947bc301ff0f806b928`
passed before publication.
- The full committed PR English audit passed for 78 source files.
Thirty-seven precisely reviewed Unicode input and expected-token
literals remain as test data; no descriptive text was exempted.
- Clang-tidy 16 reported no changed-line warnings in the modified
production sources. Three ASAN test translation units could not be
analyzed because the local clang-tidy resource headers lack the
sanitizer headers; no suppression was added.
- Exact-current-head merged-snapshot clang-format 16 and all FE reactor
Checkstyle modules passed in the shared publication gate before push.
- Local results are not a claim that fresh full CI or AI review passed.
Those runs are requested and evaluated against the published candidate.
- Behavior changed: Yes. Custom analyzers support IK tokenization; saved
policy bindings, physical index selection, source offsets, effective
duplicate-index identities, and analyzer selection remain consistent
across query and DDL paths.
- Does this need documentation: No new syntax is introduced. Existing
analyzer configuration syntax is reused; no documentation changes are
included.
Component-level replay, serialization, dispatch, reader selection,
scoring selection, and current-version end-to-end regression were
validated. A full mixed-version cluster upgrade was not run. Existing IK
internal lowercasing behavior is unchanged; the uppercase SQL probe is
not treated as proof of a lowercase-disabled runtime fix.1 parent fd4f8ac commit ef72b93
96 files changed
Lines changed: 11873 additions & 534 deletions
File tree
- be
- src
- exprs
- function
- runtime/index_policy
- storage/index/inverted
- analyzer
- ik
- core
- char_filter
- similarity
- token_filter
- tokenizer
- basic
- char
- empty
- icu
- ik
- keyword
- ngram
- pinyin
- standard
- test
- exprs/function
- runtime/index_policy
- storage
- index
- inverted
- analyzer
- char_filter
- similarity
- token_filter
- tokenizer
- segment
- fe
- fe-catalog/src/main/java/org/apache/doris/analysis
- fe-core/src
- main/java/org/apache/doris
- alter
- analysis
- invertedindex
- catalog
- indexpolicy
- nereids
- glue/translator
- trees
- expressions
- plans/commands/info
- test/java/org/apache/doris
- alter
- analysis
- invertedindex
- indexpolicy
- nereids/glue/translator
- regression-test
- data/inverted_index_p0/analyzer
- suites/inverted_index_p0/analyzer
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
307 | 307 | | |
308 | 308 | | |
309 | 309 | | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
310 | 333 | | |
311 | 334 | | |
312 | 335 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
93 | 93 | | |
94 | 94 | | |
95 | 95 | | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
96 | 106 | | |
97 | 107 | | |
98 | 108 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
62 | 62 | | |
63 | 63 | | |
64 | 64 | | |
65 | | - | |
66 | | - | |
67 | | - | |
68 | | - | |
69 | | - | |
70 | | - | |
71 | | - | |
72 | | - | |
| 65 | + | |
73 | 66 | | |
74 | 67 | | |
75 | 68 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
65 | 65 | | |
66 | 66 | | |
67 | 67 | | |
68 | | - | |
| 68 | + | |
69 | 69 | | |
70 | 70 | | |
71 | 71 | | |
| |||
90 | 90 | | |
91 | 91 | | |
92 | 92 | | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
93 | 107 | | |
94 | 108 | | |
95 | 109 | | |
| |||
415 | 429 | | |
416 | 430 | | |
417 | 431 | | |
418 | | - | |
419 | | - | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
420 | 438 | | |
421 | 439 | | |
422 | 440 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
102 | 102 | | |
103 | 103 | | |
104 | 104 | | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
105 | 110 | | |
106 | 111 | | |
107 | 112 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
58 | 58 | | |
59 | 59 | | |
60 | 60 | | |
61 | | - | |
62 | | - | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
63 | 65 | | |
64 | 66 | | |
65 | 67 | | |
| |||
81 | 83 | | |
82 | 84 | | |
83 | 85 | | |
84 | | - | |
85 | | - | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
86 | 90 | | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
87 | 109 | | |
88 | 110 | | |
89 | 111 | | |
| |||
0 commit comments