Skip to content

Commit ef72b93

Browse files
authored
[feat](inverted-index) Add IK tokenizers to custom analyzers (apache#67918)
### What problem does this PR solve? Custom inverted-index analyzers could not use IK smart or max-word tokenization with their configured character and token filters. This adds the `ik_smart` and `ik_max_word` tokenizer factories and policy validation. Composed pipelines need source-byte provenance across normalization, splitting, concatenation, trimming, and reset/reuse. Opt-in offset tracking propagates through IK, keyword, standard, ICU normalization, word-delimiter, and pinyin processing. Separate rune end offsets preserve gaps removed by delimiter concatenation. Review fixes preserve exact saved policy bindings across replay, lifecycle operations, MATCH serialization, provider dispatch, physical reader selection, and scoring-statistics collection. FE matches explicit built-in IK to its default max-word configuration. BE selection keys distinguish mode, lowercase configuration, and effective outer character filters without changing provider names, Thrift, or stored index formats. Encoded configurations cannot collide with arbitrary policy names. For backend-first upgrades, BE now binds an old FE's normalized analyzer name to the canonical saved policy and accepts a lowercase metadata key only when it resolves to the same policy. Exact metadata remains preferred. The compatibility alias cannot select a differently bound policy or a built-in analyzer with different tokenization. Named character-filter identities account for transformations absorbed by the resolved tokenizer while preserving meaningful ordered transformations. The two new global-policy suites run in `nonConcurrent` instead of contributing to concurrent P0 policy-quota exhaustion. Classic writer and query boundaries now convert lazy custom-analyzer failures to non-OK `Status` results. This includes ARRAY token-stream creation before `add_document()`, which clears the partial document and closes the writer on error instead of allowing a Doris exception to escape. FE identity construction also preserves case-folding absorption through the default `icu_normalizer` `nfkc_cf` filter, preventing equivalent lower-then-fold and fold-only pipelines from receiving different identities. FE policy validation now checks an exact replayed policy before falling back to a normalized built-in analyzer or normalizer. This prevents a legacy wrong-type `IK` or `LOWERCASE` policy from passing FE validation and failing later in BE. `CREATE ... IF NOT EXISTS` also returns for an existing replayed `ik_smart` or `ik_max_word` policy before applying the reserved built-in-name check, while fresh creations with those names remain rejected. Nested tokenizer, token-filter, and character-filter validation uses the same exact-first policy lookup as BE and enforces the expected component type. Replayed case-colliding policies can no longer bind a different saved component or the wrong filter family. When ICU token normalization changes text, its output carries a conservative source map to the original token span; unchanged tokens continue to delegate their precise upstream maps, and reset/reuse clears rewritten-map state. The latest AI-review fixes reject duplicate non-IK analyzer selectors before full identity comparison, including the implicit default selector, so outer character-filter differences cannot create two indexes that the query syntax cannot distinguish. Basic, CharGroup, Empty, ICU, and NGram tokenizers now publish document-relative source boundaries for offset-aware Pinyin. ICU resolves monotonically increasing UTF-16 break boundaries with a constant-size UTF-8/UTF-16 cursor, transformations that change rune counts use a conservative whole-source span, and NGram byte offsets preserve full `UChar32` values. The memory fixes remove full-input dense offset tables from ICU tokenization and normalization. The tokenizer preserves exact spans with its incremental cursor. The normalizer retains opt-in provenance, stores no provenance when unused, and represents a changed token with one compact conservative source span when enabled; Pinyin consumes that span without forcing dense maps. The current review fixes prevent rune-expanding ASCII folding and lowercase filters from forwarding invalid exact provenance, avoid allocating Pinyin rune and source arrays in the default ignore-offset mode, canonicalize FE analyzer identities using effective component defaults, and convert lazy scoring-analyzer failures back into `Status` results. The final CI follow-up aligns the duplicate-analyzer unit test with the intentional selector-based diagnostic introduced by those validation fixes. Current candidate: `31450230c09d6ba10b6f0c2e8de70f3765948543`. ### Release note Add IK smart and max-word tokenizers to custom inverted-index analyzers. Correct composed-tokenizer source offsets, preserve custom-analyzer MATCH bindings during backend-first upgrades, select the matching physical IK configuration, and fix effective duplicate-index identities. ### Check List (For Author) - Test: - Native FE and BE ASAN builds passed in the isolated task worktree. The BE build included the GLIBC 2.17 compatibility gate and its CDC client stage. - The official BE ASAN runner passed all 9 focused writer, reader, and analyzer failure-path tests, including the ARRAY pre-`add_document()` exception boundary. The complete `InvertedIndexWriterTest` suite passed 20/20. - The official FE runner passed all 22 `AnalyzerIdentityBuilderTest` tests. After policy-validation repairs, it also passed the replay/image coverage and all 40 `PolicyValidatorTests`; `./build.sh --fe` completed successfully with zero Checkstyle violations. - The official BE ASAN runner passed 77/77 tests across `IndexPolicyMgrTest`, `PinyinFilterTest`, and `ICUNormalizerFilterFactoryTest`, including component type collisions and ICU-normalizer-to-pinyin source spans before and after reset. - A broader official BE ASAN run passed 354 tests in 10 related suites. A later tokenizer/filter run passed 150/150, and the two new offset-focused Pinyin tests passed 2/2 on the final source content. - The memory-fix candidate passed 203/203 related ASAN tokenizer and filter tests, including deterministic large-input allocation coverage for the ICU tokenizer and both disabled and enabled ICU-normalizer provenance paths. The full ASAN BE build then passed. - An official FE run passed 118 tests in four classes, including real index selection and MATCH-to-Thrift translation. The selector-ambiguity coverage passed 58/58 in `InvertedIndexPropertiesTest`. - The current review-fix candidate passed 147/147 related ASAN BE tests covering Pinyin, ASCII folding, lowercase, and scoring collection. A full ASAN BE build passed after the C++ changes. - The latest review-fix head passed 112/112 targeted BE ASAN tests across ICU, Keyword, NGram, and Pinyin coverage. It also passed 86/86 FE tests across `AnalyzerIdentityBuilderTest` and `InvertedIndexPropertiesTest`; the full 80-module Maven reactor and Checkstyle completed successfully. - The final candidate passed all 22 `SchemaChangeHandlerTest` tests and the complete 80-module Maven reactor after updating its stale duplicate-analyzer diagnostic expectation. - Discriminating native RED/GREEN tests cover legacy exact reader metadata, normalized index metadata, policy-name collisions, a reserved built-in-name collision, canonical-key precedence, legacy normalizers, scoring selection, encoded-policy-name collisions, ambiguous non-IK selectors, document-relative Pinyin offsets for every custom tokenizer, retained allocations for large ICU inputs, rune-expansion provenance, default Pinyin allocation behavior, explicit component defaults, and scoring analyzer failures. - `test_ik_custom_analyzer`, `test_analyzer_identity_semantics`, and existing `test_ik_analyzer` passed earlier against the task-owned native ASAN cluster. The targeted analyzer-identity regression passed 1/1; the Basic CREATE/ALTER cases added by the latest review fix were not run locally. - The current candidate contains 26 PR commits relative to merge base `bf127c2d9ca1cdbb70c604320120fea1f1e33002`; exact merge-tree preflight against target snapshot `3750a819a2af157ad539d947bc301ff0f806b928` passed before publication. - The full committed PR English audit passed for 78 source files. Thirty-seven precisely reviewed Unicode input and expected-token literals remain as test data; no descriptive text was exempted. - Clang-tidy 16 reported no changed-line warnings in the modified production sources. Three ASAN test translation units could not be analyzed because the local clang-tidy resource headers lack the sanitizer headers; no suppression was added. - Exact-current-head merged-snapshot clang-format 16 and all FE reactor Checkstyle modules passed in the shared publication gate before push. - Local results are not a claim that fresh full CI or AI review passed. Those runs are requested and evaluated against the published candidate. - Behavior changed: Yes. Custom analyzers support IK tokenization; saved policy bindings, physical index selection, source offsets, effective duplicate-index identities, and analyzer selection remain consistent across query and DDL paths. - Does this need documentation: No new syntax is introduced. Existing analyzer configuration syntax is reused; no documentation changes are included. Component-level replay, serialization, dispatch, reader selection, scoring selection, and current-version end-to-end regression were validated. A full mixed-version cluster upgrade was not run. Existing IK internal lowercasing behavior is unchanged; the uppercase SQL probe is not treated as proof of a lowercase-disabled runtime fix.
1 parent fd4f8ac commit ef72b93

96 files changed

Lines changed: 11873 additions & 534 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎be/src/exprs/function/function_search.cpp‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -307,6 +307,29 @@ Status FunctionSearch::evaluate_inverted_index_with_search_param(
307307
const IndexExecContext* index_exec_ctx,
308308
const std::unordered_map<std::string, int>& field_name_to_column_id,
309309
const std::shared_ptr<IndexQueryContext>& index_query_context) const {
310+
// VSearchExpr enters here directly, outside IFunction::execute() and its exception boundary,
311+
// so this is where a failure inside the search, such as an analyzer whose first token
312+
// stream throws, has to become a Status.
313+
try {
314+
return evaluate_inverted_index_with_search_param_unguarded(
315+
search_param, data_type_with_names, std::move(iterators), num_rows, bitmap_result,
316+
enable_cache, index_exec_ctx, field_name_to_column_id, index_query_context);
317+
} catch (const CLuceneError& e) {
318+
return Status::Error<ErrorCode::INVERTED_INDEX_ANALYZER_ERROR>("search failed: {}",
319+
e.what());
320+
} catch (const Exception& e) {
321+
return e.to_status();
322+
}
323+
}
324+
325+
Status FunctionSearch::evaluate_inverted_index_with_search_param_unguarded(
326+
const TSearchParam& search_param,
327+
const std::unordered_map<std::string, IndexFieldNameAndTypePair>& data_type_with_names,
328+
std::unordered_map<std::string, IndexIterator*> iterators, uint32_t num_rows,
329+
InvertedIndexResultBitmap& bitmap_result, bool enable_cache,
330+
const IndexExecContext* index_exec_ctx,
331+
const std::unordered_map<std::string, int>& field_name_to_column_id,
332+
const std::shared_ptr<IndexQueryContext>& index_query_context) const {
310333
const bool is_nested_query = search_param.root.clause_type == "NESTED";
311334
if (is_nested_query && !is_nested_group_search_supported()) {
312335
return Status::NotSupported(

‎be/src/exprs/function/function_search.h‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,16 @@ class FunctionSearch : public IFunction {
9393
const std::unordered_map<std::string, int>& field_name_to_column_id,
9494
const std::shared_ptr<IndexQueryContext>& index_query_context = nullptr) const;
9595

96+
// The body of the overload above; it may throw, so only that overload calls it.
97+
Status evaluate_inverted_index_with_search_param_unguarded(
98+
const TSearchParam& search_param,
99+
const std::unordered_map<std::string, IndexFieldNameAndTypePair>& data_type_with_names,
100+
std::unordered_map<std::string, IndexIterator*> iterators, uint32_t num_rows,
101+
InvertedIndexResultBitmap& bitmap_result, bool enable_cache,
102+
const IndexExecContext* index_exec_ctx,
103+
const std::unordered_map<std::string, int>& field_name_to_column_id,
104+
const std::shared_ptr<IndexQueryContext>& index_query_context) const;
105+
96106
// Public methods for testing
97107
enum class ClauseTypeCategory {
98108
NON_TOKENIZED, // TERM, PREFIX, WILDCARD, REGEXP, RANGE, LIST - no tokenization, use EQUAL_QUERY

‎be/src/exprs/function/match.cpp‎

Lines changed: 1 addition & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -62,14 +62,7 @@ Status FunctionMatchBase::evaluate_inverted_index(
6262
}
6363
const std::string& function_name = get_name();
6464

65-
if (function_name == MATCH_PHRASE_FUNCTION || function_name == MATCH_PHRASE_PREFIX_FUNCTION ||
66-
function_name == MATCH_PHRASE_EDGE_FUNCTION) {
67-
auto reader = iter->get_reader(InvertedIndexReaderType::FULLTEXT);
68-
if (reader && !segment_v2::IndexReaderHelper::is_support_phrase(reader)) {
69-
return Status::Error<ErrorCode::INDEX_INVALID_PARAMETERS>(
70-
"phrase queries require setting support_phrase = true");
71-
}
72-
}
65+
// support_phrase is checked once the analyzer has selected the reader that runs the query.
7366
Field param_value;
7467
arguments[0].column->get(0, param_value);
7568
if (param_value.is_null()) {

‎be/src/exprs/function/variant_inverted_index_search.cpp‎

Lines changed: 21 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,7 @@ void add_search_binding_diagnostic(const std::shared_ptr<IndexQueryContext>& con
6565
}
6666
}
6767

68-
InvertedIndexAnalyzerCtxSPtr build_analyzer_context(
68+
InvertedIndexAnalyzerCtxSPtr build_analyzer_context_unsafe(
6969
const std::map<std::string, std::string>& properties, const std::string& analyzer_key) {
7070
InvertedIndexAnalyzerConfig config;
7171
config.analyzer_name = get_analyzer_name_from_properties(properties);
@@ -90,6 +90,20 @@ InvertedIndexAnalyzerCtxSPtr build_analyzer_context(
9090

9191
} // namespace
9292

93+
// Replayed components can collide across policy families, so building the provider throws.
94+
Result<InvertedIndexAnalyzerCtxSPtr> build_search_analyzer_context(
95+
const std::map<std::string, std::string>& properties, const std::string& analyzer_key) {
96+
try {
97+
return build_analyzer_context_unsafe(properties, analyzer_key);
98+
} catch (const CLuceneError& error) {
99+
return ResultError(Status::Error<ErrorCode::INVERTED_INDEX_ANALYZER_ERROR>(
100+
"Build search analyzer failed: {}", error.what()));
101+
} catch (const Exception& error) {
102+
return ResultError(Status::Error<ErrorCode::INVERTED_INDEX_ANALYZER_ERROR>(
103+
"Build search analyzer failed: {}", error.what()));
104+
}
105+
}
106+
93107
FieldReaderResolver::FieldReaderResolver(
94108
const std::unordered_map<std::string, IndexFieldNameAndTypePair>& data_type_with_names,
95109
const std::unordered_map<std::string, IndexIterator*>& iterators,
@@ -415,8 +429,12 @@ Status FieldReaderResolver::resolve_with_analyzer_context(const std::string& fie
415429
return Status::OK();
416430
}
417431

418-
binding->analyzer_context =
419-
build_analyzer_context(binding->index_properties, binding->analyzer_key);
432+
auto built_context =
433+
build_search_analyzer_context(binding->index_properties, binding->analyzer_key);
434+
if (!built_context.has_value()) {
435+
return built_context.error();
436+
}
437+
binding->analyzer_context = std::move(built_context.value());
420438
_cache.at(binding->binding_key).analyzer_context = binding->analyzer_context;
421439
return Status::OK();
422440
}

‎be/src/exprs/function/variant_inverted_index_search.h‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,6 +102,11 @@ struct FieldReaderBinding {
102102
}
103103
};
104104

105+
// Build the analyzer context a SEARCH binding executes with, converting a failure to build the
106+
// analyzer provider into a Status instead of letting the exception escape a Status-returning caller.
107+
Result<InvertedIndexAnalyzerCtxSPtr> build_search_analyzer_context(
108+
const std::map<std::string, std::string>& properties, const std::string& analyzer_key);
109+
105110
class FieldReaderResolver {
106111
public:
107112
FieldReaderResolver(

‎be/src/exprs/vmatch_predicate.cpp‎

Lines changed: 26 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -58,8 +58,10 @@ namespace doris {
5858
using namespace doris::segment_v2;
5959

6060
VMatchPredicate::VMatchPredicate(const TExprNode& node) : VExpr(node) {
61-
const auto resolved = AnalyzerConfigParser::parse(node.match_predicate.analyzer_name,
62-
node.match_predicate.parser_type);
61+
const auto resolved = AnalyzerConfigParser::parse(
62+
node.match_predicate.analyzer_name, node.match_predicate.parser_type,
63+
node.match_predicate.parser_mode, node.match_predicate.parser_lowercase,
64+
node.match_predicate.char_filter_map);
6365

6466
InvertedIndexAnalyzerConfig config;
6567
config.analyzer_name = resolved.provider_name;
@@ -81,9 +83,29 @@ VMatchPredicate::VMatchPredicate(const TExprNode& node) : VExpr(node) {
8183
_analyzer_ctx->parser_type = resolved.parser_type;
8284

8385
if (_analyzer_ctx->requires_analysis()) {
84-
_analyzer_provider =
85-
inverted_index::InvertedIndexAnalyzer::create_analyzer_provider(&config);
86+
std::string bound_name;
87+
std::string legacy_name;
88+
_analyzer_provider = inverted_index::InvertedIndexAnalyzer::create_analyzer_provider(
89+
&config, &bound_name, &legacy_name);
8690
_analyzer = _analyzer_provider->get_analyzer();
91+
if (!legacy_name.empty() && legacy_name != bound_name) {
92+
_analyzer_ctx->legacy_analyzer_key =
93+
AnalyzerConfigParser::parse(legacy_name, node.match_predicate.parser_type,
94+
node.match_predicate.parser_mode,
95+
node.match_predicate.parser_lowercase,
96+
node.match_predicate.char_filter_map)
97+
.analyzer_key;
98+
}
99+
if (bound_name != resolved.provider_name) {
100+
// Reader selection and query tokenization must use the same policy binding.
101+
_analyzer_ctx->analyzer_key =
102+
AnalyzerConfigParser::parse(bound_name, node.match_predicate.parser_type,
103+
node.match_predicate.parser_mode,
104+
node.match_predicate.parser_lowercase,
105+
node.match_predicate.char_filter_map)
106+
.analyzer_key;
107+
_analyzer_ctx->analyzer_name = std::move(bound_name);
108+
}
87109
}
88110

89111
_analyzer_ctx->char_filter_map = std::move(config.char_filter_map);

0 commit comments

Comments
 (0)