Skip to content

Commit 4501552

Browse files
committed
[fix](inverted-index) Preserve effective IK identities and index bindings
### What problem does this PR solve? Related PR: apache#67918 Problem Summary: Explicit built-in IK requests could select a smart or differently configured physical index after distinct IK identities were admitted on one column. Named A-to-a character replacement was not folded through IK's default lowercasing, and word delimiter concatenation discarded source gaps needed by downstream pinyin offsets. Encoded selection keys could also collide with arbitrary legal policy names. Match built-in IK indexes by effective configuration and carry the existing mode, lowercase and outer character-filter fields into BE selection keys. Separate encoded configurations from policy names without changing provider bindings, Thrift or stored index formats. Canonicalize named character filters in tokenizer context, and preserve optional per-rune source ends through concatenation and reset. Run the two new global-policy suites in the nonConcurrent group to avoid contributing to concurrent quota exhaustion. ### Release note Fix IK MATCH index selection and effective analyzer duplicate detection, and preserve pinyin source offsets through word delimiter concatenation. ### Check List (For Author) - Test - [x] Regression test Official runner: test_analyzer_identity_semantics and test_ik_custom_analyzer passed together; test_ik_analyzer passed separately. Expected output was generated by the runner and then checked in a normal run on a task-owned ASAN cluster. - [x] Unit Test Official ASAN BE runner: 178 tests across 12 related suites passed. After clang-tidy fixes, all 120 tests in the three affected suites passed again. Official FE runner: 118 tests in four classes passed. The three review defects and the encoded-name collision have discriminating native RED/GREEN evidence. - [ ] Manual test - [ ] No need to test or manual test. Explain why: - Additional validation Native build.sh --be --fe passed; BE was rebuilt after the final source change. Java Checkstyle and build hygiene passed. Clang-tidy checked all 12 newly changed C++ files and rechecked the three files subsequently corrected, with no checks disabled. The run remains qualified by pre-existing processCurrentToken size/complexity diagnostics and diagnostics in unchanged CLucene and Boost headers. A baseline-only unmatched NOLINTEND comment was removed through a temporary VFS overlay, not by changing repository code. A controlled baseline source comparison confirmed the function's complexity decreased from 75 to 69 and its length from 141 to 125 lines. These are not claimed as a clean tidy pass. - Behavior changed: - [ ] No. - [x] Yes. Explicit built-in IK uses its matching default configuration; distinct physical configurations retain separate selection keys, equivalent named character filters are rejected as duplicates, and concatenated tokens retain their original source spans. - Does this need documentation? - [x] No. This corrects existing analyzer behavior and test isolation. - [ ] Yes.
1 parent b92b994 commit 4501552

19 files changed

Lines changed: 560 additions & 77 deletions

File tree

‎be/src/exprs/vmatch_predicate.cpp‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -58,8 +58,10 @@ namespace doris {
5858
using namespace doris::segment_v2;
5959

6060
VMatchPredicate::VMatchPredicate(const TExprNode& node) : VExpr(node) {
61-
const auto resolved = AnalyzerConfigParser::parse(node.match_predicate.analyzer_name,
62-
node.match_predicate.parser_type);
61+
const auto resolved = AnalyzerConfigParser::parse(
62+
node.match_predicate.analyzer_name, node.match_predicate.parser_type,
63+
node.match_predicate.parser_mode, node.match_predicate.parser_lowercase,
64+
node.match_predicate.char_filter_map);
6365

6466
InvertedIndexAnalyzerConfig config;
6567
config.analyzer_name = resolved.provider_name;

‎be/src/storage/index/inverted/inverted_index_parser.cpp‎

Lines changed: 64 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,10 @@
1717

1818
#include "storage/index/inverted/inverted_index_parser.h"
1919

20+
#include <fmt/format.h>
21+
22+
#include <algorithm>
23+
2024
#include "common/config.h"
2125
#include "storage/index/inverted/analyzer/analyzer.h"
2226
#include "storage/tablet/tablet_schema.h"
@@ -215,28 +219,66 @@ std::string normalize_analyzer_key(std::string_view analyzer) {
215219
return std::string(analyzer);
216220
}
217221

218-
std::string build_analyzer_key_from_properties(
219-
const std::map<std::string, std::string>& properties) {
220-
const auto analyzer_name = get_analyzer_name_from_properties(properties);
221-
if (!analyzer_name.empty()) {
222-
return normalize_analyzer_key(analyzer_name);
222+
namespace {
223+
224+
constexpr std::string_view ENCODED_ANALYZER_KEY_PREFIX = "#analysis:";
225+
226+
std::string append_char_filter_key(std::string key, const CharFilterMap& char_filter_map,
227+
bool lowercase_ik) {
228+
if (char_filter_map.empty()) {
229+
return key;
230+
}
231+
DORIS_CHECK_EQ(char_filter_map.at(INVERTED_INDEX_PARSER_CHAR_FILTER_TYPE),
232+
INVERTED_INDEX_CHAR_FILTER_CHAR_REPLACE);
233+
std::string pattern = char_filter_map.at(INVERTED_INDEX_PARSER_CHAR_FILTER_PATTERN);
234+
const auto& replacement = char_filter_map.at(INVERTED_INDEX_PARSER_CHAR_FILTER_REPLACEMENT);
235+
DORIS_CHECK_EQ(replacement.size(), 1);
236+
const char replacement_byte = replacement.front();
237+
std::erase_if(pattern, [replacement_byte, lowercase_ik](char byte) {
238+
return byte == replacement_byte ||
239+
(lowercase_ik && replacement_byte >= 'a' && replacement_byte <= 'z' &&
240+
byte == replacement_byte - ('a' - 'A'));
241+
});
242+
std::ranges::sort(pattern);
243+
pattern.erase(std::ranges::unique(pattern).begin(), pattern.end());
244+
if (pattern.empty()) {
245+
return key;
223246
}
247+
return fmt::format("{}char_replace={}:{}:{}:{}:{}:{};", ENCODED_ANALYZER_KEY_PREFIX, key.size(),
248+
key, pattern.size(), pattern, replacement.size(), replacement);
249+
}
224250

225-
std::string parser;
226-
auto parser_it = properties.find(INVERTED_INDEX_PARSER_KEY);
227-
if (parser_it != properties.end()) {
228-
parser = parser_it->second;
229-
} else {
230-
parser_it = properties.find(INVERTED_INDEX_PARSER_KEY_ALIAS);
231-
if (parser_it != properties.end()) {
232-
parser = parser_it->second;
233-
}
251+
std::string build_selection_key(std::string key, const std::string& parser_mode, bool lowercase,
252+
const CharFilterMap& char_filter_map) {
253+
const bool builtin_ik = key == INVERTED_INDEX_PARSER_IK;
254+
if (builtin_ik) {
255+
key = fmt::format("{}ik|mode={}|lower_case={}", ENCODED_ANALYZER_KEY_PREFIX,
256+
parser_mode == INVERTED_INDEX_PARSER_SMART
257+
? INVERTED_INDEX_PARSER_SMART
258+
: INVERTED_INDEX_PARSER_MAX_WORD,
259+
lowercase);
260+
} else if (key.starts_with(ENCODED_ANALYZER_KEY_PREFIX)) {
261+
// Keep arbitrary policy names separate from encoded index configurations.
262+
key = fmt::format("{}name={}:{}", ENCODED_ANALYZER_KEY_PREFIX, key.size(), key);
234263
}
264+
return append_char_filter_key(std::move(key), char_filter_map, builtin_ik && lowercase);
265+
}
235266

236-
if (parser.empty()) {
237-
return INVERTED_INDEX_PARSER_NONE;
267+
} // namespace
268+
269+
std::string build_analyzer_key_from_properties(
270+
const std::map<std::string, std::string>& properties) {
271+
auto key = get_analyzer_name_from_properties(properties);
272+
if (key.empty()) {
273+
key = to_lower(get_parser_string_from_properties(properties));
274+
if (key.empty()) {
275+
key = INVERTED_INDEX_PARSER_NONE;
276+
}
238277
}
239-
return to_lower(parser);
278+
return build_selection_key(
279+
std::move(key), get_parser_mode_string_from_properties(properties),
280+
get_parser_lowercase_from_properties(properties) != INVERTED_INDEX_PARSER_FALSE,
281+
get_parser_char_filter_map_from_properties(properties));
240282
}
241283

242284
// ============================================================================
@@ -248,11 +290,14 @@ bool AnalyzerConfigParser::is_builtin_analyzer(const std::string& analyzer_name)
248290
}
249291

250292
AnalyzerConfig AnalyzerConfigParser::parse(const std::string& analyzer_name,
251-
const std::string& parser_type_str) {
293+
const std::string& parser_type_str,
294+
const std::string& parser_mode, bool lowercase,
295+
const CharFilterMap& char_filter_map) {
252296
AnalyzerConfig config;
253297

254298
if (!analyzer_name.empty()) {
255-
config.analyzer_key = normalize_analyzer_key(analyzer_name);
299+
config.analyzer_key =
300+
build_selection_key(analyzer_name, parser_mode, lowercase, char_filter_map);
256301
if (is_builtin_analyzer(analyzer_name)) {
257302
config.parser_type = get_inverted_index_parser_type_from_string(analyzer_name);
258303
} else {

‎be/src/storage/index/inverted/inverted_index_parser.h‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -196,15 +196,15 @@ std::string get_parser_dict_compression_from_properties(
196196
std::string get_analyzer_name_from_properties(const std::map<std::string, std::string>& properties);
197197

198198
// Build an exact analyzer key from index properties.
199-
// Precedence is analyzer, normalizer, then parser type. A raw index uses "none".
199+
// Include IK mode/lowercase and effective outer character filters to distinguish physical readers.
200200
std::string build_analyzer_key_from_properties(
201201
const std::map<std::string, std::string>& properties);
202202

203203
// Result structure for analyzer config parsing
204204
struct AnalyzerConfig {
205205
std::string provider_name;
206206
InvertedIndexParserType parser_type = InvertedIndexParserType::PARSER_NONE;
207-
// Physical reader selection key from the Thrift analyzer name.
207+
// Physical reader selection key from the Thrift analyzer configuration.
208208
// Empty allows fallback selection; non-empty requires an exact match.
209209
std::string analyzer_key;
210210

@@ -219,7 +219,10 @@ class AnalyzerConfigParser {
219219
// @param analyzer_name: Analyzer selection name from Thrift (custom, builtin, or empty).
220220
// @param parser_type_str: Parser type string like "chinese", "standard", etc.
221221
[[nodiscard]] static AnalyzerConfig parse(const std::string& analyzer_name,
222-
const std::string& parser_type_str);
222+
const std::string& parser_type_str,
223+
const std::string& parser_mode = "",
224+
bool lowercase = true,
225+
const CharFilterMap& char_filter_map = {});
223226

224227
// Use the writer's case-sensitive built-in dispatch.
225228
[[nodiscard]] static bool is_builtin_analyzer(const std::string& analyzer_name);

‎be/src/storage/index/inverted/token_filter/pinyin_filter.cpp‎

Lines changed: 15 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,9 @@ Token* PinyinFilter::next(Token* token) {
8282
current_end_offset_ = token->endOffset();
8383
auto source_byte_offsets = get_source_byte_offsets();
8484
current_source_byte_offsets_.assign(source_byte_offsets.begin(), source_byte_offsets.end());
85+
auto source_byte_end_offsets = get_source_byte_end_offsets();
86+
current_source_byte_end_offsets_.assign(source_byte_end_offsets.begin(),
87+
source_byte_end_offsets.end());
8588

8689
done_ = false;
8790
}
@@ -126,6 +129,7 @@ void PinyinFilter::resetVariables() {
126129
current_source_.clear();
127130
current_runes_.clear();
128131
current_source_byte_offsets_.clear();
132+
current_source_byte_end_offsets_.clear();
129133
candidate_offset_ = 0;
130134
terms_filter_.clear();
131135
last_increment_position_ = 0;
@@ -240,6 +244,8 @@ bool PinyinFilter::prepareCurrentSource(std::vector<UChar32>& source_codepoints)
240244
std::vector<UChar32> original_codepoints;
241245
const auto original_runes = convertToRunes(current_token_text_, original_codepoints);
242246
if (current_source_byte_offsets_.size() == original_runes.size() + 1) {
247+
DORIS_CHECK(current_source_byte_end_offsets_.empty() ||
248+
current_source_byte_end_offsets_.size() == original_runes.size());
243249
const auto start_rune = std::ranges::lower_bound(
244250
original_runes, static_cast<int32_t>(source_start), {}, &RuneInfo::byte_start);
245251
const auto end_rune = std::ranges::lower_bound(
@@ -249,12 +255,18 @@ bool PinyinFilter::prepareCurrentSource(std::vector<UChar32>& source_codepoints)
249255
DORIS_CHECK_EQ(end_index - start_index, current_runes_.size());
250256
const int32_t token_start_offset = current_start_offset_;
251257
current_start_offset_ += current_source_byte_offsets_[start_index];
252-
current_end_offset_ = token_start_offset + current_source_byte_offsets_[end_index];
258+
current_end_offset_ =
259+
token_start_offset + (current_source_byte_end_offsets_.empty()
260+
? current_source_byte_offsets_[end_index]
261+
: current_source_byte_end_offsets_[end_index - 1]);
253262
for (size_t i = 0; i < current_runes_.size(); ++i) {
254263
current_runes_[i].byte_start = current_source_byte_offsets_[start_index + i] -
255264
current_source_byte_offsets_[start_index];
256-
current_runes_[i].byte_end = current_source_byte_offsets_[start_index + i + 1] -
257-
current_source_byte_offsets_[start_index];
265+
current_runes_[i].byte_end =
266+
(current_source_byte_end_offsets_.empty()
267+
? current_source_byte_offsets_[start_index + i + 1]
268+
: current_source_byte_end_offsets_[start_index + i]) -
269+
current_source_byte_offsets_[start_index];
258270
}
259271
} else {
260272
current_start_offset_ += static_cast<int32_t>(source_start);

‎be/src/storage/index/inverted/token_filter/pinyin_filter.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,7 @@ class PinyinFilter : public DorisTokenFilter {
103103
int current_end_offset_;
104104
std::vector<RuneInfo> current_runes_;
105105
std::vector<int32_t> current_source_byte_offsets_;
106+
std::vector<int32_t> current_source_byte_end_offsets_;
106107
};
107108

108109
using PinyinFilterPtr = std::shared_ptr<PinyinFilter>;

‎be/src/storage/index/inverted/token_filter/token_filter.h‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,12 @@ class DorisTokenFilter : public TokenFilter, public DorisTokenStream {
3333
return source == nullptr ? std::span<const int32_t> {} : source->get_source_byte_offsets();
3434
}
3535

36+
std::span<const int32_t> get_source_byte_end_offsets() const override {
37+
const auto* source = dynamic_cast<const DorisTokenStream*>(_in.get());
38+
return source == nullptr ? std::span<const int32_t> {}
39+
: source->get_source_byte_end_offsets();
40+
}
41+
3642
void set_source_byte_offsets_enabled(bool enabled) override {
3743
auto* source = dynamic_cast<DorisTokenStream*>(_in.get());
3844
if (source != nullptr) {

‎be/src/storage/index/inverted/token_filter/word_delimiter_filter.cpp‎

Lines changed: 37 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,7 @@ Token* WordDelimiterFilter::next(Token* t) {
6969
_accum_pos_inc = 0;
7070
_first = false;
7171
_current_source_byte_offsets = _saved_source_byte_offsets;
72+
_current_source_byte_end_offsets = _saved_source_byte_end_offsets;
7273
return t;
7374
}
7475

@@ -90,6 +91,7 @@ Token* WordDelimiterFilter::next(Token* t) {
9091
_accum_pos_inc = 0;
9192
_first = false;
9293
_current_source_byte_offsets = _saved_source_byte_offsets;
94+
_current_source_byte_end_offsets = _saved_source_byte_end_offsets;
9395
return t;
9496
}
9597
}
@@ -128,6 +130,8 @@ Token* WordDelimiterFilter::next(Token* t) {
128130
t->setStartOffset(_states[_buffered_pos - 1].token_start_offset);
129131
t->setEndOffset(_states[_buffered_pos - 1].token_end_offset);
130132
_current_source_byte_offsets = _states[_buffered_pos - 1].source_byte_offsets;
133+
_current_source_byte_end_offsets =
134+
_states[_buffered_pos - 1].source_byte_end_offsets;
131135
if (_first && get_position_increment(t) == 0) {
132136
set_position_increment(t, 1);
133137
}
@@ -148,6 +152,7 @@ Token* WordDelimiterFilter::next(Token* t) {
148152
t->setStartOffset(_attribute.token_start_offset);
149153
t->setEndOffset(_attribute.token_end_offset);
150154
_current_source_byte_offsets = _attribute.source_byte_offsets;
155+
_current_source_byte_end_offsets = _attribute.source_byte_end_offsets;
151156
return t;
152157
}
153158

@@ -191,8 +196,10 @@ void WordDelimiterFilter::reset() {
191196
_buffered_len = 0;
192197
_first = true;
193198
_saved_source_byte_offsets.clear();
199+
_saved_source_byte_end_offsets.clear();
194200
_saved_token_byte_offsets.clear();
195201
_current_source_byte_offsets.clear();
202+
_current_source_byte_end_offsets.clear();
196203
}
197204

198205
void WordDelimiterFilter::save_state(const std::string_view& term) {
@@ -219,6 +226,7 @@ void WordDelimiterFilter::buffer() {
219226
_states[_buffered_len].start_off = _attribute.start_off;
220227
_states[_buffered_len].pos_inc = _attribute.pos_inc;
221228
_states[_buffered_len].source_byte_offsets = _attribute.source_byte_offsets;
229+
_states[_buffered_len].source_byte_end_offsets = _attribute.source_byte_end_offsets;
222230
_states[_buffered_len].token_start_offset = _attribute.token_start_offset;
223231
_states[_buffered_len].token_end_offset = _attribute.token_end_offset;
224232
_buffered_len++;
@@ -229,13 +237,16 @@ void WordDelimiterFilter::generate_part(bool is_single_word) {
229237
_saved_buffer.substr(_iterator->_current, _iterator->_end - _iterator->_current);
230238
_attribute.start_off = _iterator->_current;
231239
_attribute.pos_inc = position(false);
232-
set_attribute_source_byte_offsets(
233-
slice_source_byte_offsets(_iterator->_current, _iterator->_end));
240+
auto [source_byte_offsets, source_byte_end_offsets] =
241+
slice_source_byte_offsets(_iterator->_current, _iterator->_end);
242+
set_attribute_source_byte_offsets(std::move(source_byte_offsets),
243+
std::move(source_byte_end_offsets));
234244
}
235245

236246
void WordDelimiterFilter::save_source_state(std::string_view term) {
237247
auto source_byte_offsets = DorisTokenFilter::get_source_byte_offsets();
238248
_saved_source_byte_offsets.assign(source_byte_offsets.begin(), source_byte_offsets.end());
249+
_saved_source_byte_end_offsets.clear();
239250
_saved_token_byte_offsets.clear();
240251
if (_saved_source_byte_offsets.empty()) {
241252
return;
@@ -253,11 +264,21 @@ void WordDelimiterFilter::save_source_state(std::string_view term) {
253264
if (_saved_token_byte_offsets.size() != _saved_source_byte_offsets.size()) {
254265
_saved_source_byte_offsets.clear();
255266
_saved_token_byte_offsets.clear();
267+
return;
268+
}
269+
auto source_byte_end_offsets = DorisTokenFilter::get_source_byte_end_offsets();
270+
if (source_byte_end_offsets.empty()) {
271+
_saved_source_byte_end_offsets.assign(_saved_source_byte_offsets.begin() + 1,
272+
_saved_source_byte_offsets.end());
273+
} else {
274+
DORIS_CHECK_EQ(source_byte_end_offsets.size() + 1, _saved_source_byte_offsets.size());
275+
_saved_source_byte_end_offsets.assign(source_byte_end_offsets.begin(),
276+
source_byte_end_offsets.end());
256277
}
257278
}
258279

259-
std::vector<int32_t> WordDelimiterFilter::slice_source_byte_offsets(int32_t start,
260-
int32_t end) const {
280+
std::pair<std::vector<int32_t>, std::vector<int32_t>>
281+
WordDelimiterFilter::slice_source_byte_offsets(int32_t start, int32_t end) const {
261282
if (_saved_source_byte_offsets.empty()) {
262283
return {};
263284
}
@@ -269,12 +290,17 @@ std::vector<int32_t> WordDelimiterFilter::slice_source_byte_offsets(int32_t star
269290
}
270291
const auto start_index = std::distance(_saved_token_byte_offsets.begin(), start_it);
271292
const auto end_index = std::distance(_saved_token_byte_offsets.begin(), end_it);
272-
return {_saved_source_byte_offsets.begin() + start_index,
273-
_saved_source_byte_offsets.begin() + end_index + 1};
293+
std::vector<int32_t> offsets {_saved_source_byte_offsets.begin() + start_index,
294+
_saved_source_byte_offsets.begin() + end_index};
295+
std::vector<int32_t> ends {_saved_source_byte_end_offsets.begin() + start_index,
296+
_saved_source_byte_end_offsets.begin() + end_index};
297+
DORIS_CHECK(!ends.empty());
298+
offsets.push_back(ends.back());
299+
return {std::move(offsets), std::move(ends)};
274300
}
275301

276302
void WordDelimiterFilter::set_attribute_source_byte_offsets(
277-
std::vector<int32_t> source_byte_offsets) {
303+
std::vector<int32_t> source_byte_offsets, std::vector<int32_t> source_byte_end_offsets) {
278304
_attribute.token_start_offset = _saved_start_offset;
279305
_attribute.token_end_offset = _saved_end_offset;
280306
if (!source_byte_offsets.empty()) {
@@ -284,8 +310,12 @@ void WordDelimiterFilter::set_attribute_source_byte_offsets(
284310
for (int32_t& offset : source_byte_offsets) {
285311
offset -= relative_start;
286312
}
313+
for (int32_t& offset : source_byte_end_offsets) {
314+
offset -= relative_start;
315+
}
287316
}
288317
_attribute.source_byte_offsets = std::move(source_byte_offsets);
318+
_attribute.source_byte_end_offsets = std::move(source_byte_end_offsets);
289319
}
290320

291321
int32_t WordDelimiterFilter::position(bool inject) {

0 commit comments

Comments
 (0)