Skip to content

Commit 9f3b084

Browse files
authored
[refactor](index) Remove CommonGrams and realign the SNII on-disk layout with the released format (apache#67537)
### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: `CommonGrams` (the token filter, its word set, the term-key escaping codec, the per-segment metadata, the hybrid docs-only/positioned query planner and its result-cache gate, the SPIMI pair-key fast path, 17 profile counters, 4 BE switches, the FE DDL validation and the `score()` admission rule) was an experiment that never reached a released build, yet it had grown roots through the whole SNII stack and had diverged the v1 on-disk layout from what the released 3.1-based writer produces. This PR removes it end-to-end and realigns the physical format so that a node running this code can read segments written by the released writer, and vice versa, without a format version bump. What changes: - **Removed**: everything CommonGrams-specific in BE (`be/src/storage/index/inverted/common_grams/`, the `common_grams` token filter, the SNII term-key escaping codec, `SniiCommonGramsMetadataPB`, the hybrid query planner and its cost model, the SPIMI pair-key path, the frequency-less spill-run shape) and in FE (the `common_grams` token filter type, the CommonGrams policy validation, the `score()` admission rule). Loading an existing policy of that type marks it invalid with an error log instead of crashing FE. - **Term keys** are now the raw analyzed bytes everywhere. User terms that overlap the internal `\x1F` phrase-bigram namespace bypass the index (the same rule the released writer already implied). - **Scoring (BM25) is kept** and decoupled from CommonGrams: a segment is scorable iff it has positions *and* norms. Tokenized indexes with positions now always write norms (an optional region referenced from the core metadata; no new `index_config`). `tf` is the number of positions the term has in the document. Older segments without norms answer `NOT_SUPPORTED` for scoring only; a normal compaction reconstructs norms from the postings (byte-identical to a rebuild), so they become scorable after one compaction. The WAND / block-max paths are removed together with the per-window max statistics they depended on. - **On-disk layout (S1)**: `kFormatVersion` stays 1. The dict entry / dict block / frq prelude / frq pod encoders and decoders are byte-for-byte the released layout again: prelude flag bit0 = `has_prx`, no freq regions, no per-window `max_freq` / `max_norm`, no per-entry term statistics, no `frq_docs_len` / `inline_dd_disk_len` / `freq_meta`, no `kNoTermStats` dict-block flag. `sum_total_term_freq` and the norms reference live in optional proto fields the released reader ignores. Segments carrying the experimental CommonGrams metadata fields (or the experimental scoring `index_config`) are rejected as `NOT_SUPPORTED` with a rebuild hint. - Two follow-up fixes that fell out of the layout change: the phrase-query "position work" estimate now parses prx frame headers without verifying CRCs (`read_prx_frame(..., verify_crc=false)`), so a phrase query no longer checksums every candidate prx window twice; and the frequency-less spill-run record shape is gone (the production SPIMI path always materializes frequencies). Compatibility evidence (real binaries, golden corpus of 8 sample indexes / 179 query expectations / 513 checks across cache-off, cache-cold and cache-warm): - this branch reading segments written by the released 3.1-based writer: 8/8 open, 513/0 - segments written by this branch, checked against the expectations recorded before the removal: 513/0 (query semantics unchanged) - the released 3.1-based binary reading segments written by this branch: 8/8 open, 513/0 (rollback-safe: the old reader ignores the norms field) Size: 178 files, +2321 / −20981 (be/src −8184, be/test −10066, fe −946). ### Release note The experimental `common_grams` token filter and its scoring tier are removed. BM25 scoring on a tokenized SNII index now requires the norms region, which the writer always emits for tokenized indexes with positions; segments written before this change become scorable after one compaction.
1 parent e870abe commit 9f3b084

178 files changed

Lines changed: 2361 additions & 21118 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎be/src/common/config.cpp‎

Lines changed: 0 additions & 55 deletions
Original file line numberDiff line numberDiff line change
@@ -1324,25 +1324,6 @@ DEFINE_Int32(inverted_index_query_cache_shards, "256");
13241324
// inverted index match bitmap cache size
13251325
DEFINE_String(inverted_index_query_cache_limit, "10%");
13261326

1327-
namespace {
1328-
1329-
bool valid_common_grams_cost_ratio(int32_t value) {
1330-
return value >= 0 && value <= 100;
1331-
}
1332-
1333-
bool valid_common_grams_verify_factor(int32_t value) {
1334-
return value >= 0;
1335-
}
1336-
1337-
} // namespace
1338-
1339-
DEFINE_mBool(enable_common_grams_query_plan, "false");
1340-
DEFINE_mBool(enable_common_grams_index_build, "true");
1341-
DEFINE_mInt32(common_grams_plan_cost_ratio_percent, "85");
1342-
DEFINE_Validator(common_grams_plan_cost_ratio_percent, valid_common_grams_cost_ratio);
1343-
DEFINE_mInt32(common_grams_position_verify_factor, "0");
1344-
DEFINE_Validator(common_grams_position_verify_factor, valid_common_grams_verify_factor);
1345-
13461327
// condition cache limit
13471328
DEFINE_Int16(condition_cache_limit, "512");
13481329

@@ -1356,13 +1337,6 @@ DEFINE_mDouble(inverted_index_ram_buffer_size, "512");
13561337
// -1 indicates not working.
13571338
// Normally we should not change this, it's useful for testing.
13581339
DEFINE_mInt32(inverted_index_max_buffered_docs, "-1");
1359-
// G16-c: whether plain positions-tier (non-scoring) SNII indexes lay out freq
1360-
// regions. Freq bytes serve ONLY BM25 scoring, which the Doris integration
1361-
// does not reach yet (scoring_query has no production caller), so the default
1362-
// drops them (textbench: -2.2 GB index). Scoring-config indexes always write
1363-
// freq regardless. Applies at segment build (write side only); existing
1364-
// segments keep whatever layout they were written with (self-describing).
1365-
DEFINE_mBool(snii_positions_index_write_freq, "false");
13661340
// G16-h: zstd levels for the SNII dict-block compression and the .prx window
13671341
// auto mode. Level 9 (vs the historical 3) shrinks the two largest compressed
13681342
// sections -- textbench: index -457 MB (0.918x -> 0.891x V3) -- for an import
@@ -2355,33 +2329,6 @@ bool init(const char* conf_file, bool fill_conf_map, bool must_exist, bool set_t
23552329
return Status::OK(); \
23562330
}
23572331

2358-
namespace {
2359-
2360-
// UPDATE_FIELD invokes registered validators before assigning the candidate value. Validate the two
2361-
// mutable planner coefficients explicitly so their startup and runtime constraints stay identical.
2362-
Status validate_common_grams_runtime_config(const std::string& field, const std::string& value) {
2363-
bool (*validator)(int32_t) = nullptr;
2364-
if (field == "common_grams_plan_cost_ratio_percent") {
2365-
validator = valid_common_grams_cost_ratio;
2366-
} else if (field == "common_grams_position_verify_factor") {
2367-
validator = valid_common_grams_verify_factor;
2368-
} else {
2369-
return Status::OK();
2370-
}
2371-
2372-
int32_t candidate = 0;
2373-
if (!convert(value, candidate)) {
2374-
return Status::OK();
2375-
}
2376-
if (!validator(candidate)) {
2377-
return Status::Error<ErrorCode::INVALID_ARGUMENT, false>("validate {}={} failed", field,
2378-
candidate);
2379-
}
2380-
return Status::OK();
2381-
}
2382-
2383-
} // namespace
2384-
23852332
// write config to be_custom.conf
23862333
// the caller need to make sure that the given config is valid
23872334
Status persist_config(const std::string& field, const std::string& value) {
@@ -2412,8 +2359,6 @@ Status set_config(const std::string& field, const std::string& value, bool need_
24122359
"'{}' is not support to modify", field);
24132360
}
24142361

2415-
RETURN_IF_ERROR(validate_common_grams_runtime_config(field, value));
2416-
24172362
UPDATE_FIELD(it->second, value, bool, need_persist);
24182363
UPDATE_FIELD(it->second, value, int16_t, need_persist);
24192364
UPDATE_FIELD(it->second, value, int32_t, need_persist);

‎be/src/common/config.h‎

Lines changed: 0 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1375,14 +1375,9 @@ DECLARE_Int32(inverted_index_query_cache_shards);
13751375
// inverted index match bitmap cache size
13761376
DECLARE_String(inverted_index_query_cache_limit);
13771377

1378-
// Process-wide emergency switch for CommonGrams query plans.
1379-
DECLARE_mBool(enable_common_grams_query_plan);
13801378
// Build-only CommonGrams kill switch. Logical index writers snapshot it at construction; changing
13811379
// it affects only writers created after the transition and never changes query/cache semantics.
1382-
DECLARE_mBool(enable_common_grams_index_build);
13831380
// Release-calibrated query-planner coefficients. Both remain mutable for controlled recalibration.
1384-
DECLARE_mInt32(common_grams_plan_cost_ratio_percent);
1385-
DECLARE_mInt32(common_grams_position_verify_factor);
13861381

13871382
// condition cache limit
13881383
DECLARE_Int16(condition_cache_limit);
@@ -1395,11 +1390,6 @@ DECLARE_Int32(ann_index_result_cache_stale_sweep_time_sec);
13951390
// inverted index
13961391
DECLARE_mDouble(inverted_index_ram_buffer_size);
13971392
DECLARE_mInt32(inverted_index_max_buffered_docs);
1398-
// G16-c: whether plain positions-tier (non-scoring) SNII indexes lay out freq
1399-
// regions. Freq serves ONLY BM25 scoring (no production caller yet), so the
1400-
// default (false) drops the layout; scoring-config indexes always keep freq.
1401-
// Write-side only; segments are self-describing either way.
1402-
DECLARE_mBool(snii_positions_index_write_freq);
14031393
// G16-h: zstd levels for SNII dict blocks / prx windows. Default 3 (the
14041394
// all-level-3 evaluation showed level 9 buys <=6.3% index size for 17-24%
14051395
// import CPU; see the DEFINEs in config.cpp).

‎be/src/exprs/function/function_search.cpp‎

Lines changed: 4 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -819,8 +819,7 @@ Status FunctionSearch::build_leaf_query(const TSearchClause& clause,
819819
if (minimum_should_match > 0 &&
820820
inverted_index::InvertedIndexAnalyzer::should_analyzer(binding.index_properties)) {
821821
auto term_infos = inverted_index::InvertedIndexAnalyzer::get_analyse_result(
822-
value, binding.index_properties,
823-
inverted_index::AnalysisPurpose::kPlainQuery);
822+
value, binding.index_properties);
824823
if (term_infos.size() > 1) {
825824
return Status::NotSupported(
826825
"SNII native SEARCH does not support minimum_should_match for TERM "
@@ -1026,8 +1025,7 @@ Status FunctionSearch::build_leaf_query(const TSearchClause& clause,
10261025

10271026
std::vector<TermInfo> term_infos =
10281027
inverted_index::InvertedIndexAnalyzer::get_analyse_result(
1029-
value, binding.index_properties,
1030-
inverted_index::AnalysisPurpose::kPlainQuery);
1028+
value, binding.index_properties);
10311029
if (term_infos.empty()) {
10321030
LOG(WARNING) << "search: No terms found after tokenization for TERM query, field="
10331031
<< field_name << ", value='" << value
@@ -1091,8 +1089,7 @@ Status FunctionSearch::build_leaf_query(const TSearchClause& clause,
10911089

10921090
std::vector<TermInfo> term_infos =
10931091
inverted_index::InvertedIndexAnalyzer::get_analyse_result(
1094-
value, binding.index_properties,
1095-
inverted_index::AnalysisPurpose::kPlainQuery);
1092+
value, binding.index_properties);
10961093
if (term_infos.empty()) {
10971094
LOG(WARNING) << "search: No terms found after tokenization for PHRASE query, field="
10981095
<< field_name << ", value='" << value
@@ -1149,8 +1146,7 @@ Status FunctionSearch::build_leaf_query(const TSearchClause& clause,
11491146

11501147
std::vector<TermInfo> term_infos =
11511148
inverted_index::InvertedIndexAnalyzer::get_analyse_result(
1152-
value, binding.index_properties,
1153-
inverted_index::AnalysisPurpose::kPlainQuery);
1149+
value, binding.index_properties);
11541150
if (term_infos.empty()) {
11551151
LOG(WARNING) << "search: tokenization yielded no terms for clause '" << clause_type
11561152
<< "', field=" << field_name << ", returning empty BitSetQuery";

‎be/src/exprs/function/function_tokenize.cpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -187,7 +187,7 @@ Status FunctionTokenize::execute_impl(FunctionContext* /*context*/, Block& block
187187
try {
188188
analyzer_holder =
189189
doris::segment_v2::inverted_index::InvertedIndexAnalyzer::create_analyzer(
190-
&config, AnalysisPurpose::kPlainQuery);
190+
&config);
191191
} catch (CLuceneError& e) {
192192
return Status::Error<doris::ErrorCode::INVERTED_INDEX_ANALYZER_ERROR>(
193193
"inverted index create analyzer failed: {}", e.what());

‎be/src/exprs/vmatch_predicate.cpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,7 @@ VMatchPredicate::VMatchPredicate(const TExprNode& node) : VExpr(node) {
8383
if (_analyzer_ctx->requires_analysis()) {
8484
_analyzer_provider =
8585
inverted_index::InvertedIndexAnalyzer::create_analyzer_provider(&config);
86-
_analyzer = _analyzer_provider->get_analyzer(inverted_index::AnalysisPurpose::kPlainQuery);
86+
_analyzer = _analyzer_provider->get_analyzer();
8787
}
8888

8989
_analyzer_ctx->char_filter_map = std::move(config.char_filter_map);

‎be/src/runtime/index_policy/index_policy_mgr.cpp‎

Lines changed: 8 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -27,16 +27,11 @@
2727
namespace doris {
2828
namespace {
2929

30-
class PurposeInsensitiveAnalyzerProvider final
31-
: public segment_v2::inverted_index::AnalyzerProvider {
30+
class SingleAnalyzerProvider final : public segment_v2::inverted_index::AnalyzerProvider {
3231
public:
33-
explicit PurposeInsensitiveAnalyzerProvider(AnalyzerPtr analyzer)
34-
: _analyzer(std::move(analyzer)) {}
32+
explicit SingleAnalyzerProvider(AnalyzerPtr analyzer) : _analyzer(std::move(analyzer)) {}
3533

36-
AnalyzerPtr get_analyzer(
37-
segment_v2::inverted_index::AnalysisPurpose /*purpose*/) const override {
38-
return _analyzer;
39-
}
34+
AnalyzerPtr get_analyzer() const override { return _analyzer; }
4035

4136
private:
4237
const AnalyzerPtr _analyzer;
@@ -144,8 +139,7 @@ AnalyzerPtr IndexPolicyMgr::get_policy_by_name(const std::string& name) {
144139
throw Exception(ErrorCode::INVALID_ARGUMENT, "Policy not found with type: " + name);
145140
}
146141

147-
AnalyzerPtr IndexPolicyMgr::get_analyzer_by_name(
148-
const std::string& name, segment_v2::inverted_index::AnalysisPurpose purpose) {
142+
AnalyzerPtr IndexPolicyMgr::get_analyzer_by_name(const std::string& name) {
149143
std::shared_lock lock(_mutex);
150144
const std::string normalized_name = normalize_name(name);
151145
auto name_it = _name_to_id.find(normalized_name);
@@ -162,7 +156,7 @@ AnalyzerPtr IndexPolicyMgr::get_analyzer_by_name(
162156
if (policy_it->second.type == TIndexPolicyType::ANALYZER) {
163157
return build_analyzer_provider_from_config(
164158
build_analyzer_config_from_policy(policy_it->second), {})
165-
->get_analyzer(purpose);
159+
->get_analyzer();
166160
}
167161
if (policy_it->second.type == TIndexPolicyType::NORMALIZER) {
168162
return build_normalizer_from_policy(policy_it->second);
@@ -177,8 +171,7 @@ AnalyzerProviderPtr IndexPolicyMgr::get_analyzer_provider_by_name(
177171
auto name_it = _name_to_id.find(normalized_name);
178172
if (name_it == _name_to_id.end()) {
179173
if (is_builtin_normalizer(normalized_name)) {
180-
return std::make_shared<PurposeInsensitiveAnalyzerProvider>(
181-
build_builtin_normalizer(name));
174+
return std::make_shared<SingleAnalyzerProvider>(build_builtin_normalizer(name));
182175
}
183176
throw Exception(ErrorCode::INVALID_ARGUMENT, "Policy not found with name: " + name);
184177
}
@@ -191,30 +184,12 @@ AnalyzerProviderPtr IndexPolicyMgr::get_analyzer_provider_by_name(
191184
build_analyzer_config_from_policy(policy_it->second), outer_char_filter_map);
192185
}
193186
if (policy_it->second.type == TIndexPolicyType::NORMALIZER) {
194-
return std::make_shared<PurposeInsensitiveAnalyzerProvider>(
187+
return std::make_shared<SingleAnalyzerProvider>(
195188
build_normalizer_from_policy(policy_it->second));
196189
}
197190
throw Exception(ErrorCode::INVALID_ARGUMENT, "Analyzer policy not found: " + name);
198191
}
199192

200-
AnalyzerProviderPtr IndexPolicyMgr::get_analyzer_provider_by_base_fingerprint(
201-
std::string_view base_analyzer_fingerprint,
202-
const std::map<std::string, std::string>& outer_char_filter_map) {
203-
std::shared_lock lock(_mutex);
204-
for (const auto& [_, policy] : _policys) {
205-
if (policy.type != TIndexPolicyType::ANALYZER) {
206-
continue;
207-
}
208-
auto config = build_analyzer_config_from_policy(policy);
209-
if (segment_v2::inverted_index::CustomAnalyzerProvider::calculate_base_analyzer_fingerprint(
210-
config, outer_char_filter_map) != base_analyzer_fingerprint) {
211-
continue;
212-
}
213-
return build_analyzer_provider_from_config(std::move(config), outer_char_filter_map);
214-
}
215-
return nullptr;
216-
}
217-
218193
segment_v2::inverted_index::CustomAnalyzerConfigPtr
219194
IndexPolicyMgr::build_analyzer_config_from_policy(const TIndexPolicy& index_policy_analyzer) {
220195
segment_v2::inverted_index::CustomAnalyzerConfig::Builder builder;
@@ -266,17 +241,14 @@ IndexPolicyMgr::build_analyzer_config_from_policy(const TIndexPolicy& index_poli
266241
AnalyzerProviderPtr IndexPolicyMgr::build_analyzer_provider_from_config(
267242
segment_v2::inverted_index::CustomAnalyzerConfigPtr config,
268243
const std::map<std::string, std::string>& outer_char_filter_map) {
269-
// One shape for every policy: the provider sources its CommonGrams word list from the
270-
// BE-local default, so there is no per-policy word set to look up and no "not yet prepared"
271-
// state to represent.
272244
return std::make_shared<segment_v2::inverted_index::CustomAnalyzerProvider>(
273245
std::move(config), outer_char_filter_map);
274246
}
275247

276248
AnalyzerPtr IndexPolicyMgr::build_analyzer_from_policy(const TIndexPolicy& index_policy_analyzer) {
277249
return build_analyzer_provider_from_config(
278250
build_analyzer_config_from_policy(index_policy_analyzer), {})
279-
->get_analyzer(segment_v2::inverted_index::AnalysisPurpose::kIndex);
251+
->get_analyzer();
280252
}
281253

282254
AnalyzerPtr IndexPolicyMgr::build_normalizer_from_policy(

‎be/src/runtime/index_policy/index_policy_mgr.h‎

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -47,14 +47,10 @@ class IndexPolicyMgr {
4747

4848
Policys get_index_policys();
4949
AnalyzerPtr get_policy_by_name(const std::string& name);
50-
AnalyzerPtr get_analyzer_by_name(const std::string& name,
51-
segment_v2::inverted_index::AnalysisPurpose purpose);
50+
AnalyzerPtr get_analyzer_by_name(const std::string& name);
5251
AnalyzerProviderPtr get_analyzer_provider_by_name(
5352
const std::string& name,
5453
const std::map<std::string, std::string>& outer_char_filter_map = {});
55-
AnalyzerProviderPtr get_analyzer_provider_by_base_fingerprint(
56-
std::string_view base_analyzer_fingerprint,
57-
const std::map<std::string, std::string>& outer_char_filter_map = {});
5854

5955
private:
6056
segment_v2::inverted_index::CustomAnalyzerConfigPtr build_analyzer_config_from_policy(

‎be/src/storage/compaction/compaction.cpp‎

Lines changed: 6 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1214,26 +1214,12 @@ Status Compaction::do_inverted_index_compaction() {
12141214
auto* destination_writer =
12151215
inverted_index_file_writers[cast_set<int>(destination_ordinal)]
12161216
.get();
1217-
if (merge_eligibility.kind ==
1218-
snii::compaction::SniiStreamedMergeKind::kCommonGramsT3) {
1219-
merge_status = destination_writer->add_snii_index_streamed(
1220-
index_meta, dest_segment_num_rows[destination_ordinal],
1221-
merge_plan->take_destination_null_docids(destination_ordinal),
1222-
merge_plan->take_destination_encoded_norms(destination_ordinal),
1223-
merge_plan->destination_common_grams_metadata(
1224-
destination_ordinal),
1225-
merge_plan->destination_common_grams_posting_policy(),
1226-
merge_plan->destination_index_config(),
1227-
snii_merge_memory_reporter,
1228-
&destination_sessions[destination_ordinal]);
1229-
} else {
1230-
merge_status = destination_writer->add_snii_index_streamed(
1231-
index_meta, dest_segment_num_rows[destination_ordinal],
1232-
merge_plan->take_destination_null_docids(destination_ordinal),
1233-
merge_plan->destination_index_config(),
1234-
snii_merge_memory_reporter,
1235-
&destination_sessions[destination_ordinal]);
1236-
}
1217+
merge_status = destination_writer->add_snii_index_streamed(
1218+
index_meta, dest_segment_num_rows[destination_ordinal],
1219+
merge_plan->take_destination_null_docids(destination_ordinal),
1220+
merge_plan->destination_writes_norms(),
1221+
merge_plan->destination_index_config(), snii_merge_memory_reporter,
1222+
&destination_sessions[destination_ordinal]);
12371223
if (!merge_status.ok()) {
12381224
break;
12391225
}

0 commit comments

Comments
 (0)