Skip to content

Commit 14fde1c

Browse files
authored
[refactor](be) Unify MATCH and SEARCH execution across inverted index formats (apache#68758)
### What problem does this PR solve? Issue Number: None Related PR: apache#67180, apache#67538, apache#67918 Problem Summary: MATCH and SEARCH each had their own Boolean, phrase, term-expansion and scoring executors, written separately for CLucene (V1/V2) and SNII indexes. Every query optimization had to be implemented once per executor, the formats drifted apart in behavior, and cross-field queries could mix document IDs from different physical partitions. This PR runs both on one engine: - **One logical form.** MATCH (`query/logical`) and SEARCH (`exprs/function/search`) lower their queries to the same logical nodes. - **One query engine.** Boolean logic (AND/OR/NOT, minimum_should_match, WAND), term expansion (prefix, wildcard, regexp), exact and sloppy phrases and BM25 scoring are implemented once, independent of the index format. - **Formats behind an SPI.** CLucene and SNII only provide dictionaries, postings, positions, norms, statistics and batched I/O through `query/spi`. The old per-format executors are removed. - **Layered source tree.** `be/src/storage/index/` is reorganized by layer, and `build-support/check-header-deps.py` enforces the allowed include directions. Index formats and writers are unchanged, and existing indexes remain readable. FE is unchanged. #### Source layout ```text be/src/storage/index/ ├── index_*.h/.cpp shared index files: file reader/writer, compound reader, storage formats ├── ann/ bloom_filter/ zone_map/ └── inverted/ ├── analysis/ analyzers, tokenizers, char and token filters, normalizers, gram schemes ├── common/ descriptors, caches, statistics, profiles, single flight ├── reader/ text index reader, index selection, collection statistics ├── query/ format-independent query engine │ ├── logical/ MATCH lowering and logical nodes (gram/ builds gram queries) │ ├── engine/ Query/Weight/Scorer: boolean, term, phrase, expansion, WAND, union, collect │ ├── exec/ doc-id sets and sinks, term waves, chained conjunctions │ ├── phrase/ exact and sloppy phrase matchers │ ├── scoring/ BM25 │ └── spi/ the source interface each format implements ├── clucene/ CLucene (V1/V2) reader, searcher, writer, compaction │ └── spi/ CLucene index source and postings cursor └── snii/ SNII: format/, encoding/, io/, bkd/, reader/, writer/, compaction/, query/ be/src/exprs/ ├── vsearch.h/.cpp VSearchExpr, the SEARCH expression node └── function/search/ ├── function_search.* SEARCH entry: Boolean assembly, DSL result cache, NULL handling ├── search_lowering.* SEARCH DSL to logical nodes; binds each field to an index ├── function_search_leaf.* compiles each leaf into a query on its bound index ├── search_predicate_collector.* collects scored leaves for BM25 statistics └── variant_inverted_index_search.* SEARCH on VARIANT subcolumns and NESTED groups ``` #### Performance The 18 production SQL cases (V2 and SNII: point, range, absent-term and phrase queries) were compared with master on fresh dual-BE pairs: 8 pairs, 16 samples per case and binary, alternating order. Every case is within the 5% limit for both `InvertedIndexFilterTime` and client wall time; the largest ratios are 1.032 and 1.038. SNII phrase cases take 25-60% less FilterTime. TPC-H, TPC-DS and ClickBench are within noise. <details><summary>Per-case median ratios (PR / master)</summary> Run on 2026-10-14 against master 7ac0eb2 on a busy shared host (median 90% CPU), which widens the intervals; all 4,608 result checks pass. | Case | FilterTime | Client wall | | --- | ---: | ---: | | `v2/long/early/full` | 1.030 | 1.024 | | `v2/long/early/range` | 1.018 | 0.999 | | `v2/long/absent/full` | 1.032 | 1.014 | | `v2/long/absent/range` | 1.018 | 0.995 | | `v2/long/mid/full` | 1.006 | 1.038 | | `v2/long/mid/range` | 1.013 | 1.012 | | `v2/short/tiny` | 0.965 | 1.018 | | `v2/search/scored_phrase` | 1.010 | 0.990 | | `v2/search/phrase_2` | 0.954 | 0.951 | | `snii/long/early/full` | 0.950 | 0.994 | | `snii/long/early/range` | 0.961 | 1.022 | | `snii/long/absent/full` | 0.728 | 0.847 | | `snii/long/absent/range` | 0.891 | 0.983 | | `snii/long/mid/full` | 0.821 | 0.924 | | `snii/long/mid/range` | 0.975 | 1.007 | | `snii/short/tiny` | 1.020 | 0.985 | | `snii/search/scored_phrase` | 0.404 | 0.971 | | `snii/search/phrase_2` | 0.746 | 0.986 | </details>
1 parent 8f838a8 commit 14fde1c

1,013 files changed

Lines changed: 40483 additions & 35219 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.clang-format-ignore‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,5 +8,5 @@ be/src/util/sse2neo.h
88
be/src/util/sse2neon.h
99
be/src/util/mustache/mustache.h
1010
be/src/util/mustache/mustache.cc
11-
be/src/storage/index/inverted/analyzer/kuromoji/dict/darts.h
11+
be/src/storage/index/inverted/analysis/analyzer/kuromoji/dict/darts.h
1212
cloud/src/common/defer.h

‎.github/CODEOWNERS‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,7 @@ fe/fe-core/src/main/java/org/apache/doris/fs @CalvinKirs @924060929
2525
fe/fe-core/src/main/java/org/apache/doris/fsv2 @CalvinKirs
2626
be/src/olap/rowset/segment_v2/variant @eldenmoon @csun5285
2727
be/src/storage/index/inverted/ @airborne12 @eldenmoon @csun5285
28-
be/src/exprs/function/function_search.h @airborne12 @eldenmoon @csun5285
29-
be/src/exprs/function/function_search.cpp @airborne12 @eldenmoon @csun5285
28+
be/src/exprs/function/search/ @airborne12 @eldenmoon @csun5285
3029
be/src/exprs/function/match.cpp @airborne12 @eldenmoon @csun5285
3130
be/src/exprs/function/match.h @airborne12 @eldenmoon @csun5285
3231
be/src/exprs/vsearch.h @airborne12 @eldenmoon @csun5285

‎.licenserc.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -126,7 +126,7 @@ header:
126126
- "be/src/util/mustache/mustache.h"
127127
- "be/src/util/sse2neo.h"
128128
- "be/src/util/sse2neon.h"
129-
- "be/src/storage/index/inverted/analyzer/kuromoji/dict/darts.h"
129+
- "be/src/storage/index/inverted/analysis/analyzer/kuromoji/dict/darts.h"
130130
- "be/src/pch/*"
131131
- "be/test/data"
132132
- "be/test/expected_result"

‎LICENSE.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -369,7 +369,7 @@ be/src/olap/skiplist.h : BSD-style license
369369

370370
--------------------------------------------------------------------------------
371371

372-
be/src/storage/index/inverted/analyzer/kuromoji/dict/darts.h : 2-Clause BSD license (darts-clone)
372+
be/src/storage/index/inverted/analysis/analyzer/kuromoji/dict/darts.h : 2-Clause BSD license (darts-clone)
373373

374374
Copyright (c) 2008-2014 Susumu Yata. All rights reserved.
375375
Vendored from https://github.com/s-yata/darts-clone

‎NOTICE.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,7 @@ This software includes third party software subject to the following copyrights:
7575
- Jackson-dataformat-cbor - https://github.com/FasterXML/jackson-dataformats-binary
7676
- Darts-clone (double-array trie) - Copyright 2008-2014 Susumu Yata - https://github.com/s-yata/darts-clone (BSD 2-clause)
7777
- mecab-ipadic (IPADIC) Japanese morphological dictionary - Copyright 2000-2003 Nara Institute of Science and Technology (NAIST) - licensed under NAIST-2003 (BSD-style); the kuromoji analyzer bundles the UTF-8 form from https://github.com/lindera/mecab-ipadic (content of mecab-ipadic-2.7.0-20070801).
78-
- Apache Lucene - https://github.com/apache/lucene (Apache-2.0): the kuromoji Japanese analyzer under be/src/storage/index/inverted/analyzer/kuromoji is an independent C++ implementation modeled on Lucene's kuromoji analyzer (JapaneseTokenizer), including its search-mode compound-decomposition cost model.
78+
- Apache Lucene - https://github.com/apache/lucene (Apache-2.0): the kuromoji Japanese analyzer under be/src/storage/index/inverted/analysis/analyzer/kuromoji is an independent C++ implementation modeled on Lucene's kuromoji analyzer (JapaneseTokenizer), including its search-mode compound-decomposition cost model.
7979

8080
The licenses for these third party components are included in LICENSE.txt
8181

‎be/benchmark/benchmark_gram_extractor.hpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@
2424
#include <string_view>
2525
#include <vector>
2626

27-
#include "storage/index/inverted/gram/gram_extractor.h"
27+
#include "storage/index/inverted/analysis/gram/gram_extractor.h"
2828

2929
namespace doris::segment_v2::gram::benchmark_detail {
3030

‎be/src/cloud/cloud_schema_change_job.cpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@
3333
#include "common/status.h"
3434
#include "service/backend_options.h"
3535
#include "storage/delete/delete_handler.h"
36-
#include "storage/index/inverted/inverted_index_desc.h"
36+
#include "storage/index/inverted/common/inverted_index_desc.h"
3737
#include "storage/olap_define.h"
3838
#include "storage/rowset/beta_rowset.h"
3939
#include "storage/rowset/rowset.h"

‎be/src/cloud/cloud_tablet.cpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@
5151
#include "io/cache/block_file_cache_factory.h"
5252
#include "storage/compaction/compaction.h"
5353
#include "storage/compaction/cumulative_compaction_time_series_policy.h"
54-
#include "storage/index/inverted/inverted_index_desc.h"
54+
#include "storage/index/inverted/common/inverted_index_desc.h"
5555
#include "storage/olap_define.h"
5656
#include "storage/rowset/beta_rowset.h"
5757
#include "storage/rowset/rowset.h"

‎be/src/cloud/cloud_warm_up_manager.cpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@
4646
#include "runtime/cluster_info.h"
4747
#include "runtime/exec_env.h"
4848
#include "service/backend_options.h"
49-
#include "storage/index/inverted/inverted_index_desc.h"
49+
#include "storage/index/inverted/common/inverted_index_desc.h"
5050
#include "storage/rowset/beta_rowset.h"
5151
#include "storage/tablet/tablet.h"
5252
#include "util/brpc_client_cache.h" // BrpcClientCache

‎be/src/exec/operator/olap_scan_operator.cpp‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -372,7 +372,6 @@ Status OlapScanLocalState::_init_profile() {
372372
_index_filter_profile = std::make_unique<RuntimeProfile>("IndexFilter");
373373
_scanner_profile->add_child(_index_filter_profile.get(), true, nullptr);
374374
_snii_prx_profile_counters.initialize(_index_filter_profile.get());
375-
_snii_phrase_profile_counters.initialize(_index_filter_profile.get());
376375
/*
377376
SegmentIterator:
378377
- AnnIndexLoadCosts: 102.262us

0 commit comments

Comments
 (0)