Skip to content

Commit 67f31fa

Browse files
authored
[fix](be) Make SEARCH predicates consistently two-valued (apache#69347)
### What problem does this PR solve? Related PR: apache#68758 SEARCH declares a non-nullable result, but UNKNOWN leaves and root-only NULL masking can omit rows under NOT or change membership across scoring and cached execution. Evaluate the complete backend query as a two-valued match set: ordinary terms do not match NULL or proven-absent fields; AND/OR combine match sets; NOT complements them. Ordinary SQL expressions and MATCH retain their existing three-valued semantics. Filter physical NULL postings at SEARCH leaf boundaries with the existing `NullMaskedQuery`, including NULL ARRAY values with retained nested payloads. Keep cached result bitmaps independently owned and preserve source-managed document IDs across collectors. This change uses generic execution for masked terms. It does not add `excluded_rows` to term queries, weights, scorers, or grouped reads. Grouped term opening and term-specific block-max WAND are deferred for these masked leaves; no performance improvement or benchmark result is claimed. A missing iterator does not prove that a Variant path is absent. When a historical rowset contains the path but has no executable index after CREATE INDEX, fail explicitly instead of returning incomplete results. Tests write actual objects with `parse_to_variant`, check their type and extracted values, then distinguish unavailable historical indexes, newly indexed rows, and genuinely absent paths. Index backfill and scan fallback are outside this change. The historical-rowset fixture uses a non-tokenized index with `enable_add_index_for_new_data=true`, so Cloud and local mode both leave the historical rowsets unindexed. **Independent validation:** this PR contains no FE or protocol changes and was built and tested with FE source identical to master. Backend field-existence semantics are tested directly through the existing `TSearchClause` interface, including empty arrays, physical NULL postings, numeric fields, absent paths, cache, scoring, and multiple readers. SQL suites use clauses already emitted by master FE. There is no companion-PR prerequisite or required merge order.
1 parent f44fd38 commit 67f31fa

42 files changed

Lines changed: 2849 additions & 470 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎be/src/exprs/function/search/function_search.cpp‎

Lines changed: 56 additions & 76 deletions
Original file line numberDiff line numberDiff line change
@@ -114,41 +114,33 @@ static std::string extract_segment_prefix(
114114

115115
namespace {
116116

117-
// A SEARCH result is cached as two entries: its TRUE rows, and its UNKNOWN rows under the
118-
// signature with a "__null" suffix.
117+
// Keep two-valued SEARCH results separate from older SQL UNKNOWN cache entries.
119118
InvertedIndexQueryCache::CacheKey dsl_cache_key(const std::string& seg_prefix,
120119
const std::string& signature) {
121-
return {seg_prefix, "__search_dsl__", InvertedIndexQueryType::SEARCH_DSL_QUERY, signature};
120+
return {seg_prefix, "__search_dsl_2vl__", InvertedIndexQueryType::SEARCH_DSL_QUERY, signature};
122121
}
123122

124-
// Either entry can be evicted, or not yet inserted, without the other, and TRUE rows found
125-
// without their UNKNOWN rows are no result.
126123
bool lookup_dsl_result(InvertedIndexQueryCache* cache, const std::string& seg_prefix,
127124
const std::string& signature, InvertedIndexResultBitmap* result) {
128125
InvertedIndexQueryCacheHandle rows_handle;
129-
InvertedIndexQueryCacheHandle nulls_handle;
130-
if (!cache->lookup(dsl_cache_key(seg_prefix, signature), &rows_handle) ||
131-
!cache->lookup(dsl_cache_key(seg_prefix, signature + "__null"), &nulls_handle)) {
126+
if (!cache->lookup(dsl_cache_key(seg_prefix, signature), &rows_handle)) {
132127
return false;
133128
}
134129
auto rows = rows_handle.get_bitmap();
135-
auto nulls = nulls_handle.get_bitmap();
136-
if (rows == nullptr || nulls == nullptr) {
130+
if (rows == nullptr) {
137131
return false;
138132
}
139-
*result = InvertedIndexResultBitmap(std::move(rows), std::move(nulls));
133+
// SQL bitmap composition can mutate its input. The cache owns an independent copy.
134+
*result = InvertedIndexResultBitmap(std::make_shared<roaring::Roaring>(*rows),
135+
std::make_shared<roaring::Roaring>());
140136
return true;
141137
}
142138

143-
// The UNKNOWN rows go in first, so a search that finds the TRUE rows finds them as well.
144139
void insert_dsl_result(InvertedIndexQueryCache* cache, const std::string& seg_prefix,
145140
const std::string& signature, const InvertedIndexResultBitmap& result) {
146-
auto nulls = result.get_null_bitmap() != nullptr ? result.get_null_bitmap()
147-
: std::make_shared<roaring::Roaring>();
148-
InvertedIndexQueryCacheHandle nulls_handle;
149-
cache->insert(dsl_cache_key(seg_prefix, signature + "__null"), std::move(nulls), &nulls_handle);
150141
InvertedIndexQueryCacheHandle rows_handle;
151-
cache->insert(dsl_cache_key(seg_prefix, signature), result.get_data_bitmap(), &rows_handle);
142+
cache->insert(dsl_cache_key(seg_prefix, signature),
143+
std::make_shared<roaring::Roaring>(*result.get_data_bitmap()), &rows_handle);
152144
}
153145

154146
bool is_nested_group_search_supported() {
@@ -172,7 +164,7 @@ class SearchFieldCatalog final : public logical::FieldCatalog {
172164
if (!binding.is_bound()) {
173165
LOG(INFO) << "search: No inverted index for field '" << field
174166
<< "' in this segment, query_type=" << static_cast<int>(query_type)
175-
<< ", returning UNKNOWN bitmap";
167+
<< ", returning non-matching bitmap";
176168
*out = logical::FieldProps {};
177169
return Status::OK();
178170
}
@@ -244,28 +236,23 @@ Status compile_node(const logical::Node& node, const SearchLeafContext& ctx,
244236
FieldReaderResolver& resolver, query_v2::QueryPtr* out,
245237
std::string* binding_key);
246238

247-
// Marks the rows where the field is NULL as UNKNOWN for leaves that would misreport them: an array
248-
// index can list a NULL array's leftover values, and a term threshold counts its terms two-valued.
249-
Status mask_null_rows(const FieldReaderBinding& binding, const FieldReaderResolver& resolver,
250-
const std::string& field, bool threshold, query_v2::QueryPtr* query) {
251-
const bool array = binding.column_type != nullptr &&
252-
remove_nullable(binding.column_type)->get_primitive_type() == TYPE_ARRAY;
253-
if (!array && !threshold) {
254-
return Status::OK();
255-
}
239+
// Physical NULL postings can overlap a NULL ARRAY's leftover values. Exclude them
240+
// before Boolean composition without introducing SQL UNKNOWN or changing scores.
241+
Status mask_null_rows(const FieldReaderResolver& resolver, const std::string& field,
242+
query_v2::QueryPtr* query) {
256243
IndexIterator* iterator = resolver.get_iterator(field);
257-
if (iterator == nullptr) {
258-
return Status::OK();
259-
}
260-
auto has_null = iterator->has_null();
261-
if (!has_null.has_value() || !has_null.value()) {
244+
DORIS_CHECK(iterator != nullptr);
245+
const bool has_null = DORIS_TRY(iterator->has_null());
246+
if (!has_null) {
262247
return Status::OK();
263248
}
264249
InvertedIndexQueryCacheHandle handle;
265250
RETURN_IF_ERROR(iterator->read_null_bitmap(&handle));
266251
auto nulls = handle.get_bitmap();
267-
if (nulls != nullptr && !nulls->isEmpty()) {
268-
*query = std::make_shared<query_v2::NullMaskedQuery>(std::move(*query), std::move(nulls));
252+
DORIS_CHECK(nulls != nullptr);
253+
if (!nulls->isEmpty()) {
254+
*query = std::make_shared<query_v2::NullMaskedQuery>(std::move(*query), std::move(nulls),
255+
/*nulls_are_false=*/true);
269256
}
270257
return Status::OK();
271258
}
@@ -277,8 +264,7 @@ bool has_required_clause(const Clauses& clauses) {
277264
clauses, [](const auto& clause) { return clause.first == logical::Occur::kMust; });
278265
}
279266

280-
// Without scoring, optional clauses beside a required one change neither the rows nor their
281-
// UNKNOWN state, so they are left out.
267+
// Without scoring, optional clauses beside a required one do not change the match set.
282268
Clauses matching_clauses(const logical::Bool& boolean, bool scoring) {
283269
if (scoring || boolean.op != logical::BoolOp::kOccur || boolean.min_should_match != 0 ||
284270
!has_required_clause(boolean.clauses)) {
@@ -367,23 +353,30 @@ Status compile_node(const logical::Node& node, const SearchLeafContext& ctx,
367353
if (binding_key != nullptr) {
368354
*binding_key = field->binding;
369355
}
370-
if (node.as<logical::Unknown>() != nullptr) {
371-
*out = make_unknown_leaf_query(ctx.num_rows);
356+
if (node.as<logical::Empty>() != nullptr) {
357+
*out = std::make_shared<query_v2::BitSetQuery>(std::make_shared<roaring::Roaring>());
372358
return resolver.map_leaf_query(field->name, out);
373359
}
374360
const FieldReaderBinding* binding = resolver.find_binding(field->binding);
375361
if (binding == nullptr || binding->leaf_compiler == nullptr) {
376362
return Status::InternalError("search: field '{}' has no compiler for binding '{}'",
377363
field->name, field->binding);
378364
}
365+
SearchLeafContext leaf_ctx = ctx;
366+
RETURN_IF_ERROR(resolver.leaf_num_rows(field->name, &leaf_ctx.num_rows));
379367
const auto* set = node.as<logical::TermSet>();
380368
const bool threshold = set != nullptr && set->min_should_match > 0;
381-
if (threshold) {
382-
RETURN_IF_ERROR(compile_term_threshold(*set, ctx, *binding->leaf_compiler, out));
369+
if (node.as<logical::Exists>() != nullptr) {
370+
// Exists also works for BKD and uses the NestedGroup leaf's element domain.
371+
auto matches = std::make_shared<roaring::Roaring>();
372+
matches->addRange(0, leaf_ctx.num_rows);
373+
*out = std::make_shared<query_v2::BitSetQuery>(std::move(matches));
374+
} else if (threshold) {
375+
RETURN_IF_ERROR(compile_term_threshold(*set, leaf_ctx, *binding->leaf_compiler, out));
383376
} else {
384-
RETURN_IF_ERROR(binding->leaf_compiler->compile(node, ctx, out));
377+
RETURN_IF_ERROR(binding->leaf_compiler->compile(node, leaf_ctx, out));
385378
}
386-
RETURN_IF_ERROR(mask_null_rows(*binding, resolver, field->name, threshold, out));
379+
RETURN_IF_ERROR(mask_null_rows(resolver, field->name, out));
387380
return resolver.map_leaf_query(field->name, out);
388381
}
389382

@@ -453,22 +446,14 @@ Status FunctionSearch::_evaluate_search_param(
453446
"NESTED query requires NestedGroup support, which is unavailable in this build");
454447
}
455448

456-
if (!is_nested_query && (iterators.empty() || data_type_with_names.empty())) {
457-
LOG(INFO) << "No indexed columns or iterators available, returning empty result, dsl:"
458-
<< search_param.original_dsl;
459-
bitmap_result = InvertedIndexResultBitmap(std::make_shared<roaring::Roaring>(),
460-
std::make_shared<roaring::Roaring>());
461-
return Status::OK();
462-
}
463-
464449
// Track overall query time (equivalent to inverted_index_query_timer in MATCH path).
465450
// Must be declared before the DSL cache lookup so that cache-hit fast paths are
466451
// also covered by the timer.
467452
int64_t query_timer_dummy = 0;
468453
OlapReaderStatistics* outer_stats = index_query_context ? index_query_context->stats : nullptr;
469454
SCOPED_RAW_TIMER(outer_stats ? &outer_stats->inverted_index_query_timer : &query_timer_dummy);
470455

471-
// DSL result cache only stores bitmap/null bitmap. It does not store BM25 scores,
456+
// DSL result cache only stores the two-valued match bitmap. It does not store BM25 scores,
472457
// so score() queries must execute scorers to populate CollectionSimilarity.
473458
const bool enable_scoring =
474459
index_query_context != nullptr && index_query_context->collection_similarity != nullptr;
@@ -560,7 +545,7 @@ Status FunctionSearch::_evaluate_search_param(
560545
return Status::OK();
561546
}
562547

563-
VariantSearchNullBitmapAdapter null_resolver(resolver);
548+
SearchNullBitmapResolver null_resolver;
564549
query_v2::QueryExecutionContext exec_ctx =
565550
build_variant_search_query_execution_context(num_rows, resolver, &null_resolver);
566551

@@ -580,13 +565,17 @@ Status FunctionSearch::_evaluate_search_param(
580565
}
581566

582567
std::shared_ptr<roaring::Roaring> roaring = std::make_shared<roaring::Roaring>();
583-
// A three-valued root reports the rows it leaves UNKNOWN (TRUE OR NULL is TRUE, FALSE OR
584-
// NULL is NULL); a lucene-style Boolean root is two-valued: every row it does not match is
585-
// FALSE.
586-
std::shared_ptr<roaring::Roaring> null_bitmap = std::make_shared<roaring::Roaring>();
587-
const bool three_valued =
588-
exec_ctx.null_resolver != nullptr && search_param.root.clause_type != "OCCUR_BOOLEAN";
589-
const bool top_k_path = enable_scoring && !is_asc && top_k > 0;
568+
// Keyword readers can lack BM25 length statistics. Preserve their original
569+
// scores and all matches for the SQL sort when pruning cannot order them.
570+
const bool can_prune_scores = std::none_of(
571+
resolver.binding_cache().begin(), resolver.binding_cache().end(),
572+
[](const auto& entry) {
573+
const auto& binding = entry.second;
574+
DORIS_CHECK(binding.inverted_reader != nullptr);
575+
return binding.inverted_reader->type() == InvertedIndexReaderType::STRING_TYPE;
576+
});
577+
const bool top_k_path = enable_scoring && !is_asc && top_k > 0 && top_k < num_rows &&
578+
top_k <= query_v2::TopKCollector::kMaxK && can_prune_scores;
590579
exec_ctx.top_k = top_k_path;
591580
{
592581
int64_t exec_dummy = 0;
@@ -595,33 +584,24 @@ Status FunctionSearch::_evaluate_search_param(
595584
bool use_wand = index_query_context->runtime_state != nullptr &&
596585
index_query_context->runtime_state->query_options()
597586
.enable_inverted_index_wand_query;
598-
query_v2::collect_multi_segment_top_k(
599-
weight, exec_ctx, root_binding_key, top_k, roaring,
600-
index_query_context->collection_similarity, use_wand,
601-
index_query_context->delete_bitmap, three_valued ? null_bitmap.get() : nullptr);
587+
query_v2::collect_multi_segment_top_k(weight, exec_ctx, root_binding_key, top_k,
588+
roaring,
589+
index_query_context->collection_similarity,
590+
use_wand, index_query_context->delete_bitmap);
602591
} else {
603-
// The scorers listing the rows report the UNKNOWN ones as they go.
604592
query_v2::collect_multi_segment_doc_set(
605593
weight, exec_ctx, root_binding_key, roaring,
606594
index_query_context ? index_query_context->collection_similarity : nullptr,
607-
enable_scoring, three_valued ? null_bitmap.get() : nullptr);
595+
enable_scoring);
608596
}
609597
}
610598

611599
VLOG_DEBUG << "search: Query completed, matched " << roaring->cardinality() << " documents";
612600

613-
VLOG_TRACE << "search: Before mask - true_bitmap=" << roaring->cardinality()
614-
<< ", null_bitmap=" << null_bitmap->cardinality();
615-
616-
// Create result and mask out NULLs (SQL WHERE clause semantics: only TRUE rows)
617-
bitmap_result = InvertedIndexResultBitmap(std::move(roaring), std::move(null_bitmap));
618-
bitmap_result.mask_out_null();
619-
620-
VLOG_TRACE << "search: After mask - result_bitmap="
621-
<< bitmap_result.get_data_bitmap()->cardinality();
601+
// SEARCH never returns UNKNOWN; outer SQL composes this non-nullable match set.
602+
bitmap_result =
603+
InvertedIndexResultBitmap(std::move(roaring), std::make_shared<roaring::Roaring>());
622604

623-
// The cached result keeps its UNKNOWN rows, so NOT, OR and AND over a cache hit stay
624-
// three-valued.
625605
if (dsl_cache && cache_usable) {
626606
insert_dsl_result(dsl_cache, seg_prefix, dsl_sig, bitmap_result);
627607
}

‎be/src/exprs/function/search/function_search_leaf.cpp‎

Lines changed: 6 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -160,17 +160,14 @@ ScalarLeafCompiler::ScalarLeafCompiler(segment_v2::IndexIterator* iterator, Data
160160
Status ScalarLeafCompiler::compile(const logical::Node& leaf, const SearchLeafContext& ctx,
161161
query_v2::QueryPtr* out) {
162162
const auto* compare = leaf.as<logical::Compare>();
163-
if (compare == nullptr || _iterator == nullptr) {
164-
*out = make_unknown_leaf_query(ctx.num_rows);
165-
return Status::OK();
166-
}
163+
DORIS_CHECK(compare != nullptr);
164+
DORIS_CHECK(_iterator != nullptr);
167165
DataTypePtr value_type = unwrap_value_type(_column_type);
168166
Field value;
169167
if (Status parsed = parse_scalar_value(value_type, compare->value, &value); !parsed.ok()) {
170168
LOG(INFO) << "search: scalar leaf value is unsupported, field=" << compare->field.name
171169
<< ", value='" << compare->value << "', reason=" << parsed.to_string();
172-
*out = make_unknown_leaf_query(ctx.num_rows);
173-
return Status::OK();
170+
return parsed;
174171
}
175172

176173
segment_v2::InvertedIndexParam param;
@@ -182,17 +179,9 @@ Status ScalarLeafCompiler::compile(const logical::Node& leaf, const SearchLeafCo
182179
param.roaring = std::make_shared<roaring::Roaring>();
183180
RETURN_IF_ERROR(_iterator->read_from_index(segment_v2::IndexParam {&param}));
184181

185-
auto null_bitmap = std::make_shared<roaring::Roaring>();
186-
auto has_null = _iterator->has_null();
187-
if (has_null.has_value() && has_null.value()) {
188-
segment_v2::InvertedIndexQueryCacheHandle null_bitmap_cache_handle;
189-
RETURN_IF_ERROR(_iterator->read_null_bitmap(&null_bitmap_cache_handle));
190-
if (auto bitmap = null_bitmap_cache_handle.get_bitmap(); bitmap != nullptr) {
191-
null_bitmap = bitmap;
192-
}
193-
}
194-
*out = std::make_shared<query_v2::BitSetQuery>(std::move(param.roaring),
195-
std::move(null_bitmap));
182+
// Physical NULL postings are excluded by the shared SEARCH leaf mask. Never
183+
// attach SQL UNKNOWN rows or mutate the reader-owned query cache bitmap here.
184+
*out = std::make_shared<query_v2::BitSetQuery>(std::move(param.roaring));
196185
return Status::OK();
197186
}
198187

‎be/src/exprs/function/search/function_search_leaf.h‎

Lines changed: 1 addition & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -51,16 +51,6 @@ class SearchLeafCompiler {
5151
segment_v2::inverted_index::query_v2::QueryPtr* out) = 0;
5252
};
5353

54-
// UNKNOWN for every row: no row matches and every row counts as NULL.
55-
inline segment_v2::inverted_index::query_v2::QueryPtr make_unknown_leaf_query(uint32_t num_rows) {
56-
auto null_bitmap = std::make_shared<roaring::Roaring>();
57-
if (num_rows > 0) {
58-
null_bitmap->addRange(0, num_rows);
59-
}
60-
return std::make_shared<segment_v2::inverted_index::query_v2::BitSetQuery>(
61-
std::make_shared<roaring::Roaring>(), std::move(null_bitmap));
62-
}
63-
6454
// Compiles a leaf into the lazy query_v2 query the engine evaluates on the field's bound source,
6555
// after the source resolved the leaf's terms in one dictionary batch.
6656
class LazyLeafCompiler final : public SearchLeafCompiler {
@@ -78,7 +68,7 @@ class LazyLeafCompiler final : public SearchLeafCompiler {
7868
};
7969

8070
// Answers a comparison leaf from a scalar (BKD) index through its iterator.
81-
// Every other leaf on such an index is UNKNOWN.
71+
// Unsupported leaves and invalid values fail the query.
8272
class ScalarLeafCompiler final : public SearchLeafCompiler {
8373
public:
8474
ScalarLeafCompiler(segment_v2::IndexIterator* iterator, DataTypePtr column_type,

‎be/src/exprs/function/search/search_lowering.cpp‎

Lines changed: 11 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -59,15 +59,6 @@ Occur to_occur(TSearchOccur::type occur) {
5959
}
6060
}
6161

62-
NodePtr lower_direct_index_leaf(const std::string& clause_type, FieldRef field,
63-
const std::string& value) {
64-
if (clause_type == "TERM" || clause_type == "EXACT") {
65-
return make_node(
66-
Compare {.field = std::move(field), .op = CompareOp::kEqual, .value = value});
67-
}
68-
return make_node(Unknown {.field = std::move(field)});
69-
}
70-
7162
// Normalizes the literal text between a wildcard's '*' and '?' operators, so that a normalizer
7263
// never reads them and they stay operators.
7364
Status normalize_wildcard(const FieldProps& props, FieldCatalog& catalog,
@@ -104,10 +95,6 @@ Status lower_pattern_leaf(const std::string& clause_type, const FieldProps& prop
10495
.pattern = "^(" + value + ")$"});
10596
return Status::OK();
10697
}
107-
if (clause_type == "WILDCARD" && value == "*") {
108-
*out = make_node(Exists {.field = std::move(field)});
109-
return Status::OK();
110-
}
11198
const bool prefix = clause_type == "PREFIX";
11299
// The DSL keeps a PREFIX value's trailing '*'.
113100
std::string pattern =
@@ -172,11 +159,20 @@ Status lower_leaf(const TSearchClause& clause, const LoweringOptions& options,
172159
catalog.resolve(clause.field_name, search_clause_query_type(clause_type), &props));
173160
FieldRef field {.name = clause.field_name, .binding = props.binding};
174161
if (!props.bound) {
175-
*out = make_node(Unknown {.field = std::move(field)});
162+
*out = make_node(Empty {.field = std::move(field)});
163+
return Status::OK();
164+
}
165+
if (clause_type == "WILDCARD" && value == "*") {
166+
*out = make_node(Exists {.field = std::move(field)});
176167
return Status::OK();
177168
}
178169
if (props.direct_index) {
179-
*out = lower_direct_index_leaf(clause_type, std::move(field), value);
170+
if (clause_type != "TERM" && clause_type != "EXACT") {
171+
return Status::NotSupported("SEARCH clause {} is unsupported for field {}", clause_type,
172+
field.name);
173+
}
174+
*out = make_node(
175+
Compare {.field = std::move(field), .op = CompareOp::kEqual, .value = value});
180176
return Status::OK();
181177
}
182178
if (is_pattern_clause(clause_type)) {

‎be/src/exprs/function/search/search_predicate_collector.cpp‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -144,6 +144,11 @@ Status SearchPredicateCollector::collect_from_clause(const TSearchClause& clause
144144
const FieldBindingMap& field_bindings,
145145
CollectInfoMap* collect_infos) {
146146
const std::string& clause_type = clause.clause_type;
147+
// Lucene pure NOT injects a fieldless constant-score match-all child.
148+
if (clause_type == "MATCH_ALL_DOCS") {
149+
return Status::OK();
150+
}
151+
147152
if (clause_type == "NESTED") {
148153
return Status::Error<ErrorCode::INVERTED_INDEX_NOT_SUPPORTED>(
149154
"Scoring nested search clauses is not supported");

0 commit comments

Comments
 (0)