Skip to content

Commit 9fe2b52

Browse files
[fix](search) Bind NESTED aliases to physical Variant paths
### Background SEARCH dependencies can be renamed or replaced with extracted Variant slots during predicate pushdown. ### What problem does this PR solve? Issue Number: None Related PR: #69344 PR Type: Bugfix Problem Summary: NESTED queries through projected Variant aliases retained the logical array owner even after their leaves were bound to a physical scan column. The owner could not bind to the read schema. Nested leaf mapping and document-count callbacks also used logical paths and could miss deeper physical groups. ### Details - **Frequency:** NESTED over a renamed Variant root or projected Variant prefix. - **Root cause:** nested_path and nested traversal callbacks did not follow the final physical field bindings. - **Why existing tests missed it:** Owner and leaf names matched scan names; alias tests covered ordinary SEARCH rather than NESTED serialization and nested callbacks. - **Fix:** Derive the physical nested owner from a normalized leaf binding during serialization, and translate callback paths through the same binding metadata in BE. - **Other paths and branches:** Cover materialized slots and retained element_at children, root aliases and projected prefixes, and both nested leaf mapping and row counts. Logical DSL lookup keys remain unchanged. - **Behavior change:** Aliased NESTED predicates use the final physical Variant owner and paths. ### Testing - Old code: three new FE tests failed with doc.items instead of v.items/v.payload.items; the new BE test failed with doc.items.tags.name instead of v.payload.items.tags.name. - Fixed code: 40 FE tests and 180 BE tests passed using run-fe-ut.sh and run-be-ut.sh. - Built FE and ASAN BE with build.sh and passed FE Checkstyle, clang-format 16, clang-tidy, and BE build hygiene. - Deployed FE/BE libraries, verified checksums, and passed all 19 targeted SEARCH/score/Variant regression suites with zero failures or skips. - Full NESTED result-set execution remains unverified because the available community build lacks the NestedGroup module. ### Release note Fix physical Variant path binding for NESTED SEARCH predicates through projected aliases. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: No. - Pre-submission status: Partially verified: full NESTED result-set execution requires a NestedGroup-capable build; the available community build does not provide that module. ### AI Disclosure - Tool: OpenAI Codex. - Generated: Code, tests, and this description. - Human review: Pending.
1 parent 7ef8f50 commit 9fe2b52

6 files changed

Lines changed: 122 additions & 3 deletions

File tree

‎be/src/exprs/function/search/variant_inverted_index_search.cpp‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,17 @@ FieldReaderResolver::FieldReaderResolver(
123123
}
124124
}
125125

126+
std::string FieldReaderResolver::physical_field_path(const std::string& field_name) const {
127+
const auto it = _field_binding_map.find(field_name);
128+
// Index lookup keeps the logical key, but NestedGroup traversal needs the physical path.
129+
// For example, doc.items.tags.name may read v.payload.items.tags.name after pushdown.
130+
if (it != _field_binding_map.end() && it->second->__isset.parent_field_name &&
131+
it->second->__isset.subcolumn_path) {
132+
return it->second->parent_field_name + "." + it->second->subcolumn_path;
133+
}
134+
return field_name;
135+
}
136+
126137
Status FieldReaderResolver::resolve(const std::string& field_name,
127138
InvertedIndexQueryType query_type,
128139
FieldReaderBinding* binding) {

‎be/src/exprs/function/search/variant_inverted_index_search.h‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -148,8 +148,9 @@ class FieldReaderResolver {
148148
}
149149

150150
Status leaf_num_rows(const std::string& field_name, uint32_t* num_rows) const {
151-
return _leaf_num_rows_resolver ? _leaf_num_rows_resolver(field_name, num_rows)
152-
: Status::OK();
151+
return _leaf_num_rows_resolver
152+
? _leaf_num_rows_resolver(physical_field_path(field_name), num_rows)
153+
: Status::OK();
153154
}
154155

155156
void set_leaf_query_mapper(SearchLeafQueryMapper mapper) {
@@ -162,10 +163,12 @@ class FieldReaderResolver {
162163
if (!_leaf_query_mapper || query == nullptr || *query == nullptr) {
163164
return Status::OK();
164165
}
165-
return _leaf_query_mapper(field_name, query);
166+
return _leaf_query_mapper(physical_field_path(field_name), query);
166167
}
167168

168169
private:
170+
std::string physical_field_path(const std::string& field_name) const;
171+
169172
std::string binding_key_for(const std::string& stored_field_name,
170173
InvertedIndexQueryType query_type) const {
171174
return stored_field_name + "#" + std::to_string(static_cast<int>(query_type));

‎be/test/exprs/function/search/function_search_nested_test.cpp‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -182,6 +182,36 @@ class FunctionSearchNestedTest : public testing::Test {
182182
// is_nested_group_search_supported (CE / Default provider)
183183
// ===========================================================================
184184

185+
TEST_F(FunctionSearchNestedTest, NestedCallbacksUsePhysicalAliasPath) {
186+
TSearchFieldBinding binding;
187+
binding.__set_field_name("doc.items.tags.name");
188+
binding.__set_is_variant_subcolumn(true);
189+
binding.__set_parent_field_name("v");
190+
binding.__set_subcolumn_path("payload.items.tags.name");
191+
std::unordered_map<std::string, IndexFieldNameAndTypePair> types;
192+
std::unordered_map<std::string, IndexIterator*> iterators;
193+
FieldReaderResolver resolver(types, iterators, std::make_shared<IndexQueryContext>(),
194+
{binding});
195+
std::string mapped_path;
196+
resolver.set_leaf_query_mapper([&](const std::string& path, auto*) {
197+
mapped_path = path;
198+
return Status::OK();
199+
});
200+
inverted_index::query_v2::QueryPtr query =
201+
std::make_shared<inverted_index::query_v2::BitSetQuery>(roaring::Roaring());
202+
ASSERT_TRUE(resolver.map_leaf_query("doc.items.tags.name", &query).ok());
203+
EXPECT_EQ("v.payload.items.tags.name", mapped_path);
204+
resolver.set_leaf_num_rows_resolver([&](const std::string& path, uint32_t* count) {
205+
mapped_path = path;
206+
*count = 42;
207+
return Status::OK();
208+
});
209+
uint32_t count = 0;
210+
ASSERT_TRUE(resolver.leaf_num_rows("doc.items.tags.name", &count).ok());
211+
EXPECT_EQ("v.payload.items.tags.name", mapped_path);
212+
EXPECT_EQ(42, count);
213+
}
214+
185215
TEST_F(FunctionSearchNestedTest, DefaultProviderNestedGroupSearchSupport) {
186216
auto provider = segment_v2::create_nested_group_read_provider();
187217
ASSERT_NE(nullptr, provider);

‎fe/fe-core/src/main/java/org/apache/doris/analysis/ExprToThriftVisitor.java‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,7 @@
5757
import org.apache.doris.thrift.TUUIDLiteral;
5858
import org.apache.doris.thrift.TVarBinaryLiteral;
5959

60+
import com.google.common.base.Preconditions;
6061
import com.google.common.collect.Lists;
6162
import org.apache.logging.log4j.LogManager;
6263
import org.apache.logging.log4j.Logger;
@@ -693,6 +694,19 @@ static TSearchParam buildSearchThriftParam(SearchPredicate expr) {
693694
}
694695
param.setFieldBindings(bindings);
695696

697+
if ("NESTED".equals(param.root.clause_type)) {
698+
// Every NESTED leaf is relative to the same logical array. Remove its relative
699+
// suffix from the final physical leaf, preserving any projected Variant prefix:
700+
// doc.items.msg -> v.payload.items.msg implies NESTED(v.payload.items, ...).
701+
Preconditions.checkState(!bindings.isEmpty());
702+
TSearchFieldBinding binding = bindings.get(0);
703+
String suffix = binding.field_name.substring(param.root.nested_path.length());
704+
Preconditions.checkState(binding.is_variant_subcolumn && suffix.startsWith("."));
705+
String physicalField = binding.parent_field_name + "." + binding.subcolumn_path;
706+
Preconditions.checkState(physicalField.endsWith(suffix));
707+
param.root.setNestedPath(physicalField.substring(0, physicalField.length() - suffix.length()));
708+
}
709+
696710
if (qsPlan.getDefaultOperator() != null) {
697711
param.setDefaultOperator(qsPlan.getDefaultOperator());
698712
}

‎fe/fe-core/src/test/java/org/apache/doris/analysis/SearchPredicateTest.java‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@
2929
import org.junit.jupiter.api.Assertions;
3030
import org.junit.jupiter.api.Test;
3131

32+
import java.util.ArrayList;
3233
import java.util.Arrays;
3334
import java.util.Collections;
3435
import java.util.HashMap;
@@ -189,6 +190,38 @@ public void testBuildThriftParam() {
189190
Assertions.assertEquals("content", param.field_bindings.get(1).field_name);
190191
}
191192

193+
@Test
194+
public void testNestedAliasUsesPhysicalOwnerAndFullArrayPath() {
195+
String dsl = "NESTED(doc.items, msg:hello AND tags.name:world)";
196+
SearchDslParser.QsPlan plan = SearchDslParser.parseDsl(dsl);
197+
for (List<String> prefix : Arrays.asList(Collections.<String>emptyList(), Arrays.asList("payload"))) {
198+
List<String> messagePath = new ArrayList<>(prefix);
199+
messagePath.addAll(Arrays.asList("items", "msg"));
200+
List<String> tagPath = new ArrayList<>(prefix);
201+
tagPath.addAll(Arrays.asList("items", "tags", "name"));
202+
SearchPredicate predicate = new SearchPredicate(dsl, plan, Arrays.asList(
203+
createVariantSlot("v", messagePath.toArray(new String[0])),
204+
createVariantSlot("v", tagPath.toArray(new String[0]))), false);
205+
TSearchParam param = ExprToThriftVisitor.buildSearchThriftParam(predicate);
206+
Assertions.assertEquals(prefix.isEmpty() ? "v.items" : "v.payload.items", param.root.nested_path);
207+
Assertions.assertEquals("doc.items.msg", param.field_bindings.get(0).field_name);
208+
Assertions.assertEquals("doc.items.tags.name", param.field_bindings.get(1).field_name);
209+
Assertions.assertEquals("doc.items", plan.getRoot().getNestedPath());
210+
}
211+
}
212+
213+
@Test
214+
public void testNestedAliasRetainsUnmaterializedPath() {
215+
String dsl = "NESTED(doc.items, msg:hello)";
216+
Expr child = new FunctionCallExpr("element_at",
217+
Arrays.asList(createVariantSlot("v", "payload"), new StringLiteral("items.msg")), true);
218+
SearchPredicate predicate = new SearchPredicate(dsl, SearchDslParser.parseDsl(dsl),
219+
Arrays.asList(child), false);
220+
TSearchParam param = ExprToThriftVisitor.buildSearchThriftParam(predicate);
221+
Assertions.assertEquals("v.payload.items", param.root.nested_path);
222+
Assertions.assertEquals("doc.items.msg", param.field_bindings.get(0).field_name);
223+
}
224+
192225
@Test
193226
public void testNestedRelativeFieldsAreNormalizedBeforeThrift() {
194227
String dsl = "NESTED(data.items, msg:hello AND meta.channel:action)";

‎fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/SearchCteRewriteTest.java‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,18 +17,21 @@
1717

1818
package org.apache.doris.nereids.rules.rewrite;
1919

20+
import org.apache.doris.analysis.ExprToThriftVisitor;
2021
import org.apache.doris.nereids.trees.expressions.Expression;
2122
import org.apache.doris.nereids.trees.expressions.SearchExpression;
2223
import org.apache.doris.nereids.trees.expressions.functions.scalar.Search;
2324
import org.apache.doris.nereids.trees.plans.Plan;
2425
import org.apache.doris.nereids.util.PlanChecker;
26+
import org.apache.doris.thrift.TSearchParam;
2527
import org.apache.doris.utframe.TestWithFeService;
2628

2729
import org.junit.jupiter.api.Assertions;
2830
import org.junit.jupiter.api.Test;
2931

3032
import java.util.ArrayList;
3133
import java.util.List;
34+
import java.util.stream.Collectors;
3235

3336
/**
3437
* Regression test for SEARCH under a materialized CTE.
@@ -63,6 +66,31 @@ protected void runBeforeAll() throws Exception {
6366
+ "PROPERTIES ('replication_num' = '1');");
6467
}
6568

69+
@Test
70+
public void testNestedAliasPhysicalBindingAfterPlanning() throws Exception {
71+
createTable("CREATE TABLE t_nested_alias (id INT, v VARIANT, "
72+
+ "INDEX idx_v(v) USING INVERTED PROPERTIES('parser' = 'unicode')) "
73+
+ "DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 "
74+
+ "PROPERTIES ('replication_num' = '1')");
75+
for (String projection : new String[] {"v", "v['payload']"}) {
76+
String sql = "SELECT id FROM (SELECT id, " + projection + " AS doc FROM t_nested_alias) x "
77+
+ "WHERE SEARCH('NESTED(doc.items, msg:hello AND tags.name:world)')";
78+
PlanChecker.from(connectContext).checkPlannerResult(sql, planner -> {
79+
List<TSearchParam> params = planner.getScanNodes().stream()
80+
.flatMap(scan -> scan.getConjuncts().stream())
81+
.flatMap(expr -> ExprToThriftVisitor.treeToThrift(expr).nodes.stream())
82+
.filter(node -> node.isSetSearchParam())
83+
.map(node -> node.search_param).collect(Collectors.toList());
84+
Assertions.assertEquals(1, params.size());
85+
TSearchParam param = params.get(0);
86+
Assertions.assertEquals(projection.equals("v") ? "v.items" : "v.payload.items",
87+
param.root.nested_path);
88+
Assertions.assertEquals("v", param.field_bindings.get(0).parent_field_name);
89+
Assertions.assertEquals("doc.items.msg", param.field_bindings.get(0).field_name);
90+
});
91+
}
92+
}
93+
6694
@Test
6795
public void testSearchWithConstantEquality() {
6896
assertSearchRewritten(PlanChecker.from(connectContext)

0 commit comments

Comments
 (0)