Skip to content

Commit 9c71f1f

Browse files
airborne12claude
andcommitted
[fix](nereids) Keep ANN TopN and bind score filters to one scan
### What problem does this PR solve? Issue Number: None Related PR: #69292 Problem Summary: Follow-up fixes for the standalone minimum score filter (score() > x without a score TopN): 1. Binding the score range replaced the scan's ANN TopN with an empty one, so WHERE content MATCH 'x' AND score() > 0.5 ORDER BY l2_distance_approximate() LIMIT k no longer used the ANN index and computed every distance by brute force. The scan applies the score range before the ANN search, so the binding now keeps the ANN order keys and limit. On a local cluster the profile of this query shows AnnIndexTopNSearchCnt=1 and no brute-force fallback, and the nearest row that fails the score filter is not returned. 2. score() has no input slots, so join rules treated score() > x like a constant. PushDownFilterThroughJoin copied it into both children, PushFilterInsideJoin turned it into a join condition and PushDownJoinOtherCondition then pushed it into the left child. A query with an IN or EXISTS subquery failed with a misleading "must contain MATCH" error, and an inner join silently filtered whichever table was on the left. The predicate now goes only into the child whose rows all reach the output of a left/right outer, semi or anti join, as ReorderJoin already does. Inner, cross and full joins keep it above the join, where CheckScoreUsage rejects it with a message that suggests a subquery; score() in a JOIN condition fails analysis. 3. test_bm25_score still expected an error for a score() > 0.5 filter without ORDER BY and LIMIT, which is now supported. It now checks that the row count matches the score TopN path. 4. EXPLAIN prints the pushed range (SCORE RANGE: score() > 0.5), because the predicate is no longer among the scan conjuncts. 5. The new filter rules are guarded by the cached containsType check, so filters without score() skip them. 6. SearchExpression.toSql keeps the options argument of search(dsl, options). ### Release note score() filters work with IN and EXISTS subqueries and on the preserved side of left or right outer joins. With inner, cross or full joins, or in a JOIN condition, they fail with a clear error instead of filtering an arbitrary table. MATCH queries with a score() filter and an ANN TopN keep using the ANN index. EXPLAIN shows the pushed score range. ### Check List (For Author) - Test: Unit Test and Regression test - FE unit tests: CheckScoreUsageTest, PushDownFilterThroughJoinTest, PushFilterInsideJoinTest, SearchExpressionTest, RewriteSearchToSlotsTest, PullUpProjectExprUnderTopNTest, CheckAnalysisTest, ReorderJoinTest (180 tests). - Regression tests on a local ASAN cluster: test_bm25_score, test_score_filter_usage, test_score_filter_matrix, test_bm25_score_range_filter, test_search_score_topn_predicates and six more score suites in inverted_index_p0, all 36 suites in search, ann_with_fulltext and ann_topn_small_candidate_fallback. - Behavior changed: Yes. score() filters above inner, cross or full joins and in JOIN conditions are rejected; outer, semi and anti joins keep the filter on their output side; EXPLAIN shows SCORE RANGE. - Does this need documentation: Yes, with the score() documentation for standalone minimum score filters. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent bd79ec1 commit 9c71f1f

18 files changed

Lines changed: 386 additions & 31 deletions

File tree

‎fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/CheckAnalysis.java‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@
3232
import org.apache.doris.nereids.trees.expressions.functions.generator.TableGeneratingFunction;
3333
import org.apache.doris.nereids.trees.expressions.functions.generator.Unnest;
3434
import org.apache.doris.nereids.trees.expressions.functions.scalar.GroupingScalarFunction;
35+
import org.apache.doris.nereids.trees.expressions.functions.scalar.Score;
3536
import org.apache.doris.nereids.trees.expressions.typecoercion.TypeCheckResult;
3637
import org.apache.doris.nereids.trees.plans.Plan;
3738
import org.apache.doris.nereids.trees.plans.algebra.Aggregate;
@@ -91,6 +92,8 @@ public class CheckAnalysis implements AnalysisRuleFactory {
9192
.put(LogicalJoin.class, Utils.fastArray(
9293
AggregateFunction.class,
9394
GroupingScalarFunction.class,
95+
// score() belongs to the scan that evaluates MATCH or SEARCH, not to a join.
96+
Score.class,
9497
TableGeneratingFunction.class,
9598
WindowExpression.class))
9699
.put(LogicalOneRowRelation.class, Utils.fastArray(

‎fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/CheckScoreUsage.java‎

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,11 @@ public class CheckScoreUsage implements RewriteRuleFactory {
4343
private static final String SCORE_CONTEXT_ERROR =
4444
"score() function requires WHERE clause with MATCH function, ORDER BY and LIMIT for optimization, "
4545
+ "or a supported minimum score filter with MATCH or SEARCH";
46+
private static final String SCORE_FILTER_CONTEXT_ERROR =
47+
"score() filter must be in the WHERE clause of a query on one table with a MATCH or SEARCH "
48+
+ "predicate. It cannot filter the result of an inner, cross or full join, an aggregation "
49+
+ "or another operator; put the MATCH or SEARCH predicate and the score() filter in a "
50+
+ "subquery on that table";
4651

4752
@Override
4853
public List<Rule> buildRules() {
@@ -58,11 +63,13 @@ public List<Rule> buildRules() {
5863
throw new AnalysisException(SCORE_CONTEXT_ERROR);
5964
}).toRule(RuleType.CHECK_SCORE_USAGE),
6065

61-
// A filter-only score() call otherwise evaluates the scalar placeholder as zero.
66+
// A score() filter on a scan was bound by PushDownScoreTopNIntoOlapScan. One left here
67+
// filters another operator, e.g. an inner join, and would evaluate the scalar placeholder
68+
// as zero.
6269
logicalFilter(any())
6370
.when(filter -> filter.getConjuncts().stream().anyMatch(this::containsScoreFunction))
6471
.then(filter -> {
65-
throw new AnalysisException(SCORE_CONTEXT_ERROR);
72+
throw new AnalysisException(SCORE_FILTER_CONTEXT_ERROR);
6673
}).toRule(RuleType.CHECK_SCORE_USAGE),
6774

6875
logicalAggregate(any())
@@ -99,10 +106,7 @@ private boolean hasScoreInAggregate(LogicalAggregate<?> agg) {
99106
}
100107

101108
private boolean containsScoreFunction(Expression expr) {
102-
if (expr instanceof Score) {
103-
return true;
104-
}
105-
return expr.children().stream().anyMatch(this::containsScoreFunction);
109+
return expr.containsType(Score.class);
106110
}
107111

108112
private boolean isScoreAlreadyOptimized(LogicalProject<?> project) {

‎fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownFilterThroughJoin.java‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@
2323
import org.apache.doris.nereids.trees.expressions.Expression;
2424
import org.apache.doris.nereids.trees.expressions.Slot;
2525
import org.apache.doris.nereids.trees.expressions.SlotReference;
26+
import org.apache.doris.nereids.trees.expressions.functions.scalar.Score;
2627
import org.apache.doris.nereids.trees.plans.JoinType;
2728
import org.apache.doris.nereids.trees.plans.Plan;
2829
import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
@@ -126,6 +127,21 @@ public Rule build() {
126127
}
127128
Set<Slot> slots = p.collect(SlotReference.class::isInstance);
128129
if (slots.isEmpty()) {
130+
if (p.containsType(Score.class)) {
131+
// score() is the relevance of the current row computed by the scan that
132+
// evaluates MATCH or SEARCH, not a constant, so it must not be copied into
133+
// both children. Like ReorderJoin, push it only into the child whose rows
134+
// all reach the output of an outer, semi or anti join. Inner, cross and full
135+
// joins mix the rows of both children: keep it above for CheckScoreUsage.
136+
if (join.getJoinType().isLeftJoin()) {
137+
leftPredicates.add(p);
138+
} else if (join.getJoinType().isRightJoin()) {
139+
rightPredicates.add(p);
140+
} else {
141+
remainingPredicates.add(p);
142+
}
143+
continue;
144+
}
129145
leftPredicates.add(p);
130146
rightPredicates.add(p);
131147
continue;

‎fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownScoreTopNIntoOlapScan.java‎

Lines changed: 19 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -77,20 +77,21 @@ public List<Rule> buildRules() {
7777
LogicalOlapScan scan = filter.child();
7878
return pushDown(topN, project, filter, scan);
7979
}).toRule(RuleType.PUSH_DOWN_SCORE_TOPN_INTO_OLAP_SCAN),
80-
logicalProject(logicalFilter(logicalOlapScan()))
80+
logicalProject(logicalFilter(logicalOlapScan()).when(this::hasScorePredicate))
8181
.then(project -> pushDownScoreRange(project))
8282
.toRule(RuleType.PUSH_DOWN_SCORE_TOPN_INTO_OLAP_SCAN),
83-
logicalFilter(logicalOlapScan())
83+
logicalFilter(logicalOlapScan()).when(this::hasScorePredicate)
8484
.then(filter -> pushDownScoreRange(filter))
8585
.toRule(RuleType.PUSH_DOWN_SCORE_TOPN_INTO_OLAP_SCAN));
8686
}
8787

88+
private boolean hasScorePredicate(LogicalFilter<LogicalOlapScan> filter) {
89+
return filter.getConjuncts().stream().anyMatch(conjunct -> conjunct.containsType(Score.class));
90+
}
91+
8892
private LogicalProject<Plan> pushDownScoreRange(
8993
LogicalProject<LogicalFilter<LogicalOlapScan>> project) {
9094
LogicalFilter<Plan> filter = pushDownScoreRange(project.child());
91-
if (filter == null) {
92-
return null;
93-
}
9495
LogicalOlapScan scan = (LogicalOlapScan) filter.child();
9596
Expression scoreSlot = scan.getScoreOrderKeys().get(0).getExpr();
9697
Map<Expression, Expression> replaceMap = Maps.newHashMap();
@@ -104,35 +105,33 @@ private LogicalProject<Plan> pushDownScoreRange(
104105
}
105106

106107
private LogicalFilter<Plan> pushDownScoreRange(LogicalFilter<LogicalOlapScan> filter) {
107-
Optional<Expression> scorePredicateOpt = findScoreRangePredicate(filter.getConjuncts());
108-
if (!scorePredicateOpt.isPresent()) {
109-
return null;
110-
}
108+
Expression scorePredicate = findScoreRangePredicate(filter.getConjuncts()).get();
111109
boolean hasMatchOrSearch = filter.getConjuncts().stream()
112-
.anyMatch(conjunct -> !conjunct.collect(
113-
e -> e instanceof Match || e instanceof SearchExpression).isEmpty());
110+
.anyMatch(conjunct -> conjunct.containsType(Match.class, SearchExpression.class));
114111
if (!hasMatchOrSearch) {
115112
throw new AnalysisException("WHERE clause must contain at least one MATCH or SEARCH function"
116113
+ " for score() filtering");
117114
}
118-
Expression scorePredicate = scorePredicateOpt.get();
119115
Optional<ScoreRangeInfo> scoreRangeInfo = Optional.of(extractScoreRangeInfo(scorePredicate));
120116
Set<Expression> scoreFunctions = scorePredicate.collect(e -> e instanceof Score);
121117
Alias scoreAlias = new Alias(scoreFunctions.iterator().next(), "__score");
118+
LogicalOlapScan scan = filter.child();
122119
// A zero score limit enables scoring without discarding candidates before other filters.
123-
LogicalOlapScan scan = filter.child().appendVirtualColumnsAndTopN(ImmutableList.of(scoreAlias),
124-
ImmutableList.of(), Optional.empty(),
120+
// Keep an ANN TopN pushed by PushDownVectorTopNIntoOlapScan: the scan applies the score
121+
// range before the ANN search, so the ANN candidates are already filtered by score.
122+
LogicalOlapScan newScan = scan.appendVirtualColumnsAndTopN(ImmutableList.of(scoreAlias),
123+
scan.getAnnOrderKeys(), scan.getAnnLimit(),
125124
ImmutableList.of(new OrderKey(scoreAlias.toSlot(), false, false)), Optional.of(0L),
126125
scoreRangeInfo);
127126
Set<Expression> remainingConjuncts = filter.getConjuncts().stream()
128127
.filter(conjunct -> !conjunct.equals(scorePredicate))
129128
.collect(ImmutableSet.toImmutableSet());
130-
return filter.withConjunctsAndChild(remainingConjuncts, scan);
129+
return filter.withConjunctsAndChild(remainingConjuncts, newScan);
131130
}
132131

133132
private Optional<Expression> findScoreRangePredicate(Set<Expression> conjuncts) {
134133
List<Expression> scorePredicates = conjuncts.stream()
135-
.filter(conjunct -> !conjunct.collect(e -> e instanceof Score).isEmpty())
134+
.filter(conjunct -> conjunct.containsType(Score.class))
136135
.collect(ImmutableList.toImmutableList());
137136
if (scorePredicates.size() > 1) {
138137
throw new AnalysisException(
@@ -174,6 +173,10 @@ private Plan pushDown(
174173
Optional<ScoreRangeInfo> scoreRangeInfo = scorePredicate.map(this::extractScoreRangeInfo);
175174
Expression extractedScorePredicate = scorePredicate.orElse(null);
176175

176+
// When a minimum score predicate exists, a TopN that fails requirement 4 or 5 is not an
177+
// error: returning null leaves it as an ordinary TopN, and the project/filter rules below
178+
// bind the predicate with a zero score limit so that every candidate gets scored.
179+
177180
// 4. Requirement: TopN must have exactly one ordering expression.
178181
if (topN.getOrderKeys().size() != 1) {
179182
if (scoreRangeInfo.isPresent()) {

‎fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushFilterInsideJoin.java‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@
2323
import org.apache.doris.nereids.trees.expressions.Expression;
2424
import org.apache.doris.nereids.trees.expressions.MarkJoinSlotReference;
2525
import org.apache.doris.nereids.trees.expressions.Slot;
26+
import org.apache.doris.nereids.trees.expressions.functions.scalar.Score;
2627
import org.apache.doris.nereids.trees.plans.Plan;
2728
import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
2829
import org.apache.doris.nereids.util.ExpressionUtils;
@@ -59,7 +60,9 @@ public Rule build() {
5960
List<Expression> otherConditions = Lists.newArrayListWithExpectedSize(
6061
filter.getConjuncts().size() + join.getOtherJoinConjuncts().size());
6162
for (Expression expr : filter.getConjuncts()) {
62-
if (expr.containsVolatileOrNoneMovableExpression()) {
63+
// A score() predicate belongs to one scan: as a join condition without
64+
// slots, PushDownJoinOtherCondition would push it into the left child.
65+
if (expr.containsVolatileOrNoneMovableExpression() || expr.containsType(Score.class)) {
6366
remainConditions.add(expr);
6467
} else {
6568
otherConditions.add(expr);

‎fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/RewriteSearchToSlots.java‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@
4444
import java.util.HashMap;
4545
import java.util.List;
4646
import java.util.Map;
47+
import java.util.Optional;
4748
import java.util.stream.Collectors;
4849

4950
/**
@@ -172,7 +173,8 @@ private Expression rewriteSearch(Search search, LogicalOlapScan scan) {
172173
normalizePlanFields(qsPlan.getRoot(), normalizedFields);
173174

174175
// Create SearchExpression with slot children
175-
return new SearchExpression(search.getDslString(), qsPlan, slotChildren);
176+
return new SearchExpression(search.getDslString(), Optional.ofNullable(search.getOptionsJson()),
177+
qsPlan, slotChildren);
176178

177179
} catch (Exception e) {
178180
throw new AnalysisException("Failed to rewrite search expression: " + e.getMessage(), e);

‎fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/SearchExpression.java‎

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -27,18 +27,28 @@
2727

2828
import java.util.List;
2929
import java.util.Objects;
30+
import java.util.Optional;
3031

3132
/**
3233
* SearchExpression represents a search query with bound slot references.
3334
* This is created by RewriteSearchToSlots rule from Search scalar function.
3435
*/
3536
public class SearchExpression extends Expression {
3637
private final String dslString;
38+
// The second argument of search(dsl, options), kept so that toSql() reproduces the call.
39+
private final Optional<String> optionsJson;
3740
private final SearchDslParser.QsPlan qsPlan;
3841

3942
public SearchExpression(String dslString, SearchDslParser.QsPlan qsPlan, List<Expression> slotChildren) {
43+
this(dslString, Optional.empty(), qsPlan, slotChildren);
44+
}
45+
46+
/** Create a bound search expression that keeps the options argument of search(dsl, options). */
47+
public SearchExpression(String dslString, Optional<String> optionsJson, SearchDslParser.QsPlan qsPlan,
48+
List<Expression> slotChildren) {
4049
super(slotChildren);
4150
this.dslString = Objects.requireNonNull(dslString, "dslString cannot be null");
51+
this.optionsJson = Objects.requireNonNull(optionsJson, "optionsJson cannot be null");
4252
this.qsPlan = Objects.requireNonNull(qsPlan, "qsPlan cannot be null");
4353
}
4454

@@ -80,7 +90,7 @@ public SearchExpression withChildren(List<Expression> children) {
8090
"SearchExpression children must be SlotReference or ElementAt instances");
8191
}
8292
}
83-
return new SearchExpression(dslString, qsPlan, children);
93+
return new SearchExpression(dslString, optionsJson, qsPlan, children);
8494
}
8595

8696
@Override
@@ -95,7 +105,9 @@ public String toString() {
95105

96106
@Override
97107
protected String computeToSql() {
98-
return "search(" + SqlLiteralUtils.quoteStringLiteral(dslString) + ")";
108+
StringBuilder sql = new StringBuilder("search(").append(SqlLiteralUtils.quoteStringLiteral(dslString));
109+
optionsJson.ifPresent(options -> sql.append(", ").append(SqlLiteralUtils.quoteStringLiteral(options)));
110+
return sql.append(")").toString();
99111
}
100112

101113
@Override
@@ -111,11 +123,12 @@ public boolean equals(Object o) {
111123
}
112124
SearchExpression that = (SearchExpression) o;
113125
return Objects.equals(dslString, that.dslString)
126+
&& Objects.equals(optionsJson, that.optionsJson)
114127
&& Objects.equals(qsPlan, that.qsPlan);
115128
}
116129

117130
@Override
118131
public int hashCode() {
119-
return Objects.hash(super.hashCode(), dslString, qsPlan);
132+
return Objects.hash(super.hashCode(), dslString, optionsJson, qsPlan);
120133
}
121134
}

‎fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/ScoreRangeInfo.java‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,8 @@
2020
import org.apache.doris.thrift.TExprOpcode;
2121
import org.apache.doris.thrift.TScoreRangeInfo;
2222

23+
import com.google.common.base.Preconditions;
24+
2325
import java.util.Objects;
2426

2527
/**
@@ -44,6 +46,15 @@ public double getThreshold() {
4446
return threshold;
4547
}
4648

49+
/**
50+
* Render the pushed predicate for EXPLAIN, e.g. "score() > 0.5".
51+
*/
52+
public String toSql() {
53+
Preconditions.checkState(op == TExprOpcode.GT || op == TExprOpcode.GE,
54+
"score range only supports > and >=, but got %s", op);
55+
return "score() " + (op == TExprOpcode.GT ? ">" : ">=") + " " + threshold;
56+
}
57+
4758
/**
4859
* Convert to Thrift representation for sending to BE.
4960
*/

‎fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1293,6 +1293,10 @@ public String getNodeExplainString(String prefix, TExplainLevel detailLevel) {
12931293
if (scoreSortLimit != -1) {
12941294
output.append(prefix).append("SCORE SORT LIMIT: ").append(scoreSortLimit).append("\n");
12951295
}
1296+
if (scoreRangeInfo != null) {
1297+
// The pushed score predicate is no longer among the scan conjuncts.
1298+
output.append(prefix).append("SCORE RANGE: ").append(scoreRangeInfo.toSql()).append("\n");
1299+
}
12961300

12971301
if (annSortInfo != null) {
12981302
output.append(prefix).append("ANN SORT INFO:\n");

0 commit comments

Comments
 (0)