Skip to content

Commit a7ae6f4

Browse files
authored
branch-4.2: [fix](nereids) Preserve date arithmetic evaluation boundaries apache#69113 (apache#69576);;;;
### What problem does this PR solve? Issue Number: N/A (no associated GitHub issue) Related PR: apache#69113 Problem Summary: Backport apache#69113 to `branch-4.2`. An outer predicate on a projected date expression can be expanded and evaluated before an inner condition rejects the row. A filtered row containing `9999-12-31 23:59:59` can therefore cause `WEEKS_ADD(dt, 5)` to overflow instead of returning the expected empty result. Preserve potentially overflowing date arithmetic aliases above Filter, Join, Limit, EXCEPT, INTERSECT, Generate and CTE consumer boundaries in logical and physical project predicate pushdown. The branch adaptation uses its existing logical/physical EXCEPT and INTERSECT classes and retains its Project-level NoneMovableFunction and volatile-expression guards. Protect both scalar function calls and `TimestampArithmetic` (`dt +/- INTERVAL ...`). Preserve the compensating `Filter -> Limit -> Project` rewrite to `Project -> Filter -> Limit`, so OFFSET discards rows before date evaluation. Safe integer-zero offsets and ordinary UNION retain the behavior of the master fix. ### Release note Fix date arithmetic overflow errors caused by predicate pushdown evaluating expressions on rows excluded by an inner query. ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [x] Regression test - [x] Unit Test - [x] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> <details> <summary>Local validation on branch-4.2</summary> - FE/BE build and strict Checkstyle passed with JDK 17 and clang21/ASAN. The service build used the repository's `DISABLE_BUILD_UI=ON` option. - **17 FE unit tests passed**, with zero failures, errors or skips: `DateArithmeticFilterPushDownTest` (11), `PushDowFilterThroughProjectTest` (4), and `org.apache.doris.nereids.postprocess.PushDownFilterThroughProjectTest` (2). - **Three native P0 suites passed**, with zero failed/fatal/skipped suites: `nereids_rules_p0/filter_push_down/test_date_arithmetic_rejecting_boundary`, `nereids_rules_p0/date_function_rewrite/date_function_rewrite`, and `nereids_rules_p0/cte/test_cte_filter_pushdown`. - Tests ran on this branch's verified binaries in a fresh isolated 1 FE + 3 BE instance. Source, configuration, framework and process identities remained stable. The epoch-based date-rewrite suite was run with the private instance's default time zone set to `Asia/Shanghai`. - The unit tests cover logical/physical interval protection, integer and bigint zero offsets, and projection restoration above OFFSET with and without a lower Filter. The native regression adds OFFSET 1/100 empty-result checks, interval addition/subtraction and zero-offset controls, and participating-row overflow controls. Expected output files were not edited. - **10 direct SQL checks passed** against the final branch binaries: OFFSET 1/100, interval addition, subtraction of a negative interval, subtraction, both zero-interval spellings, a zero scalar offset, and scalar/interval participating-row overflow. - Source whitespace checks passed; regression outputs retain the framework's standard final separator. Before this review update, manual validation passed **18 checks covering ORC and Parquet**, in each of UTC and Asia/Shanghai. That external-file validation was not rerun for this update; the final update was verified with the unit/native/direct-SQL checks above: 1. Generate 200 rows: one row has `dt='9999-12-31 23:59:59'` and `filter_dt='2025-01-01'`; the other 199 have `dt='2020-01-01'` and `filter_dt='2020-01-01'`. Export the same input to ORC and Parquet on a fresh private HDFS fixture. 2. For WEEKS_ADD, HOURS_ADD, SECOND_CEIL, YEAR_CEIL and DAYS_ADD, run the filtered CTE/outer-alias predicate shape from the original issue. All return zero rows without overflow. A surviving-row control returns a count of 199; a shared CTE control returns 398 rows. 3. Verify that arithmetic on a participating maximum-date row still raises the expected out-of-range error, and that a NULL date returns NULL. Input row hashes and HDFS/runtime identities remained unchanged during each run. These are targeted local results, not a full TeamCity or External-group verdict. The five historical RQG executions have not been replayed because their actual external-table DDL and immutable ORC inputs were unavailable; the manual fixtures above were reconstructed. </details> - Behavior changed: - [ ] No. - [x] Yes. Preserve date arithmetic evaluation boundaries while retaining safe zero-offset predicate pushdown and partition pruning. Nonzero date predicates can remain above rejecting boundaries. - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
1 parent f58418b commit a7ae6f4

8 files changed

Lines changed: 669 additions & 6 deletions

File tree

‎fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/PushDownFilterThroughProject.java‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,8 @@ public Plan visitPhysicalFilter(PhysicalFilter<? extends Plan> filter, CascadesC
4545
PhysicalProject<? extends Plan> project = (PhysicalProject<? extends Plan>) child;
4646
Map<Slot, Expression> childAlias = project.getAliasToProducer();
4747
if (filter.getInputSlots().stream().map(childAlias::get).filter(Objects::nonNull)
48-
.anyMatch(Expression::containsVolatileExpression)) {
48+
.anyMatch(expression -> expression.containsVolatileExpression()
49+
|| ExpressionUtils.requiresDateArithmeticProtection(expression, project.child()))) {
4950
return filter;
5051
}
5152
PhysicalFilter<? extends Plan> newFilter = filter.withConjunctsAndChild(

‎fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownFilterThroughProject.java‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,7 @@ private static Plan pushDownFilterThroughProject(LogicalFilter<LogicalProject<Pl
8282
// splitConjuncts.first -> conjuncts having outer query slots which should NOT be pushed down
8383
// splitConjuncts.second -> conjuncts without any outer query slots which should be pushed down
8484
Pair<Set<Expression>, Set<Expression>> splitConjuncts =
85-
splitConjunctsByChildOutput(filter.getConjuncts(), childOutputs, childAlias);
85+
splitConjunctsByChildOutput(filter.getConjuncts(), childOutputs, childAlias, project.child(), true);
8686
Set<Expression> remainPredicates = splitConjuncts.first;
8787
Set<Expression> pushDownPredicates = splitConjuncts.second;
8888
if (pushDownPredicates.isEmpty()) {
@@ -99,9 +99,10 @@ private static Plan pushDownFilterThroughLimitProject(
9999
LogicalProject<Plan> project = limit.child();
100100
Set<Slot> childOutputs = project.getOutputSet();
101101
Map<Slot, Expression> childAlias = project.getAliasToProducer();
102-
// split the conjuncts by child's output
102+
// Restore evaluation above the enclosing Limit. The expanded filter stays above
103+
// that Limit and every lower rejecting boundary, so date arithmetic is safe here.
103104
Pair<Set<Expression>, Set<Expression>> splitConjuncts =
104-
splitConjunctsByChildOutput(filter.getConjuncts(), childOutputs, childAlias);
105+
splitConjunctsByChildOutput(filter.getConjuncts(), childOutputs, childAlias, limit, false);
105106
Set<Expression> remainPredicates = splitConjuncts.first;
106107
Set<Expression> pushDownPredicates = splitConjuncts.second;
107108
if (pushDownPredicates.isEmpty()) {
@@ -113,7 +114,8 @@ private static Plan pushDownFilterThroughLimitProject(
113114
}
114115

115116
private static Pair<Set<Expression>, Set<Expression>> splitConjunctsByChildOutput(
116-
Set<Expression> conjuncts, Set<Slot> childOutputs, Map<Slot, Expression> childAlias) {
117+
Set<Expression> conjuncts, Set<Slot> childOutputs, Map<Slot, Expression> childAlias, Plan input,
118+
boolean protectDateArithmetic) {
117119
Set<Expression> pushDownPredicates = Sets.newLinkedHashSet();
118120
Set<Expression> remainPredicates = Sets.newLinkedHashSet();
119121
for (Expression conjunct : conjuncts) {
@@ -125,7 +127,9 @@ private static Pair<Set<Expression>, Set<Expression>> splitConjunctsByChildOutpu
125127
// `project(b + random(1, 10) as a) -> filter(b + random(1, 10) > 1)`, it contains two distinct RANDOM.
126128
if (childOutputs.containsAll(conjunctSlots)
127129
&& conjunctSlots.stream().map(childAlias::get).filter(Objects::nonNull)
128-
.noneMatch(Expression::containsVolatileExpression)) {
130+
.noneMatch(expression -> expression.containsVolatileExpression()
131+
|| (protectDateArithmetic
132+
&& ExpressionUtils.requiresDateArithmeticProtection(expression, input)))) {
129133
pushDownPredicates.add(conjunct);
130134
} else {
131135
remainPredicates.add(conjunct);

‎fe/fe-core/src/main/java/org/apache/doris/nereids/util/ExpressionUtils.java‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,10 +50,13 @@
5050
import org.apache.doris.nereids.trees.expressions.Or;
5151
import org.apache.doris.nereids.trees.expressions.Slot;
5252
import org.apache.doris.nereids.trees.expressions.SlotReference;
53+
import org.apache.doris.nereids.trees.expressions.TimestampArithmetic;
5354
import org.apache.doris.nereids.trees.expressions.VolatileExpression;
5455
import org.apache.doris.nereids.trees.expressions.WhenClause;
5556
import org.apache.doris.nereids.trees.expressions.WindowExpression;
5657
import org.apache.doris.nereids.trees.expressions.functions.BoundFunction;
58+
import org.apache.doris.nereids.trees.expressions.functions.DateAddSubMonotonic;
59+
import org.apache.doris.nereids.trees.expressions.functions.DateCeilFloorMonotonic;
5760
import org.apache.doris.nereids.trees.expressions.functions.NoneMovableFunction;
5861
import org.apache.doris.nereids.trees.expressions.functions.agg.Avg;
5962
import org.apache.doris.nereids.trees.expressions.functions.agg.Count;
@@ -78,6 +81,7 @@
7881
import org.apache.doris.nereids.trees.expressions.functions.scalar.Nvl;
7982
import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
8083
import org.apache.doris.nereids.trees.expressions.literal.ComparableLiteral;
84+
import org.apache.doris.nereids.trees.expressions.literal.IntegerLikeLiteral;
8185
import org.apache.doris.nereids.trees.expressions.literal.IntegerLiteral;
8286
import org.apache.doris.nereids.trees.expressions.literal.Literal;
8387
import org.apache.doris.nereids.trees.expressions.literal.NullLiteral;
@@ -86,8 +90,18 @@
8690
import org.apache.doris.nereids.trees.expressions.visitor.DefaultExpressionVisitor;
8791
import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitors;
8892
import org.apache.doris.nereids.trees.plans.Plan;
93+
import org.apache.doris.nereids.trees.plans.algebra.Filter;
94+
import org.apache.doris.nereids.trees.plans.algebra.Generate;
95+
import org.apache.doris.nereids.trees.plans.algebra.Join;
96+
import org.apache.doris.nereids.trees.plans.algebra.Limit;
97+
import org.apache.doris.nereids.trees.plans.logical.LogicalCTEConsumer;
8998
import org.apache.doris.nereids.trees.plans.logical.LogicalEmptyRelation;
99+
import org.apache.doris.nereids.trees.plans.logical.LogicalExcept;
100+
import org.apache.doris.nereids.trees.plans.logical.LogicalIntersect;
90101
import org.apache.doris.nereids.trees.plans.logical.LogicalUnion;
102+
import org.apache.doris.nereids.trees.plans.physical.PhysicalCTEConsumer;
103+
import org.apache.doris.nereids.trees.plans.physical.PhysicalExcept;
104+
import org.apache.doris.nereids.trees.plans.physical.PhysicalIntersect;
91105
import org.apache.doris.nereids.trees.plans.visitor.ExpressionLineageReplacer;
92106
import org.apache.doris.nereids.types.DataType;
93107
import org.apache.doris.nereids.types.VariantType;
@@ -136,6 +150,42 @@ public class ExpressionUtils {
136150
// folding the conjunct). We restrict MAX_MARK_SLOT_COUNT to 4 to bound this cost.
137151
private static final int MAX_MARK_SLOT_COUNT = 4;
138152

153+
/**
154+
* Date arithmetic can throw for values outside its valid range. Keep it above a lower
155+
* condition, join, limit, rejecting set operation, generate, or CTE consumer so rejected
156+
* rows are not evaluated early.
157+
* A date add/subtract with literal zero is an identity and remains safe to push through.
158+
*/
159+
public static boolean requiresDateArithmeticProtection(Expression expression, Plan input) {
160+
return expression.anyMatch(expr -> isUnsafeDateArithmetic((Expression) expr))
161+
&& input.anyMatch(plan -> plan instanceof Filter || plan instanceof Join || plan instanceof Limit
162+
|| plan instanceof LogicalExcept || plan instanceof PhysicalExcept
163+
|| plan instanceof LogicalIntersect || plan instanceof PhysicalIntersect
164+
|| plan instanceof Generate || plan instanceof LogicalCTEConsumer
165+
|| plan instanceof PhysicalCTEConsumer);
166+
}
167+
168+
private static boolean isUnsafeDateArithmetic(Expression expression) {
169+
if (expression instanceof DateCeilFloorMonotonic) {
170+
return true;
171+
}
172+
int dateChildIndex;
173+
if (expression instanceof TimestampArithmetic) {
174+
dateChildIndex = 0;
175+
} else if (expression instanceof DateAddSubMonotonic) {
176+
dateChildIndex = ((DateAddSubMonotonic) expression).getMonotonicFunctionChildIndex();
177+
} else {
178+
return false;
179+
}
180+
for (int i = 0; i < expression.children().size(); i++) {
181+
if (i != dateChildIndex && !(expression.child(i) instanceof IntegerLikeLiteral
182+
&& ((IntegerLikeLiteral) expression.child(i)).compareTo(new IntegerLiteral(0)) == 0)) {
183+
return true;
184+
}
185+
}
186+
return false;
187+
}
188+
139189
public static List<Expression> extractConjunction(Expression expr) {
140190
return extract(And.class, expr);
141191
}

0 commit comments

Comments
 (0)