Repository navigation
Commit a7ae6f4
authored
branch-4.2: [fix](nereids) Preserve date arithmetic evaluation boundaries apache#69113 (apache#69576);;;;
### What problem does this PR solve?
Issue Number: N/A (no associated GitHub issue)
Related PR: apache#69113
Problem Summary:
Backport apache#69113 to `branch-4.2`. An outer predicate on a projected date
expression can be expanded and evaluated before an inner condition
rejects the row. A filtered row containing `9999-12-31 23:59:59` can
therefore cause `WEEKS_ADD(dt, 5)` to overflow instead of returning the
expected empty result.
Preserve potentially overflowing date arithmetic aliases above Filter,
Join, Limit, EXCEPT, INTERSECT, Generate and CTE consumer boundaries in
logical and physical project predicate pushdown. The branch adaptation
uses its existing logical/physical EXCEPT and INTERSECT classes and
retains its Project-level NoneMovableFunction and volatile-expression
guards. Protect both scalar function calls and `TimestampArithmetic`
(`dt +/- INTERVAL ...`). Preserve the compensating `Filter -> Limit ->
Project` rewrite to `Project -> Filter -> Limit`, so OFFSET discards
rows before date evaluation. Safe integer-zero offsets and ordinary
UNION retain the behavior of the master fix.
### Release note
Fix date arithmetic overflow errors caused by predicate pushdown
evaluating expressions on rows excluded by an inner query.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
<details>
<summary>Local validation on branch-4.2</summary>
- FE/BE build and strict Checkstyle passed with JDK 17 and clang21/ASAN.
The service build used the repository's `DISABLE_BUILD_UI=ON` option.
- **17 FE unit tests passed**, with zero failures, errors or skips:
`DateArithmeticFilterPushDownTest` (11),
`PushDowFilterThroughProjectTest` (4), and
`org.apache.doris.nereids.postprocess.PushDownFilterThroughProjectTest`
(2).
- **Three native P0 suites passed**, with zero failed/fatal/skipped
suites:
`nereids_rules_p0/filter_push_down/test_date_arithmetic_rejecting_boundary`,
`nereids_rules_p0/date_function_rewrite/date_function_rewrite`, and
`nereids_rules_p0/cte/test_cte_filter_pushdown`.
- Tests ran on this branch's verified binaries in a fresh isolated 1 FE
+ 3 BE instance. Source, configuration, framework and process identities
remained stable. The epoch-based date-rewrite suite was run with the
private instance's default time zone set to `Asia/Shanghai`.
- The unit tests cover logical/physical interval protection, integer and
bigint zero offsets, and projection restoration above OFFSET with and
without a lower Filter. The native regression adds OFFSET 1/100
empty-result checks, interval addition/subtraction and zero-offset
controls, and participating-row overflow controls. Expected output files
were not edited.
- **10 direct SQL checks passed** against the final branch binaries:
OFFSET 1/100, interval addition, subtraction of a negative interval,
subtraction, both zero-interval spellings, a zero scalar offset, and
scalar/interval participating-row overflow.
- Source whitespace checks passed; regression outputs retain the
framework's standard final separator.
Before this review update, manual validation passed **18 checks covering
ORC and Parquet**, in each of UTC and Asia/Shanghai. That external-file
validation was not rerun for this update; the final update was verified
with the unit/native/direct-SQL checks above:
1. Generate 200 rows: one row has `dt='9999-12-31 23:59:59'` and
`filter_dt='2025-01-01'`; the other 199 have `dt='2020-01-01'` and
`filter_dt='2020-01-01'`. Export the same input to ORC and Parquet on a
fresh private HDFS fixture.
2. For WEEKS_ADD, HOURS_ADD, SECOND_CEIL, YEAR_CEIL and DAYS_ADD, run
the filtered CTE/outer-alias predicate shape from the original issue.
All return zero rows without overflow. A surviving-row control returns a
count of 199; a shared CTE control returns 398 rows.
3. Verify that arithmetic on a participating maximum-date row still
raises the expected out-of-range error, and that a NULL date returns
NULL. Input row hashes and HDFS/runtime identities remained unchanged
during each run.
These are targeted local results, not a full TeamCity or External-group
verdict. The five historical RQG executions have not been replayed
because their actual external-table DDL and immutable ORC inputs were
unavailable; the manual fixtures above were reconstructed.
</details>
- Behavior changed:
- [ ] No.
- [x] Yes. Preserve date arithmetic evaluation boundaries while
retaining safe zero-offset predicate pushdown and partition pruning.
Nonzero date predicates can remain above rejecting boundaries.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->1 parent f58418b commit a7ae6f4
8 files changed
Lines changed: 669 additions & 6 deletions
File tree
- fe/fe-core/src
- main/java/org/apache/doris/nereids
- processor/post
- rules/rewrite
- util
- test/java/org/apache/doris/nereids/rules/rewrite
- regression-test
- data
- external_table_p0/tvf
- nereids_rules_p0/filter_push_down
- suites
- external_table_p0/tvf
- nereids_rules_p0/filter_push_down
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
48 | | - | |
| 48 | + | |
| 49 | + | |
49 | 50 | | |
50 | 51 | | |
51 | 52 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
82 | 82 | | |
83 | 83 | | |
84 | 84 | | |
85 | | - | |
| 85 | + | |
86 | 86 | | |
87 | 87 | | |
88 | 88 | | |
| |||
99 | 99 | | |
100 | 100 | | |
101 | 101 | | |
102 | | - | |
| 102 | + | |
| 103 | + | |
103 | 104 | | |
104 | | - | |
| 105 | + | |
105 | 106 | | |
106 | 107 | | |
107 | 108 | | |
| |||
113 | 114 | | |
114 | 115 | | |
115 | 116 | | |
116 | | - | |
| 117 | + | |
| 118 | + | |
117 | 119 | | |
118 | 120 | | |
119 | 121 | | |
| |||
125 | 127 | | |
126 | 128 | | |
127 | 129 | | |
128 | | - | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
129 | 133 | | |
130 | 134 | | |
131 | 135 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
| 53 | + | |
53 | 54 | | |
54 | 55 | | |
55 | 56 | | |
56 | 57 | | |
| 58 | + | |
| 59 | + | |
57 | 60 | | |
58 | 61 | | |
59 | 62 | | |
| |||
78 | 81 | | |
79 | 82 | | |
80 | 83 | | |
| 84 | + | |
81 | 85 | | |
82 | 86 | | |
83 | 87 | | |
| |||
86 | 90 | | |
87 | 91 | | |
88 | 92 | | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
89 | 98 | | |
| 99 | + | |
| 100 | + | |
90 | 101 | | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
91 | 105 | | |
92 | 106 | | |
93 | 107 | | |
| |||
136 | 150 | | |
137 | 151 | | |
138 | 152 | | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
139 | 189 | | |
140 | 190 | | |
141 | 191 | | |
| |||
0 commit comments