Repository navigation
Commit 19dfb87
authored
[fix](partition prunning) Correct partition pruning for calendar arithmetic (#69828)
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Calendar month, quarter, year and YEAR_MONTH additions
and subtractions clamp dates at month end while preserving the time of
day. Consecutive input dates can therefore produce decreasing datetime
values. Treating these functions as globally monotonic lets RANGE
partition pruning remove partitions containing matching rows, such as
January 29 filtered by months_add(datetime_column, 1) < '2023-02-28
10:00:00'. Share a calendar arithmetic monotonicity trait that retains
the optimization for DATE and single-day ranges and keeps other datetime
ranges conservatively. This also covers DATE_ADD, DATE_SUB,
TIMESTAMPADD, INTERVAL expressions and runtime-filter classification.
MONTHS_DIFF, QUARTERS_DIFF and YEARS_DIFF also fold some month-end
datetime arguments differently from BE because LocalDateTime.until
adjusts the end date before counting calendar units. Count date units
first, then compare time parts only when the remaining date fields
match, preserving BE's truncation semantics for both argument orders and
all supported date-like literal types.
### Release note
Fix missing query rows caused by calendar arithmetic partition pruning
and align month, quarter and year difference constant folding with
runtime results.
### Check List (For Author)
- Test: FE build with Checkstyle; 94 FE unit tests; regression tests
- Unit Test: CalendarAddSubMonotonicTest,
DateTimeExtractAndTransformTest, PartitionPrunerTest,
RuntimeFilterPruneClassifierTest, TimestampNsMonotonicityTest and
InferPredicateFromMonotonicFunctionTest
- Regression test: test_calendar_arithmetic_prune (73 generated result
checks) and test_date_function_prune; verify the new suite fails on the
original FE
- Behavior changed: Yes, retain matching rows and correct calendar
difference constants
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->1 parent 8e499c0 commit 19dfb87
19 files changed
Lines changed: 1102 additions & 91 deletions
File tree
- fe/fe-core/src
- main/java/org/apache/doris/nereids
- rules/expression/rules
- trees/expressions/functions
- executable
- scalar
- test/java/org/apache/doris/nereids
- rules/rewrite
- trees/expressions/functions
- executable
- scalar
- regression-test
- data/nereids_rules_p0/partition_prune
- suites/nereids_rules_p0/partition_prune
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
43 | | - | |
| 43 | + | |
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
49 | | - | |
50 | 49 | | |
51 | 50 | | |
52 | 51 | | |
| |||
69 | 68 | | |
70 | 69 | | |
71 | 70 | | |
| 71 | + | |
72 | 72 | | |
73 | 73 | | |
74 | 74 | | |
| |||
532 | 532 | | |
533 | 533 | | |
534 | 534 | | |
535 | | - | |
536 | | - | |
| 535 | + | |
| 536 | + | |
537 | 537 | | |
538 | 538 | | |
539 | 539 | | |
| |||
1005 | 1005 | | |
1006 | 1006 | | |
1007 | 1007 | | |
1008 | | - | |
1009 | | - | |
1010 | | - | |
| 1008 | + | |
| 1009 | + | |
| 1010 | + | |
1011 | 1011 | | |
1012 | 1012 | | |
1013 | | - | |
| 1013 | + | |
1014 | 1014 | | |
1015 | 1015 | | |
1016 | | - | |
1017 | | - | |
| 1016 | + | |
| 1017 | + | |
1018 | 1018 | | |
1019 | 1019 | | |
1020 | 1020 | | |
1021 | 1021 | | |
1022 | | - | |
| 1022 | + | |
1023 | 1023 | | |
1024 | 1024 | | |
1025 | 1025 | | |
| |||
1039 | 1039 | | |
1040 | 1040 | | |
1041 | 1041 | | |
1042 | | - | |
| 1042 | + | |
| 1043 | + | |
1043 | 1044 | | |
1044 | 1045 | | |
1045 | | - | |
1046 | | - | |
1047 | | - | |
1048 | | - | |
1049 | | - | |
1050 | | - | |
1051 | | - | |
1052 | | - | |
1053 | | - | |
1054 | | - | |
1055 | | - | |
1056 | | - | |
1057 | | - | |
1058 | | - | |
1059 | | - | |
1060 | | - | |
1061 | | - | |
1062 | | - | |
1063 | | - | |
1064 | | - | |
1065 | | - | |
1066 | | - | |
1067 | | - | |
1068 | | - | |
1069 | | - | |
1070 | | - | |
1071 | | - | |
1072 | | - | |
1073 | | - | |
1074 | | - | |
1075 | | - | |
1076 | | - | |
1077 | | - | |
1078 | | - | |
1079 | | - | |
1080 | 1046 | | |
1081 | | - | |
1082 | | - | |
1083 | | - | |
1084 | | - | |
| 1047 | + | |
| 1048 | + | |
| 1049 | + | |
| 1050 | + | |
| 1051 | + | |
1085 | 1052 | | |
1086 | | - | |
1087 | | - | |
| 1053 | + | |
1088 | 1054 | | |
1089 | 1055 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
20 | | - | |
| 20 | + | |
| 21 | + | |
21 | 22 | | |
22 | 23 | | |
| 24 | + | |
| 25 | + | |
23 | 26 | | |
24 | | - | |
| 27 | + | |
25 | 28 | | |
26 | 29 | | |
27 | 30 | | |
| |||
33 | 36 | | |
34 | 37 | | |
35 | 38 | | |
36 | | - | |
37 | | - | |
38 | | - | |
39 | | - | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
40 | 53 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
0 commit comments