Repository navigation
Commit f1ecc4a
authored
feat: reuse range partitioning for audited date_bin aggregation (apache#26835)
## Which issue does this PR close?
Closes apache#25344.
Closes apache#25345.
This completes the initial single-key work alongside apache#26449. Explicit
origins (apache#26913) and compound range keys (apache#24644) remain follow-ups.
## Rationale for this change
Extend the aggregate-locality optimization from apache#26449 to `date_bin`.
When bins stay within timestamp range partitions, aggregation can skip
the hash shuffle.
## What changes are included in this PR?
Pass literal argument values to the existing admission hook. `INTERVAL
'1 hour'` and `INTERVAL '1 month'` have the same `IntervalMonthDayNano`
type, so types alone cannot distinguish them. Inspecting the literal
value lets `date_bin` require a positive fixed-duration stride without
months, treating each day as 24 hours. `date_trunc` continues to rely
only on argument types.
Initially, admit only two-argument calls over timezone-less nanosecond
timestamps, using that literal stride. Explicit origins and other types
remain excluded.
For example:
```sql
SELECT
date_bin(INTERVAL '1 hour', event_time) AS hour_bin,
COUNT(*)
FROM events
GROUP BY hour_bin;
```
For ascending timestamp ranges:
- A split at `01:00` separates hourly bins, so the shuffle can be
skipped.
- A split at `01:30` cuts through the `01:00` bin, so the shuffle
remains.
Every split must pass the existing boundary proof. Evaluation errors
conservatively retain the shuffle. This follows the NULL-to-error fix in
apache#26607.
## What is the testing strategy for this PR?
Unit tests cover admission, hook forwarding, ascending/descending
boundaries, and evaluation errors. SQLLogicTests verify plans and
results for aligned bins, crossing bins, and excluded explicit origins.
## Are there any user-facing changes?
Eligible `date_bin` aggregations can avoid a hash shuffle. Function
results are unchanged.
---
Note: This PR was developed with AI assistance. All changes have been
reviewed, and I take full responsibility for this contribution.1 parent 69a7fef commit f1ecc4a
8 files changed
Lines changed: 633 additions & 58 deletions
File tree
- datafusion
- expr/src
- functions/src/datetime
- physical-expr/src
- sqllogictest
- src
- test_context
- test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
355 | 355 | | |
356 | 356 | | |
357 | 357 | | |
| 358 | + | |
358 | 359 | | |
359 | 360 | | |
360 | | - | |
| 361 | + | |
361 | 362 | | |
362 | 363 | | |
363 | 364 | | |
| |||
979 | 980 | | |
980 | 981 | | |
981 | 982 | | |
982 | | - | |
983 | | - | |
| 983 | + | |
| 984 | + | |
984 | 985 | | |
985 | 986 | | |
986 | 987 | | |
| |||
998 | 999 | | |
999 | 1000 | | |
1000 | 1001 | | |
1001 | | - | |
| 1002 | + | |
| 1003 | + | |
| 1004 | + | |
| 1005 | + | |
| 1006 | + | |
| 1007 | + | |
| 1008 | + | |
| 1009 | + | |
| 1010 | + | |
1002 | 1011 | | |
1003 | 1012 | | |
1004 | 1013 | | |
| |||
1224 | 1233 | | |
1225 | 1234 | | |
1226 | 1235 | | |
1227 | | - | |
| 1236 | + | |
| 1237 | + | |
| 1238 | + | |
| 1239 | + | |
| 1240 | + | |
1228 | 1241 | | |
1229 | | - | |
| 1242 | + | |
1230 | 1243 | | |
1231 | 1244 | | |
1232 | 1245 | | |
| |||
1314 | 1327 | | |
1315 | 1328 | | |
1316 | 1329 | | |
1317 | | - | |
| 1330 | + | |
| 1331 | + | |
| 1332 | + | |
| 1333 | + | |
1318 | 1334 | | |
1319 | 1335 | | |
1320 | 1336 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
276 | 276 | | |
277 | 277 | | |
278 | 278 | | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
279 | 307 | | |
280 | 308 | | |
281 | 309 | | |
| |||
978 | 1006 | | |
979 | 1007 | | |
980 | 1008 | | |
| 1009 | + | |
| 1010 | + | |
| 1011 | + | |
| 1012 | + | |
| 1013 | + | |
| 1014 | + | |
| 1015 | + | |
| 1016 | + | |
| 1017 | + | |
| 1018 | + | |
| 1019 | + | |
| 1020 | + | |
| 1021 | + | |
| 1022 | + | |
| 1023 | + | |
| 1024 | + | |
| 1025 | + | |
| 1026 | + | |
| 1027 | + | |
| 1028 | + | |
| 1029 | + | |
| 1030 | + | |
| 1031 | + | |
| 1032 | + | |
| 1033 | + | |
| 1034 | + | |
| 1035 | + | |
| 1036 | + | |
| 1037 | + | |
| 1038 | + | |
| 1039 | + | |
| 1040 | + | |
| 1041 | + | |
| 1042 | + | |
| 1043 | + | |
| 1044 | + | |
| 1045 | + | |
| 1046 | + | |
| 1047 | + | |
| 1048 | + | |
| 1049 | + | |
| 1050 | + | |
| 1051 | + | |
| 1052 | + | |
| 1053 | + | |
| 1054 | + | |
| 1055 | + | |
| 1056 | + | |
| 1057 | + | |
| 1058 | + | |
| 1059 | + | |
| 1060 | + | |
| 1061 | + | |
| 1062 | + | |
| 1063 | + | |
| 1064 | + | |
| 1065 | + | |
| 1066 | + | |
| 1067 | + | |
| 1068 | + | |
| 1069 | + | |
| 1070 | + | |
| 1071 | + | |
| 1072 | + | |
| 1073 | + | |
| 1074 | + | |
| 1075 | + | |
| 1076 | + | |
| 1077 | + | |
| 1078 | + | |
| 1079 | + | |
| 1080 | + | |
| 1081 | + | |
| 1082 | + | |
| 1083 | + | |
| 1084 | + | |
| 1085 | + | |
| 1086 | + | |
| 1087 | + | |
| 1088 | + | |
| 1089 | + | |
| 1090 | + | |
| 1091 | + | |
| 1092 | + | |
| 1093 | + | |
| 1094 | + | |
| 1095 | + | |
| 1096 | + | |
| 1097 | + | |
| 1098 | + | |
| 1099 | + | |
| 1100 | + | |
| 1101 | + | |
| 1102 | + | |
| 1103 | + | |
| 1104 | + | |
| 1105 | + | |
| 1106 | + | |
| 1107 | + | |
| 1108 | + | |
| 1109 | + | |
| 1110 | + | |
| 1111 | + | |
| 1112 | + | |
| 1113 | + | |
| 1114 | + | |
| 1115 | + | |
| 1116 | + | |
| 1117 | + | |
| 1118 | + | |
| 1119 | + | |
| 1120 | + | |
| 1121 | + | |
| 1122 | + | |
| 1123 | + | |
| 1124 | + | |
| 1125 | + | |
| 1126 | + | |
| 1127 | + | |
| 1128 | + | |
| 1129 | + | |
| 1130 | + | |
| 1131 | + | |
| 1132 | + | |
| 1133 | + | |
| 1134 | + | |
| 1135 | + | |
| 1136 | + | |
| 1137 | + | |
| 1138 | + | |
| 1139 | + | |
| 1140 | + | |
| 1141 | + | |
| 1142 | + | |
| 1143 | + | |
| 1144 | + | |
| 1145 | + | |
| 1146 | + | |
| 1147 | + | |
| 1148 | + | |
| 1149 | + | |
| 1150 | + | |
981 | 1151 | | |
982 | 1152 | | |
983 | 1153 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
559 | 559 | | |
560 | 560 | | |
561 | 561 | | |
562 | | - | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
563 | 567 | | |
564 | 568 | | |
565 | 569 | | |
| |||
1400 | 1404 | | |
1401 | 1405 | | |
1402 | 1406 | | |
1403 | | - | |
1404 | | - | |
1405 | | - | |
1406 | | - | |
1407 | | - | |
1408 | | - | |
1409 | | - | |
1410 | | - | |
1411 | | - | |
1412 | | - | |
1413 | | - | |
1414 | | - | |
1415 | | - | |
| 1407 | + | |
| 1408 | + | |
| 1409 | + | |
| 1410 | + | |
| 1411 | + | |
| 1412 | + | |
| 1413 | + | |
| 1414 | + | |
| 1415 | + | |
| 1416 | + | |
| 1417 | + | |
| 1418 | + | |
| 1419 | + | |
| 1420 | + | |
| 1421 | + | |
| 1422 | + | |
| 1423 | + | |
| 1424 | + | |
| 1425 | + | |
| 1426 | + | |
| 1427 | + | |
| 1428 | + | |
| 1429 | + | |
| 1430 | + | |
1416 | 1431 | | |
1417 | 1432 | | |
1418 | 1433 | | |
| |||
0 commit comments