Repository navigation
Commit 62190be
perf: preserve scalar UDF output types in property analysis (apache#25668)
## Which issue does this PR close?
- Closes apache#25667.
## Rationale for this change
Compositions such as nested `date_trunc` or `CAST(trunc(x) AS REAL)` can
retain redundant sorts even when their inputs are ordered. Scalar UDFs
using the default bounds implementation lose their output type during
property analysis, although `ScalarFunctionExpr` already knows the
resolved return type.
This implements the central fallback discussed in [the review of
apache#25575](apache#25575 (comment)).
## What changes are included in this PR?
- In `ScalarFunctionExpr::get_properties`, replace an unbounded `Null`
result with a typed unbounded interval using the resolved return type.
Preserve explicit bounds and UDF evaluation errors; retain the original
interval if the fallback type is unsupported.
- Remove the type-only `evaluate_bounds` overrides for `date_bin` and
`from_unixtime`, and document the central behavior for UDF authors.
- Keep the `PhysicalExpr::evaluate_bounds` delegation unchanged, without
adding a fallback to the constraint solver. Removing the two overrides
means their direct bounds evaluation returns the default `Null` interval
again.
- Include two equivalent benchmark cleanups required by strict Clippy: a
boolean-to-integer conversion and a local type alias.
## What is the testing strategy for this PR?
Unit tests cover resolved output types, explicit and session timezones,
unsupported interval types, preservation of existing bounds and errors,
and separation from direct bounds evaluation. Removing the fallback
makes the new type-recovery tests fail.
SQL tests cover redundant sort removal for nested datetime functions and
a numeric UDF followed by a cast, including NULL input. Negative plan
and result tests retain the required sorts for named timezone
transitions and time-of-day arithmetic across midnight. Existing
`date_bin` and `from_unixtime` plan tests remain unchanged.
Validation:
- Extended workspace test suite, including SQL logic tests.
- `cargo fmt --all`.
- `cargo clippy --all-targets --all-features -- -D warnings`.
- `uv run ./dev/rust_lint.sh`.
A local planning experiment against the base `main` commit used the same
sorted 100-row `MemTable`, 20 warmups and 200 planning iterations, with
three alternating runs of each build. The nested datetime and
numeric-cast queries each removed one redundant sort and returned
identical results. Median planning times were 679 → 645 µs and 585 → 553
µs; control and struct-return queries were essentially unchanged. These
are small local measurements using the unoptimized `ci` profile, not
production execution-speed claims.
## Are there any user-facing changes?
Safe compositions of scalar functions can avoid redundant sorts without
individual UDFs implementing bounds solely to preserve their output
type. Existing ordering rules still determine whether a sort can be
removed. No public API signatures or FFI structures change.
Co-authored-by: Andrew Lamb <[email protected]>1 parent d38f587 commit 62190be
7 files changed
Lines changed: 384 additions & 126 deletions
File tree
- datafusion
- expr/src
- functions/src/datetime
- physical-expr/src
- sqllogictest/test_files
- datetime
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
908 | 908 | | |
909 | 909 | | |
910 | 910 | | |
911 | | - | |
912 | | - | |
913 | | - | |
914 | | - | |
915 | | - | |
916 | | - | |
917 | | - | |
918 | | - | |
919 | | - | |
| 911 | + | |
| 912 | + | |
| 913 | + | |
| 914 | + | |
| 915 | + | |
| 916 | + | |
| 917 | + | |
| 918 | + | |
| 919 | + | |
920 | 920 | | |
921 | 921 | | |
922 | 922 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
41 | | - | |
42 | 41 | | |
43 | 42 | | |
44 | 43 | | |
| |||
294 | 293 | | |
295 | 294 | | |
296 | 295 | | |
297 | | - | |
298 | | - | |
299 | | - | |
300 | | - | |
301 | | - | |
302 | 296 | | |
303 | 297 | | |
304 | 298 | | |
| |||
829 | 823 | | |
830 | 824 | | |
831 | 825 | | |
832 | | - | |
| 826 | + | |
833 | 827 | | |
834 | 828 | | |
835 | 829 | | |
836 | | - | |
837 | 830 | | |
838 | 831 | | |
839 | 832 | | |
| |||
884 | 877 | | |
885 | 878 | | |
886 | 879 | | |
887 | | - | |
888 | | - | |
889 | | - | |
890 | | - | |
891 | | - | |
892 | | - | |
893 | | - | |
894 | | - | |
895 | | - | |
896 | | - | |
897 | | - | |
898 | | - | |
899 | | - | |
900 | | - | |
901 | | - | |
902 | | - | |
903 | | - | |
904 | | - | |
905 | | - | |
906 | | - | |
907 | | - | |
908 | | - | |
909 | 880 | | |
910 | 881 | | |
911 | 882 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
31 | | - | |
32 | 31 | | |
33 | 32 | | |
34 | 33 | | |
| |||
220 | 219 | | |
221 | 220 | | |
222 | 221 | | |
223 | | - | |
224 | | - | |
225 | | - | |
226 | | - | |
227 | | - | |
228 | | - | |
229 | | - | |
230 | | - | |
231 | | - | |
232 | | - | |
233 | | - | |
234 | | - | |
235 | | - | |
236 | | - | |
237 | | - | |
238 | | - | |
239 | 222 | | |
240 | 223 | | |
241 | 224 | | |
| |||
351 | 334 | | |
352 | 335 | | |
353 | 336 | | |
354 | | - | |
355 | 337 | | |
356 | 338 | | |
357 | 339 | | |
| |||
364 | 346 | | |
365 | 347 | | |
366 | 348 | | |
367 | | - | |
368 | | - | |
369 | | - | |
370 | | - | |
371 | | - | |
372 | | - | |
373 | | - | |
374 | | - | |
375 | | - | |
376 | | - | |
377 | | - | |
378 | | - | |
379 | | - | |
380 | | - | |
381 | | - | |
382 | | - | |
383 | | - | |
384 | | - | |
385 | | - | |
386 | | - | |
387 | | - | |
388 | | - | |
389 | | - | |
390 | | - | |
391 | | - | |
392 | | - | |
393 | | - | |
394 | | - | |
395 | | - | |
396 | | - | |
397 | | - | |
398 | | - | |
399 | | - | |
400 | | - | |
401 | | - | |
402 | | - | |
403 | | - | |
404 | | - | |
405 | | - | |
406 | | - | |
407 | | - | |
408 | | - | |
409 | | - | |
410 | | - | |
411 | | - | |
412 | | - | |
413 | | - | |
414 | | - | |
415 | | - | |
416 | | - | |
417 | | - | |
418 | | - | |
419 | | - | |
420 | | - | |
421 | | - | |
422 | | - | |
423 | | - | |
424 | | - | |
425 | | - | |
426 | | - | |
427 | | - | |
428 | | - | |
429 | | - | |
430 | | - | |
431 | | - | |
432 | 349 | | |
433 | 350 | | |
434 | 351 | | |
| |||
0 commit comments