Skip to content

Commit 62190be

Browse files
Toby1009alamb
authored andcommitted
perf: preserve scalar UDF output types in property analysis (apache#25668)
## Which issue does this PR close? - Closes apache#25667. ## Rationale for this change Compositions such as nested `date_trunc` or `CAST(trunc(x) AS REAL)` can retain redundant sorts even when their inputs are ordered. Scalar UDFs using the default bounds implementation lose their output type during property analysis, although `ScalarFunctionExpr` already knows the resolved return type. This implements the central fallback discussed in [the review of apache#25575](apache#25575 (comment)). ## What changes are included in this PR? - In `ScalarFunctionExpr::get_properties`, replace an unbounded `Null` result with a typed unbounded interval using the resolved return type. Preserve explicit bounds and UDF evaluation errors; retain the original interval if the fallback type is unsupported. - Remove the type-only `evaluate_bounds` overrides for `date_bin` and `from_unixtime`, and document the central behavior for UDF authors. - Keep the `PhysicalExpr::evaluate_bounds` delegation unchanged, without adding a fallback to the constraint solver. Removing the two overrides means their direct bounds evaluation returns the default `Null` interval again. - Include two equivalent benchmark cleanups required by strict Clippy: a boolean-to-integer conversion and a local type alias. ## What is the testing strategy for this PR? Unit tests cover resolved output types, explicit and session timezones, unsupported interval types, preservation of existing bounds and errors, and separation from direct bounds evaluation. Removing the fallback makes the new type-recovery tests fail. SQL tests cover redundant sort removal for nested datetime functions and a numeric UDF followed by a cast, including NULL input. Negative plan and result tests retain the required sorts for named timezone transitions and time-of-day arithmetic across midnight. Existing `date_bin` and `from_unixtime` plan tests remain unchanged. Validation: - Extended workspace test suite, including SQL logic tests. - `cargo fmt --all`. - `cargo clippy --all-targets --all-features -- -D warnings`. - `uv run ./dev/rust_lint.sh`. A local planning experiment against the base `main` commit used the same sorted 100-row `MemTable`, 20 warmups and 200 planning iterations, with three alternating runs of each build. The nested datetime and numeric-cast queries each removed one redundant sort and returned identical results. Median planning times were 679 → 645 µs and 585 → 553 µs; control and struct-return queries were essentially unchanged. These are small local measurements using the unoptimized `ci` profile, not production execution-speed claims. ## Are there any user-facing changes? Safe compositions of scalar functions can avoid redundant sorts without individual UDFs implementing bounds solely to preserve their output type. Existing ordering rules still determine whether a sort can be removed. No public API signatures or FFI structures change. Co-authored-by: Andrew Lamb <[email protected]>
1 parent d38f587 commit 62190be

7 files changed

Lines changed: 384 additions & 126 deletions

File tree

‎datafusion/expr/src/udf.rs‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -908,15 +908,15 @@ pub trait ScalarUDFImpl: Debug + DynEq + DynHash + Send + Sync + Any {
908908
/// # Output type
909909
///
910910
/// The default implementation returns an unbounded [`DataType::Null`]
911-
/// interval because the output type cannot be inferred generically.
912-
/// Implementations should override this method when they can determine the
913-
/// output type, even if they cannot compute precise bounds. Returning an
914-
/// unbounded interval with the correct type preserves information used by
915-
/// downstream physical property analysis.
916-
///
917-
/// If the output type depends on argument values, only report it when those
918-
/// values are known, such as from singleton input intervals. Otherwise,
919-
/// keep the output type unknown.
911+
/// interval because the output type cannot be inferred from input intervals
912+
/// alone. During physical property analysis, `ScalarFunctionExpr` uses its
913+
/// resolved return type to replace this default with a typed unbounded
914+
/// interval when supported. UDFs need not override this method solely to
915+
/// preserve their output type for that analysis.
916+
///
917+
/// Override this method to provide more precise bounds. Explicit bounds
918+
/// are preserved. The fallback does not apply to direct bounds evaluation
919+
/// by the constraint solver.
920920
fn evaluate_bounds(&self, _input: &[&Interval]) -> Result<Interval> {
921921
// We cannot assume the input datatype is the same of output type.
922922
Interval::make_unbounded(&DataType::Null)

‎datafusion/functions/src/datetime/date_bin.rs‎

Lines changed: 1 addition & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,6 @@ use datafusion_common::{
3838
Result, ScalarValue, exec_datafusion_err, exec_err, not_impl_err, plan_err,
3939
};
4040
use datafusion_expr::TypeSignature::Exact;
41-
use datafusion_expr::interval_arithmetic::Interval as ExprInterval;
4241
use datafusion_expr::sort_properties::{ExprProperties, SortProperties};
4342
use datafusion_expr::{
4443
ColumnarValue, Documentation, ScalarFunctionArgs, ScalarUDFImpl, Signature,
@@ -294,11 +293,6 @@ impl ScalarUDFImpl for DateBinFunc {
294293
}
295294
}
296295

297-
fn evaluate_bounds(&self, inputs: &[&ExprInterval]) -> Result<ExprInterval> {
298-
// DATE_BIN returns the same type as its source argument.
299-
ExprInterval::make_unbounded(&inputs[1].data_type())
300-
}
301-
302296
fn documentation(&self) -> Option<&Documentation> {
303297
self.doc()
304298
}
@@ -829,11 +823,10 @@ mod tests {
829823
use arrow::array::types::TimestampNanosecondType;
830824
use arrow::array::{Array, IntervalDayTimeArray, TimestampNanosecondArray};
831825
use arrow::compute::kernels::cast_utils::string_to_timestamp_nanos;
832-
use arrow::datatypes::{DataType, Field, FieldRef, IntervalUnit, TimeUnit};
826+
use arrow::datatypes::{DataType, Field, FieldRef, TimeUnit};
833827

834828
use arrow_buffer::{IntervalDayTime, IntervalMonthDayNano};
835829
use datafusion_common::{DataFusionError, ScalarValue};
836-
use datafusion_expr::interval_arithmetic::Interval as ExprInterval;
837830
use datafusion_expr::{ColumnarValue, ScalarFunctionArgs, ScalarUDFImpl};
838831

839832
use chrono::TimeDelta;
@@ -884,28 +877,6 @@ mod tests {
884877
);
885878
}
886879

887-
#[test]
888-
fn evaluate_bounds_preserves_source_type() {
889-
let stride =
890-
ExprInterval::make_unbounded(&DataType::Interval(IntervalUnit::DayTime))
891-
.unwrap();
892-
let source_types = [
893-
DataType::Timestamp(TimeUnit::Second, None),
894-
DataType::Timestamp(TimeUnit::Nanosecond, Some("UTC".into())),
895-
DataType::Time32(TimeUnit::Millisecond),
896-
DataType::Time64(TimeUnit::Microsecond),
897-
];
898-
let function = DateBinFunc::new();
899-
900-
for source_type in source_types {
901-
let source = ExprInterval::make_unbounded(&source_type).unwrap();
902-
let bounds = function.evaluate_bounds(&[&stride, &source]).unwrap();
903-
904-
assert!(bounds.is_unbounded());
905-
assert_eq!(bounds.data_type(), source_type);
906-
}
907-
}
908-
909880
#[test]
910881
fn test_date_bin() {
911882
let return_field = &Arc::new(Field::new(

‎datafusion/functions/src/datetime/from_unixtime.rs‎

Lines changed: 0 additions & 83 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,6 @@ use chrono::{DateTime, NaiveDateTime, Offset, TimeDelta, TimeZone};
2828
use datafusion_common::config::ConfigOptions;
2929
use datafusion_common::{Result, ScalarValue, exec_err, internal_err};
3030
use datafusion_expr::TypeSignature::Exact;
31-
use datafusion_expr::interval_arithmetic::Interval;
3231
use datafusion_expr::sort_properties::{ExprProperties, SortProperties};
3332
use datafusion_expr::{
3433
ColumnarValue, Documentation, ReturnFieldArgs, ScalarFunctionArgs, ScalarUDF,
@@ -220,22 +219,6 @@ impl ScalarUDFImpl for FromUnixtimeFunc {
220219
Ok(true)
221220
}
222221

223-
fn evaluate_bounds(&self, inputs: &[&Interval]) -> Result<Interval> {
224-
let data_type = match inputs {
225-
[_] => Timestamp(Second, self.timezone.clone()),
226-
// Literal properties encode a constant timezone as a singleton interval.
227-
[_, timezone] if timezone.lower() == timezone.upper() => {
228-
match timezone_from_scalar(timezone.lower()) {
229-
Some(timezone) => Timestamp(Second, Some(timezone)),
230-
None => DataType::Null,
231-
}
232-
}
233-
_ => DataType::Null,
234-
};
235-
236-
Interval::make_unbounded(&data_type)
237-
}
238-
239222
fn documentation(&self) -> Option<&Documentation> {
240223
self.doc()
241224
}
@@ -351,7 +334,6 @@ mod test {
351334
use datafusion_common::ScalarValue;
352335
use datafusion_common::ScalarValue::Int64;
353336
use datafusion_common::config::ConfigOptions;
354-
use datafusion_expr::interval_arithmetic::Interval;
355337
use datafusion_expr::{
356338
ColumnarValue, ReturnFieldArgs, ScalarFunctionArgs, ScalarUDFImpl,
357339
};
@@ -364,71 +346,6 @@ mod test {
364346
/// represent (`-262143-01-01T00:00:00`).
365347
const MIN_UTC_SECONDS: i64 = -8334601228800;
366348

367-
#[test]
368-
fn evaluate_bounds_preserves_output_type() {
369-
let seconds = Interval::make_unbounded(&DataType::Int64).unwrap();
370-
371-
let function = FromUnixtimeFunc::default();
372-
let bounds = function.evaluate_bounds(&[&seconds]).unwrap();
373-
assert!(bounds.is_unbounded());
374-
assert_eq!(bounds.data_type(), DataType::Timestamp(Second, None));
375-
376-
let mut options = ConfigOptions::default();
377-
options.execution.time_zone = Some("America/Denver".to_string());
378-
let function = FromUnixtimeFunc::new_with_config(&options);
379-
let bounds = function.evaluate_bounds(&[&seconds]).unwrap();
380-
assert_eq!(
381-
bounds.data_type(),
382-
DataType::Timestamp(Second, Some("America/Denver".into()))
383-
);
384-
385-
let timezone = ScalarValue::Utf8(Some("+08:00".to_string()));
386-
let timezone = Interval::try_new(timezone.clone(), timezone).unwrap();
387-
let bounds = function.evaluate_bounds(&[&seconds, &timezone]).unwrap();
388-
assert_eq!(
389-
bounds.data_type(),
390-
DataType::Timestamp(Second, Some("+08:00".into()))
391-
);
392-
}
393-
394-
#[test]
395-
fn evaluate_bounds_keeps_unknown_timezone_unknown() {
396-
let seconds = Interval::make_unbounded(&DataType::Int64).unwrap();
397-
let function = FromUnixtimeFunc::default();
398-
399-
let unknown = Interval::make_unbounded(&DataType::Utf8).unwrap();
400-
assert_eq!(
401-
function
402-
.evaluate_bounds(&[&seconds, &unknown])
403-
.unwrap()
404-
.data_type(),
405-
DataType::Null
406-
);
407-
408-
let varying = Interval::try_new(
409-
ScalarValue::Utf8(Some("+08:00".to_string())),
410-
ScalarValue::Utf8(Some("+09:00".to_string())),
411-
)
412-
.unwrap();
413-
assert_eq!(
414-
function
415-
.evaluate_bounds(&[&seconds, &varying])
416-
.unwrap()
417-
.data_type(),
418-
DataType::Null
419-
);
420-
421-
let empty = ScalarValue::Utf8(Some(String::new()));
422-
let empty = Interval::try_new(empty.clone(), empty).unwrap();
423-
assert_eq!(
424-
function
425-
.evaluate_bounds(&[&seconds, &empty])
426-
.unwrap()
427-
.data_type(),
428-
DataType::Null
429-
);
430-
}
431-
432349
/// Invoke the single argument form with `datafusion.execution.time_zone`
433350
/// set to `timezone`.
434351
fn from_unixtime_session_tz(

0 commit comments

Comments
 (0)