Skip to content

Commit 9547b09

Browse files
kumarUjjawalclaude
andauthored
fix: use session timezone for timestamp subtraction and comparison (apache#25094)
## Which issue does this PR close? - Closes apache#13212. ## Rationale for this change Mixed timezone-aware and timezone-naive timestamp operations ignore the configured session timezone when they interpret the naive value. For example, use midnight on `2026-07-01` in `America/New_York` and a naive midnight timestamp on the same date. Set the session timezone to `+08:00`. They currently compare equal and subtract to zero. The naive value represents midnight in the session timezone, so the values must compare unequal and differ by twelve hours. ## What changes are included in this PR? This PR: - Casts both operands of a mixed timestamp pair to the session timezone and a common timestamp precision. - Uses the same rule for subtraction and comparisons, so `a = b` and `a - b = 0` agree. - Applies comparison coercion to binary comparisons, `IN`, `BETWEEN`, simple `CASE`, `IN` subqueries, and `ANY`/`ALL`. - Applies comparison coercion recursively through dictionary, run-end encoded, list, struct, and map types. - Passes the session timezone through nested subqueries and `ExprSimplifier::coerce`, including the `SessionContext::create_physical_expr` path. The coerced timestamps use the session timezone. Casting the aware operand preserves its instant. Without a session timezone, both comparison and subtraction retain current `main` behavior and use the aware operand's timezone. The existing DST-boundary limitation in apache#25084 remains. The implicit cast can encounter that limitation for ambiguous or nonexistent local times. ## What is the testing strategy for this PR? Unit tests, SQL logic tests, and a direct physical-expression test cover: - Both operand orders, equal and different precisions, and configured and unset session timezones. - Comparisons, subtraction, nested subqueries, and encoded or nested types. - Different session and operand timezones, including EST and EDT offsets. - The configuration description in `information_schema`. The merge preserves upstream's regression tests for subtraction without a session timezone at every timestamp precision. ## Are there any user-facing changes? Yes: - Mixed timestamp subtraction and comparison results change when `datafusion.execution.time_zone` is set. - New public APIs: `BinaryTypeCoercer::with_session_time_zone` and `comparison_coercion_with_session_timezone`. - Deprecated helpers: `get_coerce_type_for_list` and `get_coerce_type_for_case_when`. Both remain available with their existing behavior. The configuration documentation and the 56.0.0 upgrade guide describe these changes. --------- Co-authored-by: Claude Opus 5 <[email protected]>
1 parent 6047791 commit 9547b09

14 files changed

Lines changed: 768 additions & 98 deletions

File tree

‎datafusion/common/src/config.rs‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -943,7 +943,9 @@ config_namespace! {
943943

944944
/// The default time zone
945945
///
946-
/// Some functions, e.g. `now` return timestamps in this time zone
946+
/// Some functions, e.g. `now`, return timestamps in this time zone.
947+
/// This is also used to interpret timezone-naive timestamps in
948+
/// comparisons and subtraction with timezone-aware timestamps.
947949
pub time_zone: Option<ConfigTimeZone>, default = None
948950

949951
/// Parquet options

‎datafusion/core/tests/expr_api/mod.rs‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717

1818
use arrow::array::{
1919
ArrayRef, Int64Array, RecordBatch, StringArray, StructArray,
20+
TimestampNanosecondArray,
2021
builder::{ListBuilder, StringBuilder},
2122
};
2223
use arrow::datatypes::{DataType, Field};
@@ -364,6 +365,43 @@ async fn test_create_physical_expr_coercion() {
364365
);
365366
}
366367

368+
#[test]
369+
fn test_create_physical_expr_timestamp_subtraction_uses_session_timezone() {
370+
const NANOS_PER_SECOND: i64 = 1_000_000_000;
371+
// 2024-11-01 00:00:00 in America/New_York (04:00 UTC).
372+
let timestamp_tz: ArrayRef = Arc::new(
373+
TimestampNanosecondArray::from(vec![1_730_433_600 * NANOS_PER_SECOND])
374+
.with_timezone("America/New_York"),
375+
);
376+
// The same wall-clock time without a timezone.
377+
let timestamp: ArrayRef = Arc::new(TimestampNanosecondArray::from(vec![
378+
1_730_419_200 * NANOS_PER_SECOND,
379+
]));
380+
let batch = RecordBatch::try_from_iter(vec![
381+
("timestamp_tz", timestamp_tz),
382+
("timestamp", timestamp),
383+
])
384+
.unwrap();
385+
let df_schema = DFSchema::try_from(batch.schema()).unwrap();
386+
let mut config = SessionConfig::new();
387+
config.options_mut().execution.time_zone = Some("+08:00".parse().unwrap());
388+
let ctx = SessionContext::new_with_config(config);
389+
390+
let physical_expr = ctx
391+
.create_physical_expr(col("timestamp_tz") - col("timestamp"), &df_schema)
392+
.unwrap();
393+
let result = physical_expr
394+
.evaluate(&batch)
395+
.unwrap()
396+
.into_array(batch.num_rows())
397+
.unwrap();
398+
399+
assert_eq!(
400+
ScalarValue::try_from_array(&result, 0).unwrap(),
401+
ScalarValue::DurationNanosecond(Some(12 * 60 * 60 * NANOS_PER_SECOND)),
402+
);
403+
}
404+
367405
/// Evaluates the specified expr as an aggregate and compares the result to the
368406
/// expected result.
369407
async fn evaluate_agg_test(expr: Expr, expected_lines: Vec<&str>) {

0 commit comments

Comments
 (0)