Skip to content

Commit b550f59

Browse files
authored
fix: Validate execution.time_zone at SET time (apache#25788)
## Which issue does this PR close? - Part of apache#17498. This covers `datafusion.execution.time_zone` only. ## Rationale for this change `datafusion.execution.time_zone` is stored as a plain string, so an invalid value such as `SET TIME ZONE = 'Asia/Taipei2'` (or `'08:00'`, `'+08:00:00'`) is accepted and shown by `SHOW TIME ZONE`. The error only appears later, on the first `::TIMESTAMPTZ` cast. Worse, `current_date()` and `current_time()` parse the value with `.ok()`, so they silently ignore a bad time zone and return results as if none was set. PostgreSQL and Spark (`spark.sql.session.timeZone` is checked with `isValidTimezone` in `SQLConf`) both reject an invalid time zone when it is set. This picks up apache#23224 by @Probablism, which was closed as stale, and answers its open review questions: - **Normalization** (the `timestamps.slt` question): apache#23224 stored a parsed `Tz`, so `+08` came back as `+08:00` and `arrow_typeof(now())` changed. Here the value is checked with `Tz::from_str` but stored exactly as written, so the only change users see is that invalid values are rejected. `timestamps.slt` is unchanged, and `set_variable.slt` now checks that `SHOW TIME ZONE` returns `+08`. - **`AEST` in tests**: `AEST` was never a valid Arrow time zone. The tests that used it only passed the string through without parsing it, so they now use `Australia/Sydney`. ## What changes are included in this PR? - Add `ConfigTimeZone` in `datafusion/common/src/config.rs`. It can only be built through `FromStr`, which checks the value with Arrow's `Tz` parser and keeps the original string. It provides `as_str()` and `Display`. - Change `ExecutionOptions::time_zone` from `Option<String>` to `Option<ConfigTimeZone>`. Its `ConfigField` impl parses before assigning, following `DFParquetStatistics`, so a rejected value leaves the previous value (or `None`) in place, and `RESET` restores `None`. - Update the readers (SQL planner, `now`, `current_date`, `current_time`, `to_timestamp*`, `from_unixtime`, Spark `cast` / `date_trunc`) to use `as_str()`. - Remove `to_timestamp_invalid_execution_timezone_behavior` and `to_timestamp_formats_invalid_execution_timezone_behavior`, along with their now-unused helper. They put an invalid string straight into the config to test the error at invoke time, and that state can no longer be constructed. The unit test and slt cases below cover the rejection instead. ## What is the testing strategy for this PR? - `set_variable.slt`: the invalid values `+08:00:00`, `08:00`, `08` and `Asia/Taipei2` now fail on `SET` (through both `SET TIME ZONE` and `SET datafusion.execution.time_zone`), and `SHOW TIME ZONE` confirms the previous value is kept. It also checks that `+08` is shown as written. - `test_execution_time_zone_validation` in `datafusion-common`: valid values round-trip exactly; invalid values error and keep the previous value; an invalid update of an unset value leaves it unset; nested keys are rejected. - Existing time zone suites pass unchanged (`timestamps.slt`, `timestamps_timezone.slt`, `current_*_timezone.slt`, `to_timestamp_timezone.slt`, `from_unixtime_timezone.slt`, Spark cast/date_trunc). Commands run (all passing): - `cargo test -p datafusion-common --lib config::tests` (with and without `--features parquet`) - `cargo test --profile=ci --test sqllogictests` (all 523 files) - `cargo test -p datafusion-functions --lib datetime` - `cargo test -p datafusion-spark --lib function::` - `cargo test -p datafusion-sql --lib` - `cargo test -p datafusion --test user_defined_integration config_options` - `cargo test -p datafusion-ffi --features integration-tests --test ffi_udf` - `cargo test -p datafusion --test parquet_integration` - `cargo fmt --all -- --check` - `cargo clippy --all-targets --all-features -- -D warnings` ## Are there any user-facing changes? Yes. `SET TIME ZONE` / `SET datafusion.execution.time_zone` now fails immediately on an invalid time zone instead of failing on a later query, or being silently ignored by `current_date()` / `current_time()`. This changes the public field type `ExecutionOptions::time_zone` from `Option<String>` to `Option<ConfigTimeZone>`. Code that reads it can use `.as_ref().map(ConfigTimeZone::as_str)`, and code that sets it can use `Some("UTC".parse()?)`. Could a maintainer please add the `api change` label? I'm happy to add a note to the 56.0.0 upgrade guide if that's wanted. cc @Jefffrey (you reviewed apache#23224), @kumarUjjawal @alamb (reviewed the other apache#17498 slices)
1 parent 801b017 commit b550f59

14 files changed

Lines changed: 198 additions & 169 deletions

File tree

‎datafusion/common/src/config.rs‎

Lines changed: 114 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -943,7 +943,7 @@ config_namespace! {
943943
/// The default time zone
944944
///
945945
/// Some functions, e.g. `now` return timestamps in this time zone
946-
pub time_zone: Option<String>, default = None
946+
pub time_zone: Option<ConfigTimeZone>, default = None
947947

948948
/// Parquet options
949949
pub parquet: ParquetOptions, default = Default::default()
@@ -2013,6 +2013,76 @@ impl From<ConfigDurationFormat> for arrow::util::display::DurationFormat {
20132013
}
20142014
}
20152015

2016+
/// A time zone that is known to parse as an Arrow
2017+
/// [`Tz`](arrow::array::timezone::Tz), used for
2018+
/// [`ExecutionOptions::time_zone`].
2019+
///
2020+
/// The value is kept as it was written, so `+08` stays `+08` rather than
2021+
/// being normalized to `+08:00`.
2022+
#[derive(Debug, Clone, PartialEq, Eq)]
2023+
pub struct ConfigTimeZone(String);
2024+
2025+
impl ConfigTimeZone {
2026+
/// Returns the time zone as it was written.
2027+
pub fn as_str(&self) -> &str {
2028+
&self.0
2029+
}
2030+
}
2031+
2032+
impl FromStr for ConfigTimeZone {
2033+
type Err = DataFusionError;
2034+
2035+
fn from_str(s: &str) -> Result<Self, Self::Err> {
2036+
match s.parse::<arrow::array::timezone::Tz>() {
2037+
Ok(_) => Ok(Self(s.to_string())),
2038+
Err(_) => _config_err!(
2039+
"Invalid time zone: {s}. Valid values are UTC offsets such as +08:00 or IANA time zone names such as Asia/Taipei"
2040+
),
2041+
}
2042+
}
2043+
}
2044+
2045+
impl Display for ConfigTimeZone {
2046+
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
2047+
f.write_str(&self.0)
2048+
}
2049+
}
2050+
2051+
/// `ConfigField` for `Option<ConfigTimeZone>` parses before assigning so an
2052+
/// invalid value leaves the current setting unchanged.
2053+
impl ConfigField for Option<ConfigTimeZone> {
2054+
fn visit<V: Visit>(&self, v: &mut V, key: &str, description: &'static str) {
2055+
match self {
2056+
Some(time_zone) => v.some(key, time_zone, description),
2057+
None => v.none(key, description),
2058+
}
2059+
}
2060+
2061+
fn set(&mut self, key: &str, value: &str) -> Result<()> {
2062+
if !key.is_empty() {
2063+
return _config_err!(
2064+
"Config field is a scalar Option<ConfigTimeZone> and does not have nested field \"{}\"",
2065+
key
2066+
);
2067+
}
2068+
2069+
*self = Some(ConfigTimeZone::from_str(value)?);
2070+
Ok(())
2071+
}
2072+
2073+
fn reset(&mut self, key: &str) -> Result<()> {
2074+
if key.is_empty() {
2075+
*self = None;
2076+
Ok(())
2077+
} else {
2078+
_config_err!(
2079+
"Config field is a scalar Option<ConfigTimeZone> and does not have nested field \"{}\"",
2080+
key
2081+
)
2082+
}
2083+
}
2084+
}
2085+
20162086
config_namespace! {
20172087
/// Options controlling the format of output when printing record batches
20182088
/// Copies [`arrow::util::display::FormatOptions`]
@@ -4061,7 +4131,6 @@ impl Display for OutputFormat {
40614131

40624132
#[cfg(test)]
40634133
mod tests {
4064-
#[cfg(feature = "parquet")]
40654134
use crate::assert_contains;
40664135
use crate::config::TableParquetOptions;
40674136
use crate::config::{
@@ -4694,6 +4763,49 @@ mod tests {
46944763
assert_eq!(scalar, DFParquetStatistics::Page);
46954764
}
46964765

4766+
#[test]
4767+
fn test_execution_time_zone_validation() {
4768+
use crate::config::ConfigOptions;
4769+
4770+
const KEY: &str = "datafusion.execution.time_zone";
4771+
let mut config = ConfigOptions::default();
4772+
assert_eq!(config.execution.time_zone, None);
4773+
4774+
// Valid values are kept exactly as written.
4775+
for value in ["+08:00", "-08:00", "+0800", "+08", "UTC", "Asia/Taipei"] {
4776+
config.set(KEY, value).unwrap();
4777+
assert_eq!(
4778+
config.execution.time_zone.as_ref().map(|tz| tz.as_str()),
4779+
Some(value)
4780+
);
4781+
}
4782+
4783+
// A rejected value leaves the previous one in place.
4784+
for value in ["+08:00:00", "08:00", "08", "Asia/Taipei2", "AEST", ""] {
4785+
let err = config.set(KEY, value).unwrap_err();
4786+
assert_contains!(err.to_string(), format!("Invalid time zone: {value}."));
4787+
assert_eq!(
4788+
config.execution.time_zone.as_ref().map(|tz| tz.as_str()),
4789+
Some("Asia/Taipei")
4790+
);
4791+
}
4792+
4793+
// An invalid update of an unset value leaves it unset.
4794+
config.reset(KEY).unwrap();
4795+
assert_eq!(config.execution.time_zone, None);
4796+
assert!(config.set(KEY, "Asia/Taipei2").is_err());
4797+
assert_eq!(config.execution.time_zone, None);
4798+
4799+
// The option has no nested fields.
4800+
config.set(KEY, "UTC").unwrap();
4801+
assert!(config.set(&format!("{KEY}.typo"), "UTC").is_err());
4802+
assert!(config.reset(&format!("{KEY}.typo")).is_err());
4803+
assert_eq!(
4804+
config.execution.time_zone.as_ref().map(|tz| tz.as_str()),
4805+
Some("UTC")
4806+
);
4807+
}
4808+
46974809
#[cfg(feature = "parquet")]
46984810
#[test]
46994811
fn test_parquet_compression_validation() {

‎datafusion/core/tests/user_defined/user_defined_scalar_functions.rs‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1965,7 +1965,12 @@ async fn test_config_options_work_for_scalar_func() -> Result<()> {
19651965
}
19661966

19671967
fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
1968-
let tz = args.config_options.execution.time_zone.clone();
1968+
let tz = args
1969+
.config_options
1970+
.execution
1971+
.time_zone
1972+
.as_ref()
1973+
.map(ToString::to_string);
19691974
Ok(ColumnarValue::Scalar(ScalarValue::from(tz)))
19701975
}
19711976
}
@@ -1975,7 +1980,7 @@ async fn test_config_options_work_for_scalar_func() -> Result<()> {
19751980
});
19761981

19771982
let mut config = SessionConfig::new();
1978-
config.options_mut().execution.time_zone = Some("AEST".into());
1983+
config.options_mut().execution.time_zone = Some("Australia/Sydney".parse().unwrap());
19791984

19801985
let ctx = SessionContext::new_with_config(config);
19811986

@@ -1988,7 +1993,7 @@ async fn test_config_options_work_for_scalar_func() -> Result<()> {
19881993
let expected_schema = Schema::new(vec![Field::new("a", DataType::Utf8, false)]);
19891994
let expected = RecordBatch::try_new(
19901995
SchemaRef::from(expected_schema),
1991-
vec![create_array!(Utf8, ["AEST"])],
1996+
vec![create_array!(Utf8, ["Australia/Sydney"])],
19921997
)?;
19931998

19941999
assert_eq!(expected, actual[0]);

‎datafusion/ffi/src/tests/udf_udaf_udwf.rs‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -102,7 +102,12 @@ impl ScalarUDFImpl for TimeZoneUDF {
102102
&self,
103103
args: ScalarFunctionArgs,
104104
) -> datafusion_common::Result<ColumnarValue> {
105-
let tz = args.config_options.execution.time_zone.clone();
105+
let tz = args
106+
.config_options
107+
.execution
108+
.time_zone
109+
.as_ref()
110+
.map(ToString::to_string);
106111
Ok(ColumnarValue::Scalar(ScalarValue::from(tz)))
107112
}
108113

‎datafusion/ffi/tests/ffi_udf.rs‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -141,7 +141,8 @@ mod tests {
141141
assert!(result[0].column(0).as_string::<i32>().is_null(0));
142142

143143
let mut config = SessionConfig::new();
144-
config.options_mut().execution.time_zone = Some("AEST".into());
144+
config.options_mut().execution.time_zone =
145+
Some("Australia/Sydney".parse().unwrap());
145146

146147
let ctx = SessionContext::new_with_config(config);
147148

@@ -154,7 +155,7 @@ mod tests {
154155
assert_eq!(result.len(), 1);
155156
assert!(!result[0].column(0).as_string::<i32>().is_null(0));
156157
let result = result[0].column(0).as_string::<i32>().value(0);
157-
assert_eq!(result, "AEST");
158+
assert_eq!(result, "Australia/Sydney");
158159

159160
Ok(())
160161
}
@@ -175,7 +176,7 @@ mod tests {
175176
);
176177

177178
let mut options = ConfigOptions::default();
178-
options.execution.time_zone = Some("AEST".into());
179+
options.execution.time_zone = Some("Australia/Sydney".parse().unwrap());
179180

180181
let updated = foreign_udf
181182
.with_updated_config(&options)

‎datafusion/functions/benches/datetime_expressions/to_timestamp.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -113,7 +113,7 @@ fn criterion_benchmark(c: &mut Criterion) {
113113
let arg_field = Field::new("a", DataType::Utf8, false).into();
114114
let arg_fields = vec![arg_field];
115115
let mut options = ConfigOptions::default();
116-
options.execution.time_zone = Some("UTC".into());
116+
options.execution.time_zone = Some("UTC".parse().unwrap());
117117
let config_options = Arc::new(options);
118118

119119
let to_timestamp_udf = to_timestamp(config_options.as_ref());

‎datafusion/functions/src/datetime/current_date.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -119,7 +119,7 @@ impl ScalarUDFImpl for CurrentDateFunc {
119119
.execution
120120
.time_zone
121121
.as_ref()
122-
.and_then(|tz| tz.parse::<Tz>().ok())
122+
.and_then(|tz| tz.as_str().parse::<Tz>().ok())
123123
.map_or_else(
124124
|| datetime_to_days(&now_ts),
125125
|tz| {

‎datafusion/functions/src/datetime/current_time.rs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -116,7 +116,7 @@ impl ScalarUDFImpl for CurrentTimeFunc {
116116
.execution
117117
.time_zone
118118
.as_ref()
119-
.and_then(|tz| tz.parse::<Tz>().ok())
119+
.and_then(|tz| tz.as_str().parse::<Tz>().ok())
120120
.map_or_else(
121121
|| datetime_to_time_nanos(&now_ts),
122122
|tz| {
@@ -158,7 +158,7 @@ mod tests {
158158
config.execution.time_zone = if tz.is_empty() {
159159
None
160160
} else {
161-
Some(tz.to_string())
161+
Some(tz.parse().unwrap())
162162
};
163163
let schema = Arc::new(DFSchema::empty());
164164
SimplifyContext::builder()

‎datafusion/functions/src/datetime/from_unixtime.rs‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -353,7 +353,7 @@ mod test {
353353
timezone: Option<&str>,
354354
) -> Result<ColumnarValue> {
355355
let mut options = ConfigOptions::default();
356-
options.execution.time_zone = timezone.map(str::to_string);
356+
options.execution.time_zone = timezone.map(|tz| tz.parse().unwrap());
357357
let func = FromUnixtimeFunc::new_with_config(&options);
358358

359359
let arg_field: FieldRef = Field::new("a", DataType::Int64, true).into();
@@ -560,7 +560,7 @@ mod test {
560560
#[test]
561561
fn test_session_timezone_is_used_without_explicit_timezone() {
562562
let mut options = ConfigOptions::default();
563-
options.execution.time_zone = Some("America/Denver".to_string());
563+
options.execution.time_zone = Some("America/Denver".parse().unwrap());
564564

565565
let func = FromUnixtimeFunc::new_with_config(&options);
566566

@@ -599,7 +599,7 @@ mod test {
599599
#[test]
600600
fn test_explicit_timezone_overrides_session_timezone() {
601601
let mut options = ConfigOptions::default();
602-
options.execution.time_zone = Some("America/Denver".to_string());
602+
options.execution.time_zone = Some("America/Denver".parse().unwrap());
603603

604604
let func = FromUnixtimeFunc::new_with_config(&options);
605605

0 commit comments

Comments
 (0)