You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
COPY ... PARTITIONED BY a Date32 / Date64 column panics on out-of-range values #24907
compute_partition_keys_by_row in datafusion/datasource/src/write/demux.rs renders Date32 and Date64 partition values as yyyy-mm-dd path segments with unchecked arithmetic and an unwrap:
Any value that does not fit a chrono::NaiveDate panics the writer task.
To Reproduce
COPY (VALUES (arrow_cast(2147483647, 'Date32'))) TO 'out1/' STORED AS parquet PARTITIONED BY (column1);
-- thread 'tokio-rt-worker' panicked at datafusion/datasource/src/write/demux.rs:427:25:-- attempt to add with overflow
COPY (VALUES (arrow_cast(-2147483648, 'Date32'))) TO 'out2/' STORED AS parquet PARTITIONED BY (column1);
-- demux.rs:429:22: called `Option::unwrap()` on a `None` value
COPY (VALUES (arrow_cast(9223372036854775807, 'Date64'))) TO 'out3/' STORED AS parquet PARTITIONED BY (column1);
-- demux.rs:443:22: called `Option::unwrap()` on a `None` value
Expected behavior
An execution error naming the value that cannot be rendered as a date, instead of a panic.
Additional context
The Date64 branch also truncates toward zero (/ 86_400_000), so a value shortly before the epoch (e.g. -1 ms) is placed in the 1970-01-01 partition rather than 1969-12-31.
Found while running a corpus of extreme-value literals against a debug build of datafusion-cli.
Activity