Skip to content

Commit 8896ab2

Browse files
fix(array): accept timestamps beyond jiff's range (vortex-data#10953)
Fixes vortex-data#8860 ## Summary `Timestamp` stores an `i64` count of its unit, but its value check used jiff's smaller range. That made a formatting library decide which timestamps Vortex could store. For example, `9999-12-31`, a common “no end date” value, fits in microsecond storage but caused a panic when the file writer wrapped its min/max statistics as timestamp scalars. Accepts every `i64` timestamp value, as Arrow does. jiff's limits apply when an operation needs a calendar representation, rather than when a value enters Vortex. Removing the value check alone is not enough: DateTimeParts previously narrowed days to `i32` unconditionally. Extreme second and millisecond timestamps would therefore wrap to different values during compression. This PR fixes that path too. ## Changes - Keep ordinary DateTimeParts days in `i32` to preserve their existing buffer size and compression behavior. If a non-null day does not fit, rebuild only the days buffer as `i64`. Seconds and subseconds remain bounded within a day and second, so they do not need wider storage. Values in null slots cannot force widening. - Make scalar reconstruction fallible. Parts produced by `split` reconstruct safely, but parts supplied through `try_new` or read from a file do not necessarily have that guarantee. Checked arithmetic and fallible part conversions return errors for those inputs. - Let the midnight comparison rewrite decline literals outside the narrowed days type. A compressed `u16` days child can hold ordinary dates while being unable to hold a literal such as `9999-12-31`. Failing that optimization should leave the comparison to the normal execution path. - Keep formatting safe for every accepted timestamp by printing the raw count when jiff cannot render it. `replace_time_zone` still needs jiff to interpret wall time, so unsupported values return an error, including through the same-zone shortcut. The logical timestamp dtype and DateTimeParts metadata format stay the same. Tests cover extreme values, null payloads, serialization, compression, comparisons, and the Java and DataFusion scalar boundaries. The current splitter trades some performance for full-range support while keeping ordinary days narrow. Millisecond splitting measured 12.9% slower than develop. This comparison does not isolate the range-check cost or establish that the slowdown is unavoidable. --------- Signed-off-by: gitRasheed <[email protected]> Signed-off-by: Connor Tsui <[email protected]> Co-authored-by: Connor Tsui <[email protected]>
1 parent 0c84f00 commit 8896ab2

13 files changed

Lines changed: 775 additions & 128 deletions

File tree

‎encodings/datetime-parts/benches/split_temporal.rs‎

Lines changed: 26 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -33,10 +33,10 @@ static SESSION: LazyLock<VortexSession> =
3333
LazyLock::new(|| VortexSession::empty().with::<KernelSession>());
3434

3535
const BENCH_ARGS: &[(usize, TimeUnit)] = &[
36-
(65_536, TimeUnit::Seconds),
37-
(65_536, TimeUnit::Milliseconds),
38-
(65_536, TimeUnit::Microseconds),
39-
(65_536, TimeUnit::Nanoseconds),
36+
(8_192, TimeUnit::Seconds),
37+
(8_192, TimeUnit::Milliseconds),
38+
(8_192, TimeUnit::Microseconds),
39+
(8_192, TimeUnit::Nanoseconds),
4040
];
4141

4242
#[divan::bench(args = BENCH_ARGS)]
@@ -63,3 +63,25 @@ fn split(bencher: Bencher, args: (usize, TimeUnit)) {
6363
.with_inputs(|| (array.clone(), SESSION.create_execution_ctx()))
6464
.bench_values(|(array, mut ctx)| split_temporal(array, &mut ctx).unwrap())
6565
}
66+
67+
const EXTREME_ARGS: &[(usize, TimeUnit)] =
68+
&[(8_192, TimeUnit::Seconds), (8_192, TimeUnit::Milliseconds)];
69+
70+
#[divan::bench(args = EXTREME_ARGS)]
71+
fn split_extreme(bencher: Bencher, args: (usize, TimeUnit)) {
72+
let (n, unit) = args;
73+
let timestamps = Buffer::from_iter((0..i64::try_from(n).unwrap()).map(|i| match i % 3 {
74+
0 => i64::MAX - i,
75+
1 => i64::MIN + i,
76+
_ => 1_700_000_000 + i,
77+
}));
78+
let array = TemporalArray::new_timestamp(
79+
PrimitiveArray::new(timestamps, Validity::NonNullable).into_array(),
80+
unit,
81+
Some("UTC".into()),
82+
);
83+
84+
bencher
85+
.with_inputs(|| (array.clone(), SESSION.create_execution_ctx()))
86+
.bench_values(|(array, mut ctx)| split_temporal(array, &mut ctx).unwrap())
87+
}

‎encodings/datetime-parts/src/canonical.rs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -128,8 +128,8 @@ impl TimePart {
128128
));
129129
}
130130

131-
// `split_temporal` produces `i32` parts, so this cast is normally a no-op. Other widths
132-
// are cast up front rather than instantiating the fused kernel for every width
131+
// `split_temporal` produces `i32` seconds and subseconds, so this cast is normally a no-op.
132+
// Other widths are cast up front rather than instantiating the fused kernel for every width
133133
// combination.
134134
let values = part
135135
.cast(DType::Primitive(PType::I32, Nullability::NonNullable))?

0 commit comments

Comments
 (0)