Repository navigation
Commit 8896ab2
fix(array): accept timestamps beyond jiff's range (vortex-data#10953)
Fixes vortex-data#8860
## Summary
`Timestamp` stores an `i64` count of its unit, but its value check used
jiff's smaller range. That made a formatting library decide which
timestamps Vortex could store. For example, `9999-12-31`, a common “no
end date” value, fits in microsecond storage but caused a panic when the
file writer wrapped its min/max statistics as timestamp scalars.
Accepts every `i64` timestamp value, as Arrow does. jiff's limits apply
when an operation needs a calendar representation, rather than when a
value enters Vortex.
Removing the value check alone is not enough: DateTimeParts previously
narrowed days to `i32` unconditionally. Extreme second and millisecond
timestamps would therefore wrap to different values during compression.
This PR fixes that path too.
## Changes
- Keep ordinary DateTimeParts days in `i32` to preserve their existing
buffer size and compression behavior. If a non-null day does not fit,
rebuild only the days buffer as `i64`. Seconds and subseconds remain
bounded within a day and second, so they do not need wider storage.
Values in null slots cannot force widening.
- Make scalar reconstruction fallible. Parts produced by `split`
reconstruct safely, but parts supplied through `try_new` or read from a
file do not necessarily have that guarantee. Checked arithmetic and
fallible part conversions return errors for those inputs.
- Let the midnight comparison rewrite decline literals outside the
narrowed days type. A compressed `u16` days child can hold ordinary
dates while being unable to hold a literal such as `9999-12-31`. Failing
that optimization should leave the comparison to the normal execution
path.
- Keep formatting safe for every accepted timestamp by printing the raw
count when jiff cannot render it. `replace_time_zone` still needs jiff
to interpret wall time, so unsupported values return an error, including
through the same-zone shortcut.
The logical timestamp dtype and DateTimeParts metadata format stay the
same. Tests cover extreme values, null payloads, serialization,
compression, comparisons, and the Java and DataFusion scalar boundaries.
The current splitter trades some performance for full-range support
while keeping ordinary days narrow. Millisecond splitting measured 12.9%
slower than develop. This comparison does not isolate the range-check
cost or establish that the slowdown is unavoidable.
---------
Signed-off-by: gitRasheed <[email protected]>
Signed-off-by: Connor Tsui <[email protected]>
Co-authored-by: Connor Tsui <[email protected]>1 parent 0c84f00 commit 8896ab2
13 files changed
Lines changed: 775 additions & 128 deletions
File tree
- encodings/datetime-parts
- benches
- src
- compute
- java/vortex-jni/src/test/java/dev/vortex/api
- vortex-array/src
- aggregate_fn/fns/min_max
- extension/datetime
- scalar_fn/fns
- vortex-btrblocks/src
- vortex-datafusion/src/convert
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
36 | | - | |
37 | | - | |
38 | | - | |
39 | | - | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
| |||
63 | 63 | | |
64 | 64 | | |
65 | 65 | | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
128 | 128 | | |
129 | 129 | | |
130 | 130 | | |
131 | | - | |
132 | | - | |
| 131 | + | |
| 132 | + | |
133 | 133 | | |
134 | 134 | | |
135 | 135 | | |
| |||
0 commit comments