Repository navigation
fix(array): accept timestamps beyond jiff's range - #10953
Conversation
Timestamp scalars were validated against `jiff::Timestamp`'s range, which stops 26 hours short of the end of year 9999, so a value Arrow, Parquet and Spark allow, such as `9999-12-31T00:00:00Z`, could neither be used as a literal nor written: the file statistics built the min/max scalar with `Scalar::extension_ref` and panicked. Every `i64` count of the unit is a valid timestamp; jiff only bounds what `Display` can render, so drop the range check, fall back to the raw count when the calendar rendering is unavailable, and build the min/max scalars fallibly so a rejected extreme is an error rather than a panic. Signed-off-by: gitRasheed <[email protected]>
Now that timestamps beyond jiff's range can be written, `split_temporal` could see second and millisecond counts more than i32::MAX days from the epoch. `split_with_divisor` truncated the days to i32 with `as_()`, so such values silently wrapped when compressed: `i64::MAX` seconds round-tripped as 50027779063807. The compress comment still claimed jiff validation made the narrowing safe, the btrblocks TemporalScheme applied the encoding with no range check, and `try_extract_days_constant` used the same truncating split, so `dtp <= lit` with such a literal was rewritten to a wrapped days constant. Microsecond and nanosecond counts cannot overflow the days part. `timestamp::split` now errors when the days do not fit, so the days pushdown declines the constant and the compare kernel falls back to the canonical comparison. `split_temporal` checks the valid min/max of second and millisecond arrays and errors instead of wrapping, and the new `can_split_temporal` lets the btrblocks TemporalScheme skip such arrays so the storage array is compressed and the data round-trips exactly. The narrowing compare test built its fixture from `i64::MAX` seconds and relied on the wrap; it now uses a value above i32::MAX whose days fit. Signed-off-by: gitRasheed <[email protected]>
…hold the literal Compression narrows the days part, for example to u16 for dates before 2149. A midnight literal past that range, such as 9999-12-31, made the comparison push-down cast its days to the narrowed type and fail with "Cannot cast i32 to u16". The rule now declines, and the comparison runs on the full timestamps. Signed-off-by: gitRasheed <[email protected]>
Signed-off-by: gitRasheed <[email protected]>
Merging this PR will improve performance by 14.83%1 warning
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
## Summary Validates temporal dtypes and returns errors from fallible scalar and import paths for invalid metadata or out-of-range values. Keeps the current timestamp range limits. #10953 proposes a wider range, so the two approaches can be reviewed together. ## Changes Checks Date/Time storage and Timestamp metadata, uses fallible Span setters, and covers Arrow, DataFusion, Python, and JNI boundaries. Pinned formatting and five-crate Clippy pass, along with 6,412 Rust tests (1 skipped), 73 Python tests, and 38 Java tests. Python type checking, Ruff, and Spotless pass. ## API Changes Adds `Timestamp::try_new_with_tz` and rejects Timestamp `Days` and unresolved time zones at construction. Existing convenience constructors retain a documented panic contract. --------- Signed-off-by: Connor Tsui <[email protected]>
split_temporal ran a min_max pass over second and millisecond values before splitting them, which made the split benchmark about a third slower for seconds when no stats were cached. The split loops now OR together each value's days plus 2^31 and check the high bits once at the end, so the check stays branch-free. When the loop trips, an exact min_max over the valid values decides the error, since null slots can hold any value. Exact min/max stats cached on the extension array or its storage settle the question up front and the loops skip the check. Microsecond and nanosecond days always fit, so those loops never check. can_split_temporal reads the same cached stats before computing min_max. Signed-off-by: gitRasheed <[email protected]>
…range Signed-off-by: Connor Tsui <[email protected]> # Conflicts: # vortex-array/src/extension/datetime/timestamp.rs
Signed-off-by: Connor Tsui <[email protected]>
Signed-off-by: Connor Tsui <[email protected]>
|
Thanks for picking this up! Added three follow-up commits:
|
Signed-off-by: Connor Tsui <[email protected]>
|
Sorry for taking over this PR, it's just that I'm working on stuff that conflicts with this and I want to make sure we dont have to keep resolving merge conflicts |
The phase graph of the extension plan now shows each PR of phase 0, its state and the order constraints. Phase 1 depends on U4 and U5, and its strict reads wait for the spatial fix. Phase 0 holds #10953, the spatial fix and the importer change, and the structural part of X29 moves out of phase 6. The PR state lists #11013 as merged and adds #11067. In the type roadmap, N6 becomes the storage declaration (X25 to X30), with hard edges from N1 and into N3, and the report matches. Signed-off-by: Connor Tsui <[email protected]>
Keep the i32 split for ordinary timestamps and rebuild only the days component as i64 when a non-null value exceeds that range. Reuse the seconds and subseconds that the split already computed. Accept the full timestamp range in scalar reads and comparisons, and let TemporalScheme use DateTimeParts for extreme second and millisecond values. Cover adaptive width selection, null slots, serialization, scalar reads, comparisons, and compression round trips. Signed-off-by: Connor Tsui <[email protected]>
all good, go ahead 👍 |
…range Signed-off-by: Connor Tsui <[email protected]> # Conflicts: # encodings/datetime-parts/src/compute/compare.rs
U2, U3, U4, U6, U7 and U8 merged on 2026-11-02 in #11012, #11014 and #11016, and the DataFusion literal fix #11067 merged right after U2. So order constraint 1 holds, and phase 1 starts from develop. The graph shows these nodes as merged, and parts 2 and 4 name the PR that fixed each finding. #10953 now keeps i32 days and widens only the days part to i64 when a value does not fit. It has no review, and it conflicts with the DateTimeParts compare kernel that U2 changed. #11056 still waits on its request for changes. #10661 replaces the closed #10482 for file sort statistics. In the type roadmap, only the N1 sentence changes. Signed-off-by: Connor Tsui <[email protected]>
A maintainer commit merged develop into #10953 again on 2026-11-02, so the PR no longer conflicts and waits for a review. Part 4 records one local run of the split_temporal benchmark, because CodSpeed cannot compare the changed cases with develop. Signed-off-by: Connor Tsui <[email protected]>
|
we should make sure that |
Check the widened days with min_max instead of the original timestamps, reuse the split days for the overflow check, and make replace_time_zone reject values outside jiff's range even when the timezone is unchanged. Signed-off-by: gitRasheed <[email protected]>
|
On |
|
I will make a few more changes and then we should be good |
Timestamp parts supplied by files or try_new can exceed the original timestamp range. Use checked arithmetic and fallible scalar conversions so these inputs return errors. Leave combine_parts in the hot decode loop unchanged. Its overflow risk predates this PR. Signed-off-by: Connor Tsui <[email protected]>
…'s range Add conversion context and require the invalid-input test to reach that error. Execute an epoch-only control for both timezone targets so unrelated expression failures cannot satisfy the regression. Signed-off-by: Connor Tsui <[email protected]>
… code Restore the full-range timestamp fixture and cover all six comparisons against an extreme constant when the days child is i32. Remove stale benchmark statistic clearing and restore the temporal compression comment. Signed-off-by: Connor Tsui <[email protected]>
The default writer computes exact bounds, but CanonicalValidity rebuilds the arrays and drops those bounds before temporal compression. The writer probe supplied no cached bounds for either seconds or milliseconds, with or without nulls. Always check the computed days and remove the cache lookup, duplicate loop variants, and cache-only test and benchmark parameters. Keep the non-null day bounds check when widening is needed. Signed-off-by: Connor Tsui <[email protected]>
Add replace_time_zone conversion context with VortexError::with_context so callers retain the jiff error as the source. Require the existing range regression to preserve that source as well as the conversion message. Signed-off-by: Connor Tsui <[email protected]>
Construct the timestamp dtype directly instead of splitting a throwaway array. Cover Gte alongside the other five comparisons when an extreme midnight literal does not fit the narrowed days child. Signed-off-by: Connor Tsui <[email protected]>
Adds EXT-V10: the `Display` of `Date` panics on valid storage values that jiff cannot hold. X31 now says so, and the plan gives the finding a follow-up PR on `develop`. Parts 4 and 5 record the review of #10953 on 2026-11-03, the local review commits, the removal of the cached-bounds fast path, and the benchmark against `develop`. Signed-off-by: Connor Tsui <[email protected]>
Use 8,192 rows, matching one default writer row block, instead of 65,536. The larger inputs took 1.5 to 4.1 ms in CodSpeed CPU simulation. Reducing the work eightfold targets sub-millisecond cases while retaining every time unit and both extreme-value cases. Signed-off-by: Connor Tsui <[email protected]>
|
hmm actually I want to make some cleanup changes |
Document the component widths, signed remainders, and bias-and-OR days range check. Name the range-check state explicitly and constrain scalar part reads to integer types. Compare the bit check against checked integer conversion and assert the remainder bounds at both timestamp storage extremes. Signed-off-by: Connor Tsui <[email protected]>
There was a problem hiding this comment.
reviewers should probably start here
SR merges into SD, because a storage declaration without the exact read check reads an old form silently wrong. SD now needs SN1, which only tightens the spatial validators. SN2 takes every Arrow importer of a known type and builds on DP, which peels dictionary fields before the importers run. V10 and DS become their own PRs. Also record that #10953 is pushed at 7943b4e and waits for an approval, that the 12.9% millisecond slowdown was measured at d6e73bd, and that #11056 merged. Signed-off-by: Connor Tsui <[email protected]>
## Summary - Tracking Issue: #11114 `Date` accepts every `i32` day count and every `i64` millisecond count, but jiff only holds the years -9999 to 9999, so formatting a valid `Date` outside that range panicked. `Display` now prints the raw value and unit instead (e.g., `10000000days`), like the `Timestamp` fallback from #10953. ## Changes Tests cover the first date after 9999-12-31, day and millisecond counts that a jiff `Span` can't hold, and an ordinary date whose output doesn't change. Signed-off-by: Connor Tsui <[email protected]>
TS (#10953) merged on 2026-11-03, and SN1 (#11183) and V10 (#11187) merged on 2026-11-04. The dependency graph marks the three merged and moves its state date to 2026-11-04. It gives SD, DP, FD and U9 their open PRs (#11184, #11185, #11186 and #11188), and it drops #10408, which closed without a merge. The text of parts 4 and 5 and the Fix lines of EXT-V9, EXT-V10, EXT-N2 and EXT-S3 now record the same state. Also record that AX, DS and SN2 are implemented on branches without a PR, and add the approved one-pass design for the masked nulls of SN2. Part 2 and decision X31 also record the fixes: #10953 made every i64 legal, #11183 made the spatial validators reject nullable nested levels, #11187 gave the Date display its fallback, and #10408 closed. Signed-off-by: Connor Tsui <[email protected]>
Fixes #8860
Summary
Timestampstores ani64count of its unit, but its value check used jiff's smaller range. That made a formatting library decide which timestamps Vortex could store. For example,9999-12-31, a common “no end date” value, fits in microsecond storage but caused a panic when the file writer wrapped its min/max statistics as timestamp scalars.Accepts every
i64timestamp value, as Arrow does. jiff's limits apply when an operation needs a calendar representation, rather than when a value enters Vortex.Removing the value check alone is not enough: DateTimeParts previously narrowed days to
i32unconditionally. Extreme second and millisecond timestamps would therefore wrap to different values during compression. This PR fixes that path too.Changes
i32to preserve their existing buffer size and compression behavior. If a non-null day does not fit, rebuild only the days buffer asi64. Seconds and subseconds remain bounded within a day and second, so they do not need wider storage. Values in null slots cannot force widening.splitreconstruct safely, but parts supplied throughtry_newor read from a file do not necessarily have that guarantee. Checked arithmetic and fallible part conversions return errors for those inputs.u16days child can hold ordinary dates while being unable to hold a literal such as9999-12-31. Failing that optimization should leave the comparison to the normal execution path.replace_time_zonestill needs jiff to interpret wall time, so unsupported values return an error, including through the same-zone shortcut.The logical timestamp dtype and DateTimeParts metadata format stay the same. Tests cover extreme values, null payloads, serialization, compression, comparisons, and the Java and DataFusion scalar boundaries.
The current splitter trades some performance for full-range support while keeping ordinary days narrow. Millisecond splitting measured 12.9% slower than develop. This comparison does not isolate the range-check cost or establish that the slowdown is unavoidable.
Benchmark methodology and results
Local arm64, rustc 1.99.1, bench profile, 16 codegen units, no LTO, and 65,536 rows. Compared the PR's develop base (
d8e7467f8b) withd6e73bd40e, using the same harness and inputs. Two runs per revision used baseline/candidate/candidate/baseline order and--min-time 2.Times are medians of the two run medians, in microseconds. These measurements cover splitting, not file-write throughput.
Extreme inputs split successfully: 63.62 microseconds for seconds and 80.16 for milliseconds. Develop does not preserve their full-range days, so it has no valid baseline result.
A cached-bounds shortcut helped milliseconds in isolation, but the default-writer probe supplied no cached bounds after canonicalization. The shortcut was removed because the tested writer could not use it.
The benchmark inputs were subsequently reduced to 8,192 rows to target sub-millisecond CPU-simulation cases. The table retains the 65,536-row measurements at
d6e73bd40e. Native runs of the smaller cases passed, but their CPU-simulation times still need CI confirmation.Checked locally
d6e73bd40e: 218 DateTimeParts/BtrBlocks tests and 159 targeted array tests passed.1c5c33932b: 52 focused timezone-replacement and DateTimeParts rewrite tests passed.-D warnings.nightly-2026-09-10, and the local Rust and TOML formatting hooks passed.Previous PR description (for reference)
Summary
Fixes #8860
Timestamp scalars were range-checked against
jiff::Timestamp, which ends 26 hours before the end of year 9999, so writing9999-12-31, a common "no end date" value, panicked while building file statistics. As discussed on #8860, these values are accepted rather than rejected. This picks up #9361, which went stale.Changes
replace_time_zoneandTemporalMetadata::to_jiffconvert through jiff, so they now return errors for valid values outside jiff's range.Timestampno longer checks the stored value against jiff's range, andunpack_nativeno longer builds ajiff::Span, which also panicked for large values.Displayprints the raw count, such as253402214400000000µs, when jiff cannot render it.i32when they fit and rebuilds only the days buffer asi64for extreme second or millisecond values. It accepts everyi64timestamp without changing the file format, and the compressor no longer skips those arrays.ts = '9999-12-31'hit it.Checked locally:
cargo nextest runforvortex-datetime-partsandvortex-btrblocks(210 passed), clippy for the same crates with all targets, all features, and-D warnings, and formatting withnightly-2026-09-10. The changed Java test is left to CI because no Java runtime is installed locally.Adaptive split benchmarks
Local arm64, rustc 1.99.1, bench profile, 65,536 rows. Libraries at
ae53147ec3compared with6d701af0ac, using the same benchmark inputs. Two runs per version in baseline/candidate/candidate/baseline order, with at least two seconds per case. Times below are medians of the two run medians, in microseconds.Microseconds and nanoseconds changed by at most 0.5%. Extreme inputs now split successfully: seconds took 64.12 microseconds without cached min/max and 54.49 with them, and milliseconds took 80.74 and 71.49 respectively. For the uncached extreme cases, min/max are cleared before each timed call.
I used AI tools to help write and review this change.