Skip to content

fix(array): accept timestamps beyond jiff's range - #10953

Merged
robert3005 merged 20 commits into
vortex-data:developfrom
gitRasheed:timestamp-full-range
Nov 3, 2026
Merged

robert3005 merged 20 commits into
vortex-data:developfrom
gitRasheed:timestamp-full-range

Conversation

@gitRasheed

@gitRasheed gitRasheed commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #8860

Summary

Timestamp stores an i64 count of its unit, but its value check used jiff's smaller range. That made a formatting library decide which timestamps Vortex could store. For example, 9999-12-31, a common “no end date” value, fits in microsecond storage but caused a panic when the file writer wrapped its min/max statistics as timestamp scalars.

Accepts every i64 timestamp value, as Arrow does. jiff's limits apply when an operation needs a calendar representation, rather than when a value enters Vortex.

Removing the value check alone is not enough: DateTimeParts previously narrowed days to i32 unconditionally. Extreme second and millisecond timestamps would therefore wrap to different values during compression. This PR fixes that path too.

Changes

  • Keep ordinary DateTimeParts days in i32 to preserve their existing buffer size and compression behavior. If a non-null day does not fit, rebuild only the days buffer as i64. Seconds and subseconds remain bounded within a day and second, so they do not need wider storage. Values in null slots cannot force widening.
  • Make scalar reconstruction fallible. Parts produced by split reconstruct safely, but parts supplied through try_new or read from a file do not necessarily have that guarantee. Checked arithmetic and fallible part conversions return errors for those inputs.
  • Let the midnight comparison rewrite decline literals outside the narrowed days type. A compressed u16 days child can hold ordinary dates while being unable to hold a literal such as 9999-12-31. Failing that optimization should leave the comparison to the normal execution path.
  • Keep formatting safe for every accepted timestamp by printing the raw count when jiff cannot render it. replace_time_zone still needs jiff to interpret wall time, so unsupported values return an error, including through the same-zone shortcut.

The logical timestamp dtype and DateTimeParts metadata format stay the same. Tests cover extreme values, null payloads, serialization, compression, comparisons, and the Java and DataFusion scalar boundaries.

The current splitter trades some performance for full-range support while keeping ordinary days narrow. Millisecond splitting measured 12.9% slower than develop. This comparison does not isolate the range-check cost or establish that the slowdown is unavoidable.

Benchmark methodology and results

Local arm64, rustc 1.99.1, bench profile, 16 codegen units, no LTO, and 65,536 rows. Compared the PR's develop base (d8e7467f8b) with d6e73bd40e, using the same harness and inputs. Two runs per revision used baseline/candidate/candidate/baseline order and --min-time 2.

Times are medians of the two run medians, in microseconds. These measurements cover splitting, not file-write throughput.

Unit Develop Candidate Change
Seconds 19.62 18.89 -3.7%
Milliseconds 30.66 34.60 +12.9%
Microseconds 30.73 30.68 -0.1%
Nanoseconds 30.70 30.66 -0.1%

Extreme inputs split successfully: 63.62 microseconds for seconds and 80.16 for milliseconds. Develop does not preserve their full-range days, so it has no valid baseline result.

A cached-bounds shortcut helped milliseconds in isolation, but the default-writer probe supplied no cached bounds after canonicalization. The shortcut was removed because the tested writer could not use it.

The benchmark inputs were subsequently reduced to 8,192 rows to target sub-millisecond CPU-simulation cases. The table retains the 65,536-row measurements at d6e73bd40e. Native runs of the smaller cases passed, but their CPU-simulation times still need CI confirmation.

Checked locally
  • At d6e73bd40e: 218 DateTimeParts/BtrBlocks tests and 159 targeted array tests passed.
  • At 1c5c33932b: 52 focused timezone-replacement and DateTimeParts rewrite tests passed.
  • Scoped Clippy passed with all targets, all features, and -D warnings.
  • Formatting passed with nightly-2026-09-10, and the local Rust and TOML formatting hooks passed.
  • Java, Python, CUDA, doctests, and workspace-wide tests were not run.
Previous PR description (for reference)

Summary

Fixes #8860

Timestamp scalars were range-checked against jiff::Timestamp, which ends 26 hours before the end of year 9999, so writing 9999-12-31, a common "no end date" value, panicked while building file statistics. As discussed on #8860, these values are accepted rather than rejected. This picks up #9361, which went stale.

Changes

  • replace_time_zone and TemporalMetadata::to_jiff convert through jiff, so they now return errors for valid values outside jiff's range.
  • Timestamp no longer checks the stored value against jiff's range, and unpack_native no longer builds a jiff::Span, which also panicked for large values. Display prints the raw count, such as 253402214400000000µs, when jiff cannot render it.
  • DateTimeParts keeps days in i32 when they fit and rebuilds only the days buffer as i64 for extreme second or millisecond values. It accepts every i64 timestamp without changing the file format, and the compressor no longer skips those arrays.
  • A separate bug on develop that this made easy to hit: compression narrows the days part, to u16 for dates before 2149, and comparing it with a midnight literal past that range failed with "Cannot cast i32 to u16" in the days pushdown. The pushdown now declines. This affected in-range literals such as 2200-01-01 too, and with this change ts = '9999-12-31' hit it.

Checked locally: cargo nextest run for vortex-datetime-parts and vortex-btrblocks (210 passed), clippy for the same crates with all targets, all features, and -D warnings, and formatting with nightly-2026-09-10. The changed Java test is left to CI because no Java runtime is installed locally.

Adaptive split benchmarks

Local arm64, rustc 1.99.1, bench profile, 65,536 rows. Libraries at ae53147ec3 compared with 6d701af0ac, using the same benchmark inputs. Two runs per version in baseline/candidate/candidate/baseline order, with at least two seconds per case. Times below are medians of the two run medians, in microseconds.

Unit Min/max cached Before Adaptive
Seconds No 19.06 19.16
Seconds Yes 19.74 19.73
Milliseconds No 34.72 34.72
Milliseconds Yes 30.43 30.95

Microseconds and nanoseconds changed by at most 0.5%. Extreme inputs now split successfully: seconds took 64.12 microseconds without cached min/max and 54.49 with them, and milliseconds took 80.74 and 71.49 respectively. For the uncached extreme cases, min/max are cleared before each timed call.

I used AI tools to help write and review this change.

Timestamp scalars were validated against `jiff::Timestamp`'s range, which stops 26
hours short of the end of year 9999, so a value Arrow, Parquet and Spark allow, such as
`9999-12-31T00:00:00Z`, could neither be used as a literal nor written: the file
statistics built the min/max scalar with `Scalar::extension_ref` and panicked. Every
`i64` count of the unit is a valid timestamp; jiff only bounds what `Display` can render,
so drop the range check, fall back to the raw count when the calendar rendering is
unavailable, and build the min/max scalars fallibly so a rejected extreme is an error
rather than a panic.

Signed-off-by: gitRasheed <[email protected]>
Now that timestamps beyond jiff's range can be written, `split_temporal` could
see second and millisecond counts more than i32::MAX days from the epoch.
`split_with_divisor` truncated the days to i32 with `as_()`, so such values
silently wrapped when compressed: `i64::MAX` seconds round-tripped as
50027779063807. The compress comment still claimed jiff validation made the
narrowing safe, the btrblocks TemporalScheme applied the encoding with no range
check, and `try_extract_days_constant` used the same truncating split, so
`dtp <= lit` with such a literal was rewritten to a wrapped days constant.
Microsecond and nanosecond counts cannot overflow the days part.

`timestamp::split` now errors when the days do not fit, so the days pushdown
declines the constant and the compare kernel falls back to the canonical
comparison. `split_temporal` checks the valid min/max of second and millisecond
arrays and errors instead of wrapping, and the new `can_split_temporal` lets
the btrblocks TemporalScheme skip such arrays so the storage array is
compressed and the data round-trips exactly. The narrowing compare test built
its fixture from `i64::MAX` seconds and relied on the wrap; it now uses a value
above i32::MAX whose days fit.

Signed-off-by: gitRasheed <[email protected]>
…hold the literal

Compression narrows the days part, for example to u16 for dates before 2149. A midnight
literal past that range, such as 9999-12-31, made the comparison push-down cast its days
to the narrowed type and fail with "Cannot cast i32 to u16". The rule now declines, and the
comparison runs on the full timestamps.

Signed-off-by: gitRasheed <[email protected]>
Signed-off-by: gitRasheed <[email protected]>
@codspeed

codspeed Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by 14.83%

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚡ 6 improved benchmarks
✅ 1872 untouched benchmarks
🆕 6 new benchmarks
⏩ 473 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ CPU Simulation bench_compare_varbin[(50000, 2)] 162.3 µs 132.4 µs +22.59%
⚡ CPU Simulation bench_compare_sliced_dict_varbinview[(1000, 10000)] 90.5 µs 78.3 µs +15.58%
⚡ Walltime add_constant_shapes_neon[(32768, PerRowConstant)] 16 µs 14 µs +13.85%
⚡ CPU Simulation bench_compare_sliced_dict_varbinview[(9999, 10000)] 233.4 µs 205.8 µs +13.39%
⚡ Walltime add_constant_shapes_neon[(32768, PerRowNullableConstant)] 16.5 µs 14.6 µs +13.26%
⚡ Walltime add_constant_shapes_neon[(32768, ConstantPerRow)] 15.5 µs 14 µs +10.68%
🆕 CPU Simulation split_extreme[(8192, Milliseconds)] N/A 572.3 µs N/A
🆕 CPU Simulation split_extreme[(8192, Seconds)] N/A 587.6 µs N/A
🆕 CPU Simulation split[(8192, Microseconds)] N/A 299.6 µs N/A
🆕 CPU Simulation split[(8192, Milliseconds)] N/A 336.8 µs N/A
🆕 CPU Simulation split[(8192, Nanoseconds)] N/A 307.3 µs N/A
🆕 CPU Simulation split[(8192, Seconds)] N/A 205 µs N/A

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing gitRasheed:timestamp-full-range (7943b4e) with develop (41dec20)

Open in CodSpeed

Footnotes

  1. 473 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

connortsui20 added a commit that referenced this pull request Oct 30, 2026
## Summary

Validates temporal dtypes and returns errors from fallible scalar and
import paths for invalid metadata or out-of-range values. Keeps the
current timestamp range limits. #10953 proposes a wider range, so the
two approaches can be reviewed together.

## Changes

Checks Date/Time storage and Timestamp metadata, uses fallible Span
setters, and covers Arrow, DataFusion, Python, and JNI boundaries.
Pinned formatting and five-crate Clippy pass, along with 6,412 Rust
tests (1 skipped), 73 Python tests, and 38 Java tests. Python type
checking, Ruff, and Spotless pass.

## API Changes

Adds `Timestamp::try_new_with_tz` and rejects Timestamp `Days` and
unresolved time zones at construction. Existing convenience constructors
retain a documented panic contract.

---------

Signed-off-by: Connor Tsui <[email protected]>
gitRasheed and others added 4 commits October 30, 2026 14:33
split_temporal ran a min_max pass over second and millisecond values
before splitting them, which made the split benchmark about a third
slower for seconds when no stats were cached.

The split loops now OR together each value's days plus 2^31 and check
the high bits once at the end, so the check stays branch-free. When the
loop trips, an exact min_max over the valid values decides the error,
since null slots can hold any value. Exact min/max stats cached on the
extension array or its storage settle the question up front and the
loops skip the check. Microsecond and nanosecond days always fit, so
those loops never check.

can_split_temporal reads the same cached stats before computing min_max.

Signed-off-by: gitRasheed <[email protected]>
…range

Signed-off-by: Connor Tsui <[email protected]>

# Conflicts:
#	vortex-array/src/extension/datetime/timestamp.rs
@connortsui20

connortsui20 commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Thanks for picking this up! Added three follow-up commits:

@connortsui20 connortsui20 added the changelog/fix A bug fix label Oct 30, 2026
@connortsui20

Copy link
Copy Markdown
Member

Sorry for taking over this PR, it's just that I'm working on stuff that conflicts with this and I want to make sure we dont have to keep resolving merge conflicts

connortsui20 added a commit that referenced this pull request Oct 30, 2026
The phase graph of the extension plan now shows each PR of phase 0, its
state and the order constraints. Phase 1 depends on U4 and U5, and its
strict reads wait for the spatial fix. Phase 0 holds #10953, the spatial
fix and the importer change, and the structural part of X29 moves out of
phase 6. The PR state lists #11013 as merged and adds #11067.

In the type roadmap, N6 becomes the storage declaration (X25 to X30),
with hard edges from N1 and into N3, and the report matches.

Signed-off-by: Connor Tsui <[email protected]>
Keep the i32 split for ordinary timestamps and rebuild only the days
component as i64 when a non-null value exceeds that range. Reuse the
seconds and subseconds that the split already computed.

Accept the full timestamp range in scalar reads and comparisons, and let
TemporalScheme use DateTimeParts for extreme second and millisecond values.
Cover adaptive width selection, null slots, serialization, scalar reads,
comparisons, and compression round trips.

Signed-off-by: Connor Tsui <[email protected]>
@gitRasheed

Copy link
Copy Markdown
Contributor Author

Sorry for taking over this PR, it's just that I'm working on stuff that conflicts with this and I want to make sure we dont have to keep resolving merge conflicts

all good, go ahead 👍

@connortsui20 connortsui20 mentioned this pull request Nov 2, 2026
9 tasks done
…range

Signed-off-by: Connor Tsui <[email protected]>

# Conflicts:
#	encodings/datetime-parts/src/compute/compare.rs
connortsui20 added a commit that referenced this pull request Nov 2, 2026
U2, U3, U4, U6, U7 and U8 merged on 2026-11-02 in #11012, #11014 and
#11016, and the DataFusion literal fix #11067 merged right after U2. So
order constraint 1 holds, and phase 1 starts from develop. The graph
shows these nodes as merged, and parts 2 and 4 name the PR that fixed
each finding.

#10953 now keeps i32 days and widens only the days part to i64 when a
value does not fit. It has no review, and it conflicts with the
DateTimeParts compare kernel that U2 changed. #11056 still waits on its
request for changes. #10661 replaces the closed #10482 for file sort
statistics.

In the type roadmap, only the N1 sentence changes.

Signed-off-by: Connor Tsui <[email protected]>
connortsui20 added a commit that referenced this pull request Nov 2, 2026
A maintainer commit merged develop into #10953 again on 2026-11-02, so the PR no longer conflicts and waits for a review. Part 4 records one local run of the split_temporal benchmark, because CodSpeed cannot compare the changed cases with develop.

Signed-off-by: Connor Tsui <[email protected]>
@connortsui20 connortsui20 mentioned this pull request Nov 2, 2026
12 of 49 tasks
@robert3005

Copy link
Copy Markdown
Contributor

we should make sure that replace_time_zone errors on these inputs

Check the widened days with min_max instead of the original timestamps,
reuse the split days for the overflow check, and make replace_time_zone
reject values outside jiff's range even when the timezone is unchanged.

Signed-off-by: gitRasheed <[email protected]>
@gitRasheed

Copy link
Copy Markdown
Contributor Author

On replace_time_zone: fixed the one gap, same-zone raise skipped the conversion. Added a test.

@connortsui20

Copy link
Copy Markdown
Member

I will make a few more changes and then we should be good

Timestamp parts supplied by files or try_new can exceed the original timestamp range. Use checked arithmetic and fallible scalar conversions so these inputs return errors.

Leave combine_parts in the hot decode loop unchanged. Its overflow risk predates this PR.

Signed-off-by: Connor Tsui <[email protected]>
…'s range

Add conversion context and require the invalid-input test to reach that error. Execute an epoch-only control for both timezone targets so unrelated expression failures cannot satisfy the regression.

Signed-off-by: Connor Tsui <[email protected]>
… code

Restore the full-range timestamp fixture and cover all six comparisons against an extreme constant when the days child is i32. Remove stale benchmark statistic clearing and restore the temporal compression comment.

Signed-off-by: Connor Tsui <[email protected]>
The default writer computes exact bounds, but CanonicalValidity rebuilds the arrays and drops those bounds before temporal compression. The writer probe supplied no cached bounds for either seconds or milliseconds, with or without nulls.

Always check the computed days and remove the cache lookup, duplicate loop variants, and cache-only test and benchmark parameters. Keep the non-null day bounds check when widening is needed.

Signed-off-by: Connor Tsui <[email protected]>
Add replace_time_zone conversion context with VortexError::with_context so callers retain the jiff error as the source. Require the existing range regression to preserve that source as well as the conversion message.

Signed-off-by: Connor Tsui <[email protected]>
Construct the timestamp dtype directly instead of splitting a throwaway array. Cover Gte alongside the other five comparisons when an extreme midnight literal does not fit the narrowed days child.

Signed-off-by: Connor Tsui <[email protected]>
connortsui20 added a commit that referenced this pull request Nov 3, 2026
Adds EXT-V10: the `Display` of `Date` panics on valid storage values
that jiff cannot hold. X31 now says so, and the plan gives the finding
a follow-up PR on `develop`. Parts 4 and 5 record the review of #10953
on 2026-11-03, the local review commits, the removal of the
cached-bounds fast path, and the benchmark against `develop`.

Signed-off-by: Connor Tsui <[email protected]>
Use 8,192 rows, matching one default writer row block, instead of 65,536. The larger inputs took 1.5 to 4.1 ms in CodSpeed CPU simulation. Reducing the work eightfold targets sub-millisecond cases while retaining every time unit and both extreme-value cases.

Signed-off-by: Connor Tsui <[email protected]>
@connortsui20

Copy link
Copy Markdown
Member

hmm actually I want to make some cleanup changes

Document the component widths, signed remainders, and bias-and-OR days range check. Name the range-check state explicitly and constrain scalar part reads to integer types.

Compare the bit check against checked integer conversion and assert the remainder bounds at both timestamp storage extremes.

Signed-off-by: Connor Tsui <[email protected]>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reviewers should probably start here

connortsui20 added a commit that referenced this pull request Nov 3, 2026
SR merges into SD, because a storage declaration without the exact read
check reads an old form silently wrong. SD now needs SN1, which only
tightens the spatial validators. SN2 takes every Arrow importer of a
known type and builds on DP, which peels dictionary fields before the
importers run. V10 and DS become their own PRs.

Also record that #10953 is pushed at 7943b4e and waits for an
approval, that the 12.9% millisecond slowdown was measured at
d6e73bd, and that #11056 merged.

Signed-off-by: Connor Tsui <[email protected]>
@robert3005
robert3005 merged commit 8896ab2 into vortex-data:develop Nov 3, 2026
79 checks passed
connortsui20 added a commit that referenced this pull request Nov 4, 2026
## Summary

- Tracking Issue: #11114

`Date` accepts every `i32` day count and every `i64` millisecond count,
but jiff only holds the years -9999 to 9999, so formatting a valid
`Date` outside that range panicked. `Display` now prints the raw value
and unit instead (e.g., `10000000days`), like the `Timestamp` fallback
from #10953.

## Changes

Tests cover the first date after 9999-12-31, day and millisecond counts
that a jiff `Span` can't hold, and an ordinary date whose output doesn't
change.

Signed-off-by: Connor Tsui <[email protected]>
connortsui20 added a commit that referenced this pull request Nov 4, 2026
TS (#10953) merged on 2026-11-03, and SN1 (#11183) and V10 (#11187)
merged on 2026-11-04. The dependency graph marks the three merged and
moves its state date to 2026-11-04. It gives SD, DP, FD and U9 their
open PRs (#11184, #11185, #11186 and #11188), and it drops #10408,
which closed without a merge. The text of parts 4 and 5 and the Fix
lines of EXT-V9, EXT-V10, EXT-N2 and EXT-S3 now record the same state.

Also record that AX, DS and SN2 are implemented on branches without a
PR, and add the approved one-pass design for the masked nulls of SN2.

Part 2 and decision X31 also record the fixes: #10953 made every i64
legal, #11183 made the spatial validators reject nullable nested
levels, #11187 gave the Date display its fallback, and #10408 closed.

Signed-off-by: Connor Tsui <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Writing a Timestamp in the last ~26h of year 9999 panics during Min/Max stats

3 participants