Skip to content

feat(array): add Interval extension dtype - #10468

Closed
mhk197 wants to merge 1 commit into
mk/duration-dtypefrom
mk/interval-dtype
Closed

mhk197 wants to merge 1 commit into
mk/duration-dtypefrom
mk/interval-dtype

Conversation

@mhk197

@mhk197 mhk197 commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds vortex.interval, a builtin extension dtype for calendar intervals of months, days, and nanoseconds, and maps it to Arrow Interval and DataFusion ScalarValue::Interval*. It joins AnyTemporal alongside Duration.

Changes

  • Interval ext type in vortex-array/src/extension/datetime/interval.rs: empty metadata, storage Struct { months: i32, days: i32, nanoseconds: i64 } (non-nullable fields), so each part compresses on its own. Values display as ISO 8601 with a sign per part.
  • Intervals compare field by field (months, then days, then nanoseconds), which matches Arrow's ordering of IntervalMonthDayNano.
  • Arrow: all three interval units import without loss; export is MonthDayNano, or YearMonth/DayTime when requested and no value would lose a part or precision.
  • DataFusion scalars convert in both directions. DuckDB and CUDA reject Interval for now.

API Changes

  • TemporalMetadata gains an Interval variant.
  • TemporalMetadata::time_unit() now returns Option<TimeUnit> (None for Interval).
  • TemporalArray::new_interval constructor.

Testing

  • Rust: ext type validation/display/scalar round trip; field-by-field comparison; Arrow import of all three units and narrowing export (including lossy-value errors); scalar export; DataFusion scalar conversion.
  • Python: pa.month_day_nano_interval() round trip.

Open question

vortex.interval is in no edition yet, the same open question as vortex.duration in #10467.

Follow-up

Map DuckDB INTERVAL (DuckDB compares normalized intervals, so interval comparisons should not be pushed down from DuckDB), push down interval literals from DataFusion, ts ± interval.

@mhk197 mhk197 added the changelog/break A breaking API change label Oct 13, 2026
@mhk197
mhk197 added this pull request to stack #10469 October 13, 2026 21:33
@codspeed

codspeed Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 2.22%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ 1 benchmark measured no execution time

Nothing ran under measurement, usually because the compiler removed the code under test. This result is not comparable, so it counts as unchanged.

Preventing compiler optimizations

⚠️ 12 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 4 improved benchmarks
❌ 3 regressed benchmarks
✅ 2212 untouched benchmarks
🗄️ 31 archived benchmarks run1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ WallTime threshold_neon[64] 4.3 µs 6.6 µs -34.74%
❌ WallTime threshold_neon[8] 2.9 µs 4.1 µs -28.81%
❌ WallTime threshold_neon[80] 4.7 µs 5.8 µs -18.29%
⚡ WallTime mul_u64_nonnull_neon 40.4 µs 28.4 µs +42.04%
⚡ WallTime multiply_shapes_neon[(32768, PerRowPerRow)] 38.4 µs 32.3 µs +18.74%
⚡ WallTime mul_i64_nonnull_neon 38.4 µs 32.4 µs +18.58%
⚡ WallTime bitpack_blocked_compress_avx2 7.6 µs 6.7 µs +12.56%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing mk/interval-dtype (3a74ca5) with mk/duration-dtype (7261620)

Open in CodSpeed

Footnotes

  1. 31 benchmarks were run, but are now archived. If they were deleted in another branch, consider rebasing to remove them from the report. Instead if they were added back, click here to restore them. ↩

@mhk197
mhk197 force-pushed the mk/interval-dtype branch from 677444b to 58dfb85 Compare October 14, 2026 14:08
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days

@github-actions github-actions Bot added the stale This PR is stale and will be auto-closed soon label Nov 3, 2026
@github-actions

Copy link
Copy Markdown
Contributor

This PR was closed because it has been inactive for 7 days since being marked as stale.

@github-actions github-actions Bot closed this Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/break A breaking API change stale This PR is stale and will be auto-closed soon

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant