Repository navigation
DataFusion support for TimestampWithOffset #21116
Description
Is your feature request related to a problem or challenge?
DataFusion has no support for the arrow.timestamp_with_offset which is an Arrow extension type introduced in arrow-rs 58.0.0 (apache/arrow-rs#8743). This type represents SQL TIMESTAMP WITH TIME ZONE by storing a UTC timestamp alongside a per row offset in minutes as a Struct { timestamp: Timestamp(unit, "UTC"), offset_minutes: Int16 }.
Describe the solution you'd like
Support for this type in the existing date functions: date_part, date_trunc, to_char, to_unixtime
Describe alternatives you've considered
The main concern I have on implementing this is about efficiency. TimestampWithOffset stores a per-row offset, therefore existing datetime functions cannot be applied to the array as a whole, since they require a uniform timezone for the entire array.
One approach is to just loop row by row applying each function individually, which is simple but does not take advantage of arrow's kernels. A more efficient alternative is to group rows by their offset_minutes value (which in theory should be low cardinality), process each group as a sub-array using the existing kernels, then reassemble the results in original row order. However, this grouping approach adds implementation complexity.
Additional context
For substrait queries we will need substrait support to specify the offset and a different timezone to UTC -- substrait-io/substrait#841
Activity
coderfender commentedon Mar 30, 2026
IMHO there is no right way to represent timestamps with various offsets and effectively leverage vector (SIMD) advantages. We could probably go with a hybrid approach and fork implementation based on
CARDINALITY_FACTORkind of parameter (say 16 / 32 etc) and decide between grouping and row-by-row process. Let me raise a draft PR in the following week to provide a POC with some benchmarksLiaCastaneda commentedon Mar 30, 2026
Thanks @coderfender! Yeah, having benchmarks on this would be great. My rule of thumb is to always prefer SIMD and avoid going row-by-row, but honestly I'm not sure it would make a huge difference here. Internally we have a few UDFs that work row-by-row just fine since the core operation isn't that expensive, I'm not sure if optimizing these date functions would be micro optimizing or not.
coderfender commentedon Apr 8, 2026
take
coderfender commentedon Apr 8, 2026
Ill take a stab at it sometime unless you are @LiaCastaneda ?
LiaCastaneda commentedon Apr 8, 2026
Yeah, feel free to take this on. I probably won’t have time to work on it myself until next week anyway.
coderfender commentedon Apr 14, 2026
Okay.! I think I have a rough draft of the data type here . Let me try and see if this compiles and has basic functionality up