You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Use arrow cast for hand-rolled temporal unit conversion loops #5090
Several hand-rolled per-row loops implement temporal unit conversions that arrow's cast kernel performs bit-identically (verified against the arrow-cast 58.4.0 source):
native/core/src/parquet/cast_column.rs (cast_timestamp_micros_to_millis_array / _scalar): unary(|v| v / 1000) plus manual timezone re-attachment. Arrow's Timestamp-to-Timestamp cast performs the same truncating division and attaches the target tz.
native/spark-expr/src/conversion_funcs/temporal.rs (NTZ arm of cast_date_to_timestamp): per-row (d as i64) * 86_400 * 1_000_000 into a builder. Arrow's (Date32, Timestamp(Microsecond, _)) cast computes exactly (x as i64) * MICROSECONDS_IN_DAY.
native/spark-expr/src/utils.rs (millis-to-micros): arity::unary(|v| v * 1000). Arrow's timestamp-unit cast performs the same multiply.
native/spark-expr/src/datetime_funcs/date_from_unix_date.rs: manual Int32-to-Date32 reinterpretation plus a scalar match arm. Arrow dispatches (Int32, Date32) to a zero-copy reinterpret cast; the scalar path can use ScalarValue::cast_to.
Describe the potential solution
Replace each loop with arrow::compute::cast_with_options to the target type.
Additional context
Caveats to preserve:
micros-to-millis: safe when source and target tz are both Some or both None. If the source is NTZ and the target has a tz, arrow wall-clock-adjusts values, so that combination must keep the current relabel behavior.
Date32-to-Timestamp: only the target_tz.is_none() branch is replaceable. The session-tz branch implements Spark's DST ambiguity/gap rules (earlier occurrence, pre-transition offset), while arrow's adjustment returns null on both, so it must stay.
Found during an audit of native code that replicates existing arrow-rs kernels.
Activity
peterxcli commentedon Jul 31, 2026
take