Skip to content

Use arrow cast for hand-rolled temporal unit conversion loops #5090

Closed

Description

@andygrove

What is the problem the feature request solves?

Several hand-rolled per-row loops implement temporal unit conversions that arrow's cast kernel performs bit-identically (verified against the arrow-cast 58.4.0 source):

  • native/core/src/parquet/cast_column.rs (cast_timestamp_micros_to_millis_array / _scalar): unary(|v| v / 1000) plus manual timezone re-attachment. Arrow's Timestamp-to-Timestamp cast performs the same truncating division and attaches the target tz.
  • native/spark-expr/src/conversion_funcs/temporal.rs (NTZ arm of cast_date_to_timestamp): per-row (d as i64) * 86_400 * 1_000_000 into a builder. Arrow's (Date32, Timestamp(Microsecond, _)) cast computes exactly (x as i64) * MICROSECONDS_IN_DAY.
  • native/spark-expr/src/utils.rs (millis-to-micros): arity::unary(|v| v * 1000). Arrow's timestamp-unit cast performs the same multiply.
  • native/spark-expr/src/datetime_funcs/date_from_unix_date.rs: manual Int32-to-Date32 reinterpretation plus a scalar match arm. Arrow dispatches (Int32, Date32) to a zero-copy reinterpret cast; the scalar path can use ScalarValue::cast_to.

Describe the potential solution

Replace each loop with arrow::compute::cast_with_options to the target type.

Additional context

Caveats to preserve:

  • micros-to-millis: safe when source and target tz are both Some or both None. If the source is NTZ and the target has a tz, arrow wall-clock-adjusts values, so that combination must keep the current relabel behavior.
  • Date32-to-Timestamp: only the target_tz.is_none() branch is replaceable. The session-tz branch implements Spark's DST ambiguity/gap rules (earlier occurrence, pre-transition offset), while arrow's adjustment returns null on both, so it must stay.

Found during an audit of native code that replicates existing arrow-rs kernels.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions