Skip to content

[datafusion-spark] unix_millis / unix_seconds truncate toward zero instead of flooring for pre-epoch timestamps #26128

Closed

Description

@harsh-ande

Describe the bug

The Spark-compatible unix_millis and unix_seconds functions in datafusion-spark return different results from Spark for timestamps before 1970-01-01 that have a sub-unit component. Spark floors the result; DataFusion truncates toward zero.

For example, one microsecond before the epoch gives unix_millis = -1 in Spark but 0 in DataFusion.

unix_micros is not affected, since it does no unit conversion.

To Reproduce

DataFusion (datafusion-cli --spark):

SELECT
  unix_millis(arrow_cast(-1, 'Timestamp(Microsecond, None)'))        AS millis_a,
  unix_millis(arrow_cast(-1001, 'Timestamp(Microsecond, None)'))     AS millis_b,
  unix_seconds(arrow_cast(-1, 'Timestamp(Microsecond, None)'))       AS secs_a,
  unix_seconds(arrow_cast(-1000001, 'Timestamp(Microsecond, None)')) AS secs_b;

Spark 4.0.0 (spark.sql.session.timeZone=UTC):

SELECT
  unix_millis(timestamp_micros(-1)),
  unix_millis(timestamp_micros(-1001)),
  unix_seconds(timestamp_micros(-1)),
  unix_seconds(timestamp_micros(-1000001));
| input (µs) | function       | Spark 4.0.0 | DataFusion |
|-----------:|----------------|------------:|-----------:|
| -1         | `unix_millis`  | -1          | 0          |
| -1001      | `unix_millis`  | -2          | -1         |
| -1         | `unix_seconds` | -1          | 0          |
| -1000001   | `unix_seconds` | -2          | -1         |

Exact multiples of the unit (e.g. -1000 µs → -1 ms) and non-negative values (e.g. 1500 µs → 1 ms) already match

Expected behavior

Match Spark: floor-divide the microsecond value by 1,000 for unix_millis and by 1,000,000 for unix_seconds.

Additional context

The functions were added in #19892. SparkUnixTimestamp::simplify rewrites the call as CAST(CAST(ts AS Timestamp(<unit>, "UTC")) AS Int64). The Arrow cast to a coarser timestamp unit truncates toward zero, which is wrong for negative values with a remainder.

Spark computes these in TimestampToLongBase (datetimeExpressions.scala) using Math.floorDiv.

I searched existing issues and PRs and didn't find this reported. I have a fix with sqllogictest coverage and will open a PR shortly.

Activity

  1. added theissue type on Oct 7, 2026
  2. harsh-ande commented on Oct 7, 2026

    @harsh-ande
    ContributorAuthor

    take

  3. added a commit that references this issue on Oct 30, 2026
    4ec76ad
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions