You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[datafusion-spark] unix_millis / unix_seconds truncate toward zero instead of flooring for pre-epoch timestamps #26128
The Spark-compatible unix_millis and unix_seconds functions in datafusion-spark return different results from Spark for timestamps before 1970-01-01 that have a sub-unit component. Spark floors the result; DataFusion truncates toward zero.
For example, one microsecond before the epoch gives unix_millis = -1 in Spark but 0 in DataFusion.
unix_micros is not affected, since it does no unit conversion.
To Reproduce
DataFusion (datafusion-cli --spark):
SELECT
unix_millis(arrow_cast(-1, 'Timestamp(Microsecond, None)')) AS millis_a,
unix_millis(arrow_cast(-1001, 'Timestamp(Microsecond, None)')) AS millis_b,
unix_seconds(arrow_cast(-1, 'Timestamp(Microsecond, None)')) AS secs_a,
unix_seconds(arrow_cast(-1000001, 'Timestamp(Microsecond, None)')) AS secs_b;
Exact multiples of the unit (e.g. -1000 µs → -1 ms) and non-negative values (e.g. 1500 µs → 1 ms) already match
Expected behavior
Match Spark: floor-divide the microsecond value by 1,000 for unix_millis and by 1,000,000 for unix_seconds.
Additional context
The functions were added in #19892. SparkUnixTimestamp::simplify rewrites the call as CAST(CAST(ts AS Timestamp(<unit>, "UTC")) AS Int64). The Arrow cast to a coarser timestamp unit truncates toward zero, which is wrong for negative values with a remainder.
Spark computes these in TimestampToLongBase (datetimeExpressions.scala) using Math.floorDiv.
I searched existing issues and PRs and didn't find this reported. I have a fix with sqllogictest coverage and will open a PR shortly.
Activity
harsh-ande commentedon Oct 7, 2026
take
fix(spark): floor instead of truncate in unix_millis/unix_seconds for…