Skip to content

[Bug] next_day rejects STRING and TIMESTAMP start dates that Spark accepts #23889

Closed

Description

@andygrove

Describe the bug

datafusion-spark's next_day declares
Signature::exact(vec![DataType::Date32, DataType::Utf8], Volatility::Immutable)
in datafusion/spark/src/function/datetime/next_day.rs.

Spark's NextDay extends ImplicitCastInputTypes with
inputTypes = Seq(DateType, StringTypeWithCollation(...)), so Spark casts the
first argument to DATE before evaluating. A STRING or a TIMESTAMP start
date is accepted. DataFusion raises a planning error instead.

This affects Spark 3.5.8 through 4.2.0 identically. The behavior has not
changed across those versions.

To Reproduce

Spark SQL, verified against a live pyspark==4.2.0 under both
spark.sql.ansi.enabled=true and =false:

SELECT next_day('2015-07-23', 'Mon');
-- 2015-07-27

SELECT next_day(TIMESTAMP '2015-07-23 12:12:12', 'Mon');
-- 2015-07-27

SELECT next_day('2015-07-23 12:12:12', 'Mon');
-- 2015-07-27

Spark's own golden file
sql/core/src/test/resources/sql-tests/results/date.sql.out asserts all three.

DataFusion SQL:

SELECT next_day('2015-07-23'::string, 'Mon'::string);
-- Error during planning: Failed to coerce arguments to satisfy a call to
-- 'next_day' function: coercion from Utf8View, Utf8View to the signature
-- Exact(Date32, Utf8) failed. No function matches the given name and argument
-- types 'next_day(Utf8View, Utf8View)'.

SELECT next_day('2015-07-23 12:12:12'::timestamp, 'Mon'::string);
-- Error during planning: Failed to coerce arguments to satisfy a call to
-- 'next_day' function: coercion from Timestamp(ns), Utf8View to the signature
-- Exact(Date32, Utf8) failed.

Expected behavior

next_day should accept the argument types Spark accepts and cast them the way
Spark does.

Additional context

Not fixed in the audit that surfaced this because it is a semantics decision
rather than a mechanical correction. Widening the signature raises questions
the maintainers should answer first:

  1. Should datafusion-spark model Spark's ImplicitCastInputTypes generally,
    or does it expect the caller to insert the casts? The answer applies to
    every function in the crate, not just next_day, so a one-off widening here
    would set an undeclared precedent.
  2. Spark's STRING-to-DATE cast is itself ANSI dependent. next_day('xx', 'Mon')
    raises CAST_INVALID_INPUT under ANSI and returns NULL otherwise. Any
    widening has to route through a Spark-compatible cast, not
    arrow::compute::cast.
  3. datafusion/sqllogictest/test_files/spark/datetime/next_day.slt currently
    asserts the coercion error as expected behavior. That assertion has to be
    replaced, not merely extended.

The blocked cases are checked in as commented-out queries in
datafusion/sqllogictest/test_files/spark/datetime/next_day.slt, carrying the
Spark 4.2.0 results above. Uncommenting them is the contract for verifying a
fix.

Relevant file and line:
datafusion/spark/src/function/datetime/next_day.rs, SparkNextDay::new.

Surfaced by the audit-datafusion-spark-expression skill.

Activity

  1. added
    bugSomething isn't working
    sparkChanges to datafusion-spark crate
    on Jul 25, 2026
  2. harsh-ande commented on Oct 7, 2026

    @harsh-ande
    Contributor

    take

  3. added a commit that references this issue on Oct 29, 2026
    d2bd93c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingsparkChanges to datafusion-spark crate

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions