Repository navigation
Description
Describe the bug
datafusion-spark's next_day declares
Signature::exact(vec![DataType::Date32, DataType::Utf8], Volatility::Immutable)
in datafusion/spark/src/function/datetime/next_day.rs.
Spark's NextDay extends ImplicitCastInputTypes with
inputTypes = Seq(DateType, StringTypeWithCollation(...)), so Spark casts the
first argument to DATE before evaluating. A STRING or a TIMESTAMP start
date is accepted. DataFusion raises a planning error instead.
This affects Spark 3.5.8 through 4.2.0 identically. The behavior has not
changed across those versions.
To Reproduce
Spark SQL, verified against a live pyspark==4.2.0 under both
spark.sql.ansi.enabled=true and =false:
SELECT next_day('2015-07-23', 'Mon');
-- 2015-07-27
SELECT next_day(TIMESTAMP '2015-07-23 12:12:12', 'Mon');
-- 2015-07-27
SELECT next_day('2015-07-23 12:12:12', 'Mon');
-- 2015-07-27Spark's own golden file
sql/core/src/test/resources/sql-tests/results/date.sql.out asserts all three.
DataFusion SQL:
SELECT next_day('2015-07-23'::string, 'Mon'::string);
-- Error during planning: Failed to coerce arguments to satisfy a call to
-- 'next_day' function: coercion from Utf8View, Utf8View to the signature
-- Exact(Date32, Utf8) failed. No function matches the given name and argument
-- types 'next_day(Utf8View, Utf8View)'.
SELECT next_day('2015-07-23 12:12:12'::timestamp, 'Mon'::string);
-- Error during planning: Failed to coerce arguments to satisfy a call to
-- 'next_day' function: coercion from Timestamp(ns), Utf8View to the signature
-- Exact(Date32, Utf8) failed.Expected behavior
next_day should accept the argument types Spark accepts and cast them the way
Spark does.
Additional context
Not fixed in the audit that surfaced this because it is a semantics decision
rather than a mechanical correction. Widening the signature raises questions
the maintainers should answer first:
- Should
datafusion-sparkmodel Spark'sImplicitCastInputTypesgenerally,
or does it expect the caller to insert the casts? The answer applies to
every function in the crate, not justnext_day, so a one-off widening here
would set an undeclared precedent. - Spark's STRING-to-DATE cast is itself ANSI dependent.
next_day('xx', 'Mon')
raisesCAST_INVALID_INPUTunder ANSI and returns NULL otherwise. Any
widening has to route through a Spark-compatible cast, not
arrow::compute::cast. datafusion/sqllogictest/test_files/spark/datetime/next_day.sltcurrently
asserts the coercion error as expected behavior. That assertion has to be
replaced, not merely extended.
The blocked cases are checked in as commented-out queries in
datafusion/sqllogictest/test_files/spark/datetime/next_day.slt, carrying the
Spark 4.2.0 results above. Uncommenting them is the contract for verifying a
fix.
Relevant file and line:
datafusion/spark/src/function/datetime/next_day.rs, SparkNextDay::new.
Surfaced by the audit-datafusion-spark-expression skill.
Activity
docs: add an agent skill for auditing datafusion-spark expressions (a…
docs: add an agent skill for auditing datafusion-spark expressions (a…
harsh-ande commentedon Oct 7, 2026
take
fix(spark): accept STRING and TIMESTAMP start dates in next_day (apac…