Repository navigation
[Bug] next_day cross-version divergence: ANSI default flip and error class change between Spark 3.5 and 4.x #25
Description
Describe the bug
next_day behaves differently across the Spark versions datafusion-spark
targets, and DataFusion's .slt files have no way to express a
version-specific expectation. This issue records the divergence so it can be
scheduled once that mechanism exists.
Companion to https://github.com/apache/datafusion/issues/23887.
The divergence
Given a non-NULL start date and an unparseable day_of_week:
| Spark version | Default spark.sql.ansi.enabled |
Default result | Error class when ANSI is on |
|---|---|---|---|
| 3.5.8 | false |
NULL |
_LEGACY_ERROR_TEMP_2000, wrapped by QueryExecutionErrors.ansiIllegalArgumentError |
| 4.0.4 | true |
error | ILLEGAL_DAY_OF_WEEK, SQLSTATE 22009 |
| 4.1.3 | true |
error | ILLEGAL_DAY_OF_WEEK, SQLSTATE 22009 |
| 4.2.0 | true |
error | ILLEGAL_DAY_OF_WEEK, SQLSTATE 22009 |
Two independent changes landed in Spark 4.0:
- ANSI mode became the default, so the default result for the same query
flipped fromNULLto an error. DateTimeUtils.getDayOfWeekFromStringchanged from
throw new IllegalArgumentException("Illegal input for day of week: ...")
tothrow new SparkIllegalArgumentException(errorClass = "ILLEGAL_DAY_OF_WEEK", ...),
andNextDay.nullSafeEvalchanged from catchingIllegalArgumentException
and re-wrapping to catchingSparkIllegalArgumentExceptionand rethrowing
unchanged.
A third 4.0 change is not currently observable: inputTypes moved from
Seq(DateType, StringType) to
Seq(DateType, StringTypeWithCollation(supportsTrimCollation = true)).
datafusion-spark has no collation support, so there is nothing to diverge
from yet, but a non-default collation on the day_of_week argument is a
divergence axis that exists only in Spark 4.x.
Current state in DataFusion
Both modes are now implemented and tested behind
datafusion.execution.enable_ansi_mode, whose default is false. Anyone
comparing against a stock Spark 4.x will see DataFusion's default disagree with
Spark's default. The .slt file asserts Spark 4.2.0 behavior in each mode and
carries a comment naming the versions.
DataFusion does not model Spark error classes, so
_LEGACY_ERROR_TEMP_2000 versus ILLEGAL_DAY_OF_WEEK is not represented at
all. datafusion-spark raises
Execution error: Illegal input for day of week: <input>, matching Spark's
message text but not its error class or SQLSTATE.
Additional context
Relevant files:
datafusion/spark/src/function/datetime/next_day.rsdatafusion/sqllogictest/test_files/spark/datetime/next_day.slt- Spark:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala,case class NextDay - Spark:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala,getDayOfWeekFromString
Surfaced by the audit-datafusion-spark-expression skill.
Activity
fix: honor ANSI mode in next_day, and fix a panic and a missing argum…
fix: honor ANSI mode in next_day, and fix a panic and a missing argum…
docs: add an agent skill for auditing datafusion-spark expressions (a…
docs: add an agent skill for auditing datafusion-spark expressions (a…
fix: honor ANSI mode in next_day, and fix a missing argument shape
fix(spark): accept STRING and TIMESTAMP start dates in next_day (apac…
andygrove commentedon Nov 6, 2026
Running
next_day.slton Spark 4.2.0 with the runner in apache/datafusion#27165 found another divergence that follows the same split. An unparseable start date is NULL in Spark when ANSI mode is off, but an error in DataFusion even withenable_ansi_mode = false:Spark casts the string to a
DATEbeforenext_dayruns. That cast returns NULL when ANSI mode is off, the default in Spark 3.4 and 3.5, and raises an error when it is on, the default since 4.0 (from the Spark source, not run). Like theday_of_weekcase above,next_dayshould followenable_ansi_mode, and its documentation should say which Spark version the default matches until #16 lets it follow the version.