Repository navigation
Description
What is the problem the feature request solves?
The scalar-function wiring makes it easy to silently lose ANSI semantics, and it has already happened once (MakeDecimal). Two independent mechanisms both fail "open" toward non-ANSI behavior:
CometScalarFunction.convertuses the plainscalarFunctionExprToProtohelper, which hardcodesbuilder.setFailOnError(false)(spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala:1070-1075). Any ANSI-sensitive Spark expression wired via plainCometScalarFunctiondrops the flag at serialization time.- On the native side,
create_comet_physical_funthreadsfail_on_erroronly into the hand-written match arms; the catch-all_ => registry.udf(fun_name)(native/spark-expr/src/comet_scalar_funcs.rs:240) drops it for every registry-resolved UDF, which includes all functions wired in from the datafusion-spark crate.create_comet_physical_fun_with_eval_modealso defaultsfail_on_error.unwrap_or(false).
Today no wired datafusion-spark function is ANSI-sensitive (verified against Spark 4.1.1: date_add, date_sub, last_day, from_utc_timestamp, to_utc_timestamp, factorial, chr, substring, space, luhn_check, url_encode, hex, rint, etc. never consult ANSI), so there is no live gap besides MakeDecimal (tracked separately). But the failure mode is silent and will recur as more expressions are wired.
Describe the potential solution
Options, not mutually exclusive:
- Make
scalarFunctionExprToProtoWithReturnType(the only helper that can setfail_on_error) reject or assert when the native registration cannot receive the flag, or conversely make the native side return an error whenfail_on_error=Some(true)reaches the registry catch-all instead of silently ignoring it. - Add a debug assertion or serde-level check: expressions whose Spark class has a
failOnError/evalModefield must not be registered via plainCometScalarFunction. - Document the constraint in the contributor guide for wiring datafusion-spark functions.
Additional context
Found by an ANSI-mode audit of all native expressions against Spark 4.1.1. The audit relied on name-based variant selection (parse_url vs try_parse_url, url_decode vs try_url_decode) as the sound pattern for ANSI-sensitive scalar functions; a guard would enforce that pattern.
Activity
sam-1112 commentedon Aug 6, 2026
take
fix: guard against silent fail_on_error loss in scalar wiring (apache…
fix: guard against silent fail_on_error loss in scalar wiring (apache…
fix: guard against silent fail_on_error loss in scalar wiring (apache…
test: reject nullOnOverflow via plain CometScalarFunction (apache#5074)
fix: guard against silent fail_on_error loss in scalar wiring (apache…
fix: guard against silent fail_on_error loss in scalar wiring (#5074) (…