Skip to content

Commit 36caf8e

Browse files
sam-1112Hung
andauthored
fix: guard against silent fail_on_error loss in scalar wiring (#5074) (#5359)
* fix: guard against silent fail_on_error loss in scalar wiring (#5074) Fail closed when fail_on_error=true reaches a registry-only UDF, reject ANSI-sensitive expressions from plain CometScalarFunction, and document the name-based ANSI/try wiring constraint. * fix: address scalar wiring review feedback Register CometScalarFunctionSuite in CI, use Scala 2.12-compatible ANSI field detection, keep make_time dispatch for fail_on_error=true, and recognize ansiEnabled/evalContext. --------- Co-authored-by: Hung <[email protected]>
1 parent e4ec630 commit 36caf8e

8 files changed

Lines changed: 412 additions & 2 deletions

File tree

‎.github/workflows/pr_build_linux.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -409,6 +409,7 @@ jobs:
409409
org.apache.comet.CometStringDecodeSuite
410410
org.apache.comet.CometWidthBucketSuite
411411
org.apache.comet.CometUuidExpressionSuite
412+
org.apache.comet.serde.CometScalarFunctionSuite
412413
fail-fast: false
413414
name: ${{ matrix.profile.name }} [${{ matrix.suite.name }}]
414415
runs-on: ubuntu-24.04

‎.github/workflows/pr_build_macos.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -225,6 +225,7 @@ jobs:
225225
org.apache.comet.CometStringDecodeSuite
226226
org.apache.comet.CometWidthBucketSuite
227227
org.apache.comet.CometUuidExpressionSuite
228+
org.apache.comet.serde.CometScalarFunctionSuite
228229
229230
fail-fast: false
230231
name: ${{ matrix.os }}/${{ matrix.profile.name }} [${{ matrix.suite.name }}]

‎docs/source/contributor-guide/adding_a_new_expression.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,37 @@ For simple scalar functions that map directly to a DataFusion function, you can
8787
classOf[Cos] -> CometScalarFunction("cos")
8888
```
8989

90+
### ANSI / `fail_on_error` constraints
91+
92+
`CometScalarFunction` uses `scalarFunctionExprToProto`, which always serializes
93+
`fail_on_error=false`.
94+
95+
On the native side, `create_comet_physical_fun` delegates to
96+
`create_comet_physical_fun_with_eval_mode`, whose registry catch-all **fails
97+
closed**. If `fail_on_error=Some(true)` reaches a UDF that can only be resolved
98+
from the function registry—including functions registered by
99+
`datafusion-spark`—Comet returns an error instead of silently ignoring the flag.
100+
101+
Follow these rules when wiring scalar expressions:
102+
103+
- **Do not** register a Spark expression class that exposes `failOnError`,
104+
`evalMode`, `nullOnOverflow`, `ansiEnabled`, or `evalContext` using plain
105+
`CometScalarFunction`. `CometScalarFunction.convert` rejects such expressions
106+
by returning `None`, allowing the planner to fall back to Spark.
107+
108+
- **Prefer name-based ANSI and try variants** so that the error semantics are
109+
encoded in the function name while the proto `fail_on_error` flag remains
110+
`false`. Existing examples include `parse_url` / `try_parse_url`
111+
(`CometParseUrl`) and `url_decode` / `try_url_decode`
112+
(`CometUrlDecodeStaticInvoke`).
113+
114+
- If a function must receive `fail_on_error` through the proto, use
115+
`scalarFunctionExprToProtoWithReturnType(..., failOnError, ...)`. The function
116+
must also have a dedicated match arm in
117+
`create_comet_physical_fun_with_eval_mode` that actually consumes the flag;
118+
it must not fall through to registry lookup. Existing native consumers
119+
include `make_decimal`, `make_date`, `make_time`, and `next_day`.
120+
90121
#### When to set the return type explicitly
91122

92123
`CometScalarFunction(name)` and the lower-level `scalarFunctionExprToProto(name, args)` helper both produce a protobuf `ScalarFunc` message **without** a `return_type` field. That is fine when the function name does not collide with a DataFusion built-in, or when it does collide and the Spark and DataFusion versions take the same arity and types. In that case the native planner consults DataFusion's UDF registry only to resolve the return type, then swaps in Comet's UDF for execution.

‎native/spark-expr/src/comet_scalar_funcs.rs‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -260,6 +260,17 @@ pub fn create_comet_physical_fun_with_eval_mode(
260260
let func = Arc::new(crate::string_funcs::spark_levenshtein);
261261
make_comet_scalar_udf!("levenshtein", func, without data_type)
262262
}
263+
// Spark 4.1+ serde always sets fail_on_error=true (always-throw semantics).
264+
// SparkMakeTime already throws on invalid input, so accept the flag here rather
265+
// than falling through to the registry fail-closed path.
266+
"make_time" => Ok(Arc::new(ScalarUDF::new_from_impl(SparkMakeTime::new()))),
267+
// Registry UDFs (including datafusion-spark) cannot receive fail_on_error.
268+
_ if fail_on_error => Err(DataFusionError::Execution(format!(
269+
"Function '{fun_name}' is resolved from the UDF registry and cannot \
270+
honor fail_on_error=true. Use a name-based ANSI/try variant \
271+
(e.g. parse_url / try_parse_url) or a dedicated match arm that \
272+
consumes the flag."
273+
))),
263274
_ => registry.udf(fun_name).map_err(|e| {
264275
DataFusionError::Execution(format!(
265276
"Function {fun_name} not found in the registry: {e}",

‎native/spark-expr/tests/spark_expr_reg.rs‎

Lines changed: 65 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717

1818
#[cfg(test)]
1919
mod tests {
20-
use arrow::datatypes::DataType;
20+
use arrow::datatypes::{DataType, TimeUnit};
2121
use datafusion::error::Result;
2222
use datafusion::execution::session_state::SessionStateBuilder;
2323
use datafusion::execution::FunctionRegistry;
@@ -91,4 +91,68 @@ mod tests {
9191

9292
Ok(())
9393
}
94+
95+
/// Dedicated match arm must honor fail_on_error=true at execution time (#5074).
96+
#[tokio::test]
97+
async fn test_make_date_ansi_invalid_input_errors() -> Result<()> {
98+
let ctx = SessionContext::new();
99+
let udf = {
100+
let session_state = ctx.state();
101+
create_comet_physical_fun("make_date", DataType::Date32, &session_state, Some(true))?
102+
};
103+
ctx.register_udf(udf.as_ref().clone());
104+
105+
let df = ctx.sql("SELECT make_date(2023, 0, 15)").await?;
106+
let result = df.collect().await;
107+
assert!(
108+
result.is_err(),
109+
"make_date with fail_on_error=true must error on invalid date, got Ok"
110+
);
111+
112+
Ok(())
113+
}
114+
115+
/// Registry catch-all must fail closed when fail_on_error=true (#5074).
116+
#[tokio::test]
117+
async fn test_registry_udf_rejects_fail_on_error() -> Result<()> {
118+
// SessionContext registers DataFusion built-ins (e.g. acos) used by the catch-all.
119+
let ctx = SessionContext::new();
120+
let session_state = ctx.state();
121+
// "acos" is a DataFusion built-in resolved only via the registry catch-all.
122+
let err = create_comet_physical_fun("acos", DataType::Float64, &session_state, Some(true));
123+
assert!(
124+
err.is_err(),
125+
"registry UDF must reject fail_on_error=true, got Ok"
126+
);
127+
let msg = err.unwrap_err().to_string();
128+
assert!(msg.contains("acos"), "unexpected error message: {msg}");
129+
assert!(
130+
msg.contains("fail_on_error=true"),
131+
"unexpected error message: {msg}"
132+
);
133+
134+
let udf_false =
135+
create_comet_physical_fun("acos", DataType::Float64, &session_state, Some(false))?;
136+
assert_eq!(udf_false.name(), "acos");
137+
138+
let udf_none = create_comet_physical_fun("acos", DataType::Float64, &session_state, None)?;
139+
assert_eq!(udf_none.name(), "acos");
140+
141+
// Dedicated match arms that consume the flag still accept fail_on_error=true.
142+
let make_date =
143+
create_comet_physical_fun("make_date", DataType::Date32, &session_state, Some(true))?;
144+
assert_eq!(make_date.name(), "make_date");
145+
146+
// Spark 4.1+ make_time serde always passes fail_on_error=true; the dedicated arm
147+
// must accept it even though SparkMakeTime does not take the flag as a constructor arg.
148+
let make_time = create_comet_physical_fun(
149+
"make_time",
150+
DataType::Time64(TimeUnit::Nanosecond),
151+
&session_state,
152+
Some(true),
153+
)?;
154+
assert_eq!(make_time.name(), "make_time");
155+
156+
Ok(())
157+
}
94158
}

‎spark/src/main/scala/org/apache/comet/serde/CometScalarFunction.scala‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,14 +21,60 @@ package org.apache.comet.serde
2121

2222
import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression}
2323

24+
import org.apache.comet.CometSparkSessionExtensions.withFallbackReason
2425
import org.apache.comet.serde.ExprOuterClass.Expr
2526
import org.apache.comet.serde.QueryPlanSerde.{exprToProtoInternal, scalarFunctionExprToProto}
2627

2728
/** Serde for scalar function. */
2829
case class CometScalarFunction[T <: Expression](name: String) extends CometExpressionSerde[T] {
2930
override def convert(expr: T, inputs: Seq[Attribute], binding: Boolean): Option[Expr] = {
31+
if (CometScalarFunction.isAnsiSensitive(expr)) {
32+
withFallbackReason(
33+
expr,
34+
s"${expr.nodeName} carries failOnError/evalMode/nullOnOverflow/ansiEnabled/evalContext " +
35+
s"and cannot use CometScalarFunction('$name'). Prefer name-based ANSI/try variants " +
36+
"(e.g. parse_url / try_parse_url), or a custom serde with " +
37+
"scalarFunctionExprToProtoWithReturnType plus a native match arm that " +
38+
"consumes fail_on_error.")
39+
return None
40+
}
3041
val childExpr = expr.children.map(exprToProtoInternal(_, inputs, binding))
3142
val optExpr = scalarFunctionExprToProto(name, childExpr: _*)
3243
optExpr
3344
}
3445
}
46+
47+
object CometScalarFunction {
48+
49+
/**
50+
* Product / Java field names that indicate ANSI / eval-mode sensitive Spark expressions.
51+
*
52+
* Detection uses Java reflection (not `Product.productElementNames`) so the check compiles on
53+
* Scala 2.12 used by Spark 3.4 / 3.5.
54+
*/
55+
private val AnsiSensitiveFields: Set[String] =
56+
Set("failOnError", "evalMode", "nullOnOverflow", "ansiEnabled", "evalContext")
57+
58+
/**
59+
* True when the Spark expression case class declares an ANSI-related constructor field. Used to
60+
* reject miswiring via plain [[CometScalarFunction]].
61+
*/
62+
private[serde] def isAnsiSensitive(expr: Expression): Boolean = {
63+
isAnsiSensitive(expr.getClass)
64+
}
65+
66+
/**
67+
* Class-level check used by registration audits: the Spark expression type carries an
68+
* ANSI-related field regardless of any particular instance's flag value.
69+
*/
70+
private[serde] def isAnsiSensitive(clazz: Class[_]): Boolean = {
71+
var current: Class[_] = clazz
72+
while (current != null && current != classOf[Object]) {
73+
if (current.getDeclaredFields.exists(f => AnsiSensitiveFields.contains(f.getName))) {
74+
return true
75+
}
76+
current = current.getSuperclass
77+
}
78+
false
79+
}
80+
}

0 commit comments

Comments
 (0)