Skip to content

Commit 3e999c4

Browse files
committed
fix: reject deterministic = false when registering a Rust UDF
`CometRustUDF.register` accepted a `deterministic` flag, installed a nondeterministic Spark stub for it, and carried it in the RustUdfCall proto, but nothing on the native side ever read it: ImportedCScalarUdf hardcodes Volatility::Immutable. A UDF the caller declared nondeterministic was therefore planned as pure and could be constant-folded, evaluated once and reused, or eliminated as a common subexpression. Honoring the flag needs the cached per-library signature to become per registration, which is #5249. Until then, fail the registration rather than let the parameter lie about what Comet does with it. Raised in review: does the proto field's value mean RustUdfs are always immutable?
1 parent f20bc31 commit 3e999c4

4 files changed

Lines changed: 63 additions & 2 deletions

File tree

‎native/core/src/execution/rust_udf/imported_c.rs‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,12 @@ impl ImportedCScalarUdf {
8181
// Use UserDefined signature: per-call init() is what decides
8282
// whether the input types are acceptable. `coerce_types` is not
8383
// implemented; user must pass exact types from the JVM register call.
84+
//
85+
// Volatility is always Immutable. The signature is built once per
86+
// library load, while determinism is declared per registration, so
87+
// the two do not line up: `CometRustUDF.register` rejects
88+
// `deterministic = false` rather than let a volatile function be
89+
// planned as if it were pure.
8490
let signature = Signature::new(TypeSignature::UserDefined, Volatility::Immutable);
8591

8692
Ok(Self {

‎native/proto/src/proto/expr.proto‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -636,5 +636,10 @@ message RustUdfCall {
636636
// Expected return type, declared at register time on the JVM side.
637637
DataType return_type = 4;
638638
// Whether the call is deterministic (mirrors Spark's deterministic flag).
639+
//
640+
// Always true today: registration rejects deterministic = false, because the
641+
// native side plans every Rust UDF with Volatility::Immutable and has no way
642+
// to express a volatile one. The field is carried so that honoring it later
643+
// does not need a wire change.
639644
bool deterministic = 5;
640645
}

‎spark/src/main/scala/org/apache/comet/udf/CometRustUDF.scala‎

Lines changed: 21 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,13 +30,17 @@ import com.fasterxml.jackson.databind.ObjectMapper
3030
import com.fasterxml.jackson.databind.node.ObjectNode
3131

3232
/**
33-
* Public entry point for registering Rust scalar UDFs with Comet.
33+
* Entry point for registering Rust scalar UDFs with Comet.
3434
*
3535
* The UDF cdylib is built against the `comet-udf-sdk` crate and exposes its functions through an
3636
* ABI built only on the Arrow C Data Interface, so a compiled UDF is not tied to Comet's
3737
* DataFusion version.
3838
*
39-
* This is an experimental API and may change without the usual deprecation cycle.
39+
* This is an experimental API. It is deliberately not annotated
40+
* `org.apache.comet.annotation.Public`, so it sits outside the enumerated public API in Comet's
41+
* [[https://datafusion.apache.org/comet/about/versioning_policy.html versioning policy]] and
42+
* carries no compatibility guarantee: it may change or be removed in any release, including a
43+
* patch release, with no deprecation cycle.
4044
*/
4145
object CometRustUDF {
4246

@@ -52,6 +56,10 @@ object CometRustUDF {
5256
* Executors do not consult the driver's registry: the library path travels with the plan in the
5357
* `RustUdfCall` proto, and each executor loads the library itself on first use. The path must
5458
* therefore be valid on every executor, not just the driver.
59+
*
60+
* `deterministic` must be `true`. Comet plans every imported kernel as immutable, so a
61+
* nondeterministic UDF cannot yet be expressed; passing `false` fails here rather than silently
62+
* planning the function as pure.
5563
*/
5664
def register(
5765
spark: SparkSession,
@@ -60,6 +68,17 @@ object CometRustUDF {
6068
inputTypes: Seq[DataType],
6169
returnType: DataType,
6270
deterministic: Boolean = true): Unit = {
71+
if (!deterministic) {
72+
// The native signature is built once per library load with
73+
// Volatility::Immutable, while determinism is declared per registration, so the
74+
// flag cannot be honored without reworking how kernels are cached. Until then a
75+
// `false` here would let DataFusion constant-fold or CSE a call the user told us
76+
// was not safe to reuse.
77+
throw new IllegalArgumentException(
78+
s"Rust UDF '$name': deterministic = false is not supported yet. Comet plans Rust UDFs " +
79+
"as immutable, so a nondeterministic function may be constant-folded or eliminated " +
80+
"as a common subexpression. See https://github.com/apache/datafusion-comet/issues/5249")
81+
}
6382
val described = describeOne(libraryPath, name)
6483
require(described.name == name, s"unexpected name from native: ${described.name}")
6584
installCatalogStub(spark, name, inputTypes, returnType, deterministic)

‎spark/src/test/scala/org/apache/comet/CometRustUdfSuite.scala‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -248,6 +248,37 @@ class CometRustUdfSuite extends CometTestBase {
248248
spark.range(0, 2).selectExpr("echo_c(id, id) AS y").collect()
249249
}
250250
}
251+
252+
test("registering a nondeterministic UDF is refused") {
253+
// Comet plans every Rust UDF as immutable, so accepting this would let the optimizer
254+
// constant-fold or CSE a call the caller told us was not safe to reuse. Refuse at
255+
// registration rather than silently ignore the flag.
256+
val e = intercept[IllegalArgumentException] {
257+
CometRustUDF.register(
258+
spark,
259+
"echo_c",
260+
libPath,
261+
Seq(LongType),
262+
LongType,
263+
deterministic = false)
264+
}
265+
assert(e.getMessage.contains("deterministic = false is not supported"), s"unclear: $e")
266+
267+
// The check runs before any library work, so it fires on a path that does not exist
268+
// rather than reporting a load failure first.
269+
val early = intercept[IllegalArgumentException] {
270+
CometRustUDF.register(
271+
spark,
272+
"echo_c",
273+
"/no/such/library.so",
274+
Seq(LongType),
275+
LongType,
276+
deterministic = false)
277+
}
278+
assert(
279+
early.getMessage.contains("deterministic = false is not supported"),
280+
s"unclear: $early")
281+
}
251282
}
252283

253284
object CometRustUdfSuite {

0 commit comments

Comments
 (0)