Skip to content

Commit b0f0de3

Browse files
authored
fix: dispatch StaticInvoke and Invoke only into Spark's own classes (#6542)
* fix: dispatch StaticInvoke and Invoke only into Spark's own classes Since #5692, CometStaticInvoke and CometInvoke send every call they do not otherwise handle to the codegen dispatcher, including calls to DataSource V2 catalog functions. Such a function can return a Decimal at a different scale from its declared type, or one that does not fit it. Spark corrects that only when it writes a row, and an expression around the call reads the value as returned, so the dispatcher's Arrow output cannot match it (#6425). The dispatcher now runs a StaticInvoke or Invoke only when it calls one of Spark's own classes, or the predicate of a typed Dataset.filter, and never an ApplyFunctionExpression. It checks the whole tree it is given, since the kernel runs every node in it. Anything else falls back to Spark, as in 1.0.0. Closes #6425. * test: tidy the #6425 fallback reason, fixture and docs The decline reason no longer claims that only Spark's own code runs in the dispatcher, which the typed filter predicate contradicts. The DSv2 fixture no longer forces a single file, which only #6455's writer tests needed. The dispatch benchmark's description and the Iceberg guide match the allow list. * test: assert that a DSv2 call in a shuffle's partitioning falls back On main's writer the partition ids happen to match Spark for these values, so comparing answers alone could not catch the call being dispatched. * refactor: simplify the dispatcher allow list and its #6425 tests Fold the typed filter rule into the callee lookup and name the DSv2 check once. The tests use Spark's InMemoryCatalog and the suite's withTypedCol instead of their own catalog and table setup, derive the static-invoke fixture from the shared one, drop a leftover Invoke target, and pin how each fixture lowers.
1 parent 486be2c commit b0f0de3

7 files changed

Lines changed: 355 additions & 64 deletions

File tree

‎docs/source/user-guide/latest/expressions.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -705,7 +705,7 @@ Comet also accelerates a number of Catalyst expressions that have no Spark SQL f
705705
- **Accessor expressions (subscript and field access, not functions):** struct field access (`col.field`), array element access (`arr[i]`), and map value access (`map[key]`).
706706
- **Internal decimal arithmetic:** `CheckOverflow`, `MakeDecimal`, and `UnscaledValue`, which the analyzer inserts around decimal operations.
707707
- **User-defined functions:** Scala UDFs registered through the DataFrame or SQL API.
708-
- **DataSource V2 catalog functions:** Iceberg's system functions `bucket`, `truncate`, `years`, `months`, `days`, and `hours` (for example `system.bucket(16, id)`) run natively; see [Iceberg system functions](iceberg.md#iceberg-system-functions).
708+
- **DataSource V2 catalog functions:** Iceberg's system functions `bucket`, `truncate`, `years`, `months`, `days`, and `hours` (for example `system.bucket(16, id)`) run natively; see [Iceberg system functions](iceberg.md#iceberg-system-functions). Other DataSource V2 catalog functions run in Spark.
709709
- **Lowered built-ins:** Spark lowers some built-in functions to `StaticInvoke` or `Invoke` calls. Those without a native mapping run through the JVM codegen dispatcher when their input and output types are supported.
710710
- **Structural expressions:** aliases, attribute references, literals, sort orders, and `CASE WHEN`.
711711

‎docs/source/user-guide/latest/iceberg.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -224,7 +224,9 @@ Iceberg's system functions `bucket`, `truncate`, `years`, `months`, `days`, and
224224
form of its partition transforms, for example `SELECT system.bucket(16, id) FROM t`) run natively.
225225
Spark binds them as static invocations of Iceberg's per-type implementations under
226226
`org.apache.iceberg.spark.functions`, and Comet recognizes those classes wherever the expression
227-
appears: in a projection, a filter, a sort key, or the hash partitioning of a shuffle.
227+
appears: in a projection, a filter, a sort key, or the hash partitioning of a shuffle. The
228+
exception is a call nested inside an expression that Comet runs through the JVM codegen
229+
dispatcher, such as `map(...)`. That makes the operator fall back to Spark.
228230

229231
The native kernels reproduce Iceberg's Java semantics exactly rather than approximately:
230232

‎spark/src/main/scala/org/apache/comet/serde/CometScalaUDF.scala‎

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -65,9 +65,10 @@ object CometScalaUDF extends CometExpressionSerde[ScalaUDF] {
6565
* batch kernel on first invocation per task.
6666
*
6767
* Returns `None` (with `withFallbackReason` tagging the reason) when the dispatcher is disabled
68-
* via [[CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED]], when [[CometBatchKernelCodegen.canHandle]]
69-
* refuses the expression tree, or when the bound tree cannot be closure-serialized. Callers
70-
* should treat `None` as a clean Spark-fallback signal; this method never throws.
68+
* via [[CometConf.COMET_SCALA_UDF_CODEGEN_ENABLED]], when the tree calls into code other than
69+
* Spark's own (see [[CometInvokeTargets]]), when [[CometBatchKernelCodegen.canHandle]] refuses
70+
* the expression tree, or when the bound tree cannot be closure-serialized. Callers should
71+
* treat `None` as a clean Spark-fallback signal; this method never throws.
7172
*/
7273
def emitJvmCodegenDispatch(
7374
expr: Expression,
@@ -92,6 +93,15 @@ object CometScalaUDF extends CometExpressionSerde[ScalaUDF] {
9293
case other => other
9394
}
9495

96+
// The kernel runs every call in the tree, not only the root, so a DataSource V2 function
97+
// under a dispatched `map(...)` would run in it too. Check the whole tree.
98+
CometInvokeTargets.declineReason(target) match {
99+
case Some(reason) =>
100+
withFallbackReason(expr, s"$exprName: codegen dispatch: $reason")
101+
return None
102+
case None =>
103+
}
104+
95105
// Bind against only the AttributeReferences the tree actually reads, so ordinals align with
96106
// the data args we ship.
97107
val attrs = target.collect { case a: AttributeReference => a }.distinct

‎spark/src/main/scala/org/apache/comet/serde/statics.scala‎

Lines changed: 85 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -19,11 +19,14 @@
1919

2020
package org.apache.comet.serde
2121

22+
import org.apache.spark.api.java.function.FilterFunction
2223
import org.apache.spark.sql.catalyst.expressions.{ApplyFunctionExpression, Attribute, Base64, Expression, ExpressionImplUtils, Literal, StringDecode, TryEval, UrlCodec}
2324
import org.apache.spark.sql.catalyst.expressions.objects.{Invoke, StaticInvoke}
2425
import org.apache.spark.sql.catalyst.util.CharVarcharCodegenUtils
25-
import org.apache.spark.sql.types.StringType
26+
import org.apache.spark.sql.connector.catalog.functions.ScalarFunction
27+
import org.apache.spark.sql.types.{BooleanType, ObjectType, StringType}
2628

29+
import org.apache.comet.CometExplainInfo
2730
import org.apache.comet.CometSparkSessionExtensions.withFallbackReason
2831
import org.apache.comet.serde.QueryPlanSerde.{exprToProtoInternal, scalarFunctionExprToProto, scalarFunctionExprToProtoWithReturnType}
2932

@@ -84,7 +87,8 @@ object CometStaticInvoke extends CometExpressionSerde[StaticInvoke] {
8487
* Arrow `Decimal128(precision, scale)` vector just like a native kernel does, so it produces
8588
* the out-of-range value instead of a null. The mixin's contract ("the case must be something
8689
* `doGenCode` can compile") does not cover a limit that lives at the Arrow output boundary, so
87-
* enrollment stays with the individual handlers.
90+
* enrollment stays with the individual handlers. [[CometInvokeTargets]] now keeps Iceberg's
91+
* classes out of the dispatcher as well.
8892
*/
8993
override def getSupportLevel(expr: StaticInvoke): SupportLevel =
9094
handlerFor(expr)
@@ -119,7 +123,9 @@ object CometStaticInvoke extends CometExpressionSerde[StaticInvoke] {
119123
// `StaticInvoke.doGenCode` emits a static method call, so the kernel matches Spark by
120124
// construction. Spark 4.x keeps lowering more `RuntimeReplaceable` functions this way
121125
// (`encode`, `is_valid_utf8`, the `TIME` family, ...) and `lpad` / `rpad` on binary
122-
// has lowered to `StaticInvoke(ByteArray, ...)` since Spark 3.4.
126+
// has lowered to `StaticInvoke(ByteArray, ...)` since Spark 3.4. The dispatcher
127+
// declines a call into any class that is not Spark's, such as a DataSource V2
128+
// function, so that falls back (see [[CometInvokeTargets]]).
123129
//
124130
// The encoder and deserializer trees that make up most `StaticInvoke` usage in typed
125131
// Dataset operations are unaffected: their arguments are `ObjectType`, which
@@ -167,17 +173,92 @@ object CometApplyFunctionExpression extends CometExpressionSerde[ApplyFunctionEx
167173
}
168174

169175
/**
170-
* Catch-all for `Invoke`, which has no allowlist at all. Spark 4.x lowers a growing number of
176+
* Catch-all for `Invoke`, which has no allowlist of its own. Spark 4.x lowers a growing number of
171177
* `RuntimeReplaceable` expressions to evaluator-backed `Invoke` nodes; the Spark 4.x shim
172178
* reconstructs the handful it recognizes and everything else lands here. `Invoke.doGenCode` emits
173179
* a method call on the target object, so the dispatcher runs Spark's own implementation inside
174180
* the Comet pipeline rather than failing the operator back to Spark.
175181
*
182+
* A DataSource V2 function with an instance `invoke` method lowers to `Invoke` too. The
183+
* dispatcher declines that, and any other call into a class that is not Spark's, so the operator
184+
* falls back (see [[CometInvokeTargets]]).
185+
*
176186
* As with [[CometStaticInvoke]], the object-typed `Invoke` nodes in encoder / deserializer trees
177187
* are rejected by `CometBatchKernelCodegen.canHandle` and fall back as before.
178188
*/
179189
object CometInvoke extends CometCodegenDispatch[Invoke]
180190

191+
/**
192+
* The classes the codegen dispatcher may call into from a `StaticInvoke` or an `Invoke`: Spark's
193+
* own, other than a DataSource V2 function, plus the predicate of a typed `Dataset.filter`.
194+
* [[CometScalaUDF.emitJvmCodegenDispatch]] checks every node of a dispatched tree.
195+
*
196+
* Spark's helpers return values that already have their declared type. Other code need not, and
197+
* Spark corrects such a value only when it writes a row. A DataSource V2 function can return a
198+
* `Decimal` at a different scale from the type it declares, or one that does not fit it: Spark
199+
* rescales it, or writes null, when it writes the row, and an expression around the call reads
200+
* the value as returned (#6425). The kernel has to write an Arrow vector of the declared type, so
201+
* a native consumer of its output would read something else, and where Spark writes rows depends
202+
* on whole-stage codegen, so Comet cannot reproduce it. Any other call, including every
203+
* `ApplyFunctionExpression`, makes the operator fall back to Spark instead, as it did before
204+
* #5692 routed these calls through the dispatcher.
205+
*/
206+
object CometInvokeTargets {
207+
208+
/** The packages of Spark's expression helpers and evaluators, and of its internal values. */
209+
private val SparkPackages = Seq("org.apache.spark.sql.catalyst.", "org.apache.spark.unsafe.")
210+
211+
/**
212+
* The types Spark's `TypedFilter` calls the predicate of a typed `Dataset.filter` through. The
213+
* predicate is user code, but it returns a boolean, which a row write has nothing to correct
214+
* in.
215+
*/
216+
private val TypedFilterPredicates: Set[Class[_]] =
217+
Set(classOf[scala.Function1[_, _]], classOf[FilterFunction[_]])
218+
219+
/**
220+
* Why the dispatcher must not run `expr`, naming the first call in it into a class the
221+
* dispatcher does not allow, or `None` when it allows them all.
222+
*/
223+
def declineReason(expr: Expression): Option[String] =
224+
expr.find(checkedCallee(_).exists(!isSparkCode(_))).map { node =>
225+
val cls = checkedCallee(node).get
226+
val target =
227+
if (isV2Function(cls)) {
228+
s"the DataSource V2 function ${cls.getName}"
229+
} else {
230+
s"${cls.getName}, which is not part of Spark"
231+
}
232+
s"${CometExplainInfo.exprDisplayName(node)} calls $target"
233+
}
234+
235+
private def isV2Function(cls: Class[_]): Boolean =
236+
classOf[ScalarFunction[_]].isAssignableFrom(cls)
237+
238+
private def isSparkCode(cls: Class[_]): Boolean =
239+
!isV2Function(cls) && SparkPackages.exists(p => cls.getName.startsWith(p))
240+
241+
/**
242+
* The class whose code `expr` calls, unless it calls a method of a Catalyst value or is the
243+
* predicate of a typed filter.
244+
*/
245+
private def checkedCallee(expr: Expression): Option[Class[_]] = expr match {
246+
case s: StaticInvoke => Some(s.staticObject)
247+
case i: Invoke =>
248+
i.targetObject.dataType match {
249+
case ObjectType(cls)
250+
if TypedFilterPredicates.contains(cls) && i.dataType == BooleanType =>
251+
None
252+
case ObjectType(cls) => Some(cls)
253+
// Any other target is a Catalyst value, such as the `UTF8String` that Spark 4 lowers
254+
// `is_valid_utf8` to call `isValid` on.
255+
case _ => None
256+
}
257+
case a: ApplyFunctionExpression => Some(a.function.getClass)
258+
case _ => None
259+
}
260+
}
261+
181262
object CometUrlEncodeStaticInvoke extends CometExpressionSerde[StaticInvoke] {
182263
override def convert(
183264
expr: StaticInvoke,

0 commit comments

Comments
 (0)