Skip to content

Commit 26994c9

Browse files
committed
review: express default dispatcher routing declaratively
Replace the hand-rolled getSupportLevel/convert branching with the `CodegenDispatchFallback` mixin, which is what the framework already provides for exactly this shape. `exprToProtoInternal` routes an `Unsupported` result to the dispatcher plus a logDebug, a non-opted-in `Incompatible` to the dispatcher plus the [COMET-INFO] opt-in hint, and an opted-in `Incompatible` to the native proto with a logWarning. For `CometStringTranslate` that makes the whole diff one mixin on the original declaration: the pre-existing `Incompatible(Some(incompatReason))` was already the right classification. This also restores the `CometScalarFunction` base, so the function name "translate" stops being duplicated and the opted-in logWarning comes back. For `CometStructsToCsv` the original three-way `getSupportLevel` was likewise already correct -- `Unsupported` for complex field types, `Incompatible` for the #3232 types, `Incompatible()` otherwise -- so the +30/-30 rewrite collapses to the mixin. That restores `getUnsupportedReasons`, drops the duplicated `nativeSupported` predicate that had to stay in sync between two call sites, and picks up the logDebug on the opted-in-but-unsupported path for free. Also drops `CometCodegenDispatch` as the base, which is for expressions with no native path at all. Tests: - string_translate.sql gains the two rows that actually distinguish the paths: "e" + U+0301 (one grapheme, two code points) and U+0000 as the `to` argument. Built with decode(X'65CC81') / decode(X'00') to keep the fixture ASCII. Verified these discriminate: adding them to string_translate_enabled.sql, which opts into the native path, fails. - to_csv.sql gains options coverage (`sep`, and `timestampFormat`/`dateFormat` over date and timestamp fields), which the dispatcher path had nowhere. - CometCsvExpressionSuite "to_csv - default options" drops the allowIncompatible=true wrapper and covers c6 Double, c10 Timestamp and c11 TimestampNTZ, which it previously had to skip. It now tests what its name says. Two field types are deliberately left unasserted, both Spark behavior rather than Comet's, confirmed against Spark 3.5 with Comet disabled: - Binary: Spark's CSV converter renders it with Java's default toString, e.g. "[B@731af74", an identity hash that differs between any two evaluations, so it is not comparable by any engine including Spark against itself. - Complex types: a non-null array/map/struct renders as "org.apache.spark.sql.vectorized.ColumnarArray@1ada50f0" and any null complex value throws NullPointerException inside Spark's UnsafeWriter.write. Before this change these schemas fell back to Spark; now they reach the dispatcher, which runs the same Spark code, so the observable result is unchanged.
1 parent 3abb614 commit 26994c9

5 files changed

Lines changed: 106 additions & 74 deletions

File tree

‎spark/src/main/scala/org/apache/comet/serde/strings.scala‎

Lines changed: 7 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -109,38 +109,20 @@ object CometOctetLength extends CometScalarFunction[OctetLength]("octet_length")
109109
}
110110
}
111111

112+
// Routes through the JVM codegen dispatcher by default via `CodegenDispatchFallback`: the
113+
// `Incompatible` result below reaches the dispatcher (Spark's own `doGenCode`, bit-exact) instead
114+
// of falling the projection back to Spark. The native path stays available via `allowIncompatible`.
112115
object CometStringTranslate
113-
extends CometExpressionSerde[StringTranslate]
114-
with NativeOptInAvailable {
116+
extends CometScalarFunction[StringTranslate]("translate")
117+
with CodegenDispatchFallback {
115118
private val incompatReason =
116119
"DataFusion's translate iterates over Unicode graphemes (Spark uses code points) and" +
117120
" substitutes U+0000 instead of treating it as a deletion sentinel"
118121

119122
override def getIncompatibleReasons(): Seq[String] = Seq(incompatReason)
120123

121-
override def getSupportLevel(expr: StringTranslate): SupportLevel =
122-
if (!CometConf.isExprAllowIncompat(getExprConfigName(expr))) {
123-
Compatible(nativeOptIn =
124-
Some(NativeOptIn(CometConf.getExprAllowIncompatConfigKey(getExprConfigName(expr)))))
125-
} else {
126-
Compatible()
127-
}
128-
129-
override def convert(
130-
expr: StringTranslate,
131-
inputs: Seq[Attribute],
132-
binding: Boolean): Option[Expr] = {
133-
if (CometConf.isExprAllowIncompat(getExprConfigName(expr))) {
134-
val childExprs = expr.children.map(exprToProtoInternal(_, inputs, binding))
135-
val optExpr = scalarFunctionExprToProto("translate", childExprs: _*)
136-
optExprWithFallbackReason(optExpr, expr, expr.children: _*)
137-
} else {
138-
// Default: route through the codegen dispatcher so Spark's own doGenCode runs inside the
139-
// Comet pipeline (bit-exact). The native path differs on graphemes vs code points, so it is
140-
// opt-in via allowIncompatible.
141-
CometScalaUDF.emitJvmCodegenDispatch(expr, inputs, binding)
142-
}
143-
}
124+
override def getSupportLevel(expr: StringTranslate): SupportLevel = Incompatible(
125+
Some(incompatReason))
144126
}
145127

146128
object CometLevenshtein extends CometExpressionSerde[Levenshtein] {

‎spark/src/main/scala/org/apache/comet/serde/structs.scala‎

Lines changed: 35 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -259,50 +259,55 @@ object CometJsonToStructs extends CometCodegenDispatch[JsonToStructs] with Nativ
259259
}
260260
}
261261

262-
object CometStructsToCsv extends CometCodegenDispatch[StructsToCsv] with NativeOptInAvailable {
262+
// Routes through the JVM codegen dispatcher by default: the `Unsupported` (complex field types)
263+
// and non-opted-in `Incompatible` (#3232 field types) results below are both handled by
264+
// `CodegenDispatchFallback`, which runs Spark's own `doGenCode` inside the Comet pipeline so the
265+
// projection stays native and bit-exact. The native ToCsv path remains available via
266+
// `allowIncompatible`, for the field types it can actually handle.
267+
object CometStructsToCsv extends CometExpressionSerde[StructsToCsv] with CodegenDispatchFallback {
263268

264269
private val incompatibleDataTypes = Seq(DateType, TimestampType, TimestampNTZType, BinaryType)
265270

266271
override def getIncompatibleReasons(): Seq[String] = Seq(
267272
"Date, Timestamp, TimestampNTZ, and Binary data types may produce different results" +
268273
" (https://github.com/apache/datafusion-comet/issues/3232)")
269274

270-
// The native ToCsv path only supports non-complex, compatible field types. Everything else
271-
// (and the default, unless opted in) runs through the codegen dispatcher, which is bit-exact.
272-
private def nativeSupported(expr: StructsToCsv): Boolean = {
273-
val dataTypes = expr.inputSchema.fields.map(_.dataType)
274-
!dataTypes.exists(DataTypeSupport.isComplexType) &&
275-
!dataTypes.exists(incompatibleDataTypes.contains)
276-
}
275+
override def getUnsupportedReasons(): Seq[String] = Seq(
276+
"Complex types (arrays, maps, structs) in the schema are not supported")
277277

278-
override def getSupportLevel(expr: StructsToCsv): SupportLevel =
279-
if (!CometConf.isExprAllowIncompat(getExprConfigName(expr)) && nativeSupported(expr)) {
280-
Compatible(nativeOptIn =
281-
Some(NativeOptIn(CometConf.getExprAllowIncompatConfigKey(getExprConfigName(expr)))))
282-
} else {
283-
Compatible()
278+
override def getSupportLevel(expr: StructsToCsv): SupportLevel = {
279+
val dataTypes = expr.inputSchema.fields.map(_.dataType)
280+
val containsComplexType = dataTypes.exists(DataTypeSupport.isComplexType)
281+
if (containsComplexType) {
282+
return Unsupported(
283+
Some(
284+
s"The schema ${expr.inputSchema} is not supported because it includes a complex type"))
285+
}
286+
val containsIncompatibleDataTypes = dataTypes.exists(incompatibleDataTypes.contains)
287+
if (containsIncompatibleDataTypes) {
288+
return Incompatible(
289+
Some(
290+
s"The schema ${expr.inputSchema} is not supported because " +
291+
s"it includes a incompatible data types: $incompatibleDataTypes"))
284292
}
293+
// https://github.com/apache/datafusion-comet/issues/3232
294+
Incompatible()
295+
}
285296

286297
override def convert(
287298
expr: StructsToCsv,
288299
inputs: Seq[Attribute],
289300
binding: Boolean): Option[ExprOuterClass.Expr] = {
290-
if (CometConf.isExprAllowIncompat(getExprConfigName(expr)) && nativeSupported(expr)) {
291-
for {
292-
childProto <- exprToProtoInternal(expr.child, inputs, binding)
293-
} yield {
294-
val optionsProto = options2Proto(expr.options, expr.timeZoneId)
295-
val toCsv = ExprOuterClass.ToCsv
296-
.newBuilder()
297-
.setChild(childProto)
298-
.setOptions(optionsProto)
299-
.build()
300-
ExprOuterClass.Expr.newBuilder().setToCsv(toCsv).build()
301-
}
302-
} else {
303-
// Default: route through the codegen dispatcher so Spark's own doGenCode runs inside the
304-
// Comet pipeline (bit-exact). The native path is opt-in via allowIncompatible.
305-
super.convert(expr, inputs, binding)
301+
for {
302+
childProto <- exprToProtoInternal(expr.child, inputs, binding)
303+
} yield {
304+
val optionsProto = options2Proto(expr.options, expr.timeZoneId)
305+
val toCsv = ExprOuterClass.ToCsv
306+
.newBuilder()
307+
.setChild(childProto)
308+
.setOptions(optionsProto)
309+
.build()
310+
ExprOuterClass.Expr.newBuilder().setToCsv(toCsv).build()
306311
}
307312
}
308313

‎spark/src/test/resources/sql-tests/expressions/csv/to_csv.sql‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,3 +39,39 @@ query
3939
SELECT
4040
to_csv(named_struct('a', 1, 'b', 'x', 'c', 2.5)),
4141
to_csv(named_struct('s', 'a,b', 'n', CAST(NULL AS INT)))
42+
43+
-- Options are the main axis of behavior difference for this expression, and the dispatcher path
44+
-- had no options coverage anywhere (CometCsvExpressionSuite only varies them under
45+
-- allowIncompatible=true). `sep` also changes which values need quoting.
46+
query
47+
SELECT to_csv(named_struct('a', a, 'b', b, 'c', c), map('sep', ';')) FROM test_to_csv
48+
49+
-- timestampFormat over date and timestamp fields: these are the #3232 types that were
50+
-- Incompatible before this change and now route through the dispatcher.
51+
-- BinaryType is the other #3232 type but is deliberately absent: Spark's CSV converter renders it
52+
-- with Java's default Object.toString(), e.g. "[B@10bc15e4", an identity hash that differs between
53+
-- any two evaluations, so the value is not assertable by any engine including Spark itself.
54+
statement
55+
CREATE TABLE test_to_csv_temporal(d date, t timestamp) USING parquet
56+
57+
statement
58+
INSERT INTO test_to_csv_temporal VALUES
59+
(DATE '2024-01-31', TIMESTAMP '2024-01-31 12:34:56.789'),
60+
(DATE '1970-01-01', TIMESTAMP '1970-01-01 00:00:00'),
61+
(NULL, NULL)
62+
63+
query
64+
SELECT to_csv(named_struct('d', d, 't', t)) FROM test_to_csv_temporal
65+
66+
query
67+
SELECT to_csv(named_struct('d', d, 't', t), map('timestampFormat', 'yyyy/MM/dd HH:mm', 'dateFormat', 'dd-MM-yyyy')) FROM test_to_csv_temporal
68+
69+
-- Complex field types (arrays, maps, nested structs) are deliberately not asserted here.
70+
-- StructsToCsv.checkInputDataTypes accepts them, so they reach the converter, but Spark's own
71+
-- output for them is not a value that can be compared: a non-null array/map/struct renders as a
72+
-- Java identity string such as "org.apache.spark.sql.vectorized.ColumnarArray@1ada50f0", and any
73+
-- null complex value throws NullPointerException inside Spark's UnsafeWriter.write. Both were
74+
-- confirmed against Spark 3.5 with Comet disabled entirely, so they are Spark behavior, not
75+
-- Comet's. Before this change these schemas were Unsupported and fell the projection back to
76+
-- Spark; now they reach the dispatcher, which runs the same Spark code, so the observable result
77+
-- is unchanged either way.

‎spark/src/test/resources/sql-tests/expressions/string/string_translate.sql‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,8 +24,20 @@
2424
statement
2525
CREATE TABLE test_translate(s string, from_str string, to_str string) USING parquet
2626

27+
-- The last two rows are the regression test for this file's routing change: they are the inputs
28+
-- where the native path is known to disagree with Spark, so they are only assertable now that the
29+
-- default is the bit-exact dispatcher. Built with decode() rather than literal characters to keep
30+
-- the fixture ASCII.
31+
-- decode(X'65CC81') is "e" + U+0301 COMBINING ACUTE ACCENT: one grapheme, two code points.
32+
-- DataFusion iterates graphemes, Spark iterates code points, so translating "e" differs.
33+
-- decode(X'00') as the `to` argument is U+0000, which Spark treats as a deletion sentinel and
34+
-- the native path substitutes literally.
2735
statement
28-
INSERT INTO test_translate VALUES ('hello', 'el', 'ip'), ('hello', 'aeiou', '12345'), ('', 'a', 'b'), (NULL, 'a', 'b'), ('hello', '', ''), ('abc', 'abc', 'x')
36+
INSERT INTO test_translate VALUES
37+
('hello', 'el', 'ip'), ('hello', 'aeiou', '12345'), ('', 'a', 'b'), (NULL, 'a', 'b'),
38+
('hello', '', ''), ('abc', 'abc', 'x'),
39+
(concat('caf', decode(X'65CC81', 'UTF-8')), 'e', 'E'),
40+
('hello', 'l', decode(X'00', 'UTF-8'))
2941

3042
query
3143
SELECT translate(s, from_str, to_str) FROM test_translate

‎spark/src/test/scala/org/apache/comet/CometCsvExpressionSuite.scala‎

Lines changed: 15 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -47,24 +47,21 @@ class CometCsvExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper
4747
SchemaGenOptions(generateArray = false, generateStruct = false, generateMap = false),
4848
DataGenOptions(allowNull = true, generateNegativeZero = true))
4949
}
50-
withSQLConf(CometConf.getExprAllowIncompatConfigKey(classOf[StructsToCsv]) -> "true") {
51-
val df = spark.read
52-
.parquet(filename)
53-
.select(
54-
to_csv(
55-
struct(
56-
col("c0"),
57-
col("c1"),
58-
col("c2"),
59-
col("c3"),
60-
col("c4"),
61-
col("c5"),
62-
col("c7"),
63-
col("c8"),
64-
col("c9"),
65-
col("c12"))))
66-
checkSparkAnswerAndOperator(df)
67-
}
50+
// Every column in the fuzz schema except c13 Binary, with no allowIncompatible opt-in:
51+
// to_csv now routes through the codegen dispatcher by default, so Spark's own converter runs
52+
// and all field types match by construction. This previously ran under
53+
// allowIncompatible=true and had to skip c6 Double, c10 Timestamp and c11 TimestampNTZ,
54+
// which are types the native path cannot match Spark on (the latter two are #3232).
55+
//
56+
// c13 Binary stays out, and not because of a divergence: Spark's CSV converter renders
57+
// BinaryType with Java's default Object.toString(), so a row comes out as
58+
// "...,[B@731af74". That is an identity hash of the byte array instance, which differs
59+
// between any two evaluations, so the value can never be asserted against a second run --
60+
// by Comet or by Spark against itself.
61+
val df = spark.read
62+
.parquet(filename)
63+
.select(to_csv(struct((0 to 12).map(i => col(s"c$i")): _*)))
64+
checkSparkAnswerAndOperator(df)
6865
}
6966
}
7067

0 commit comments

Comments
 (0)