Skip to content

Commit 5e302d9

Browse files
authored
feat: route translate and to_csv through codegen dispatch by default (#5032)
* feat: route translate and to_csv through codegen dispatch by default `translate` (StringTranslate) and `to_csv` (StructsToCsv) have native implementations that are incompatible with Spark by default, so today the whole enclosing operator falls back to Spark unless allowIncompatible is set. Route them through the JVM codegen dispatcher by default instead, using the same NativeOptInAvailable pattern as `to_json` and `split`: the dispatcher runs Spark's own generated code (bit-exact) while keeping the operator on the native pipeline, and the faster native path stays available via allowIncompatible. Update the translate SQL test (previously asserting fallback) to verify native dispatch, and add a to_csv test. * review: express default dispatcher routing declaratively Replace the hand-rolled getSupportLevel/convert branching with the `CodegenDispatchFallback` mixin, which is what the framework already provides for exactly this shape. `exprToProtoInternal` routes an `Unsupported` result to the dispatcher plus a logDebug, a non-opted-in `Incompatible` to the dispatcher plus the [COMET-INFO] opt-in hint, and an opted-in `Incompatible` to the native proto with a logWarning. For `CometStringTranslate` that makes the whole diff one mixin on the original declaration: the pre-existing `Incompatible(Some(incompatReason))` was already the right classification. This also restores the `CometScalarFunction` base, so the function name "translate" stops being duplicated and the opted-in logWarning comes back. For `CometStructsToCsv` the original three-way `getSupportLevel` was likewise already correct -- `Unsupported` for complex field types, `Incompatible` for the #3232 types, `Incompatible()` otherwise -- so the +30/-30 rewrite collapses to the mixin. That restores `getUnsupportedReasons`, drops the duplicated `nativeSupported` predicate that had to stay in sync between two call sites, and picks up the logDebug on the opted-in-but-unsupported path for free. Also drops `CometCodegenDispatch` as the base, which is for expressions with no native path at all. Tests: - string_translate.sql gains the two rows that actually distinguish the paths: "e" + U+0301 (one grapheme, two code points) and U+0000 as the `to` argument. Built with decode(X'65CC81') / decode(X'00') to keep the fixture ASCII. Verified these discriminate: adding them to string_translate_enabled.sql, which opts into the native path, fails. - to_csv.sql gains options coverage (`sep`, and `timestampFormat`/`dateFormat` over date and timestamp fields), which the dispatcher path had nowhere. - CometCsvExpressionSuite "to_csv - default options" drops the allowIncompatible=true wrapper and covers c6 Double, c10 Timestamp and c11 TimestampNTZ, which it previously had to skip. It now tests what its name says. Two field types are deliberately left unasserted, both Spark behavior rather than Comet's, confirmed against Spark 3.5 with Comet disabled: - Binary: Spark's CSV converter renders it with Java's default toString, e.g. "[B@731af74", an identity hash that differs between any two evaluations, so it is not comparable by any engine including Spark against itself. - Complex types: a non-null array/map/struct renders as "org.apache.spark.sql.vectorized.ColumnarArray@1ada50f0" and any null complex value throws NullPointerException inside Spark's UnsafeWriter.write. Before this change these schemas fell back to Spark; now they reach the dispatcher, which runs the same Spark code, so the observable result is unchanged. * review: pin to_csv dispatcher behavior for ANSI casts and complex field types Addresses review feedback on #5032. - to_csv_ansi.sql: a raising Cast inside the named_struct, under ANSI, in both the single-ordinal and the two-ordinal (#5219) shape, plus a try_cast counterpart and the sentinel query the ExpectError convention requires. - to_csv.sql: complex field types (array / map / nested struct) reach the runtime dispatch because isSupportedDataType recurses, so assert the dispatcher hands the nested InternalRow / ArrayData / MapData through without raising. On Spark 3.4 / 3.5 that is the CometSpecializedGettersDispatch generic get; verified the queries fail when its complex-type branches are removed. - to_csv_nested.sql (Spark 4.0+): Spark 4.0 gave UnivocityGenerator real array / map / struct converters, so from 4.0 on the rendered value is deterministic and can be asserted in full rather than only for nullness. - string_translate.sql: the two divergence rows used decode(X'..') inside an inline table, which Spark 4.1 rejects with INVALID_INLINE_TABLE. Use \u escapes instead; verified the rows still fail under allowIncompatible=true.
1 parent bdd4b90 commit 5e302d9

8 files changed

Lines changed: 275 additions & 32 deletions

File tree

‎docs/source/user-guide/latest/expressions.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -239,7 +239,7 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci
239239
| --- | --- | --- | --- |
240240
| `from_csv` | ✅ | Codegen dispatch | |
241241
| `schema_of_csv` | ✅ | Codegen dispatch | |
242-
| `to_csv` | ✅ | Native | |
242+
| `to_csv` | ✅ | Hybrid | Codegen dispatch by default; the native path is opt-in via allowIncompatible |
243243

244244
---
245245

@@ -610,7 +610,7 @@ The type-name conversion functions (`bigint`, `binary`, `boolean`, `date`, `deci
610610
| `to_char` | ✅ | Codegen dispatch | |
611611
| `to_number` | ✅ | Codegen dispatch | |
612612
| `to_varchar` | ✅ | Codegen dispatch | |
613-
| `translate` | ✅ | Native | DataFusion's `translate` iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel, so the native path is opt-in via allowIncompatible |
613+
| `translate` | ✅ | Hybrid | Codegen dispatch by default: DataFusion's `translate` iterates over Unicode graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion sentinel, so the native path is opt-in via allowIncompatible |
614614
| `trim` | ✅ | Native | |
615615
| `try_to_binary` | ✅ | — | Runs natively (rewrites to `try_eval(to_binary(...))`) |
616616
| `try_to_number` | ✅ | Codegen dispatch | Routed through the JVM codegen dispatcher |

‎spark/src/main/scala/org/apache/comet/serde/strings.scala‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -109,7 +109,12 @@ object CometOctetLength extends CometScalarFunction[OctetLength]("octet_length")
109109
}
110110
}
111111

112-
object CometStringTranslate extends CometScalarFunction[StringTranslate]("translate") {
112+
// Routes through the JVM codegen dispatcher by default via `CodegenDispatchFallback`: the
113+
// `Incompatible` result below reaches the dispatcher (Spark's own `doGenCode`, bit-exact) instead
114+
// of falling the projection back to Spark. The native path stays available via `allowIncompatible`.
115+
object CometStringTranslate
116+
extends CometScalarFunction[StringTranslate]("translate")
117+
with CodegenDispatchFallback {
113118
private val incompatReason =
114119
"DataFusion's translate iterates over Unicode graphemes (Spark uses code points) and" +
115120
" substitutes U+0000 instead of treating it as a deletion sentinel"

‎spark/src/main/scala/org/apache/comet/serde/structs.scala‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -258,7 +258,12 @@ object CometJsonToStructs extends CometCodegenDispatch[JsonToStructs] with Nativ
258258
}
259259
}
260260

261-
object CometStructsToCsv extends CometExpressionSerde[StructsToCsv] {
261+
// Routes through the JVM codegen dispatcher by default: the `Unsupported` (complex field types)
262+
// and non-opted-in `Incompatible` (#3232 field types) results below are both handled by
263+
// `CodegenDispatchFallback`, which runs Spark's own `doGenCode` inside the Comet pipeline so the
264+
// projection stays native and bit-exact. The native ToCsv path remains available via
265+
// `allowIncompatible`, for the field types it can actually handle.
266+
object CometStructsToCsv extends CometExpressionSerde[StructsToCsv] with CodegenDispatchFallback {
262267

263268
private val incompatibleDataTypes = Seq(DateType, TimestampType, TimestampNTZType, BinaryType)
264269

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
-- Licensed to the Apache Software Foundation (ASF) under one
2+
-- or more contributor license agreements. See the NOTICE file
3+
-- distributed with this work for additional information
4+
-- regarding copyright ownership. The ASF licenses this file
5+
-- to you under the Apache License, Version 2.0 (the
6+
-- "License"); you may not use this file except in compliance
7+
-- with the License. You may obtain a copy of the License at
8+
--
9+
-- http://www.apache.org/licenses/LICENSE-2.0
10+
--
11+
-- Unless required by applicable law or agreed to in writing,
12+
-- software distributed under the License is distributed on an
13+
-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
-- KIND, either express or implied. See the License for the
15+
-- specific language governing permissions and limitations
16+
-- under the License.
17+
18+
-- to_csv runs through the codegen dispatcher by default so results match Spark exactly, including
19+
-- quoting and escaping. The native path is opt-in via
20+
-- spark.comet.expression.StructsToCsv.allowIncompatible.
21+
22+
statement
23+
CREATE TABLE test_to_csv(a int, b string, c double) USING parquet
24+
25+
statement
26+
INSERT INTO test_to_csv VALUES
27+
(1, 'x', 2.5),
28+
(-3, 'hello,world', 0.0),
29+
(0, 'has "quote"', -1.5),
30+
(NULL, NULL, NULL),
31+
(7, '', 3.0)
32+
33+
-- column struct: values with delimiters and quotes exercise Spark's CSV quoting rules
34+
query
35+
SELECT to_csv(named_struct('a', a, 'b', b, 'c', c)) FROM test_to_csv
36+
37+
-- literal struct (constant folding is disabled by the test suite)
38+
query
39+
SELECT
40+
to_csv(named_struct('a', 1, 'b', 'x', 'c', 2.5)),
41+
to_csv(named_struct('s', 'a,b', 'n', CAST(NULL AS INT)))
42+
43+
-- Options are the main axis of behavior difference for this expression, and the dispatcher path
44+
-- had no options coverage anywhere (CometCsvExpressionSuite only varies them under
45+
-- allowIncompatible=true). `sep` also changes which values need quoting.
46+
query
47+
SELECT to_csv(named_struct('a', a, 'b', b, 'c', c), map('sep', ';')) FROM test_to_csv
48+
49+
-- timestampFormat over date and timestamp fields: these are the #3232 types that were
50+
-- Incompatible before this change and now route through the dispatcher.
51+
-- BinaryType is the other #3232 type but is deliberately absent: on Spark 3.4 / 3.5 the CSV
52+
-- converter has no binary branch and renders it with Java's default Object.toString(), e.g.
53+
-- "[B@10bc15e4", an identity hash that differs between any two evaluations, so the value is not
54+
-- assertable by any engine including Spark itself. (Spark 4.0 added a real binary formatter, but
55+
-- this fixture runs on every supported version.)
56+
statement
57+
CREATE TABLE test_to_csv_temporal(d date, t timestamp) USING parquet
58+
59+
statement
60+
INSERT INTO test_to_csv_temporal VALUES
61+
(DATE '2024-01-31', TIMESTAMP '2024-01-31 12:34:56.789'),
62+
(DATE '1970-01-01', TIMESTAMP '1970-01-01 00:00:00'),
63+
(NULL, NULL)
64+
65+
query
66+
SELECT to_csv(named_struct('d', d, 't', t)) FROM test_to_csv_temporal
67+
68+
query
69+
SELECT to_csv(named_struct('d', d, 't', t), map('timestampFormat', 'yyyy/MM/dd HH:mm', 'dateFormat', 'dd-MM-yyyy')) FROM test_to_csv_temporal
70+
71+
-- Complex field types (arrays, maps, nested structs). `StructsToCsv.checkInputDataTypes` accepts
72+
-- them and `CometBatchKernelCodegen.isSupportedDataType` recurses into them, so they pass the
73+
-- plan-time gate and reach the runtime dispatch. That is the accepted-at-plan-time /
74+
-- rejected-at-runtime shape #5219 found for TIME types: `canHandle` greenlights the expression
75+
-- before the plan commits, so a runtime gap is an execute-time failure with no fallback left.
76+
--
77+
-- Only non-nullness is asserted here, because the rendered value is not comparable on every
78+
-- supported Spark version. Spark 3.4 / 3.5's `UnivocityGenerator.makeConverter` has no branch for
79+
-- complex types and lands on `getter.get(ordinal, dataType).toString`, an identity string such as
80+
-- "org.apache.spark.sql.vectorized.ColumnarArray@1ada50f0" whose hash differs between any two
81+
-- evaluations and whose class differs between Spark's converter input (`ColumnarArray` /
82+
-- `UnsafeArrayData`) and the kernel's (`InputArray_*`). That generic `get` is exactly what
83+
-- `CometSpecializedGettersDispatch` implements for `CometInternalRow` / `CometArrayData`, so on
84+
-- those versions these queries are the coverage for it. Spark 4.0 added real array/map/struct
85+
-- converters, which render deterministically; `to_csv_nested.sql` asserts those values in full.
86+
statement
87+
CREATE TABLE test_to_csv_nested(s struct<i: int, arr: array<int>, m: map<string, int>, n: struct<x: int, y: string>>) USING parquet
88+
89+
statement
90+
INSERT INTO test_to_csv_nested VALUES
91+
(named_struct('i', 1, 'arr', array(1, 2, 3), 'm', map('k', 10), 'n', named_struct('x', 5, 'y', 'z'))),
92+
(named_struct('i', NULL, 'arr', CAST(NULL AS array<int>), 'm', CAST(NULL AS map<string, int>), 'n', CAST(NULL AS struct<x: int, y: string>))),
93+
(CAST(NULL AS struct<i: int, arr: array<int>, m: map<string, int>, n: struct<x: int, y: string>>))
94+
95+
-- Struct column straight from the scan: the kernel reads it with getStruct, so the converter's
96+
-- per-field reads of the array / map / struct fields all land on `CometInternalRow`. The all-null
97+
-- and null-struct rows keep the assertion from collapsing to a constant true.
98+
query
99+
SELECT to_csv(s) IS NOT NULL FROM test_to_csv_nested
100+
101+
-- named_struct over the complex fields: the kernel's array / map / struct readers produce the
102+
-- values that CreateNamedStruct stores into the row the converter then reads.
103+
query
104+
SELECT to_csv(named_struct('arr', s.arr, 'm', s.m, 'n', s.n)) IS NOT NULL FROM test_to_csv_nested
105+
106+
-- Complex types nested inside complex types, so the converter recurses through the kernel's
107+
-- element getters rather than stopping at the first level.
108+
query
109+
SELECT to_csv(named_struct('aa', array(array(1, 2), array(3)), 'ms', map('k', named_struct('x', 1)))) IS NOT NULL FROM test_to_csv_nested
Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
-- Licensed to the Apache Software Foundation (ASF) under one
2+
-- or more contributor license agreements. See the NOTICE file
3+
-- distributed with this work for additional information
4+
-- regarding copyright ownership. The ASF licenses this file
5+
-- to you under the Apache License, Version 2.0 (the
6+
-- "License"); you may not use this file except in compliance
7+
-- with the License. You may obtain a copy of the License at
8+
--
9+
-- http://www.apache.org/licenses/LICENSE-2.0
10+
--
11+
-- Unless required by applicable law or agreed to in writing,
12+
-- software distributed under the License is distributed on an
13+
-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
-- KIND, either express or implied. See the License for the
15+
-- specific language governing permissions and limitations
16+
-- under the License.
17+
18+
-- ANSI mode with a raising subtree inside the struct: `to_csv` routes through the codegen
19+
-- dispatcher by default, and the whole tree (StructsToCsv over CreateNamedStruct over Cast) is
20+
-- bound into one kernel, so the Cast's ANSI throw has to survive the dispatcher rather than being
21+
-- swallowed into a NULL. That is the failure mode #5219 fixed for `CometBatchKernelCodegen`'s null
22+
-- short-circuit, which skipped a subtree Spark would have evaluated when every input ordinal was
23+
-- tested up front.
24+
--
25+
-- `to_csv` does not reach the short-circuit today (CreateNamedStruct is not NullIntolerant, so
26+
-- `allNullIntolerant` already fails), but the shape is one wire-up change away from mattering and
27+
-- was untested for this expression, so these queries pin it. The second query is the exact #5219
28+
-- shape: two input ordinals where the non-cast one is NULL on the row whose cast raises.
29+
-- Config: spark.sql.ansi.enabled=true
30+
-- Config: spark.comet.exec.scalaUDF.codegen.enabled=true
31+
32+
statement
33+
CREATE TABLE test_to_csv_ansi(s string, i int) USING parquet
34+
35+
statement
36+
INSERT INTO test_to_csv_ansi VALUES
37+
('1', 10),
38+
('notanint', NULL)
39+
40+
query expect_error(CAST_INVALID_INPUT)
41+
SELECT to_csv(named_struct('a', CAST(s AS INT))) FROM test_to_csv_ansi
42+
43+
query expect_error(CAST_INVALID_INPUT)
44+
SELECT to_csv(named_struct('a', CAST(s AS INT), 'b', i)) FROM test_to_csv_ansi
45+
46+
-- The struct field's cast is the only raising node, so restricting to the well-formed row makes
47+
-- the expression succeed. Doubles as the sentinel required by `ExpectError` fixtures: it uses
48+
-- checkSparkAnswerAndOperator, so a silent dispatcher rejection (which would make the queries
49+
-- above pass vacuously via a Spark fallback that raises the same error) fails here instead.
50+
query
51+
SELECT to_csv(named_struct('a', CAST(s AS INT), 'b', i)) FROM test_to_csv_ansi WHERE i IS NOT NULL
52+
53+
-- try_cast does not raise under ANSI, so the surrounding to_csv must render the NULL field with
54+
-- the CSV nullValue rather than propagating an error.
55+
query
56+
SELECT to_csv(named_struct('a', TRY_CAST(s AS INT), 'b', i)) FROM test_to_csv_ansi
Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
-- Licensed to the Apache Software Foundation (ASF) under one
2+
-- or more contributor license agreements. See the NOTICE file
3+
-- distributed with this work for additional information
4+
-- regarding copyright ownership. The ASF licenses this file
5+
-- to you under the Apache License, Version 2.0 (the
6+
-- "License"); you may not use this file except in compliance
7+
-- with the License. You may obtain a copy of the License at
8+
--
9+
-- http://www.apache.org/licenses/LICENSE-2.0
10+
--
11+
-- Unless required by applicable law or agreed to in writing,
12+
-- software distributed under the License is distributed on an
13+
-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
-- KIND, either express or implied. See the License for the
15+
-- specific language governing permissions and limitations
16+
-- under the License.
17+
18+
-- Full-value coverage of `to_csv` over complex field types, which `to_csv.sql` can only assert for
19+
-- non-nullness because Spark 3.4 / 3.5 render a complex field as a Java identity string. Spark 4.0
20+
-- gave `UnivocityGenerator.makeConverter` real array / map / struct branches ("[1, 2, 3]",
21+
-- "{k -> 10}", "{5, z}"), built from the typed `getArray` / `getMap` / `getStruct` getters, so from
22+
-- 4.0 on the value is deterministic and Comet's kernel-side `CometInternalRow` / `CometArrayData` /
23+
-- `CometMapData` getters have to agree with Spark's row-side ones element for element.
24+
-- MinSparkVersion: 4.0
25+
26+
statement
27+
CREATE TABLE test_to_csv_nested4(s struct<i: int, arr: array<int>, m: map<string, int>, n: struct<x: int, y: string>>) USING parquet
28+
29+
statement
30+
INSERT INTO test_to_csv_nested4 VALUES
31+
(named_struct('i', 1, 'arr', array(1, 2, 3), 'm', map('k', 10), 'n', named_struct('x', 5, 'y', 'z'))),
32+
(named_struct('i', 2, 'arr', array(), 'm', map(), 'n', named_struct('x', NULL, 'y', NULL))),
33+
(named_struct('i', 3, 'arr', array(1, CAST(NULL AS int)), 'm', map('k', CAST(NULL AS int)), 'n', named_struct('x', 7, 'y', 'a,b'))),
34+
(named_struct('i', NULL, 'arr', CAST(NULL AS array<int>), 'm', CAST(NULL AS map<string, int>), 'n', CAST(NULL AS struct<x: int, y: string>))),
35+
(CAST(NULL AS struct<i: int, arr: array<int>, m: map<string, int>, n: struct<x: int, y: string>>))
36+
37+
-- Struct column straight from the scan: the converter reads the array / map / struct fields off
38+
-- the kernel's CometInternalRow. Empty collections, null elements and a value needing CSV quoting
39+
-- are all in the data above.
40+
query
41+
SELECT to_csv(s) FROM test_to_csv_nested4
42+
43+
-- nullValue changes how a null *element* inside a complex field renders (`appendNull` only emits
44+
-- when the option was set explicitly), which is a separate code path from a null top-level field.
45+
query
46+
SELECT to_csv(s, map('nullValue', 'NIL')) FROM test_to_csv_nested4
47+
48+
-- named_struct over the complex fields, so the values pass through CreateNamedStruct's row before
49+
-- the converter reads them back.
50+
query
51+
SELECT to_csv(named_struct('arr', s.arr, 'm', s.m, 'n', s.n)) FROM test_to_csv_nested4
52+
53+
-- Complex inside complex: the converter recurses, so an array-of-array read goes through the
54+
-- kernel's element getters rather than stopping at the first level.
55+
query
56+
SELECT to_csv(named_struct('aa', array(array(1, 2), array(3)), 'ms', map('k', named_struct('x', 1)))) FROM test_to_csv_nested4

‎spark/src/test/resources/sql-tests/expressions/string/string_translate.sql‎

Lines changed: 25 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -15,29 +15,44 @@
1515
-- specific language governing permissions and limitations
1616
-- under the License.
1717

18-
-- translate is gated as Incompatible by default. DataFusion's translate iterates over Unicode
19-
-- graphemes (Spark uses code points) and substitutes U+0000 instead of treating it as a deletion
20-
-- sentinel, so the native path silently diverges from Spark for combining-mark inputs and for
21-
-- to=NUL. These default-config tests assert that the expression falls back cleanly to Spark.
22-
-- See string_translate_enabled.sql for the opt-in native path.
18+
-- translate runs through the codegen dispatcher by default so results match Spark exactly. The
19+
-- native path diverges from Spark (DataFusion iterates over Unicode graphemes where Spark uses code
20+
-- points, and substitutes U+0000 instead of treating it as a deletion sentinel), so it is opt-in
21+
-- via spark.comet.expression.StringTranslate.allowIncompatible. See string_translate_enabled.sql
22+
-- for the opt-in native path.
2323

2424
statement
2525
CREATE TABLE test_translate(s string, from_str string, to_str string) USING parquet
2626

27+
-- The last two rows are the regression test for this file's routing change: they are the inputs
28+
-- where the native path is known to disagree with Spark, so they are only assertable now that the
29+
-- default is the bit-exact dispatcher. Written with \u escapes rather than literal characters to
30+
-- keep the fixture ASCII. They have to be plain string literals: an inline table only accepts
31+
-- expressions the analyzer can evaluate, and `decode(X'..')` is not one of them on Spark 4.1
32+
-- (INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE).
33+
-- 'cafe\u0301' ends in "e" + U+0301 COMBINING ACUTE ACCENT: one grapheme, two code
34+
-- points. DataFusion iterates graphemes, Spark iterates code points, so translating "e"
35+
-- differs.
36+
-- '\u0000' as the `to` argument is U+0000, which Spark treats as a deletion sentinel and
37+
-- the native path substitutes it literally.
2738
statement
28-
INSERT INTO test_translate VALUES ('hello', 'el', 'ip'), ('hello', 'aeiou', '12345'), ('', 'a', 'b'), (NULL, 'a', 'b'), ('hello', '', ''), ('abc', 'abc', 'x')
39+
INSERT INTO test_translate VALUES
40+
('hello', 'el', 'ip'), ('hello', 'aeiou', '12345'), ('', 'a', 'b'), (NULL, 'a', 'b'),
41+
('hello', '', ''), ('abc', 'abc', 'x'),
42+
('cafe\u0301', 'e', 'E'),
43+
('hello', 'l', '\u0000')
2944

30-
query expect_fallback(is not fully compatible with Spark)
45+
query
3146
SELECT translate(s, from_str, to_str) FROM test_translate
3247

3348
-- column + literal + literal
34-
query expect_fallback(is not fully compatible with Spark)
49+
query
3550
SELECT translate(s, 'el', 'ip') FROM test_translate
3651

3752
-- literal + column + column
38-
query expect_fallback(is not fully compatible with Spark)
53+
query
3954
SELECT translate('hello', from_str, to_str) FROM test_translate
4055

4156
-- literal + literal + literal
42-
query expect_fallback(is not fully compatible with Spark)
57+
query
4358
SELECT translate('hello', 'el', 'ip'), translate('hello', 'aeiou', '12345'), translate('', 'a', 'b'), translate(NULL, 'a', 'b')

‎spark/src/test/scala/org/apache/comet/CometCsvExpressionSuite.scala‎

Lines changed: 15 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -47,24 +47,21 @@ class CometCsvExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper
4747
SchemaGenOptions(generateArray = false, generateStruct = false, generateMap = false),
4848
DataGenOptions(allowNull = true, generateNegativeZero = true))
4949
}
50-
withSQLConf(CometConf.getExprAllowIncompatConfigKey(classOf[StructsToCsv]) -> "true") {
51-
val df = spark.read
52-
.parquet(filename)
53-
.select(
54-
to_csv(
55-
struct(
56-
col("c0"),
57-
col("c1"),
58-
col("c2"),
59-
col("c3"),
60-
col("c4"),
61-
col("c5"),
62-
col("c7"),
63-
col("c8"),
64-
col("c9"),
65-
col("c12"))))
66-
checkSparkAnswerAndOperator(df)
67-
}
50+
// Every column in the fuzz schema except c13 Binary, with no allowIncompatible opt-in:
51+
// to_csv now routes through the codegen dispatcher by default, so Spark's own converter runs
52+
// and all field types match by construction. This previously ran under
53+
// allowIncompatible=true and had to skip c6 Double, c10 Timestamp and c11 TimestampNTZ,
54+
// which are types the native path cannot match Spark on (the latter two are #3232).
55+
//
56+
// c13 Binary stays out, and not because of a divergence: Spark's CSV converter renders
57+
// BinaryType with Java's default Object.toString(), so a row comes out as
58+
// "...,[B@731af74". That is an identity hash of the byte array instance, which differs
59+
// between any two evaluations, so the value can never be asserted against a second run --
60+
// by Comet or by Spark against itself.
61+
val df = spark.read
62+
.parquet(filename)
63+
.select(to_csv(struct((0 to 12).map(i => col(s"c$i")): _*)))
64+
checkSparkAnswerAndOperator(df)
6865
}
6966
}
7067

0 commit comments

Comments
 (0)