Skip to content

Commit 7a7942c

Browse files
committed
review: pin to_csv dispatcher behavior for ANSI casts and complex field types
Addresses review feedback on #5032. - to_csv_ansi.sql: a raising Cast inside the named_struct, under ANSI, in both the single-ordinal and the two-ordinal (#5219) shape, plus a try_cast counterpart and the sentinel query the ExpectError convention requires. - to_csv.sql: complex field types (array / map / nested struct) reach the runtime dispatch because isSupportedDataType recurses, so assert the dispatcher hands the nested InternalRow / ArrayData / MapData through without raising. On Spark 3.4 / 3.5 that is the CometSpecializedGettersDispatch generic get; verified the queries fail when its complex-type branches are removed. - to_csv_nested.sql (Spark 4.0+): Spark 4.0 gave UnivocityGenerator real array / map / struct converters, so from 4.0 on the rendered value is deterministic and can be asserted in full rather than only for nullness. - string_translate.sql: the two divergence rows used decode(X'..') inside an inline table, which Spark 4.1 rejects with INVALID_INLINE_TABLE. Use \u escapes instead; verified the rows still fail under allowIncompatible=true.
1 parent e00ad56 commit 7a7942c

4 files changed

Lines changed: 167 additions & 20 deletions

File tree

‎spark/src/test/resources/sql-tests/expressions/csv/to_csv.sql‎

Lines changed: 44 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -48,9 +48,11 @@ SELECT to_csv(named_struct('a', a, 'b', b, 'c', c), map('sep', ';')) FROM test_t
4848

4949
-- timestampFormat over date and timestamp fields: these are the #3232 types that were
5050
-- Incompatible before this change and now route through the dispatcher.
51-
-- BinaryType is the other #3232 type but is deliberately absent: Spark's CSV converter renders it
52-
-- with Java's default Object.toString(), e.g. "[B@10bc15e4", an identity hash that differs between
53-
-- any two evaluations, so the value is not assertable by any engine including Spark itself.
51+
-- BinaryType is the other #3232 type but is deliberately absent: on Spark 3.4 / 3.5 the CSV
52+
-- converter has no binary branch and renders it with Java's default Object.toString(), e.g.
53+
-- "[B@10bc15e4", an identity hash that differs between any two evaluations, so the value is not
54+
-- assertable by any engine including Spark itself. (Spark 4.0 added a real binary formatter, but
55+
-- this fixture runs on every supported version.)
5456
statement
5557
CREATE TABLE test_to_csv_temporal(d date, t timestamp) USING parquet
5658

@@ -66,12 +68,42 @@ SELECT to_csv(named_struct('d', d, 't', t)) FROM test_to_csv_temporal
6668
query
6769
SELECT to_csv(named_struct('d', d, 't', t), map('timestampFormat', 'yyyy/MM/dd HH:mm', 'dateFormat', 'dd-MM-yyyy')) FROM test_to_csv_temporal
6870

69-
-- Complex field types (arrays, maps, nested structs) are deliberately not asserted here.
70-
-- StructsToCsv.checkInputDataTypes accepts them, so they reach the converter, but Spark's own
71-
-- output for them is not a value that can be compared: a non-null array/map/struct renders as a
72-
-- Java identity string such as "org.apache.spark.sql.vectorized.ColumnarArray@1ada50f0", and any
73-
-- null complex value throws NullPointerException inside Spark's UnsafeWriter.write. Both were
74-
-- confirmed against Spark 3.5 with Comet disabled entirely, so they are Spark behavior, not
75-
-- Comet's. Before this change these schemas were Unsupported and fell the projection back to
76-
-- Spark; now they reach the dispatcher, which runs the same Spark code, so the observable result
77-
-- is unchanged either way.
71+
-- Complex field types (arrays, maps, nested structs). `StructsToCsv.checkInputDataTypes` accepts
72+
-- them and `CometBatchKernelCodegen.isSupportedDataType` recurses into them, so they pass the
73+
-- plan-time gate and reach the runtime dispatch. That is the accepted-at-plan-time /
74+
-- rejected-at-runtime shape #5219 found for TIME types: `canHandle` greenlights the expression
75+
-- before the plan commits, so a runtime gap is an execute-time failure with no fallback left.
76+
--
77+
-- Only non-nullness is asserted here, because the rendered value is not comparable on every
78+
-- supported Spark version. Spark 3.4 / 3.5's `UnivocityGenerator.makeConverter` has no branch for
79+
-- complex types and lands on `getter.get(ordinal, dataType).toString`, an identity string such as
80+
-- "org.apache.spark.sql.vectorized.ColumnarArray@1ada50f0" whose hash differs between any two
81+
-- evaluations and whose class differs between Spark's converter input (`ColumnarArray` /
82+
-- `UnsafeArrayData`) and the kernel's (`InputArray_*`). That generic `get` is exactly what
83+
-- `CometSpecializedGettersDispatch` implements for `CometInternalRow` / `CometArrayData`, so on
84+
-- those versions these queries are the coverage for it. Spark 4.0 added real array/map/struct
85+
-- converters, which render deterministically; `to_csv_nested.sql` asserts those values in full.
86+
statement
87+
CREATE TABLE test_to_csv_nested(s struct<i: int, arr: array<int>, m: map<string, int>, n: struct<x: int, y: string>>) USING parquet
88+
89+
statement
90+
INSERT INTO test_to_csv_nested VALUES
91+
(named_struct('i', 1, 'arr', array(1, 2, 3), 'm', map('k', 10), 'n', named_struct('x', 5, 'y', 'z'))),
92+
(named_struct('i', NULL, 'arr', CAST(NULL AS array<int>), 'm', CAST(NULL AS map<string, int>), 'n', CAST(NULL AS struct<x: int, y: string>))),
93+
(CAST(NULL AS struct<i: int, arr: array<int>, m: map<string, int>, n: struct<x: int, y: string>>))
94+
95+
-- Struct column straight from the scan: the kernel reads it with getStruct, so the converter's
96+
-- per-field reads of the array / map / struct fields all land on `CometInternalRow`. The all-null
97+
-- and null-struct rows keep the assertion from collapsing to a constant true.
98+
query
99+
SELECT to_csv(s) IS NOT NULL FROM test_to_csv_nested
100+
101+
-- named_struct over the complex fields: the kernel's array / map / struct readers produce the
102+
-- values that CreateNamedStruct stores into the row the converter then reads.
103+
query
104+
SELECT to_csv(named_struct('arr', s.arr, 'm', s.m, 'n', s.n)) IS NOT NULL FROM test_to_csv_nested
105+
106+
-- Complex types nested inside complex types, so the converter recurses through the kernel's
107+
-- element getters rather than stopping at the first level.
108+
query
109+
SELECT to_csv(named_struct('aa', array(array(1, 2), array(3)), 'ms', map('k', named_struct('x', 1)))) IS NOT NULL FROM test_to_csv_nested
Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
-- Licensed to the Apache Software Foundation (ASF) under one
2+
-- or more contributor license agreements. See the NOTICE file
3+
-- distributed with this work for additional information
4+
-- regarding copyright ownership. The ASF licenses this file
5+
-- to you under the Apache License, Version 2.0 (the
6+
-- "License"); you may not use this file except in compliance
7+
-- with the License. You may obtain a copy of the License at
8+
--
9+
-- http://www.apache.org/licenses/LICENSE-2.0
10+
--
11+
-- Unless required by applicable law or agreed to in writing,
12+
-- software distributed under the License is distributed on an
13+
-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
-- KIND, either express or implied. See the License for the
15+
-- specific language governing permissions and limitations
16+
-- under the License.
17+
18+
-- ANSI mode with a raising subtree inside the struct: `to_csv` routes through the codegen
19+
-- dispatcher by default, and the whole tree (StructsToCsv over CreateNamedStruct over Cast) is
20+
-- bound into one kernel, so the Cast's ANSI throw has to survive the dispatcher rather than being
21+
-- swallowed into a NULL. That is the failure mode #5219 fixed for `CometBatchKernelCodegen`'s null
22+
-- short-circuit, which skipped a subtree Spark would have evaluated when every input ordinal was
23+
-- tested up front.
24+
--
25+
-- `to_csv` does not reach the short-circuit today (CreateNamedStruct is not NullIntolerant, so
26+
-- `allNullIntolerant` already fails), but the shape is one wire-up change away from mattering and
27+
-- was untested for this expression, so these queries pin it. The second query is the exact #5219
28+
-- shape: two input ordinals where the non-cast one is NULL on the row whose cast raises.
29+
-- Config: spark.sql.ansi.enabled=true
30+
-- Config: spark.comet.exec.scalaUDF.codegen.enabled=true
31+
32+
statement
33+
CREATE TABLE test_to_csv_ansi(s string, i int) USING parquet
34+
35+
statement
36+
INSERT INTO test_to_csv_ansi VALUES
37+
('1', 10),
38+
('notanint', NULL)
39+
40+
query expect_error(CAST_INVALID_INPUT)
41+
SELECT to_csv(named_struct('a', CAST(s AS INT))) FROM test_to_csv_ansi
42+
43+
query expect_error(CAST_INVALID_INPUT)
44+
SELECT to_csv(named_struct('a', CAST(s AS INT), 'b', i)) FROM test_to_csv_ansi
45+
46+
-- The struct field's cast is the only raising node, so restricting to the well-formed row makes
47+
-- the expression succeed. Doubles as the sentinel required by `ExpectError` fixtures: it uses
48+
-- checkSparkAnswerAndOperator, so a silent dispatcher rejection (which would make the queries
49+
-- above pass vacuously via a Spark fallback that raises the same error) fails here instead.
50+
query
51+
SELECT to_csv(named_struct('a', CAST(s AS INT), 'b', i)) FROM test_to_csv_ansi WHERE i IS NOT NULL
52+
53+
-- try_cast does not raise under ANSI, so the surrounding to_csv must render the NULL field with
54+
-- the CSV nullValue rather than propagating an error.
55+
query
56+
SELECT to_csv(named_struct('a', TRY_CAST(s AS INT), 'b', i)) FROM test_to_csv_ansi
Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
-- Licensed to the Apache Software Foundation (ASF) under one
2+
-- or more contributor license agreements. See the NOTICE file
3+
-- distributed with this work for additional information
4+
-- regarding copyright ownership. The ASF licenses this file
5+
-- to you under the Apache License, Version 2.0 (the
6+
-- "License"); you may not use this file except in compliance
7+
-- with the License. You may obtain a copy of the License at
8+
--
9+
-- http://www.apache.org/licenses/LICENSE-2.0
10+
--
11+
-- Unless required by applicable law or agreed to in writing,
12+
-- software distributed under the License is distributed on an
13+
-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
-- KIND, either express or implied. See the License for the
15+
-- specific language governing permissions and limitations
16+
-- under the License.
17+
18+
-- Full-value coverage of `to_csv` over complex field types, which `to_csv.sql` can only assert for
19+
-- non-nullness because Spark 3.4 / 3.5 render a complex field as a Java identity string. Spark 4.0
20+
-- gave `UnivocityGenerator.makeConverter` real array / map / struct branches ("[1, 2, 3]",
21+
-- "{k -> 10}", "{5, z}"), built from the typed `getArray` / `getMap` / `getStruct` getters, so from
22+
-- 4.0 on the value is deterministic and Comet's kernel-side `CometInternalRow` / `CometArrayData` /
23+
-- `CometMapData` getters have to agree with Spark's row-side ones element for element.
24+
-- MinSparkVersion: 4.0
25+
26+
statement
27+
CREATE TABLE test_to_csv_nested4(s struct<i: int, arr: array<int>, m: map<string, int>, n: struct<x: int, y: string>>) USING parquet
28+
29+
statement
30+
INSERT INTO test_to_csv_nested4 VALUES
31+
(named_struct('i', 1, 'arr', array(1, 2, 3), 'm', map('k', 10), 'n', named_struct('x', 5, 'y', 'z'))),
32+
(named_struct('i', 2, 'arr', array(), 'm', map(), 'n', named_struct('x', NULL, 'y', NULL))),
33+
(named_struct('i', 3, 'arr', array(1, CAST(NULL AS int)), 'm', map('k', CAST(NULL AS int)), 'n', named_struct('x', 7, 'y', 'a,b'))),
34+
(named_struct('i', NULL, 'arr', CAST(NULL AS array<int>), 'm', CAST(NULL AS map<string, int>), 'n', CAST(NULL AS struct<x: int, y: string>))),
35+
(CAST(NULL AS struct<i: int, arr: array<int>, m: map<string, int>, n: struct<x: int, y: string>>))
36+
37+
-- Struct column straight from the scan: the converter reads the array / map / struct fields off
38+
-- the kernel's CometInternalRow. Empty collections, null elements and a value needing CSV quoting
39+
-- are all in the data above.
40+
query
41+
SELECT to_csv(s) FROM test_to_csv_nested4
42+
43+
-- nullValue changes how a null *element* inside a complex field renders (`appendNull` only emits
44+
-- when the option was set explicitly), which is a separate code path from a null top-level field.
45+
query
46+
SELECT to_csv(s, map('nullValue', 'NIL')) FROM test_to_csv_nested4
47+
48+
-- named_struct over the complex fields, so the values pass through CreateNamedStruct's row before
49+
-- the converter reads them back.
50+
query
51+
SELECT to_csv(named_struct('arr', s.arr, 'm', s.m, 'n', s.n)) FROM test_to_csv_nested4
52+
53+
-- Complex inside complex: the converter recurses, so an array-of-array read goes through the
54+
-- kernel's element getters rather than stopping at the first level.
55+
query
56+
SELECT to_csv(named_struct('aa', array(array(1, 2), array(3)), 'ms', map('k', named_struct('x', 1)))) FROM test_to_csv_nested4

‎spark/src/test/resources/sql-tests/expressions/string/string_translate.sql‎

Lines changed: 11 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -26,18 +26,21 @@ CREATE TABLE test_translate(s string, from_str string, to_str string) USING parq
2626

2727
-- The last two rows are the regression test for this file's routing change: they are the inputs
2828
-- where the native path is known to disagree with Spark, so they are only assertable now that the
29-
-- default is the bit-exact dispatcher. Built with decode() rather than literal characters to keep
30-
-- the fixture ASCII.
31-
-- decode(X'65CC81') is "e" + U+0301 COMBINING ACUTE ACCENT: one grapheme, two code points.
32-
-- DataFusion iterates graphemes, Spark iterates code points, so translating "e" differs.
33-
-- decode(X'00') as the `to` argument is U+0000, which Spark treats as a deletion sentinel and
34-
-- the native path substitutes literally.
29+
-- default is the bit-exact dispatcher. Written with \u escapes rather than literal characters to
30+
-- keep the fixture ASCII. They have to be plain string literals: an inline table only accepts
31+
-- expressions the analyzer can evaluate, and `decode(X'..')` is not one of them on Spark 4.1
32+
-- (INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE).
33+
-- 'cafe\u0301' ends in "e" + U+0301 COMBINING ACUTE ACCENT: one grapheme, two code
34+
-- points. DataFusion iterates graphemes, Spark iterates code points, so translating "e"
35+
-- differs.
36+
-- '\u0000' as the `to` argument is U+0000, which Spark treats as a deletion sentinel and
37+
-- the native path substitutes it literally.
3538
statement
3639
INSERT INTO test_translate VALUES
3740
('hello', 'el', 'ip'), ('hello', 'aeiou', '12345'), ('', 'a', 'b'), (NULL, 'a', 'b'),
3841
('hello', '', ''), ('abc', 'abc', 'x'),
39-
(concat('caf', decode(X'65CC81', 'UTF-8')), 'e', 'E'),
40-
('hello', 'l', decode(X'00', 'UTF-8'))
42+
('cafe\u0301', 'e', 'E'),
43+
('hello', 'l', '\u0000')
4144

4245
query
4346
SELECT translate(s, from_str, to_str) FROM test_translate

0 commit comments

Comments
 (0)