Repository navigation
Commit 7d8d9b1
authored
feat: normalize floats in native array_distinct and array_union (#6563)
* feat: normalize floats in native array_distinct and array_union
Spark 4.2.0 normalizes the arguments of array_distinct and array_union in
the plan (SPARK-54918), and 4.0.5, 4.1.4 and 4.2.1 normalize inside the
expressions instead (SPARK-59602), so those releases pass Comet the raw
input. DataFusion folds -0.0 into 0.0 only in a flat float array and
compares NaNs by their bits, which is why #5750 limited native execution
of float elements to 4.2.0.
Arrays with a float at any depth now go to spark_array_distinct and
spark_array_union, which run normalize_nested_floats on their inputs and
then delegate to DataFusion, as SparkArrayExtrema does. That matches every
release that merges signed zeros, so float elements run natively by
default on 4.0.5+, 4.1.4+ and 4.2+, and only 3.4, 3.5, 4.0.0 to 4.0.4 and
4.1.0 to 4.1.3 fall back. Older Spark merges NaNs and nested zeros too, so
the opt-in now differs from it only on signed zeros.
Within 4.0 and 4.1 the default path now depends on the patch release, so
the 4.0/4.1 signed-zero fixture checks answers only and
CometArrayExpressionSuite checks the routing. The 4.2+ fixture checks the
default path instead of the opt-ins.
* fix: keep collated struct fields out of native array_distinct and array_union
The spark_ variants normalize floats and then let DataFusion compare the
elements, which compares strings by their bytes. Spark compares a
collated string under its collation, so under UTF8_LCASE two structs
that differ only in 'a' and 'A' are one element. Widening the gate to
4.0.5+, 4.1.4+ and 4.2.x would have sent such elements to the native
kernels, and 4.2.0 already did.
Elements that hold both a float and a non-UTF8_BINARY collated string
now fall back on every version. Collated strings without floats are
tracked separately in #6470.
* fix: skip array_union's right operand on rows whose left array is NULL
Spark's ArrayUnion returns NULL for a NULL left array without evaluating
the right operand. The native function evaluates both over the whole
batch, so a right operand that throws, such as slice(b, 0, 1) or an ANSI
cast, failed on rows that Spark never evaluates it for. That already
happened on main for every element type that runs natively, and this PR
would have extended it to float elements on 4.0.5+ and 4.1.4+.
CometArrayUnion now guards the call with CASE WHEN <left> IS NOT NULL, as
CometElementAt does, unless the right operand is a column or a literal,
which cannot throw. A nullable nondeterministic left operand, which the
guard would evaluate twice, stays on Spark.
* revert: drop the array_union NULL guard in favor of #6716
The CASE WHEN <left> IS NOT NULL guard serialized the left operand twice,
in the IS NOT NULL test and in the call, so nested unions grew
exponentially: twelve nested array_union calls became 4,095 union nodes,
and each level evaluated its left operand twice.
#6716 adds a native NullShortCircuit that evaluates the left operand once
and the right operand only on the rows where the left is not NULL. It
covers array_union for every element type, so it fixes the eager
evaluation this guard was for without the duplication.
This reverts commit 88ec629.
* perf: normalize only the child range a sliced list references
normalize_nested_floats rebuilt a list's whole child array, but a sliced list
keeps its child whole, and explode hands out slices of one child array for a
whole input batch. Rebuild only the range the offsets reference, at every
depth, and rebase the offsets onto it.
The float_arrays bench gains array_distinct and array_union cases over whole
and sliced lists.
* test: cover array_union over ARRAY<DOUBLE> with a NULL left array
#6716's array_null_short_circuit*.sql fixtures only use array<int>, which
runs through DataFusion's array_union. A float element type goes through
spark_array_union, so nothing checked that withNullShortCircuit reaches
it. Without the guard, native evaluation raises on rows Spark skips:
INVALID_PARAMETER_VALUE.START for slice(b, 0, 1) over a NULL array, and
CAST_INVALID_INPUT for an ANSI cast of a malformed string.
The fixture opts ArrayUnion in so every Spark version runs the float path
natively, not only 4.0.5+, 4.1.4+ and 4.2+. The data has no signed zero,
which earlier releases keep apart.1 parent 33f4f88 commit 7d8d9b1
15 files changed
Lines changed: 695 additions & 120 deletions
File tree
- docs/source
- contributor-guide/expression-audits
- user-guide/latest
- compatibility
- native/spark-expr
- benches
- src
- array_funcs
- float_semantics
- spark/src
- main/scala/org/apache/comet/serde
- test
- resources/sql-tests/expressions/array
- scala/org/apache/comet
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
56 | 56 | | |
57 | 57 | | |
58 | 58 | | |
59 | | - | |
| 59 | + | |
60 | 60 | | |
61 | 61 | | |
62 | 62 | | |
| |||
141 | 141 | | |
142 | 142 | | |
143 | 143 | | |
144 | | - | |
| 144 | + | |
145 | 145 | | |
146 | 146 | | |
147 | 147 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
113 | 113 | | |
114 | 114 | | |
115 | 115 | | |
116 | | - | |
117 | | - | |
118 | | - | |
119 | | - | |
120 | | - | |
121 | | - | |
122 | | - | |
123 | | - | |
124 | | - | |
125 | | - | |
126 | | - | |
127 | | - | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
128 | 134 | | |
129 | 135 | | |
130 | 136 | | |
| |||
133 | 139 | | |
134 | 140 | | |
135 | 141 | | |
136 | | - | |
137 | | - | |
138 | | - | |
139 | | - | |
140 | | - | |
141 | | - | |
142 | | - | |
143 | | - | |
144 | | - | |
145 | | - | |
146 | | - | |
147 | | - | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
148 | 150 | | |
149 | 151 | | |
150 | 152 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
156 | 156 | | |
157 | 157 | | |
158 | 158 | | |
159 | | - | |
| 159 | + | |
160 | 160 | | |
161 | 161 | | |
162 | 162 | | |
| |||
167 | 167 | | |
168 | 168 | | |
169 | 169 | | |
170 | | - | |
| 170 | + | |
171 | 171 | | |
172 | 172 | | |
173 | 173 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
21 | 25 | | |
22 | 26 | | |
23 | 27 | | |
| |||
26 | 30 | | |
27 | 31 | | |
28 | 32 | | |
| 33 | + | |
29 | 34 | | |
30 | 35 | | |
31 | | - | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
32 | 39 | | |
33 | 40 | | |
34 | 41 | | |
| |||
41 | 48 | | |
42 | 49 | | |
43 | 50 | | |
44 | | - | |
| 51 | + | |
45 | 52 | | |
46 | | - | |
| 53 | + | |
47 | 54 | | |
48 | | - | |
| 55 | + | |
49 | 56 | | |
50 | | - | |
| 57 | + | |
51 | 58 | | |
52 | 59 | | |
53 | 60 | | |
54 | | - | |
| 61 | + | |
55 | 62 | | |
56 | | - | |
| 63 | + | |
57 | 64 | | |
58 | 65 | | |
59 | 66 | | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
60 | 73 | | |
61 | 74 | | |
62 | 75 | | |
| |||
81 | 94 | | |
82 | 95 | | |
83 | 96 | | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
84 | 100 | | |
85 | 101 | | |
86 | | - | |
| 102 | + | |
87 | 103 | | |
88 | 104 | | |
89 | 105 | | |
| |||
121 | 137 | | |
122 | 138 | | |
123 | 139 | | |
124 | | - | |
| 140 | + | |
125 | 141 | | |
126 | 142 | | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
127 | 151 | | |
128 | 152 | | |
129 | 153 | | |
| |||
160 | 184 | | |
161 | 185 | | |
162 | 186 | | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
163 | 215 | | |
164 | 216 | | |
165 | 217 | | |
| |||
0 commit comments