You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit f80042f
Browse filesBrowse the repository at this point in the historyBrowse files
fix: Route collated array element comparisons through the codegen dispatcher (#6471)
* fix: Route collated array element comparisons through the codegen dispatcher
array_contains, arrays_overlap, array_distinct and array_union compare
array elements by raw bytes in their native kernels, so they returned
wrong results for strings with a non-UTF8_BINARY collation (for example
UTF8_LCASE or UTF8_BINARY_RTRIM) when a dispatched collated cast fed
them.
Report these cases as Incompatible and mix in CodegenDispatchFallback so
they run Spark's collation-aware comparison through the JVM codegen
dispatcher by default. ArrayDistinct gets its own serde for this.
* docs: Mark collation-routed array functions as Hybrid in expressions.md
Copy file name to clipboardExpand all lines: docs/source/user-guide/latest/expressions.md
+4-4Lines changed: 4 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -155,8 +155,8 @@ The tables below list every Spark built-in expression with its current status.
155
155
|`array`| ✅ | Native ||
156
156
|`array_append`| ✅ | Native ||
157
157
|`array_compact`| ✅ | — ||
158
-
|`array_contains`| ✅ |Native| Float/double element arrays route through the JVM codegen dispatcher by default; the native path is opt-in via allowIncompatible |
159
-
|`array_distinct`| ✅ | Native | Floating-point elements fall back on Spark versions other than 4.2.0; signed-zero and NaN results may differ with native opt-in ([details](compatibility/floating-point.md)) |
158
+
|`array_contains`| ✅ |Hybrid| Float/double element arrays and non-default string collations route through the JVM codegen dispatcher by default; the native path is opt-in via allowIncompatible |
159
+
|`array_distinct`| ✅ | Native | Floating-point elements fall back on Spark versions other than 4.2.0; signed-zero and NaN results may differ with native opt-in ([details](compatibility/floating-point.md)). Non-default string collations fall back|
160
160
|`array_except`| ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; the incompatible native path is opt-in via allowIncompatible ([details](compatibility/expressions/array.md)) |
161
161
|`array_insert`| ✅ | Native ||
162
162
|`array_intersect`| ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; the incompatible native path is opt-in via allowIncompatible ([details](compatibility/expressions/array.md)) |
@@ -167,8 +167,8 @@ The tables below list every Spark built-in expression with its current status.
167
167
|`array_prepend`| ✅ | — ||
168
168
|`array_remove`| ✅ | Native ||
169
169
|`array_repeat`| ✅ | Native ||
170
-
|`array_union`| ✅ | Native | Floating-point elements fall back on Spark versions other than 4.2.0; signed-zero and NaN results may differ with native opt-in ([details](compatibility/floating-point.md)) |
171
-
|`arrays_overlap`| ✅ |Native||
170
+
|`array_union`| ✅ | Native | Floating-point elements fall back on Spark versions other than 4.2.0; signed-zero and NaN results may differ with native opt-in ([details](compatibility/floating-point.md)). Non-default string collations fall back|
171
+
|`arrays_overlap`| ✅ |Hybrid|Non-default string collations use the JVM codegen dispatcher|
172
172
|`arrays_zip`| ✅ | Native ||
173
173
|`element_at`| ✅ | Native ||
174
174
|`flatten`| ✅ | Native | Binary/struct/map elements fall back |
0 commit comments