Skip to content

Commit 390fc91

Browse files
peterxcliclaude
andcommitted
docs: clarify nested-float fallback wording and map unreachability
- Comment in hasNestedFloatElements that map elements are rejected by Spark's analyzer (TypeUtils.checkForOrderingExpr) before planning - Reword arrays_overlap/array_position notes in expressions.md to state the nested-float case falls back to Spark by default, with the native path opt-in via allowIncompatible Co-Authored-By: Claude Fable 5 <[email protected]>
1 parent db077df commit 390fc91

2 files changed

Lines changed: 5 additions & 2 deletions

File tree

‎docs/source/user-guide/latest/expressions.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -158,12 +158,12 @@ The tables below list every Spark built-in expression with its current status.
158158
| `array_join` | ✅ | Hybrid | Routes through the JVM codegen dispatcher by default; the incompatible native path is opt-in via allowIncompatible ([details](compatibility/expressions/array.md)) |
159159
| `array_max` | ✅ | Native | NaN ordering may differ ([details](compatibility/floating-point.md)) |
160160
| `array_min` | ✅ | Native | NaN ordering may differ ([details](compatibility/floating-point.md)) |
161-
| `array_position` | ✅ | Native | Binary/struct/map/null elements fall back; nested floating-point signed-zero handling differs ([#5191](https://github.com/apache/datafusion-comet/issues/5191)) |
161+
| `array_position` | ✅ | Native | Binary/struct/map/null elements fall back; arrays with nested floating-point elements fall back to Spark by default, and the incompatible native path is opt-in via allowIncompatible ([#5191](https://github.com/apache/datafusion-comet/issues/5191)) |
162162
| `array_prepend` | ✅ | — | |
163163
| `array_remove` | ✅ | Native | |
164164
| `array_repeat` | ✅ | Native | |
165165
| `array_union` | ✅ | Native | NaN/signed-zero handling may differ ([details](compatibility/floating-point.md)) |
166-
| `arrays_overlap` | ✅ | Native | Nested floating-point signed-zero handling differs ([#5191](https://github.com/apache/datafusion-comet/issues/5191)) |
166+
| `arrays_overlap` | ✅ | Native | Arrays with nested floating-point elements fall back to Spark by default, and the incompatible native path is opt-in via allowIncompatible ([#5191](https://github.com/apache/datafusion-comet/issues/5191)) |
167167
| `arrays_zip` | ✅ | Native | |
168168
| `element_at` | ✅ | Native | |
169169
| `flatten` | ✅ | Native | Binary/struct/map elements fall back |

‎spark/src/main/scala/org/apache/comet/serde/arrays.scala‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -851,6 +851,9 @@ trait ArraysBase {
851851
SupportLevel.containsType(elementType, classOf[FloatType], classOf[DoubleType])
852852
case ArrayType(elementType: StructType, _) =>
853853
SupportLevel.containsType(elementType, classOf[FloatType], classOf[DoubleType])
854+
// Map elements cannot reach here: ArraysOverlap and ArrayPosition call
855+
// TypeUtils.checkForOrderingExpr on the element type, and maps are not orderable,
856+
// so Spark's analyzer rejects them before planning.
854857
case _ => false
855858
}
856859

0 commit comments

Comments
 (0)