You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit ee66473
Browse filesBrowse the repository at this point in the historyBrowse files
fix: route map_from_arrays float keys through the codegen dispatcher
Address review feedback on the float map key gate:
- Mix CodegenDispatchFallback into CometMapFromArrays so FLOAT or
DOUBLE keys and LAST_WIN stay in the Comet pipeline, as they do for
map_from_entries.
- Use expect_error_class for the dispatched DUPLICATED_MAP_KEY queries,
and fix the array-key zeros query to read the row that holds zeros.
- Rename the fixtures to map_builders_float_key_gate* so they do not
collide with the ones in #6682.
- Correct the MapKeySupport reason, which described the lookup as
Spark normalizing keys, and trim the repeated explanation of the
builder rules down to MapBuilderKeySupport.floatingPointReason.
Copy file name to clipboardExpand all lines: docs/source/contributor-guide/expression-audits/map_funcs.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -48,7 +48,7 @@
48
48
- Spark 3.5.8 (audited 2026-05-27): baseline. `MapFromArrays(left, right) extends BinaryExpression with NullIntolerant`; Spark uses `ArrayBasedMapBuilder` to detect duplicate keys (subject to `spark.sql.mapKeyDedupPolicy`) and rejects null keys with `RuntimeException("Cannot use null as map key")`. Comet `CometMapFromArrays` emits nested `CaseWhen(IsNotNull(left), CaseWhen(IsNotNull(right), map(left, right), null), null)` guards so NULL-array inputs return NULL rather than triggering the previously reported native crash ([#3327](https://github.com/apache/datafusion-comet/issues/3327)). The outer key guard preserves Spark's left-before-right evaluation: one `IsNotNull(left) AND IsNotNull(right)` predicate can still evaluate the values child for a NULL-key row in a mixed batch. The native `SparkMapFromArrays` wrapper also validates key/value lengths per non-null row before DataFusion constructs the map, so matching flattened totals cannot pair values across row boundaries.
49
49
- Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` trait replaced by `nullIntolerant: Boolean`. `ArrayBasedMapBuilder` gains the `keyNormalizer` described below.
50
50
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
51
-
- Floating-point keys (2026-10-14, [#6549](https://github.com/apache/datafusion-comet/issues/6549)): `ArrayBasedMapBuilder` finds a duplicate `FLOAT` or `DOUBLE` key in a `HashMap` of boxed values (`Double.equals`, so NaNs with different bits are one key) and a duplicate struct or array key in a `TreeMap` in the SQL ordering (`-0.0` equals `0.0`). From Spark 4.0 its `keyNormalizer` also normalizes a `FLOAT` or `DOUBLE` key before the lookup, unless `spark.sql.legacy.disableMapKeyNormalization` is set. The native builders compare the keys' bits, so a key type that holds a float at any depth is `Incompatible` on every version (`MapBuilderKeySupport`). `map_from_arrays`falls back to Spark for them. `from` returns its input arrays when no key repeats, so a lone `-0.0` key stays `-0.0` there.
51
+
- Floating-point keys (2026-10-14, [#6549](https://github.com/apache/datafusion-comet/issues/6549)): `ArrayBasedMapBuilder` finds a duplicate `FLOAT` or `DOUBLE` key in a `HashMap` of boxed values (`Double.equals`, so NaNs with different bits are one key) and a duplicate struct or array key in a `TreeMap` in the SQL ordering (`-0.0` equals `0.0`). From Spark 4.0 its `keyNormalizer` also normalizes a `FLOAT` or `DOUBLE` key before the lookup, unless `spark.sql.legacy.disableMapKeyNormalization` is set. The native builders compare the keys' bits, so a key type that holds a float at any depth is `Incompatible` on every version (`MapBuilderKeySupport`). `map_from_arrays`routes them through the codegen dispatcher. `from` returns its input arrays when no key repeats, so a lone `-0.0` key stays `-0.0` there.
Copy file name to clipboardExpand all lines: docs/source/user-guide/latest/compatibility/floating-point.md
+4-8Lines changed: 4 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -168,11 +168,7 @@ array whose elements cannot be null in ascending order, Spark's generated code p
168
168
## Map construction
169
169
170
170
`map_from_arrays` and `map_from_entries` don't build a map natively when its key type holds a
171
-
`FLOAT` or `DOUBLE` at any depth. Spark finds duplicate keys by value, so NaNs with different bits
172
-
are one key, and inside a struct or array key `-0.0` and `0.0` are one key too. From Spark 4.0,
173
-
unless `spark.sql.legacy.disableMapKeyNormalization` is set, Spark also normalizes a top-level key,
174
-
so `-0.0` and `0.0` are one key and `map_from_entries` stores `0.0`. Comet's native map
175
-
construction compares the keys' bits
176
-
([#6549](https://github.com/apache/datafusion-comet/issues/6549)), so `map_from_entries` runs
177
-
through the codegen dispatcher instead, and `map_from_arrays` falls back to Spark. `FLOAT` and
178
-
`DOUBLE` values don't affect either function.
171
+
`FLOAT` or `DOUBLE` at any depth. They run Spark's code through the codegen dispatcher instead, so
172
+
duplicate keys, including NaNs and zeros of either sign, behave as in Spark. `FLOAT` and `DOUBLE`
173
+
values don't affect either function. The expressions' entries in the
174
+
[expression compatibility guide](expressions/map.md) give the reason.
Copy file name to clipboardExpand all lines: docs/source/user-guide/latest/expressions.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -408,7 +408,7 @@ to Spark ([#2837](https://github.com/apache/datafusion-comet/issues/2837)). Enab
408
408
|`map_concat`| ✅ | Codegen dispatch ||
409
409
|`map_contains_key`| ✅ | — ||
410
410
|`map_entries`| ✅ | Native ||
411
-
|`map_from_arrays`| ✅ |Native|`FLOAT` or `DOUBLE` keys and `spark.sql.mapKeyDedupPolicy=LAST_WIN`fall back to Spark ([details](compatibility/expressions/map.md)) |
411
+
|`map_from_arrays`| ✅ |Hybrid|`FLOAT` or `DOUBLE` keys and `spark.sql.mapKeyDedupPolicy=LAST_WIN`route through the JVM codegen dispatcher ([details](compatibility/expressions/map.md)) |
412
412
|`map_from_entries`| ✅ | Hybrid | BinaryType keys/values, `FLOAT` or `DOUBLE` keys and `spark.sql.mapKeyDedupPolicy=LAST_WIN` route through the JVM codegen dispatcher ([details](compatibility/expressions/map.md)) |
0 commit comments