You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 46faab0
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: .ai/skills/review-comet-shuffle-pr/SKILL.md
+8-6Lines changed: 8 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -55,12 +55,14 @@ keys**, and it is not the same rule for the two partitionings:
55
55
and rejects collated strings, because Comet compares raw bytes. That is the whole rule. Scalar
56
56
float and double are accepted, under `spark.comet.exec.strictFloatingPoint` as well: since
57
57
[#5981](https://github.com/apache/datafusion-comet/pull/5981) the native range partitioner
58
-
normalizes its comparison keys and its sampled boundary rows the same way the native sort does,
59
-
and `CometSortOrder.getSupportLevel` returns `Compatible()` for them regardless of strict mode.
60
-
What strict mode still governs is floating point _nested_ in an array, struct, or map
61
-
([#5507](https://github.com/apache/datafusion-comet/issues/5507)), which is already out as a
62
-
range key for being nested. `CometNativeShuffleSuite` runs "range partitioning on floating-point
63
-
uses native shuffle" under both settings of the config.
58
+
normalizes its comparison keys and its sampled boundary rows the same way the native sort does.
59
+
`CometSortOrder` is `Compatible()` for scalar floats regardless of strict mode. It is for floats
60
+
nested in arrays and structs too, because the native sort normalizes them, unless the key's type
[#6477](https://github.com/apache/datafusion-comet/issues/6477)), which strict mode declines.
64
+
A nested key is already out as a range key for being nested. `CometNativeShuffleSuite` runs
65
+
"range partitioning on floating-point uses native shuffle" under both settings of the config.
64
66
-**`HashPartitioning` is primitive-only by default only.** With
Copy file name to clipboardExpand all lines: docs/source/contributor-guide/expression-audits/array_funcs.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -53,10 +53,10 @@
53
53
## array_distinct
54
54
55
55
- Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8.
56
-
- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayDistinct(child)` over `ArraySetLike`; uses `SQLOpenHashSet` so NaN and `+0.0`/`-0.0`are canonicalized. Wired as `CometScalarFunction("array_distinct")`.
56
+
- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayDistinct(child)` over `ArraySetLike`; uses `SQLOpenHashSet`, which merges NaNs but keeps `+0.0` and `-0.0`distinct. Wired as `CometScalarFunction("array_distinct")`.
57
57
- Spark 4.0.1 (audited 2026-05-27): `NullIntolerant` -> `nullIntolerant` field refactor.
58
58
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
59
-
- Float/double arrays containing NaN and signed zero match Spark; DataFusion canonicalizes them like Spark's `SQLOpenHashSet`.
59
+
- Float/double element types run natively only on Spark 4.2.0, which normalizes the arguments in the plan (SPARK-54918); every other version falls back by default. See [Array distinct and union](../../user-guide/latest/compatibility/floating-point.md#array-distinct-and-union).
60
60
61
61
## array_except
62
62
@@ -141,7 +141,7 @@
141
141
- Spark 3.5.8 (audited 2026-05-27): baseline. `ArrayUnion(left, right) extends ArrayBinaryLike with ComplexTypeMergingExpression`; result is left-side distinct elements followed by new right-side elements. Wired as `CometScalarFunction("array_union")`.
- Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1.
144
-
- Float/double NaN and signed-zero canonicalization matches`array_distinct`. Result element ordering also matches Spark (left-side distinct elements followed by new right-side elements).
144
+
- Float/double element types follow the same Spark 4.2.0-only gate as`array_distinct`. Result element ordering matches Spark (left-side distinct elements followed by new right-side elements).
0 commit comments