You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 77aa7f3
Browse filesBrowse the repository at this point in the historyBrowse files
fix: match Spark's float ordering for sort and window keys nested in arrays and structs (apache#6475)
* fix: match Spark's float ordering for sort and window keys nested in arrays and structs
Spark orders floats with SQLOrderingUtil.compareDoubles at every depth of
an array or struct. Comet normalized scalar float sort and window keys so
that Arrow's total order agrees with Spark's, but compared nested keys raw,
so -0.0 sorted below 0.0 and a NaN with the sign bit set below -Infinity.
Wrap array and struct keys with a float leaf in NormalizeNestedFloats in
create_normalized_key_expr, which builds the keys of Sort, TopK, Window and
WindowGroupLimit. Only the comparison key is normalized, so returned values
keep their bits. CometSortOrder is compatible for every key type, so strict
floating-point mode no longer falls back for these keys.
Closesapache#5507.
* test: link the nested-null issues from the nested float key fixture
* fix: decline nullable nested float keys in strict mode and drop the nested float rank-limit guard
Strict floating-point mode fell back for every float nested in a sort or window key, before
those keys were normalized. Removing that fallback also admitted keys that can hold a null
element or field, which the native sort and RANGE window frames order differently from Spark
(apache#6476, apache#6477). CometSortOrder now keeps the fallback for a key that has a float and whose
type can hold a nested null, and every other key stays native.
apache#6468 made RANK and DENSE_RANK limits fall back for nested float order keys, because the
native operator compared them raw. The native planner now normalizes those keys, so drop that
guard and make its test expect native execution.
Split the nested float order key fixture: the existing one runs with the default settings, and
a new one runs in strict mode and pins both halves of the policy.
* test: drop the sweep suite's known gap for nested float sort keys
With floats nested in array and struct sort keys normalized, the nested
ORDER BY cases match Spark (apache#5507), so the suite now requires them to.
* fix: fall back for RANGE frames over ORDER BY keys DataFusion cannot compare
DataFusion finds a RANGE frame's CURRENT ROW bound by comparing ORDER BY
values with ScalarValue::partial_cmp, which cannot order an array of arrays
or structs, or a struct holding an array (apache/datafusion#24937), so the
query fails with "Uncomparable values". Strict floating-point mode declined
these keys while nested float keys were compared raw, and once null-free
nested float keys became compatible they reached this failure in strict
mode too. The same shapes fail with the default settings whatever their
element type.
CometWindowExec now falls back for a RANGE frame bounded by CURRENT ROW over
such a key. Ranking functions, CUME_DIST, ROWS frames and an unbounded RANGE
frame never compare values this way and stay native.
Copy file name to clipboardExpand all lines: .ai/skills/review-comet-shuffle-pr/SKILL.md
+8-6Lines changed: 8 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -55,12 +55,14 @@ keys**, and it is not the same rule for the two partitionings:
55
55
and rejects collated strings, because Comet compares raw bytes. That is the whole rule. Scalar
56
56
float and double are accepted, under `spark.comet.exec.strictFloatingPoint` as well: since
57
57
[#5981](https://github.com/apache/datafusion-comet/pull/5981) the native range partitioner
58
-
normalizes its comparison keys and its sampled boundary rows the same way the native sort does,
59
-
and `CometSortOrder.getSupportLevel` returns `Compatible()` for them regardless of strict mode.
60
-
What strict mode still governs is floating point _nested_ in an array, struct, or map
61
-
([#5507](https://github.com/apache/datafusion-comet/issues/5507)), which is already out as a
62
-
range key for being nested. `CometNativeShuffleSuite` runs "range partitioning on floating-point
63
-
uses native shuffle" under both settings of the config.
58
+
normalizes its comparison keys and its sampled boundary rows the same way the native sort does.
59
+
`CometSortOrder` is `Compatible()` for scalar floats regardless of strict mode. It is for floats
60
+
nested in arrays and structs too, because the native sort normalizes them, unless the key's type
[#6477](https://github.com/apache/datafusion-comet/issues/6477)), which strict mode declines.
64
+
A nested key is already out as a range key for being nested. `CometNativeShuffleSuite` runs
65
+
"range partitioning on floating-point uses native shuffle" under both settings of the config.
64
66
-**`HashPartitioning` is primitive-only by default only.** With
[#6477](https://github.com/apache/datafusion-comet/issues/6477)). Strict mode makes those keys fall
68
+
back to Spark. A key whose type cannot hold a null, such as `array(coalesce(x, 0.0D))`, stays
69
+
native.
65
70
66
71
`array_min` and `array_max` use Spark-compatible native comparisons in both strict and non-strict
67
72
floating-point modes. Signed zeros compare equal, and all NaN representations compare equal and
-`lag` / `lead` with a non-literal default value ([#4268](https://github.com/apache/datafusion-comet/issues/4268)).
@@ -102,20 +106,10 @@ runs natively; it is controlled by `spark.comet.exec.windowGroupLimit.enabled` (
102
106
- Any `PARTITION BY` or `ORDER BY` key whose type carries a non-default `StringType` collation
103
107
(e.g. `UTF8_LCASE`). The native operator detects partitions and order-key peer groups by
104
108
comparing Arrow row-encoded keys for byte equality, which splits peers that Spark ties.
105
-
-`RANK` and `DENSE_RANK` whose `ORDER BY` key has a `FLOAT` or `DOUBLE` nested in an array or
106
-
struct. The same byte equality decides their ties, and nested floating-point values aren't
107
-
normalized, so `-0.0` and `+0.0`, or two NaN representations, would get different ranks and the
-`ROW_NUMBER` over such a key still runs natively and follows the native sort, which compares
114
-
nested floating-point values with Arrow's raw total ordering. Which of two rows that differ only
115
-
in `-0.0` and `+0.0`, or in their NaN representation, gets the lower row number can therefore
116
-
differ from Spark ([#5507](https://github.com/apache/datafusion-comet/issues/5507)). Scalar
117
-
`FLOAT` and `DOUBLE` keys are normalized and match Spark; see
118
-
[floating-point ordering](./floating-point.md).
109
+
110
+
Floating-point `ORDER BY` keys, including floats nested in arrays and structs, are normalized
111
+
and match Spark's ranks; see [floating-point ordering](./floating-point.md), which also covers
0 commit comments