Skip to content

Commit 6471565

Browse files
committed
fix: decline nullable nested float keys in strict mode and drop the nested float rank-limit guard
Strict floating-point mode fell back for every float nested in a sort or window key, before those keys were normalized. Removing that fallback also admitted keys that can hold a null element or field, which the native sort and RANGE window frames order differently from Spark (#6476, #6477). CometSortOrder now keeps the fallback for a key that has a float and whose type can hold a nested null, and every other key stays native. #6468 made RANK and DENSE_RANK limits fall back for nested float order keys, because the native operator compared them raw. The native planner now normalizes those keys, so drop that guard and make its test expect native execution. Split the nested float order key fixture: the existing one runs with the default settings, and a new one runs in strict mode and pins both halves of the policy.
1 parent e9578ca commit 6471565

11 files changed

Lines changed: 225 additions & 59 deletions

File tree

‎.ai/skills/review-comet-shuffle-pr/SKILL.md‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -56,10 +56,13 @@ keys**, and it is not the same rule for the two partitionings:
5656
float and double are accepted, under `spark.comet.exec.strictFloatingPoint` as well: since
5757
[#5981](https://github.com/apache/datafusion-comet/pull/5981) the native range partitioner
5858
normalizes its comparison keys and its sampled boundary rows the same way the native sort does.
59-
`CometSortOrder` is `Compatible()` for every key type regardless of strict mode, because the
60-
native sort also normalizes floats nested in arrays and structs, but a nested key is already out
61-
as a range key for being nested. `CometNativeShuffleSuite` runs "range partitioning on
62-
floating-point uses native shuffle" under both settings of the config.
59+
`CometSortOrder` is `Compatible()` for scalar floats regardless of strict mode. It is for floats
60+
nested in arrays and structs too, because the native sort normalizes them, unless the key's type
61+
can hold a null element or field
62+
([#6476](https://github.com/apache/datafusion-comet/issues/6476),
63+
[#6477](https://github.com/apache/datafusion-comet/issues/6477)), which strict mode declines.
64+
A nested key is already out as a range key for being nested. `CometNativeShuffleSuite` runs
65+
"range partitioning on floating-point uses native shuffle" under both settings of the config.
6366
- **`HashPartitioning` is primitive-only by default only.** With
6467
`spark.comet.shuffle.native.partitioning.hash.nested.enabled=true` (default `false`),
6568
`supportedHashPartitioningDataType` admits structs and arrays recursively, and maps on Spark

‎docs/source/user-guide/latest/compatibility/floating-point.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,16 @@ Because those comparison keys match Spark, `spark.comet.exec.strictFloatingPoint
5858
force a fallback for them: sort keys, window and rank order keys, and range partitioning keys all
5959
stay native under strict mode, whether the floats in them are scalar or nested.
6060

61+
The exception is a key that nests floats in an array or struct whose type can hold a null element
62+
or field. Spark orders such a null below every other value, whatever the key's `NULLS FIRST` or
63+
`NULLS LAST`. The native sort places it by that null order, so `ASC NULLS LAST` and
64+
`DESC NULLS FIRST` can differ from Spark, and a `RANGE` window frame orders it above every other
65+
value, so a running aggregate can span the whole partition
66+
([#6476](https://github.com/apache/datafusion-comet/issues/6476),
67+
[#6477](https://github.com/apache/datafusion-comet/issues/6477)). Strict mode makes those keys fall
68+
back to Spark. A key whose type cannot hold a null, such as `array(coalesce(x, 0.0D))`, stays
69+
native.
70+
6171
`array_min` and `array_max` use Spark-compatible native comparisons in both strict and non-strict
6272
floating-point modes. Signed zeros compare equal, and all NaN representations compare equal and
6373
greater than non-NaN values. The original first equal element is retained: for example,

‎docs/source/user-guide/latest/compatibility/operators.md‎

Lines changed: 2 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -102,14 +102,10 @@ runs natively; it is controlled by `spark.comet.exec.windowGroupLimit.enabled` (
102102
- Any `PARTITION BY` or `ORDER BY` key whose type carries a non-default `StringType` collation
103103
(e.g. `UTF8_LCASE`). The native operator detects partitions and order-key peer groups by
104104
comparing Arrow row-encoded keys for byte equality, which splits peers that Spark ties.
105-
- `RANK` and `DENSE_RANK` whose `ORDER BY` key has a `FLOAT` or `DOUBLE` nested in an array or
106-
struct. The same byte equality decides their ties, and nested floating-point values aren't
107-
normalized, so `-0.0` and `+0.0`, or two NaN representations, would get different ranks and the
108-
cutoff would drop rows that Spark keeps
109-
([#5507](https://github.com/apache/datafusion-comet/issues/5507)).
110105

111106
Floating-point `ORDER BY` keys, including floats nested in arrays and structs, are normalized
112-
and match Spark's ranks; see [floating-point ordering](./floating-point.md).
107+
and match Spark's ranks; see [floating-point ordering](./floating-point.md), which also covers
108+
strict floating-point mode.
113109

114110
## Round-Robin Partitioning
115111

‎docs/source/user-guide/latest/tuning/operators.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -154,6 +154,13 @@ nested in arrays and structs, so `ORDER BY`, window ordering and range partition
154154
even with `spark.comet.exec.strictFloatingPoint=true`. Only the comparison key is normalized; returned values keep their
155155
original NaN representation and zero sign.
156156

157+
The exception is a key that nests floating-point values in an array or struct whose type can hold a null element or
158+
field. Spark orders such a null below every other value, and the native sort and `RANGE` window frames do not
159+
([#6476](https://github.com/apache/datafusion-comet/issues/6476),
160+
[#6477](https://github.com/apache/datafusion-comet/issues/6477)), so
161+
`spark.comet.exec.strictFloatingPoint=true` makes those keys fall back to Spark. They can be forced back onto the
162+
native path with `spark.comet.expression.SortOrder.allowIncompatible=true`.
163+
157164
`sort_array` is separate. It sorts array elements rather than ordering rows, and its elements are compared with Arrow's
158165
raw total ordering, so `spark.comet.exec.strictFloatingPoint=true` makes it fall back even for a scalar floating-point
159166
element type. Use `spark.comet.expression.SortArray.allowIncompatible=true` to keep it native.

‎spark/src/main/scala/org/apache/comet/CometConf.scala‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1082,7 +1082,9 @@ object CometConf extends ShimCometConf {
10821082
"Spark, such as comparing -0.0 and 0.0, or sorting the elements of a floating-point " +
10831083
"array with `sort_array`. `ORDER BY`, window ordering and range partitioning keys " +
10841084
"are unaffected, including floating-point values nested in arrays and structs, " +
1085-
"because Comet normalizes those comparison keys to match Spark. " +
1085+
"because Comet normalizes those comparison keys to match Spark. The exception is a " +
1086+
"nested key whose type can hold a null element or field, which falls back until the " +
1087+
"native sort and window frames order those nulls as Spark does. " +
10861088
s"$COMPAT_GUIDE.")
10871089
.booleanConf
10881090
.createWithDefault(false)

‎spark/src/main/scala/org/apache/comet/serde/CometSortOrder.scala‎

Lines changed: 51 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -20,20 +20,66 @@
2020
package org.apache.comet.serde
2121

2222
import org.apache.spark.sql.catalyst.expressions.{Ascending, Attribute, Descending, NullsFirst, NullsLast, SortOrder}
23+
import org.apache.spark.sql.types.{ArrayType, DataType, StructType}
2324

25+
import org.apache.comet.CometConf
2426
import org.apache.comet.serde.QueryPlanSerde.exprToProtoInternal
2527

2628
/**
2729
* The key of a Sort, TopK, Window, WindowGroupLimit or range partitioning.
2830
*
29-
* Every key type is compatible, in strict floating-point mode too. Arrow orders floats by IEEE
30-
* 754 total order, so the native side normalizes a `FLOAT` or `DOUBLE` key, and an array or
31-
* struct key with a float at any depth, before comparing: NaN payloads fold together and signed
32-
* zeros tie, as in Spark's `SQLOrderingUtil`. Only the comparison key is normalized; returned
33-
* values keep their original NaN representation and zero sign.
31+
* Arrow orders floats by IEEE 754 total order, so the native side normalizes a `FLOAT` or
32+
* `DOUBLE` key, and an array or struct key with a float at any depth, before comparing: NaN
33+
* payloads fold together and signed zeros tie, as in Spark's `SQLOrderingUtil`. Only the
34+
* comparison key is normalized; returned values keep their original NaN representation and zero
35+
* sign. So every key is compatible, in strict floating-point mode too, except one that has a
36+
* float and whose type can hold a null element or field.
37+
*
38+
* Spark orders a null element or field below every other value, whatever the key's null order.
39+
* The native sort takes its place from the key's null order, and a `RANGE` window frame orders it
40+
* above every other value, so those keys can sort or frame differently from Spark, with floats or
41+
* without. Strict mode keeps the fallback it applied to every nested float key before they were
42+
* normalized, for the keys that can hit this.
43+
* https://github.com/apache/datafusion-comet/issues/6476
44+
* https://github.com/apache/datafusion-comet/issues/6477
3445
*/
3546
object CometSortOrder extends CometExpressionSerde[SortOrder] {
3647

48+
/**
49+
* Subject of both the runtime fallback reason and the generated compatibility docs. Shared so
50+
* the two cannot describe the policy differently.
51+
*/
52+
private val nullableNestedFloatingPointSort =
53+
"Sorting on floating-point values nested in an array or struct that can hold a null " +
54+
"element or field"
55+
56+
override def getIncompatibleReasons(): Seq[String] = Seq(
57+
s"$nullableNestedFloatingPointSort is not 100% compatible with Spark when " +
58+
s"`${CometConf.COMET_EXEC_STRICT_FLOATING_POINT.key}=true`")
59+
60+
override def getSupportLevel(expr: SortOrder): SupportLevel = {
61+
val dataType = expr.child.dataType
62+
if (canHoldNestedNull(dataType)) {
63+
SupportLevel
64+
.strictFloatingPointReason(dataType, nullableNestedFloatingPointSort)
65+
.map(reason => Incompatible(Some(reason)))
66+
.getOrElse(Compatible())
67+
} else {
68+
Compatible()
69+
}
70+
}
71+
72+
/**
73+
* Whether a value of `dataType` can hold a null below its top level: an array element or a
74+
* struct field, at any depth. The key being null is not one: both engines place that by the
75+
* key's null order.
76+
*/
77+
private def canHoldNestedNull(dataType: DataType): Boolean = dataType match {
78+
case ArrayType(elementType, containsNull) => containsNull || canHoldNestedNull(elementType)
79+
case StructType(fields) => fields.exists(f => f.nullable || canHoldNestedNull(f.dataType))
80+
case _ => false
81+
}
82+
3783
override def convert(
3884
expr: SortOrder,
3985
inputs: Seq[Attribute],

‎spark/src/main/scala/org/apache/spark/sql/comet/CometWindowGroupLimitExec.scala‎

Lines changed: 6 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -24,13 +24,12 @@ import scala.jdk.CollectionConverters._
2424
import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression, SortOrder}
2525
import org.apache.spark.sql.catalyst.plans.physical.Partitioning
2626
import org.apache.spark.sql.execution.SparkPlan
27-
import org.apache.spark.sql.types.{DoubleType, FloatType}
2827

2928
import com.google.common.base.Objects
3029

3130
import org.apache.comet.{CometConf, ConfigEntry}
3231
import org.apache.comet.CometSparkSessionExtensions.withFallbackReason
33-
import org.apache.comet.serde.{CometOperatorSerde, OperatorOuterClass, SupportLevel}
32+
import org.apache.comet.serde.{CometOperatorSerde, OperatorOuterClass}
3433
import org.apache.comet.serde.OperatorOuterClass.{Operator, RankLikeFunction}
3534
import org.apache.comet.serde.QueryPlanSerde.{exprToProto, hasNonDefaultStringCollation}
3635
import org.apache.comet.shims.ShimCometWindowGroupLimit
@@ -100,32 +99,11 @@ object CometWindowGroupLimitExec extends CometOperatorSerde[SparkPlan] {
10099
return None
101100
}
102101

103-
// The same byte equality decides RANK and DENSE_RANK ties. Scalar FLOAT and DOUBLE order keys
104-
// are normalized natively first, but a float nested in an array or struct keeps its raw bits,
105-
// so -0.0 and 0.0, or two NaN encodings, would split a tie that Spark keeps and the cutoff
106-
// would drop rows. ROW_NUMBER never compares peers, and Spark normalizes floating-point
107-
// partition keys before the limit is planned.
108-
// https://github.com/apache/datafusion-comet/issues/5507
109-
if (fields.rankLikeFunction != RankLikeFunction.RowNumber) {
110-
val nestedFloat = fields.orderSpec.map(_.child).filter { e =>
111-
e.dataType match {
112-
case _: FloatType | _: DoubleType => false
113-
case dt => SupportLevel.containsType(dt, classOf[FloatType], classOf[DoubleType])
114-
}
115-
}
116-
if (nestedFloat.nonEmpty) {
117-
withFallbackReason(
118-
op,
119-
nestedFloat
120-
.map(_.sql)
121-
.mkString(
122-
"WindowGroupLimit: RANK and DENSE_RANK compare nested floating-point values " +
123-
"exactly, so they fall back for order key(s): ",
124-
", ",
125-
""))
126-
return None
127-
}
128-
}
102+
// The same byte equality decides RANK and DENSE_RANK ties, but it runs on keys that the native
103+
// planner has normalized: a `FLOAT` or `DOUBLE` order key, scalar or nested in an array or
104+
// struct, has its NaNs folded together and its signed zeros tied first, as Spark does, so no
105+
// float key needs a guard here. Strict floating-point mode declines the nested keys that can
106+
// hold a null element or field, through `CometSortOrder`.
129107

130108
val childOutput = op.children.head.output
131109
val partitionProtos = fields.partitionSpec.map(e => e -> exprToProto(e, childOutput))

‎spark/src/test/resources/sql-tests/windows/nested_float_order_keys.sql‎

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -24,10 +24,8 @@
2424
-- total order sorts below -Infinity. Each ORDER BY ends with a unique tiebreaker, so peers come
2525
-- out in the tiebreaker's order.
2626

27-
-- Strict floating-point mode no longer needs to fall back for these keys. The test harness
28-
-- admits incompatible sort orders by default, so turn that off to check the shipped policy.
29-
-- ConfigMatrix: spark.comet.exec.strictFloatingPoint=false,true
30-
-- Config: spark.comet.expression.SortOrder.allowIncompatible=false
27+
-- Strict floating-point mode declines these keys, because their types can hold a null element or
28+
-- field: see nested_float_order_keys_strict.sql.
3129

3230
statement
3331
CREATE TABLE nested_float_keys(id INT, g INT, d DOUBLE, f FLOAT, s BOOLEAN) USING parquet
Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
-- Licensed to the Apache Software Foundation (ASF) under one
2+
-- or more contributor license agreements. See the NOTICE file
3+
-- distributed with this work for additional information
4+
-- regarding copyright ownership. The ASF licenses this file
5+
-- to you under the Apache License, Version 2.0 (the
6+
-- "License"); you may not use this file except in compliance
7+
-- with the License. You may obtain a copy of the License at
8+
--
9+
-- http://www.apache.org/licenses/LICENSE-2.0
10+
--
11+
-- Unless required by applicable law or agreed to in writing,
12+
-- software distributed under the License is distributed on an
13+
-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
-- KIND, either express or implied. See the License for the
15+
-- specific language governing permissions and limitations
16+
-- under the License.
17+
18+
-- Strict floating-point mode and sort or window keys that nest floats in arrays and structs.
19+
-- nested_float_order_keys.sql checks that Comet normalizes those floats to match Spark. Strict
20+
-- mode still declines a key whose type can hold a null element or field, because Spark orders
21+
-- such a null below every value whatever the key's null order, the native sort places it by the
22+
-- null order (#6476), and a RANGE window frame orders it above every value (#6477). A key whose
23+
-- type cannot hold a null stays native.
24+
--
25+
-- The test harness admits incompatible sort orders by default, so turn that off to check the
26+
-- shipped policy.
27+
-- Config: spark.comet.exec.strictFloatingPoint=true
28+
-- Config: spark.comet.expression.SortOrder.allowIncompatible=false
29+
30+
statement
31+
CREATE TABLE nested_float_strict(id INT, g INT, d DOUBLE, f FLOAT, s BOOLEAN) USING parquet
32+
33+
statement
34+
INSERT INTO nested_float_strict VALUES
35+
(1, 1, 0.0D, float('0.0'), true),
36+
(2, 1, 0.0D, float('0.0'), false),
37+
(3, 1, 1.0D, float('1.0'), false),
38+
(4, 2, double('NaN'), float('NaN'), false),
39+
(5, 2, double('NaN'), float('NaN'), true),
40+
(6, 2, double('Infinity'), float('Infinity'), false),
41+
(7, 2, -1.0D, float('-1.0'), false),
42+
(8, 1, NULL, NULL, false),
43+
(9, 2, double('-Infinity'), float('-Infinity'), false)
44+
45+
-- Keys over the nullable columns can hold a null element or field, whether or not any row does,
46+
-- so they fall back, and Spark's answers come back. Row 8 holds one. Native, ORDER BY array(d)
47+
-- NULLS LAST puts row 8 last, where Spark puts it first, and the running sum spans the whole
48+
-- partition on row 8 and every row after it.
49+
query expect_fallback(can hold a null element or field)
50+
SELECT id FROM nested_float_strict ORDER BY array(d) NULLS LAST, id
51+
52+
query expect_fallback(can hold a null element or field)
53+
SELECT id, SUM(id) OVER (ORDER BY array(d)) AS running FROM nested_float_strict
54+
55+
query expect_fallback(can hold a null element or field)
56+
SELECT id FROM nested_float_strict ORDER BY named_struct('x', f) DESC NULLS FIRST, id
57+
58+
query expect_fallback(can hold a null element or field)
59+
SELECT id FROM nested_float_strict ORDER BY array(IF(s, -d, d)), id LIMIT 4
60+
61+
query expect_fallback(can hold a null element or field)
62+
SELECT id,
63+
RANK() OVER (PARTITION BY g ORDER BY named_struct('x', IF(s, -d, d))) AS r,
64+
DENSE_RANK() OVER (ORDER BY array(IF(s, -f, f)) DESC) AS dr
65+
FROM nested_float_strict
66+
67+
query expect_fallback(can hold a null element or field)
68+
SELECT id FROM (
69+
SELECT id, RANK() OVER (ORDER BY array(IF(s, -d, d))) AS r FROM nested_float_strict
70+
) WHERE r <= 3
71+
72+
-- coalesce with a literal is never null, so an array or struct of it cannot hold a null, and
73+
-- these keys stay native. Row 8 becomes a third zero. Each ORDER BY ends with a unique
74+
-- tiebreaker, so peers come out in its order, and the running sums need no null row to leave out.
75+
query
76+
SELECT id FROM nested_float_strict ORDER BY array(coalesce(IF(s, -d, d), 0.0D)), id DESC
77+
78+
query
79+
SELECT id FROM nested_float_strict ORDER BY named_struct('x', coalesce(IF(s, -f, f), 0.0F)) DESC, id
80+
81+
query
82+
SELECT id FROM nested_float_strict ORDER BY array(coalesce(IF(s, -d, d), 0.0D)) DESC, id LIMIT 4
83+
84+
query
85+
SELECT id,
86+
RANK() OVER (ORDER BY array(coalesce(IF(s, -d, d), 0.0D))) AS r,
87+
SUM(id) OVER (ORDER BY array(coalesce(IF(s, -d, d), 0.0D))) AS running,
88+
DENSE_RANK() OVER (PARTITION BY g ORDER BY named_struct('x', coalesce(IF(s, -f, f), 0.0F)) DESC) AS dr
89+
FROM nested_float_strict
90+
91+
-- A rank limit keeps every peer of the last rank it admits
92+
query
93+
SELECT id FROM (
94+
SELECT id, RANK() OVER (ORDER BY array(coalesce(IF(s, -d, d), 0.0D))) AS r
95+
FROM nested_float_strict
96+
) WHERE r <= 3
97+
98+
query
99+
SELECT id FROM (
100+
SELECT id,
101+
DENSE_RANK() OVER (PARTITION BY g ORDER BY named_struct('x', coalesce(IF(s, -f, f), 0.0F)) DESC) AS r
102+
FROM nested_float_strict
103+
) WHERE r <= 1

0 commit comments

Comments
 (0)