Skip to content

Commit c312b1c

Browse files
committed
fix: fall back to Spark for RANK and DENSE_RANK limits over nested floating-point keys
The native WindowGroupLimit finds RANK and DENSE_RANK ties by comparing row-encoded ORDER BY keys byte for byte. Scalar float keys are normalized first, but a float nested in an array or struct keeps its raw bits, so [-0.0] and [0.0] got different ranks and the cutoff dropped rows that Spark keeps as ties. In 1.0.0 the limit ran in Spark, so this is a regression in 1.1.0. Fall back for RANK and DENSE_RANK when an order key contains a nested FLOAT or DOUBLE. ROW_NUMBER never compares peers and stays native, and Spark already normalizes floating-point partition keys. Part of apache#5507.
1 parent 9f68a41 commit c312b1c

3 files changed

Lines changed: 93 additions & 5 deletions

File tree

‎docs/source/user-guide/latest/compatibility/operators.md‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -102,13 +102,19 @@ runs natively; it is controlled by `spark.comet.exec.windowGroupLimit.enabled` (
102102
- Any `PARTITION BY` or `ORDER BY` key whose type carries a non-default `StringType` collation
103103
(e.g. `UTF8_LCASE`). The native operator detects partitions and order-key peer groups by
104104
comparing Arrow row-encoded keys for byte equality, which splits peers that Spark ties.
105+
- `RANK` and `DENSE_RANK` whose `ORDER BY` key has a `FLOAT` or `DOUBLE` nested in an array or
106+
struct. The same byte equality decides their ties, and nested floating-point values aren't
107+
normalized, so `-0.0` and `+0.0`, or two NaN representations, would get different ranks and the
108+
cutoff would drop rows that Spark keeps
109+
([#5507](https://github.com/apache/datafusion-comet/issues/5507)).
105110

106111
**Known incompatibilities:**
107112

108-
- Floating-point values nested in array or struct `ORDER BY` keys are compared with Arrow's raw
109-
total ordering, so ranks can differ from Spark when the data mixes `-0.0` and `+0.0` or more
110-
than one NaN representation ([#5507](https://github.com/apache/datafusion-comet/issues/5507)).
111-
Scalar `FLOAT` and `DOUBLE` keys are normalized and match Spark; see
113+
- `ROW_NUMBER` over such a key still runs natively and follows the native sort, which compares
114+
nested floating-point values with Arrow's raw total ordering. Which of two rows that differ only
115+
in `-0.0` and `+0.0`, or in their NaN representation, gets the lower row number can therefore
116+
differ from Spark ([#5507](https://github.com/apache/datafusion-comet/issues/5507)). Scalar
117+
`FLOAT` and `DOUBLE` keys are normalized and match Spark; see
112118
[floating-point ordering](./floating-point.md).
113119

114120
## Round-Robin Partitioning

‎spark/src/main/scala/org/apache/spark/sql/comet/CometWindowGroupLimitExec.scala‎

Lines changed: 29 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,12 +24,13 @@ import scala.jdk.CollectionConverters._
2424
import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression, SortOrder}
2525
import org.apache.spark.sql.catalyst.plans.physical.Partitioning
2626
import org.apache.spark.sql.execution.SparkPlan
27+
import org.apache.spark.sql.types.{DoubleType, FloatType}
2728

2829
import com.google.common.base.Objects
2930

3031
import org.apache.comet.{CometConf, ConfigEntry}
3132
import org.apache.comet.CometSparkSessionExtensions.withFallbackReason
32-
import org.apache.comet.serde.{CometOperatorSerde, OperatorOuterClass}
33+
import org.apache.comet.serde.{CometOperatorSerde, OperatorOuterClass, SupportLevel}
3334
import org.apache.comet.serde.OperatorOuterClass.{Operator, RankLikeFunction}
3435
import org.apache.comet.serde.QueryPlanSerde.{exprToProto, hasNonDefaultStringCollation}
3536
import org.apache.comet.shims.ShimCometWindowGroupLimit
@@ -99,6 +100,33 @@ object CometWindowGroupLimitExec extends CometOperatorSerde[SparkPlan] {
99100
return None
100101
}
101102

103+
// The same byte equality decides RANK and DENSE_RANK ties. Scalar FLOAT and DOUBLE order keys
104+
// are normalized natively first, but a float nested in an array or struct keeps its raw bits,
105+
// so -0.0 and 0.0, or two NaN encodings, would split a tie that Spark keeps and the cutoff
106+
// would drop rows. ROW_NUMBER never compares peers, and Spark normalizes floating-point
107+
// partition keys before the limit is planned.
108+
// https://github.com/apache/datafusion-comet/issues/5507
109+
if (fields.rankLikeFunction != RankLikeFunction.RowNumber) {
110+
val nestedFloat = fields.orderSpec.map(_.child).filter { e =>
111+
e.dataType match {
112+
case _: FloatType | _: DoubleType => false
113+
case dt => SupportLevel.containsType(dt, classOf[FloatType], classOf[DoubleType])
114+
}
115+
}
116+
if (nestedFloat.nonEmpty) {
117+
withFallbackReason(
118+
op,
119+
nestedFloat
120+
.map(_.sql)
121+
.mkString(
122+
"WindowGroupLimit: RANK and DENSE_RANK compare nested floating-point values " +
123+
"exactly, so they fall back for order key(s): ",
124+
", ",
125+
""))
126+
return None
127+
}
128+
}
129+
102130
val childOutput = op.children.head.output
103131
val partitionProtos = fields.partitionSpec.map(e => e -> exprToProto(e, childOutput))
104132
val orderProtos = fields.orderSpec.map(e => e -> exprToProto(e, childOutput))

‎spark/src/test/scala/org/apache/comet/exec/CometWindowExecSuite.scala‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -210,6 +210,60 @@ class CometWindowExecSuite extends CometTestBase {
210210
}
211211
}
212212

213+
test("window group limit: floating-point values nested in the order key") {
214+
assume(isSpark35Plus, "WindowGroupLimit was added in Spark 3.5")
215+
// The native RANK and DENSE_RANK find ties by byte equality, and nested floats aren't
216+
// normalized, so [-0.0] and [0.0] would get different ranks and the cutoff would drop a row
217+
// that Spark keeps (#5507). The limit falls back to Spark for them instead. ROW_NUMBER never
218+
// compares peers, and Spark normalizes nested floating-point partition keys itself.
219+
withTempDir { dir =>
220+
val path = new Path(dir.toString, "nested_float_order").toString
221+
Seq((1, -0.0f, -0.0d), (2, 0.0f, 0.0d), (3, 1.0f, 1.0d))
222+
.toDF("id", "f", "d")
223+
.write
224+
.parquet(path)
225+
spark.read.parquet(path).createOrReplaceTempView("nested_float_order")
226+
227+
withSQLConf(
228+
SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false",
229+
SQLConf.SHUFFLE_PARTITIONS.key -> "1",
230+
CometConf.COMET_EXEC_STRICT_FLOATING_POINT.key -> "false",
231+
CometConf.getExprAllowIncompatConfigKey("SortOrder") -> "false",
232+
CometConf.COMET_EXEC_WINDOW_GROUP_LIMIT_ENABLED.key -> "true") {
233+
for {
234+
rankFunction <- Seq("RANK", "DENSE_RANK")
235+
// A struct needs a second key, because a lone struct sort key falls back on its own.
236+
orderBy <- Seq("array(f)", "array(d)", "named_struct('x', d), id > 0")
237+
} {
238+
val query = sql(s"""
239+
|SELECT id FROM (
240+
| SELECT id, $rankFunction() OVER (ORDER BY $orderBy) AS rnk
241+
| FROM nested_float_order
242+
|) WHERE rnk <= 1
243+
|""".stripMargin)
244+
checkSparkAnswerAndFallbackReason(query, "compare nested floating-point values exactly")
245+
assert(query.collect().map(_.getInt(0)).sorted.toSeq == Seq(1, 2))
246+
}
247+
248+
checkSparkAnswerAndOperator(
249+
sql("""
250+
|SELECT id FROM (
251+
| SELECT id, ROW_NUMBER() OVER (PARTITION BY id > 0 ORDER BY array(d), id) AS rn
252+
| FROM nested_float_order
253+
|) WHERE rn <= 2
254+
|""".stripMargin),
255+
Seq(classOf[CometWindowGroupLimitExec]))
256+
257+
checkSparkAnswer(sql("""
258+
|SELECT id FROM (
259+
| SELECT id, RANK() OVER (PARTITION BY array(d) ORDER BY id) AS rnk
260+
| FROM nested_float_order
261+
|) WHERE rnk <= 1
262+
|""".stripMargin))
263+
}
264+
}
265+
}
266+
213267
for (orderColumn <- Seq("f", "d")) {
214268
test(s"window: scalar floating-point order key under strict floating point ($orderColumn)") {
215269
// Deliberately not gated on Spark 3.5, unlike the WindowGroupLimit tests above, so the

0 commit comments

Comments
 (0)