Skip to content

Commit 2c37b8a

Browse files
committed
test: sort a real null element in the strict nested float sort test
The IF key in the strict nested sort test never produced a null, because id runs from 0 to 4. It now nulls row 4's element, so the test checks that a null element sorts first in strict mode, ahead of the signed zeros and NaN payloads, with that row's bits check skipped. Also update two comments that still described the strict-only decline: nested_float_order_keys.sql and CometWindowGroupLimitExec now say that strict mode runs these keys natively, and that CometSortOrder declines ASC NULLS LAST and DESC NULLS FIRST on them in every mode.
1 parent 4335062 commit 2c37b8a

3 files changed

Lines changed: 24 additions & 16 deletions

File tree

‎spark/src/main/scala/org/apache/spark/sql/comet/CometWindowGroupLimitExec.scala‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -102,8 +102,8 @@ object CometWindowGroupLimitExec extends CometOperatorSerde[SparkPlan] {
102102
// The same byte equality decides RANK and DENSE_RANK ties, but it runs on keys that the native
103103
// planner has normalized: a `FLOAT` or `DOUBLE` order key, scalar or nested in an array or
104104
// struct, has its NaNs folded together and its signed zeros tied first, as Spark does, so no
105-
// float key needs a guard here. Strict floating-point mode declines the nested keys that can
106-
// hold a null element or field, through `CometSortOrder`.
105+
// float key needs a guard here. `CometSortOrder` declines `ASC NULLS LAST` and
106+
// `DESC NULLS FIRST` on a nested key that can hold a null element or field, in every mode.
107107

108108
val childOutput = op.children.head.output
109109
val partitionProtos = fields.partitionSpec.map(e => e -> exprToProto(e, childOutput))

‎spark/src/test/resources/sql-tests/windows/nested_float_order_keys.sql‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,8 +24,8 @@
2424
-- total order sorts below -Infinity. Each ORDER BY ends with a unique tiebreaker, so peers come
2525
-- out in the tiebreaker's order.
2626

27-
-- Strict floating-point mode declines these keys, because their types can hold a null element or
28-
-- field: see nested_float_order_keys_strict.sql.
27+
-- Strict floating-point mode runs these keys natively too. nested_float_order_keys_strict.sql covers
28+
-- that, along with the null orders and RANGE frames that fall back in every mode.
2929

3030
statement
3131
CREATE TABLE nested_float_keys(id INT, g INT, d DOUBLE, f FLOAT, s BOOLEAN) USING parquet

‎spark/src/test/scala/org/apache/comet/CometExpressionSuite.scala‎

Lines changed: 20 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -294,9 +294,9 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper {
294294
test("strict floating point: nested sort keeps NaN payloads and zero signs unchanged") {
295295
// As in the scalar test, a local relation keeps the raw bits that Parquet would canonicalize.
296296
// `d` is a primitive `Double`, so it is not nullable, and neither are the element of
297-
// `array(d)` and the field of `named_struct('v', d)`. The `IF` key's element can be null, and
298-
// strict mode admits it too, because its default null order places a null element where
299-
// Spark does.
297+
// `array(d)` and the field of `named_struct('v', d)`. The `IF` key's element is null on row
298+
// 4, and strict mode admits it too, because its default null order sorts a null element first,
299+
// where Spark does.
300300
val negNan = java.lang.Double.longBitsToDouble(0xfff8000000000002L)
301301
val posNan = java.lang.Double.longBitsToDouble(0x7ff8000000000002L)
302302
val rows = Seq((0, negNan), (1, posNan), (2, -0.0d), (3, 0.0d), (4, 1.0d))
@@ -309,10 +309,14 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper {
309309
withSQLConf(
310310
CometConf.getExprAllowIncompatConfigKey("SortOrder") -> "false",
311311
CometConf.COMET_EXEC_STRICT_FLOATING_POINT.key -> "true") {
312-
for ((key, value) <- Seq[(String, Row => Double)](
313-
"array(d)" -> (_.getSeq[Double](1).head),
314-
"named_struct('v', d)" -> (_.getStruct(1).getDouble(0)),
315-
"array(IF(id < 0, CAST(NULL AS DOUBLE), d))" -> (_.getSeq[Double](1).head))) {
312+
// The zeros are peers, and so are the two NaNs, which sort last.
313+
for ((key, value, order) <- Seq[(String, Row => Option[Double], Seq[Int])](
314+
("array(d)", r => Some(r.getSeq[Double](1).head), Seq(2, 3, 4, 0, 1)),
315+
("named_struct('v', d)", r => Some(r.getStruct(1).getDouble(0)), Seq(2, 3, 4, 0, 1)),
316+
(
317+
"array(IF(id = 4, CAST(NULL AS DOUBLE), d))",
318+
r => Option(r.getSeq[java.lang.Double](1).head).map(_.doubleValue),
319+
Seq(4, 2, 3, 0, 1)))) {
316320
val query = s"SELECT id, $key AS k FROM strict_fp_nested_bits ORDER BY k, id"
317321
checkSparkAnswerAndOperator(
318322
sql(query),
@@ -321,12 +325,16 @@ class CometExpressionSuite extends CometTestBase with AdaptiveSparkPlanHelper {
321325

322326
val actual = sql(query).collect().toSeq
323327
actual.foreach { row =>
324-
assert(
325-
java.lang.Double.doubleToRawLongBits(value(row)) == expected(row.getInt(0)),
326-
s"row ${row.getInt(0)} had its floating-point bits rewritten by the sort on $key")
328+
val id = row.getInt(0)
329+
value(row) match {
330+
case Some(v) =>
331+
assert(
332+
java.lang.Double.doubleToRawLongBits(v) == expected(id),
333+
s"row $id had its floating-point bits rewritten by the sort on $key")
334+
case None => assert(id == 4, s"row $id lost its value in the sort on $key")
335+
}
327336
}
328-
// The zeros are peers, and so are the two NaNs, which sort last.
329-
assert(actual.map(_.getInt(0)) == Seq(2, 3, 4, 0, 1), s"sort on $key")
337+
assert(actual.map(_.getInt(0)) == order, s"sort on $key")
330338
}
331339
}
332340
}

0 commit comments

Comments
 (0)