Skip to content

Commit 4cf0b1c

Browse files
committed
fix: fall back for RANGE frames over ORDER BY keys DataFusion cannot compare
DataFusion finds a RANGE frame's CURRENT ROW bound by comparing ORDER BY values with ScalarValue::partial_cmp, which cannot order an array of arrays or structs, or a struct holding an array (apache/datafusion#24937), so the query fails with "Uncomparable values". Strict floating-point mode declined these keys while nested float keys were compared raw, and once null-free nested float keys became compatible they reached this failure in strict mode too. The same shapes fail with the default settings whatever their element type. CometWindowExec now falls back for a RANGE frame bounded by CURRENT ROW over such a key. Ranking functions, CUME_DIST, ROWS frames and an unbounded RANGE frame never compare values this way and stay native.
1 parent 6fa5836 commit 4cf0b1c

4 files changed

Lines changed: 119 additions & 1 deletion

File tree

‎docs/source/user-guide/latest/compatibility/operators.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,10 @@ incorrect result. When any single window expression in a `WindowExec` falls back
8888
overflow instead of returning Spark's `NULL`.
8989
- `RANGE` frame with an explicit offset when the `ORDER BY` column is `DATE` or `DECIMAL`
9090
([#4834](https://github.com/apache/datafusion-comet/issues/4834)).
91+
- `RANGE` frame bounded by `CURRENT ROW` when an `ORDER BY` key is an array of arrays or structs, or a struct
92+
holding an array, such as `array(named_struct('x', x))`. DataFusion cannot compare those values to find the
93+
frame's bounds ([apache/datafusion#24937](https://github.com/apache/datafusion/issues/24937)). Ranking functions
94+
and `ROWS` frames over the same keys run natively.
9195
- `first_value` / `last_value` on a `RANGE` frame with a literal offset
9296
([#4835](https://github.com/apache/datafusion-comet/issues/4835)).
9397
- `lag` / `lead` with a non-literal default value ([#4268](https://github.com/apache/datafusion-comet/issues/4268)).

‎spark/src/main/scala/org/apache/spark/sql/comet/CometWindowExec.scala‎

Lines changed: 37 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ import org.apache.spark.sql.execution.SparkPlan
2828
import org.apache.spark.sql.execution.metric.{SQLMetric, SQLMetrics}
2929
import org.apache.spark.sql.execution.window.WindowExec
3030
import org.apache.spark.sql.internal.SQLConf
31-
import org.apache.spark.sql.types.{DataType, DateType, Decimal, DecimalType, DoubleType, LongType, NumericType}
31+
import org.apache.spark.sql.types.{ArrayType, DataType, DateType, Decimal, DecimalType, DoubleType, LongType, MapType, NumericType, StructType}
3232

3333
import com.google.common.base.Objects
3434

@@ -433,6 +433,27 @@ object CometWindowExec extends CometOperatorSerde[WindowExec] {
433433
case _ =>
434434
}
435435

436+
// DataFusion finds a RANGE frame bound other than UNBOUNDED by comparing ORDER BY values,
437+
// and its comparison cannot order an array of arrays or structs, or a struct holding an
438+
// array, so execution fails with "Uncomparable values". Ranking functions and ROWS frames
439+
// never compare values that way and stay native over the same keys. That includes CUME_DIST,
440+
// whose RANGE frame DataFusion never reads.
441+
// https://github.com/apache/datafusion/issues/24937
442+
f match {
443+
case SpecifiedWindowFrame(RangeFrame, lb, ub)
444+
if (lb != UnboundedPreceding || ub != UnboundedFollowing) &&
445+
!windowExpr.windowFunction.isInstanceOf[CumeDist] =>
446+
windowExpr.windowSpec.orderSpec.map(_.dataType).find(!isRangeComparable(_)) match {
447+
case Some(dt) =>
448+
withFallbackReason(
449+
windowExpr,
450+
s"RANGE frame on ${dt.catalogString} ORDER BY is not supported")
451+
return None
452+
case None =>
453+
}
454+
case _ =>
455+
}
456+
436457
val (frameType, lowerBound, upperBound) = f match {
437458
case SpecifiedWindowFrame(frameType, lBound, uBound) =>
438459
val frameProto = frameType match {
@@ -601,6 +622,21 @@ object CometWindowExec extends CometOperatorSerde[WindowExec] {
601622
SerializedPlan(None))
602623
}
603624

625+
// Whether DataFusion can compare two ORDER BY values of `dataType` to find a RANGE frame
626+
// bound. `ScalarValue::partial_cmp` compares an array's elements, and a struct's fields with
627+
// nested structs flattened, using Arrow comparison kernels that reject nested values.
628+
private def isRangeComparable(dataType: DataType): Boolean = dataType match {
629+
case ArrayType(_: ArrayType | _: StructType | _: MapType, _) => false
630+
case StructType(fields) =>
631+
fields.forall { field =>
632+
field.dataType match {
633+
case _: ArrayType | _: MapType => false
634+
case dt => isRangeComparable(dt)
635+
}
636+
}
637+
case _ => true
638+
}
639+
604640
// Folds a RANGE frame bound expression to a constant and serializes its
605641
// magnitude as a typed Literal proto. Spark encodes PRECEDING/FOLLOWING via
606642
// the sign of the literal (negative => PRECEDING, positive => FOLLOWING),

‎spark/src/test/resources/sql-tests/windows/nested_float_order_keys_strict.sql‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,3 +101,25 @@ SELECT id FROM (
101101
DENSE_RANK() OVER (PARTITION BY g ORDER BY named_struct('x', coalesce(IF(s, -f, f), 0.0F)) DESC) AS r
102102
FROM nested_float_strict
103103
) WHERE r <= 1
104+
105+
-- DataFusion cannot compare an array of structs or arrays, or a struct holding an array, to find
106+
-- the CURRENT ROW bound of a RANGE frame (apache/datafusion#24937), so a running aggregate over
107+
-- such a key falls back even when the key cannot hold a null. window_functions.sql checks the
108+
-- same without floats.
109+
query expect_fallback(RANGE frame on array<struct<x:double>> ORDER BY is not supported)
110+
SELECT id, COUNT(id) OVER (ORDER BY array(named_struct('x', coalesce(d, 0.0D))), id) AS running
111+
FROM nested_float_strict
112+
113+
query expect_fallback(RANGE frame on struct<a:array<float>> ORDER BY is not supported)
114+
SELECT id,
115+
SUM(id) OVER (PARTITION BY g
116+
ORDER BY named_struct('a', array(coalesce(IF(s, -f, f), 0.0F))) DESC) AS by_group
117+
FROM nested_float_strict
118+
119+
-- Ranks and ROWS frames over the same keys stay native
120+
query
121+
SELECT id,
122+
RANK() OVER (PARTITION BY g ORDER BY array(named_struct('x', coalesce(IF(s, -d, d), 0.0D)))) AS r,
123+
SUM(id) OVER (PARTITION BY g ORDER BY array(named_struct('x', coalesce(IF(s, -d, d), 0.0D))), id
124+
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS running
125+
FROM nested_float_strict

‎spark/src/test/resources/sql-tests/windows/window_functions.sql‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -673,6 +673,62 @@ SELECT player, game, score,
673673
RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS range_sum
674674
FROM scores
675675

676+
-- ============================================================
677+
-- 5.6: RANGE frame over array and struct ORDER BY keys
678+
-- DataFusion finds a CURRENT ROW bound of a RANGE frame by comparing ORDER BY
679+
-- values. It compares an array's elements and a struct's fields, flattening
680+
-- nested structs, but its comparison rejects anything nested below that, so
681+
-- over an array of arrays or structs, or a struct holding an array, the query
682+
-- fails with "Uncomparable values"
683+
-- (https://github.com/apache/datafusion/issues/24937). Comet falls back for
684+
-- those keys. Ranking functions, ROWS frames and an unbounded RANGE frame
685+
-- never compare values that way and stay native over the same keys.
686+
-- ============================================================
687+
688+
query
689+
SELECT player, game, score,
690+
SUM(score) OVER (PARTITION BY player ORDER BY array(score)) AS by_array,
691+
SUM(score) OVER (PARTITION BY player
692+
ORDER BY named_struct('a', named_struct('b', score)) DESC) AS by_struct
693+
FROM scores
694+
695+
query expect_fallback(RANGE frame on array<struct<x:int>> ORDER BY is not supported)
696+
SELECT player, game, score,
697+
SUM(score) OVER (PARTITION BY player ORDER BY array(named_struct('x', score))) AS run_sum
698+
FROM scores
699+
700+
query expect_fallback(RANGE frame on struct<a:array<int>> ORDER BY is not supported)
701+
SELECT player, game, score,
702+
COUNT(score) OVER (PARTITION BY player ORDER BY named_struct('a', array(score)) DESC
703+
RANGE BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING) AS run_count
704+
FROM scores
705+
706+
query expect_fallback(RANGE frame on array<array<int>> ORDER BY is not supported)
707+
SELECT player, game, score,
708+
LAST_VALUE(game) OVER (PARTITION BY player ORDER BY array(array(score)), game) AS lv
709+
FROM scores
710+
711+
query
712+
SELECT player, game, score,
713+
RANK() OVER (PARTITION BY player ORDER BY array(named_struct('x', score))) AS rk,
714+
SUM(score) OVER (PARTITION BY player ORDER BY array(named_struct('x', score))
715+
RANGE BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING) AS total
716+
FROM scores
717+
718+
-- Spark gives CUME_DIST a RANGE frame, but DataFusion computes it from peer
719+
-- groups without reading the frame.
720+
query tolerance=1e-6
721+
SELECT player, game, score,
722+
PERCENT_RANK() OVER (PARTITION BY player ORDER BY named_struct('a', array(score))) AS pr,
723+
CUME_DIST() OVER (PARTITION BY player ORDER BY named_struct('a', array(score))) AS cd
724+
FROM scores
725+
726+
query
727+
SELECT player, game, score,
728+
SUM(score) OVER (PARTITION BY player ORDER BY named_struct('a', array(score)), game
729+
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS rows_sum
730+
FROM scores
731+
676732
-- ############################################################
677733
-- Section 6: Other window / aggregate functions
678734
-- ############################################################

0 commit comments

Comments
 (0)