Skip to content

Commit 45b179f

Browse files
committed
fix: fall back for DECIMAL RANGE offsets that can overflow Decimal128
1 parent cc5ae3a commit 45b179f

4 files changed

Lines changed: 124 additions & 21 deletions

File tree

‎docs/source/user-guide/latest/compatibility/operators.md‎

Lines changed: 5 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -150,6 +150,11 @@ incorrect result. When any single window expression in a `WindowExec` falls back
150150
([#7007](https://github.com/apache/datafusion-comet/issues/7007)). From precision 28 the native average also
151151
differs from Spark once the running sum overflows precision 38
152152
([#5418](https://github.com/apache/datafusion-comet/issues/5418)).
153+
- `RANGE` frame with an offset when the `ORDER BY` column is `DECIMAL` and the bound can overflow `Decimal128`, that
154+
is, when the largest value of the `ORDER BY` type plus the offset exceeds 2^127 - 1 unscaled. Only `DECIMAL(38, s)`
155+
can reach that, and only with an offset above about 7.01 * 10^37 unscaled. Spark computes the bound exactly, while
156+
the native frame treats an overflowing bound as the edge of the partition and so takes in `NULL` keys sorted
157+
there ([#5022](https://github.com/apache/datafusion-comet/issues/5022)).
153158
- `RANGE` frame with an offset above 2144550751 days when the `ORDER BY` column is `DATE`. Spark adds the offset
154159
to the date as an `Int` that wraps on overflow, while the native frame treats an overflowing bound as the edge of
155160
the partition. Smaller offsets cannot wrap for any date between 0001-01-01 and 9999-12-31
@@ -169,13 +174,6 @@ incorrect result. When any single window expression in a `WindowExec` falls back
169174
- A `ROWS` offset that is not an integer or long, or a `RANGE` offset that is not numeric.
170175
- Any `PARTITION BY` or `ORDER BY` expression that Comet cannot serialize.
171176

172-
**Known differences (runs natively but may diverge from Spark on edge cases):**
173-
174-
- `RANGE` frame boundary arithmetic on `DECIMAL` near max precision: when `current +/- offset` overflows the
175-
underlying integer, DataFusion collapses the bound to the partition edge, whereas Spark evaluates the boundary
176-
through normal decimal arithmetic (returning `NULL` under non-ANSI, throwing under ANSI). The two behaviors
177-
agree for values away from `Decimal128` limits.
178-
179177
A `RANGE` frame with an explicit offset on a `DATE` `ORDER BY` runs natively, but over a date outside Spark's
180178
supported range of 0001-01-01 to 9999-12-31, such as one produced by `date_add` with a very large day count, the
181179
native frame can treat the bound as the edge of the partition where Spark compares against it

‎spark/src/main/scala/org/apache/spark/sql/comet/CometWindowExec.scala‎

Lines changed: 47 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -393,22 +393,31 @@ object CometWindowExec extends CometOperatorSerde[WindowExec] {
393393
}
394394
}
395395

396-
// The RANGE frame offset type follows Spark's WindowFrameTypeCoercion:
397-
// DECIMAL ORDER BY casts the offset to the same Decimal type, so it runs
398-
// natively. DATE is handled natively by wrapping the offset as
399-
// IntervalDayTime in the planner. UNBOUNDED / CURRENT ROW bounds skip the
400-
// arithmetic and stay native.
396+
// DataFusion computes a RANGE frame bound as `current +/- offset` and treats
397+
// a bound that overflows as the edge of the partition, which also takes in
398+
// NULL keys sorted at that edge. Spark's bound never overflows that way, so
399+
// fall back for the offsets that can overflow. UNBOUNDED / CURRENT ROW
400+
// bounds skip the arithmetic and stay native.
401401
//
402402
// For DATE, Spark computes the bound with DateAdd, an Int addition that
403-
// wraps silently on overflow, while DataFusion treats an overflowing bound
404-
// as the partition edge. Over dates in Spark's supported range the two only
405-
// disagree when the offset is large enough to wrap, so fall back for those
406-
// offsets. https://github.com/apache/datafusion-comet/issues/5022
403+
// wraps silently on overflow. Over dates in Spark's supported range the two
404+
// only disagree when the offset is large enough to wrap, so fall back for
405+
// those offsets.
406+
//
407+
// For DECIMAL, Spark's DecimalAddNoOverflowCheck keeps the exact bound,
408+
// while DataFusion adds in Decimal128 and overflows past i128.
409+
// https://github.com/apache/datafusion-comet/issues/5022
407410
f match {
408411
case SpecifiedWindowFrame(RangeFrame, lb, ub)
409412
if !lb.isInstanceOf[SpecialFrameBoundary] ||
410413
!ub.isInstanceOf[SpecialFrameBoundary] =>
411414
windowExpr.windowSpec.orderSpec.headOption.map(_.dataType) match {
415+
case Some(dt: DecimalType) if !Seq(lb, ub).forall(isDecimalRangeOffsetSafe(_, dt)) =>
416+
withFallbackReason(
417+
windowExpr,
418+
s"RANGE frame on ${dt.catalogString} ORDER BY with an offset whose bound can " +
419+
"overflow Decimal128 is not supported")
420+
return None
412421
case Some(DateType) if !Seq(lb, ub).forall(isDateRangeOffsetSafe) =>
413422
withFallbackReason(
414423
windowExpr,
@@ -691,6 +700,35 @@ object CometWindowExec extends CometOperatorSerde[WindowExec] {
691700
}
692701
}
693702

703+
private val MaxDecimal128Unscaled: BigInt = BigInt(2).pow(127) - 1
704+
705+
// DataFusion rescales both operands to the larger scale and adds them as
706+
// i128, so the bound cannot overflow when the largest value the ORDER BY
707+
// type holds plus the offset fits in an i128.
708+
private def isDecimalRangeOffsetSafe(bound: Expression, orderType: DecimalType): Boolean =
709+
bound match {
710+
case _: SpecialFrameBoundary => true
711+
case e =>
712+
val offset =
713+
try {
714+
e.eval()
715+
} catch {
716+
// Rejected when the bound is serialized.
717+
case _: Exception => return true
718+
}
719+
offset match {
720+
case d: Decimal if d.scale >= 0 && orderType.scale >= 0 =>
721+
val scale = math.max(orderType.scale, d.scale)
722+
val maxValue = (BigInt(10).pow(orderType.precision) - 1) *
723+
BigInt(10).pow(scale - orderType.scale)
724+
val offsetValue =
725+
BigInt(d.toJavaBigDecimal.unscaledValue).abs * BigInt(10).pow(scale - d.scale)
726+
maxValue + offsetValue <= MaxDecimal128Unscaled
727+
case _: Decimal => false
728+
case _ => true
729+
}
730+
}
731+
694732
// Folds a RANGE frame bound expression to a constant and serializes its
695733
// magnitude as a typed Literal proto. Spark encodes PRECEDING/FOLLOWING via
696734
// the sign of the literal (negative => PRECEDING, positive => FOLLOWING),

‎spark/src/test/resources/sql-tests/windows/window_functions.sql‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -251,8 +251,8 @@ ORDER BY cate, val_date
251251
-- Runs natively: Spark's WindowFrameTypeCoercion casts the frame offset to
252252
-- the same Decimal type as the ORDER BY column, so the boundary arithmetic
253253
-- keeps the current row's precision and DataFusion's comparator has matching
254-
-- operands. Contrast with the DATE case in 1.11, which still falls back
255-
-- because the offset stays IntegerType and arrow-arith rejects Date32 + Int32.
254+
-- operands. The DATE case in 1.11 keeps an IntegerType offset instead, which
255+
-- the native planner coerces to IntervalDayTime.
256256
-- ============================================================
257257

258258
query

‎spark/src/test/scala/org/apache/comet/exec/CometWindowExecSuite.scala‎

Lines changed: 70 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1860,9 +1860,9 @@ class CometWindowExecSuite extends CometTestBase {
18601860
// offset -> Decimal(11,0) vs current Decimal(10,0), "Uncomparable values").
18611861
// The upstream fix (apache/datafusion#22174, closes #22113) makes
18621862
// ScalarValue::partial_cmp ignore precision for Decimal128 when scales match,
1863-
// and is included in DataFusion 54.0.0 which Comet already pins. These tests
1864-
// pin the behavior with `checkSparkAnswerAndOperator` so results must match
1865-
// Spark AND run natively.
1863+
// and was first released in DataFusion 54.0.0. These tests pin the behavior
1864+
// with `checkSparkAnswerAndOperator` so results must match Spark AND run
1865+
// natively.
18661866
// ---------------------------------------------------------------------------
18671867

18681868
private def writeDecimalWindowFixture(dir: java.io.File, decimalType: String): Unit = {
@@ -2107,4 +2107,71 @@ class CometWindowExecSuite extends CometTestBase {
21072107
checkSparkAnswerAndOperator(df)
21082108
}
21092109
}
2110+
2111+
private def writeDecimalOverflowFixture(dir: java.io.File, values: Seq[String]): Unit = {
2112+
values
2113+
.map(Option(_))
2114+
.:+(Option.empty[String])
2115+
.toDF("d_str")
2116+
.selectExpr("CAST(d_str AS DECIMAL(38, 0)) AS d")
2117+
.repartition(1)
2118+
.write
2119+
.mode("overwrite")
2120+
.parquet(dir.toString)
2121+
}
2122+
2123+
test("window: RANGE offset on DECIMAL(38, 0) ORDER BY that overflows Decimal128 falls back") {
2124+
// Spark's DecimalAddNoOverflowCheck keeps the exact bound, so the bound 2N of the row N stays
2125+
// below the trailing NULL. DataFusion's i128 bound overflows and becomes the partition edge,
2126+
// which takes the NULL in, so this falls back.
2127+
// https://github.com/apache/datafusion-comet/issues/5022
2128+
val n = "90000000000000000000000000000000000000"
2129+
withTempDir { dir =>
2130+
writeDecimalOverflowFixture(dir, Seq("0", n))
2131+
spark.read.parquet(dir.toString).createOrReplaceTempView("dec_overflow")
2132+
for (ansi <- Seq("true", "false")) {
2133+
withSQLConf(SQLConf.ANSI_ENABLED.key -> ansi) {
2134+
val df = sql(s"""
2135+
SELECT d, COUNT(*) OVER (ORDER BY d ASC NULLS LAST
2136+
RANGE BETWEEN $n PRECEDING AND $n FOLLOWING) AS cnt
2137+
FROM dec_overflow
2138+
""")
2139+
val (_, cometPlan) = checkSparkAnswerAndFallbackReason(
2140+
df,
2141+
"RANGE frame on decimal(38,0) ORDER BY with an offset whose bound can overflow " +
2142+
"Decimal128 is not supported")
2143+
assert(collect(cometPlan) { case w: CometWindowExec => w }.isEmpty, cometPlan)
2144+
checkAnswer(df, Seq(Row(BigDecimal(0), 2L), Row(BigDecimal(n), 2L), Row(null, 1L)))
2145+
}
2146+
}
2147+
}
2148+
}
2149+
2150+
test(
2151+
"window: largest RANGE offset on DECIMAL(38, 0) ORDER BY that cannot overflow runs natively") {
2152+
// The limit is 2^127 - 1 minus the largest DECIMAL(38, 0), that is
2153+
// 70141183460469231731687303715884105728, so `current +/- offset` stays in an i128 for every
2154+
// value of the type, including the type's extremes next to NULLs at either edge. Spark negates
2155+
// a PRECEDING offset with 34 significant digits, which would round the limit itself past it,
2156+
// so use the largest 34-digit offset under it.
2157+
val maxValue = "9" * 38
2158+
val offset = "70141183460469231731687303715884100000"
2159+
withTempDir { dir =>
2160+
writeDecimalOverflowFixture(dir, Seq(s"-$maxValue", "0", maxValue))
2161+
spark.read.parquet(dir.toString).createOrReplaceTempView("dec_edge")
2162+
for {
2163+
ansi <- Seq("true", "false")
2164+
order <- Seq("ASC NULLS LAST", "ASC NULLS FIRST", "DESC NULLS LAST", "DESC NULLS FIRST")
2165+
} {
2166+
withSQLConf(SQLConf.ANSI_ENABLED.key -> ansi) {
2167+
val df = sql(s"""
2168+
SELECT d, COUNT(*) OVER (ORDER BY d $order
2169+
RANGE BETWEEN $offset PRECEDING AND $offset FOLLOWING) AS cnt
2170+
FROM dec_edge
2171+
""")
2172+
checkSparkAnswerAndOperator(df)
2173+
}
2174+
}
2175+
}
2176+
}
21102177
}

0 commit comments

Comments
 (0)