Skip to content

Commit c94ab88

Browse files
joelrobin1818Isaac
andcommitted
Restore exact TRY rewrites and keep inexact non-ANSI decimal steps nested
ReorderAssociativeOperator again removes TRY steps that add 0 or multiply by 1, and folds the literals on the path from the top of a TRY chain through steps with one literal operand into one step above the sub-expression where that path ends, when the literals are not NULL, move away from zero, and fold to a value that fits the type. These rewrites cannot change which written step overflows, and master already applied them, so filters such as try_add(a, 0) > 5 can be pushed down again. Add and Multiply canonicalization now keeps a nested non-ANSI decimal step whose result type is bounded by the maximum precision as one operand, as it does for TRY steps, because such a step returns NULL on its own overflow or rounds. Differently grouped chains of such steps are no longer semantically equal. Also add plan tests for the BIGINT, SMALLINT and TINYINT fit checks and the NULL-literal guard, replace the TRY GROUP BY SQL tests, which already passed on master, with projections that select both groupings, add non-ANSI decimal SQL tests, reword the rule's scaladoc, and document the behavior changes in the SQL and Structured Streaming migration guides. Co-authored-by: Isaac <[email protected]>
1 parent fa6a662 commit c94ab88

16 files changed

Lines changed: 633 additions & 136 deletions

File tree

‎docs/sql-migration-guide.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,8 @@ license: |
2626

2727
- Since Spark 5.0, the Spark Thrift Server refuses to start when `hive.server2.enable.doAs` is `true` and `hive.server2.authentication` verifies user identities (any recognized type other than `NONE`/`NOSASL`). It impersonates the connecting user for Hive metastore calls and driver-side file system access, but executor-side data access still runs as the server's own service identity, so storage-level permissions are checked against the service principal rather than the connecting user (SPARK-5159). Spark 4.4 logs a warning for the same configuration. To restore the previous behavior, set `spark.sql.hive.thriftServer.allowIneffectiveDoAs` to `true`. Setting `hive.server2.enable.doAs` to `false` also starts the server, but note that it stops impersonating metastore calls as well, so it is not a no-op.
2828
- Since Spark 5.0, Spark requires Java 21 or later. Users who ran Spark 4.x on Java 17 can see two differences that come from Java. The datetime pattern letter `F` is the aligned week of month instead of the aligned day of week in month (JDK-8282081). For example, `date_format(date '2019-12-31', 'F')` returns `5` instead of `3`. Converting a `FLOAT` or `DOUBLE` value to a string gives the shortest decimal that represents the value (JDK-4511638). For example, `CAST(CAST(4567890123456789 AS FLOAT) AS STRING)` returns `4.56789E15` instead of `4.5678899E15`. A cast from `FLOAT` or `DOUBLE` to `DECIMAL` goes through this string, so its result can change as well. For example, `CAST(1.0751133388005187E17D AS DECIMAL(38, 0))` returns `107511333880051870` instead of `107511333880051872`.
29+
- Since Spark 5.0, `try_add` and `try_multiply` combined with regular `+` or `*` follow each operator's own overflow behavior, and a chain of `try_add` or `try_multiply` returns `NULL` exactly when one of its written steps overflows. Previously, the optimizer could regroup such an expression and evaluate its operations in the mode of the outermost one. For example, with `spark.sql.ansi.enabled` set to `true`, `try_add(x + y, 1)` fails with `ARITHMETIC_OVERFLOW` when `x + y` overflows instead of returning `NULL`, `try_add(x, y) + 1` returns `NULL` when `x + y` overflows instead of failing, and `try_add(try_add(x, -5), y)` returns `x - 5 + y` when only `x + y` overflows instead of `NULL`. The previous results were wrong, and only expressions over non-constant inputs were affected. To get `NULL` when an inner `+` or `*` overflows, use `try_add` or `try_multiply` for it as well, for example `try_add(try_add(x, y), 1)`.
30+
- Since Spark 5.0, when `spark.sql.ansi.enabled` is `false`, differently grouped chains of decimal `+` or `*` are no longer the same expression if one of their steps can overflow or round, because such a step returns `NULL` on its own overflow or rounds its own result. For example, for `DECIMAL(38, 0)` columns `a`, `b` and `c`, `SELECT (a + c) + b FROM t GROUP BY (a + b) + c` fails with `MISSING_AGGREGATION` instead of returning the value of `(a + b) + c`, which is `NULL` while `(a + c) + b` is not when `a + b` overflows. To keep such a query, write the chain in the select list as it is written in the `GROUP BY` clause.
2931

3032
## Upgrading from Spark SQL 4.3 to 4.4
3133

‎docs/streaming/ss-migration-guide.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,16 @@ Note that this migration guide describes the items specific to Structured Stream
2323
Many items of SQL migration can be applied when migrating Structured Streaming to higher versions.
2424
Please refer [Migration Guide: SQL, Datasets and DataFrame](../sql-migration-guide.html).
2525

26+
## Upgrading from Structured Streaming 4.4 to 5.0
27+
28+
- Since Spark 5.0, `try_add` and `try_multiply` combined with regular `+` or `*` follow each
29+
operator's own overflow behavior (see the SQL migration guide), so such an expression can be
30+
nullable where the optimizer previously rewrote it into a non-nullable one. For example,
31+
`try_add(coalesce(value, 0), 1) + 1` was rewritten to `coalesce(value, 0) + 2`. A stateful
32+
query started before Spark 4.3 that keys its state on such an expression fails the state
33+
schema compatibility check when it restarts. To recover, restart the query with a new
34+
checkpoint location.
35+
2636
## Upgrading from Structured Streaming 4.3 to 4.4
2737

2838
- Since Spark 4.4, streaming queries using `EXCEPT` with a streaming left input are rejected

‎sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala‎

Lines changed: 41 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -442,14 +442,33 @@ case class Add(
442442
// scalastyle:on
443443
override def resultDecimalType(p1: Int, s1: Int, p2: Int, s2: Int): DecimalType = {
444444
val resultScale = max(s1, s2)
445-
val resultPrecision = max(p1 - s1, p2 - s2) + resultScale + 1
445+
val resultPrecision = unboundedResultPrecision(p1, s1, p2, s2)
446446
if (allowPrecisionLoss) {
447447
DecimalType.adjustPrecisionScale(resultPrecision, resultScale)
448448
} else {
449449
DecimalType.bounded(resultPrecision, resultScale)
450450
}
451451
}
452452

453+
// The result precision before it is bounded by the maximum precision. A result type of this
454+
// precision holds every exact sum.
455+
private def unboundedResultPrecision(p1: Int, s1: Int, p2: Int, s2: Int): Int =
456+
max(p1 - s1, p2 - s2) + max(s1, s2) + 1
457+
458+
// Whether canonicalization can regroup this step with an enclosing Add of the same evaluation
459+
// context. A TRY step returns NULL exactly when its own operation overflows. So does a non-ANSI
460+
// decimal step whose result type is bounded by the maximum precision, which can also round under
461+
// precision loss. Regrouping such a step can change the result.
462+
private def canBeRegrouped: Boolean = evalMode match {
463+
case EvalMode.TRY => false
464+
case EvalMode.LEGACY if childrenResolved => (left.dataType, right.dataType) match {
465+
case (DecimalType.Fixed(p1, s1), DecimalType.Fixed(p2, s2)) =>
466+
unboundedResultPrecision(p1, s1, p2, s2) <= DecimalType.MAX_PRECISION
467+
case _ => true
468+
}
469+
case _ => true
470+
}
471+
453472
override def calendarIntervalMethod: String = if (failOnError) "addExact" else "add"
454473

455474
private lazy val numeric = TypeUtils.getNumeric(dataType, failOnError)
@@ -481,9 +500,8 @@ case class Add(
481500

482501
override lazy val canonicalized: Expression = {
483502
val reorderResult = buildCanonicalizedPlan(
484-
// TRY arithmetic is commutative but not associative: each step returns NULL only when its
485-
// own operation overflows. Keep the written nesting and order only the direct operands.
486-
{ case a @ Add(l, r, em) if em == evalContext && (evalMode != EvalMode.TRY || (a eq this)) =>
503+
// A nested step that cannot be regrouped stays one operand, canonicalized on its own.
504+
{ case a @ Add(l, r, em) if em == evalContext && ((a eq this) || a.canBeRegrouped) =>
487505
Seq(l, r) },
488506
{ case (l: Expression, r: Expression) => Add(l, r, evalContext)},
489507
Some(evalContext)
@@ -632,14 +650,29 @@ case class Multiply(
632650
// scalastyle:on
633651
override def resultDecimalType(p1: Int, s1: Int, p2: Int, s2: Int): DecimalType = {
634652
val resultScale = s1 + s2
635-
val resultPrecision = p1 + p2 + 1
653+
val resultPrecision = unboundedResultPrecision(p1, p2)
636654
if (allowPrecisionLoss) {
637655
DecimalType.adjustPrecisionScale(resultPrecision, resultScale)
638656
} else {
639657
DecimalType.bounded(resultPrecision, resultScale)
640658
}
641659
}
642660

661+
// The result precision before it is bounded by the maximum precision. A result type of this
662+
// precision holds every exact product.
663+
private def unboundedResultPrecision(p1: Int, p2: Int): Int = p1 + p2 + 1
664+
665+
// See Add.canBeRegrouped.
666+
private def canBeRegrouped: Boolean = evalMode match {
667+
case EvalMode.TRY => false
668+
case EvalMode.LEGACY if childrenResolved => (left.dataType, right.dataType) match {
669+
case (DecimalType.Fixed(p1, _), DecimalType.Fixed(p2, _)) =>
670+
unboundedResultPrecision(p1, p2) <= DecimalType.MAX_PRECISION
671+
case _ => true
672+
}
673+
case _ => true
674+
}
675+
643676
private lazy val numeric = TypeUtils.getNumeric(dataType, failOnError)
644677

645678
protected override def nullSafeEval(input1: Any, input2: Any): Any = dataType match {
@@ -665,9 +698,9 @@ case class Multiply(
665698

666699
override lazy val canonicalized: Expression = {
667700
buildCanonicalizedPlan(
668-
// See Add.canonicalized for why TRY chains keep their written nesting.
669-
{ case m @ Multiply(l, r, ec)
670-
if ec == evalContext && (evalMode != EvalMode.TRY || (m eq this)) => Seq(l, r) },
701+
// A nested step that cannot be regrouped stays one operand, canonicalized on its own.
702+
{ case m @ Multiply(l, r, ec) if ec == evalContext && ((m eq this) || m.canBeRegrouped) =>
703+
Seq(l, r) },
671704
{ case (l: Expression, r: Expression) => Multiply(l, r, evalContext) },
672705
Some(evalContext)
673706
)

‎sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala‎

Lines changed: 61 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717

1818
package org.apache.spark.sql.catalyst.optimizer
1919

20+
import scala.annotation.tailrec
2021
import scala.collection.immutable.HashSet
2122
import scala.collection.mutable.{ArrayBuffer, Stack}
2223
import scala.util.control.NonFatal
@@ -367,14 +368,14 @@ object ConstantPropagation extends Rule[LogicalPlan] with PredicateHelper {
367368
}
368369

369370
/**
370-
* Reorder associative integral-type operators and fold all constants into one.
371+
* Reorders the operands of integral [[Add]] and [[Multiply]] chains whose operators share an
372+
* evaluation mode and folds their literals into one, keeping an operator of another mode as one
373+
* operand. TRY-mode chains keep their written grouping apart from rewrites that cannot change the
374+
* result, see `rewriteTryChain`.
371375
*/
372376
object ReorderAssociativeOperator extends Rule[LogicalPlan] {
373377
// Flatten only operators evaluated in the same mode as the root: rebuilding a TRY-mode Add or
374378
// Multiply (try_add, try_multiply) in the enclosing mode would change its overflow behavior.
375-
// TRY arithmetic is not regrouped in general either, because each TRY step returns NULL only
376-
// when its own operation overflows, so regrouping or folding its literals can change which step
377-
// overflows. See foldTryChain for the one shape where folding is exact.
378379
private def flattenAdd(
379380
expression: Expression,
380381
groupSet: ExpressionSet,
@@ -405,32 +406,61 @@ object ReorderAssociativeOperator extends Rule[LogicalPlan] {
405406
}
406407

407408
/**
408-
* Folds the literals of a flattened TRY chain into one TRY operation only when that cannot change
409-
* the result: the chain has exactly one non-literal operand, `literalsMoveAwayFromZero` holds for
410-
* its literals, and the folded literal fits the result type. The partial results then move
411-
* monotonically from the operand to the final value, so a written step overflows exactly when the
412-
* folded one does.
409+
* Rewrites a TRY-mode chain only where that cannot change its result. A TRY step returns NULL
410+
* exactly when its own operation overflows, so regrouping the chain can change which step
411+
* overflows. Two rewrites are exact:
412+
* - A step whose literal operand is the `identity` never overflows, so it is replaced by its
413+
* other operand.
414+
* - The literals on the path from the top of the chain through steps with one literal and one
415+
* non-literal operand are folded into one step above the sub-expression X where the path ends,
416+
* for example a step with two non-literal operands. If these literals are not NULL, satisfy
417+
* `literalsMoveAwayFromZero`, and fold to a value that fits the result type, every written
418+
* partial result lies between X and the final result, so some written step overflows exactly
419+
* when the folded step does.
420+
* As in `flattenAdd`, steps that are grouping expressions are kept.
413421
*/
414-
private def foldTryChain(
415-
operands: Seq[Expression],
416-
dataType: DataType,
422+
private def rewriteTryChain(
423+
root: Expression,
424+
groupSet: ExpressionSet,
425+
tryStepOperands: PartialFunction[Expression, (Expression, Expression)],
426+
identity: Int,
417427
literalsMoveAwayFromZero: Seq[BigInt] => Boolean,
418428
combine: (BigInt, BigInt) => BigInt,
419-
build: (Expression, Expression) => Expression): Option[Expression] = {
420-
val (literals, others) = operands.partition(_.isInstanceOf[Literal])
421-
val values = literals.map(_.asInstanceOf[Literal].value)
422-
if (others.length != 1 || literals.length < 2 || values.contains(null)) return None
429+
build: (Expression, Expression) => Expression): Expression = {
430+
def operands(e: Expression): Option[(Expression, Expression)] =
431+
if (groupSet.contains(e)) None else tryStepOperands.lift(e)
432+
433+
@tailrec
434+
def removeIdentitySteps(e: Expression): Expression = operands(e) match {
435+
case Some((l, r)) if isSameInteger(l, identity) => removeIdentitySteps(r)
436+
case Some((l, r)) if isSameInteger(r, identity) => removeIdentitySteps(l)
437+
case _ => e
438+
}
439+
440+
@tailrec
441+
def splitLiteralPath(
442+
e: Expression,
443+
literals: List[Literal]): (Expression, List[Literal]) = operands(e) match {
444+
case Some((l: Literal, r)) if !r.isInstanceOf[Literal] => splitLiteralPath(r, l :: literals)
445+
case Some((l, r: Literal)) if !l.isInstanceOf[Literal] => splitLiteralPath(l, r :: literals)
446+
case _ => (e, literals)
447+
}
448+
449+
val chain = removeIdentitySteps(root)
450+
val (rest, literals) = splitLiteralPath(chain, Nil)
451+
val values = literals.map(_.value)
452+
if (literals.length < 2 || values.contains(null)) return chain
423453
val integers = values.map(v => BigInt(v.asInstanceOf[Number].longValue))
424-
if (!literalsMoveAwayFromZero(integers)) return None
454+
if (!literalsMoveAwayFromZero(integers)) return chain
425455
val folded = integers.reduce(combine)
426-
val foldedLiteral = dataType match {
456+
val foldedLiteral = root.dataType match {
427457
case ByteType if folded.isValidByte => Some(Literal(folded.toByte))
428458
case ShortType if folded.isValidShort => Some(Literal(folded.toShort))
429459
case IntegerType if folded.isValidInt => Some(Literal(folded.toInt))
430460
case LongType if folded.isValidLong => Some(Literal(folded.toLong))
431461
case _ => None
432462
}
433-
foldedLiteral.map(build(others.head, _))
463+
foldedLiteral.map(build(rest, _)).getOrElse(chain)
434464
}
435465

436466
def apply(plan: LogicalPlan): LogicalPlan = plan.transformWithPruning(
@@ -443,20 +473,24 @@ object ReorderAssociativeOperator extends Rule[LogicalPlan] {
443473
q.transformExpressionsDownWithPruning(_.containsPattern(BINARY_ARITHMETIC)) {
444474
case a @ Add(_, _, f) if f.evalMode == EvalMode.TRY && a.deterministic &&
445475
a.dataType.isInstanceOf[IntegralType] =>
446-
foldTryChain(
447-
flattenAdd(a, groupingExpressionSet, f.evalMode),
448-
a.dataType,
476+
rewriteTryChain(
477+
a,
478+
groupingExpressionSet,
479+
{ case Add(l, r, c) if c.evalMode == EvalMode.TRY => (l, r) },
480+
identity = 0,
449481
literals => literals.forall(_ >= 0) || literals.forall(_ <= 0),
450482
_ + _,
451-
(x, y) => Add(x, y, f)).getOrElse(a)
483+
(x, y) => Add(x, y, f))
452484
case m @ Multiply(_, _, f) if f.evalMode == EvalMode.TRY && m.deterministic &&
453485
m.dataType.isInstanceOf[IntegralType] =>
454-
foldTryChain(
455-
flattenMultiply(m, groupingExpressionSet, f.evalMode),
456-
m.dataType,
486+
rewriteTryChain(
487+
m,
488+
groupingExpressionSet,
489+
{ case Multiply(l, r, c) if c.evalMode == EvalMode.TRY => (l, r) },
490+
identity = 1,
457491
_.forall(_ >= 1),
458492
_ * _,
459-
(x, y) => Multiply(x, y, f)).getOrElse(m)
493+
(x, y) => Multiply(x, y, f))
460494
case a @ Add(_, _, f) if f.evalMode != EvalMode.TRY && a.deterministic &&
461495
a.dataType.isInstanceOf[IntegralType] =>
462496
val (literals, others) = flattenAdd(a, groupingExpressionSet, f.evalMode)

‎sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CanonicalizeSuite.scala‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -517,6 +517,48 @@ class CanonicalizeSuite extends SparkFunSuite {
517517
}
518518
}
519519

520+
test("Non-ANSI decimal Add and Multiply steps that can overflow keep their nesting") {
521+
def add(l: Expression, r: Expression): Expression = Add(l, r, EvalMode.LEGACY)
522+
def multiply(l: Expression, r: Expression): Expression = Multiply(l, r, EvalMode.LEGACY)
523+
val a = $"a".decimal(38, 0)
524+
val b = $"b".decimal(38, 0)
525+
val c = $"c".decimal(38, 0)
526+
527+
val default = SQLConf.get.getConf(MULTI_COMMUTATIVE_OP_OPT_THRESHOLD)
528+
Seq(2, 100).foreach { threshold =>
529+
SQLConf.get.setConfString(MULTI_COMMUTATIVE_OP_OPT_THRESHOLD.key, threshold.toString)
530+
try {
531+
// A non-ANSI decimal step whose result type is bounded by the maximum precision returns
532+
// NULL on its own overflow, so regrouping it changes which step overflows.
533+
assert(!add(add(a, b), c).semanticEquals(add(add(a, c), b)))
534+
assert(!multiply(multiply(a, b), c).semanticEquals(multiply(multiply(a, c), b)))
535+
536+
// Swapping the operands of each step is still allowed.
537+
assert(add(add(a, b), c).semanticEquals(add(c, add(b, a))))
538+
assert(multiply(multiply(a, b), c).semanticEquals(multiply(c, multiply(b, a))))
539+
540+
// Steps whose result type holds every exact result, up to the maximum precision, are
541+
// still regrouped.
542+
val p = $"p".decimal(37, 0)
543+
val q = $"q".decimal(37, 0)
544+
val r = $"r".decimal(37, 0)
545+
assert(add(add(p, q), r).semanticEquals(add(add(p, r), q)))
546+
val m = $"m".decimal(18, 0)
547+
val n = $"n".decimal(19, 0)
548+
val o = $"o".decimal(19, 0)
549+
assert(multiply(multiply(m, n), o).semanticEquals(multiply(multiply(m, o), n)))
550+
551+
// ANSI chains raise on overflow and are still treated as associative.
552+
assert(Add(Add(a, b, EvalMode.ANSI), c, EvalMode.ANSI).semanticEquals(
553+
Add(Add(a, c, EvalMode.ANSI), b, EvalMode.ANSI)))
554+
assert(Multiply(Multiply(a, b, EvalMode.ANSI), c, EvalMode.ANSI).semanticEquals(
555+
Multiply(Multiply(a, c, EvalMode.ANSI), b, EvalMode.ANSI)))
556+
} finally {
557+
SQLConf.get.setConfString(MULTI_COMMUTATIVE_OP_OPT_THRESHOLD.key, default.toString)
558+
}
559+
}
560+
}
561+
520562
test("unit test for gatherCommutative()") {
521563
val addExpression = Add(
522564
Literal(1),

0 commit comments

Comments
 (0)