Skip to content

Commit f05fce0

Browse files
committed
[SPARK-60178][SQL] Restore percentile fusion guard coverage and document GROUP BY change
Add decimal-based CombineApproximatePercentilesSuite tests that reach the percentage-evaluation and evaluated-accuracy guards, which the float tests no longer reach. Pin the GROUP BY analysis change in SQLQuerySuite and record it in the SQL migration guide.
1 parent 42346f7 commit f05fce0

3 files changed

Lines changed: 73 additions & 1 deletion

File tree

‎docs/sql-migration-guide.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ license: |
4646
- Since Spark 4.4, `spark.sql.sources.v2.bucketing.partition.filter.enabled` defaults to `true`. In a storage-partitioned join, the partition key groups pushed down to both sides may now be narrowed to those that can produce output for the join type, instead of always taking the union of the two sides' groups: an inner or semi join may keep only the groups present on both sides, a join that keeps or tests every left row (left outer, left anti, left single, existence) keeps the left side's groups, a right outer join keeps the right side's, and a full outer join is unaffected; a CROSS join carrying an equality condition is treated like an inner join. The narrowing is skipped when either side's partitioning may contain unknown partition keys, as after a shuffle on one side, so such a join keeps the full union. A group that is dropped is not scanned at all, so a query may read fewer files. To restore the previous behavior, set `spark.sql.sources.v2.bucketing.partition.filter.enabled` to `false`.
4747
- Since Spark 4.4, `spark.sql.sources.v2.bucketing.partitionKeyOrdering.enabled` defaults to `true`. A V2 scan that reports a keyed partitioning but no explicit ordering now also reports itself sorted by its partition key expressions, because every row of such a partition evaluates them to the same value. Partition transforms such as `days(ts)` or `bucket(8, id)` are left out of that ordering. This lets Spark drop a `Sort` it would otherwise place above the scan. An aggregate that groups by a leading run of the other key expressions, in key order, may also be planned as a sort aggregate rather than a hash aggregate. This is because `spark.sql.execution.replaceHashWithSortAgg` keys off the reported ordering. To restore the previous behavior, set `spark.sql.sources.v2.bucketing.partitionKeyOrdering.enabled` to `false`.
4848
- Since Spark 4.4, `spark.sql.sources.v2.bucketing.preserveKeyOrderingOnCoalesce.enabled` defaults to `true`. When `GroupPartitionsExec` merges several input partitions that share one partition key value into a single output partition, it now keeps sort orders over the partition key expressions in the ordering it reports, since those expressions are constant within the merged partition. Sort orders over partition transforms such as `days(ts)` or `bucket(8, id)` are left out. Orders over other columns are still dropped, as the concatenation invalidates them, and a join that reduced the partition keys onto a common key space (see `spark.sql.sources.v2.bucketing.allowCompatibleTransforms.enabled`) reports no order, since the merged partitions then share only the reduced key. This can remove a `Sort` downstream of the merge. To restore the previous behavior, set `spark.sql.sources.v2.bucketing.preserveKeyOrderingOnCoalesce.enabled` to `false`.
49+
- Since Spark 4.4, chains of `FLOAT` or `DOUBLE` addition or multiplication that are associated differently are no longer treated as semantically equal, because floating-point addition and multiplication are not associative and such expressions can evaluate to different values. Reordering the two operands of a single operator, such as `a + b` and `b + a`, is still treated as equal. As a result, a query that selects a floating-point expression grouped by a differently associated form of it, for example `SELECT a + b + c FROM t GROUP BY c + b + a` on `DOUBLE` columns, now fails analysis with `MISSING_AGGREGATION`. Previously, the query passed analysis and returned the grouping expression's value, which could differ from the selected expression's value. To keep such a query working, write the selected expression the same way as the grouping expression.
4950

5051
## Upgrading from Spark SQL 4.2 to 4.3
5152

‎sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/CombineApproximatePercentilesSuite.scala‎

Lines changed: 55 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LocalRelation, Lo
2727
import org.apache.spark.sql.catalyst.rules.RuleExecutor
2828
import org.apache.spark.sql.catalyst.util.ArrayData
2929
import org.apache.spark.sql.internal.SQLConf
30-
import org.apache.spark.sql.types.IntegerType
30+
import org.apache.spark.sql.types.{Decimal, DecimalType, DoubleType, IntegerType}
3131

3232
class CombineApproximatePercentilesSuite extends PlanTest {
3333

@@ -80,6 +80,11 @@ class CombineApproximatePercentilesSuite extends PlanTest {
8080
assert(!aggregate.aggregateExpressions.exists(_.exists(_.isInstanceOf[GetArrayItem])))
8181
}
8282

83+
private def decimalLiteral(value: String, scale: Int): Literal = {
84+
Literal(Decimal(BigDecimal(value), DecimalType.MAX_PRECISION, scale),
85+
DecimalType(DecimalType.MAX_PRECISION, scale))
86+
}
87+
8388
test("do not combine approximate percentiles when disabled") {
8489
assert(!new SQLConf().getConf(SQLConf.COMBINE_APPROXIMATE_PERCENTILES_ENABLED))
8590
val original = Aggregate(
@@ -297,6 +302,55 @@ class CombineApproximatePercentilesSuite extends PlanTest {
297302
assert(ordinals(arrayResult) == Seq(None, Some(0), Some(1)))
298303
}
299304

305+
// Decimal Add is still re-associated by canonicalization, but rounds intermediate results when
306+
// precision loss is allowed, so differently associated sums can be canonically equal while
307+
// evaluating differently.
308+
test("do not fuse canonically equal decimal-derived accuracies that evaluate differently") {
309+
val a = decimalLiteral("1.9999994", 7)
310+
val b = decimalLiteral("0.0000001", 7)
311+
val c = decimalLiteral("1.0000000", 7)
312+
val firstAccuracy = ((a + b) + c).cast(IntegerType)
313+
val secondAccuracy = (a + (b + c)).cast(IntegerType)
314+
315+
def percentileWithAccuracy(
316+
percentage: Double,
317+
accuracy: Expression): AggregateExpression = {
318+
new ApproximatePercentile(value, Literal(percentage), accuracy).toAggregateExpression()
319+
}
320+
321+
val result = optimizedAggregate(
322+
Alias(percentileWithAccuracy(0.5, firstAccuracy), "first_p50")(),
323+
Alias(percentileWithAccuracy(0.5, secondAccuracy), "second_p50")(),
324+
Alias(percentileWithAccuracy(0.9, secondAccuracy), "second_p90")(),
325+
Alias(percentileWithAccuracy(0.9, firstAccuracy), "first_p90")())
326+
327+
assert(firstAccuracy.eval() == 3)
328+
assert(secondAccuracy.eval() == 2)
329+
assert(firstAccuracy.canonicalized == secondAccuracy.canonicalized)
330+
assertNotCombined(result)
331+
}
332+
333+
test("do not fuse canonically equal decimal-derived percentages that evaluate differently") {
334+
val a = decimalLiteral("0.0000000004", 10)
335+
val b = decimalLiteral("0.0000000001", 10)
336+
val c = decimalLiteral("0.0000000040", 10)
337+
val firstPercentage = ((a + b) + c).cast(DoubleType)
338+
val secondPercentage = (a + (b + c)).cast(DoubleType)
339+
def percentileWithPercentage(percentage: Expression): AggregateExpression = {
340+
new ApproximatePercentile(value, percentage, Literal(10000)).toAggregateExpression()
341+
}
342+
343+
val result = optimizedAggregate(
344+
Alias(percentileWithPercentage(firstPercentage), "first")(),
345+
Alias(percentileWithPercentage(secondPercentage), "second")(),
346+
Alias(percentile(value, 0.9), "p90")())
347+
348+
assert(firstPercentage.eval() == 1.0e-8)
349+
assert(secondPercentage.eval() == 0.0d)
350+
assert(firstPercentage.canonicalized == secondPercentage.canonicalized)
351+
assertNotCombined(result)
352+
}
353+
300354
test("preserve existing arrays while combining compatible scalars") {
301355
val arrayPercentile = new ApproximatePercentile(
302356
value,

‎sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3408,6 +3408,23 @@ class SQLQuerySuite extends SharedSparkSession with AdaptiveSparkPlanHelper
34083408
}
34093409
}
34103410

3411+
test("SPARK-60178: GROUP BY does not match re-associated floating-point arithmetic") {
3412+
withTempView("t") {
3413+
sql("SELECT CAST(id AS DOUBLE) AS a, CAST(id AS DOUBLE) AS b, CAST(id AS DOUBLE) AS c, " +
3414+
"id AS i, id AS j, id AS k FROM range(0, 2)").createOrReplaceTempView("t")
3415+
3416+
val ex = intercept[AnalysisException] {
3417+
sql("SELECT a + b + c FROM t GROUP BY c + b + a").collect()
3418+
}
3419+
assert(ex.getCondition === "MISSING_AGGREGATION")
3420+
3421+
// Commuting the operands of a single operator is still semantically equal.
3422+
checkAnswer(sql("SELECT a + b FROM t GROUP BY b + a"), Seq(Row(0.0), Row(2.0)))
3423+
// Integral arithmetic keeps re-associating.
3424+
checkAnswer(sql("SELECT i + j + k FROM t GROUP BY k + j + i"), Seq(Row(0L), Row(3L)))
3425+
}
3426+
}
3427+
34113428
test("SPARK-29213: FilterExec should not throw NPE") {
34123429
// Under ANSI mode, casting string '' as numeric will cause runtime error
34133430
if (!conf.ansiEnabled) {

0 commit comments

Comments
 (0)