You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit f05fce0
Browse filesBrowse the repository at this point in the historyBrowse files
[SPARK-60178][SQL] Restore percentile fusion guard coverage and document GROUP BY change
Add decimal-based CombineApproximatePercentilesSuite tests that reach the
percentage-evaluation and evaluated-accuracy guards, which the float tests
no longer reach. Pin the GROUP BY analysis change in SQLQuerySuite and
record it in the SQL migration guide.
Copy file name to clipboardExpand all lines: docs/sql-migration-guide.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -46,6 +46,7 @@ license: |
46
46
- Since Spark 4.4, `spark.sql.sources.v2.bucketing.partition.filter.enabled` defaults to `true`. In a storage-partitioned join, the partition key groups pushed down to both sides may now be narrowed to those that can produce output for the join type, instead of always taking the union of the two sides' groups: an inner or semi join may keep only the groups present on both sides, a join that keeps or tests every left row (left outer, left anti, left single, existence) keeps the left side's groups, a right outer join keeps the right side's, and a full outer join is unaffected; a CROSS join carrying an equality condition is treated like an inner join. The narrowing is skipped when either side's partitioning may contain unknown partition keys, as after a shuffle on one side, so such a join keeps the full union. A group that is dropped is not scanned at all, so a query may read fewer files. To restore the previous behavior, set `spark.sql.sources.v2.bucketing.partition.filter.enabled` to `false`.
47
47
- Since Spark 4.4, `spark.sql.sources.v2.bucketing.partitionKeyOrdering.enabled` defaults to `true`. A V2 scan that reports a keyed partitioning but no explicit ordering now also reports itself sorted by its partition key expressions, because every row of such a partition evaluates them to the same value. Partition transforms such as `days(ts)` or `bucket(8, id)` are left out of that ordering. This lets Spark drop a `Sort` it would otherwise place above the scan. An aggregate that groups by a leading run of the other key expressions, in key order, may also be planned as a sort aggregate rather than a hash aggregate. This is because `spark.sql.execution.replaceHashWithSortAgg` keys off the reported ordering. To restore the previous behavior, set `spark.sql.sources.v2.bucketing.partitionKeyOrdering.enabled` to `false`.
48
48
- Since Spark 4.4, `spark.sql.sources.v2.bucketing.preserveKeyOrderingOnCoalesce.enabled` defaults to `true`. When `GroupPartitionsExec` merges several input partitions that share one partition key value into a single output partition, it now keeps sort orders over the partition key expressions in the ordering it reports, since those expressions are constant within the merged partition. Sort orders over partition transforms such as `days(ts)` or `bucket(8, id)` are left out. Orders over other columns are still dropped, as the concatenation invalidates them, and a join that reduced the partition keys onto a common key space (see `spark.sql.sources.v2.bucketing.allowCompatibleTransforms.enabled`) reports no order, since the merged partitions then share only the reduced key. This can remove a `Sort` downstream of the merge. To restore the previous behavior, set `spark.sql.sources.v2.bucketing.preserveKeyOrderingOnCoalesce.enabled` to `false`.
49
+
- Since Spark 4.4, chains of `FLOAT` or `DOUBLE` addition or multiplication that are associated differently are no longer treated as semantically equal, because floating-point addition and multiplication are not associative and such expressions can evaluate to different values. Reordering the two operands of a single operator, such as `a + b` and `b + a`, is still treated as equal. As a result, a query that selects a floating-point expression grouped by a differently associated form of it, for example `SELECT a + b + c FROM t GROUP BY c + b + a` on `DOUBLE` columns, now fails analysis with `MISSING_AGGREGATION`. Previously, the query passed analysis and returned the grouping expression's value, which could differ from the selected expression's value. To keep such a query working, write the selected expression the same way as the grouping expression.
Copy file name to clipboardExpand all lines: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/CombineApproximatePercentilesSuite.scala
+55-1Lines changed: 55 additions & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -27,7 +27,7 @@ import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LocalRelation, Lo
0 commit comments