You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit c94ab88
Browse filesBrowse the repository at this point in the historyBrowse files
ReorderAssociativeOperator again removes TRY steps that add 0 or multiply
by 1, and folds the literals on the path from the top of a TRY chain
through steps with one literal operand into one step above the
sub-expression where that path ends, when the literals are not NULL, move
away from zero, and fold to a value that fits the type. These rewrites
cannot change which written step overflows, and master already applied
them, so filters such as try_add(a, 0) > 5 can be pushed down again.
Add and Multiply canonicalization now keeps a nested non-ANSI decimal step
whose result type is bounded by the maximum precision as one operand, as
it does for TRY steps, because such a step returns NULL on its own
overflow or rounds. Differently grouped chains of such steps are no longer
semantically equal.
Also add plan tests for the BIGINT, SMALLINT and TINYINT fit checks and
the NULL-literal guard, replace the TRY GROUP BY SQL tests, which already
passed on master, with projections that select both groupings, add
non-ANSI decimal SQL tests, reword the rule's scaladoc, and document the
behavior changes in the SQL and Structured Streaming migration guides.
Co-authored-by: Isaac <[email protected]>
Copy file name to clipboardExpand all lines: docs/sql-migration-guide.md
+2Lines changed: 2 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -26,6 +26,8 @@ license: |
26
26
27
27
- Since Spark 5.0, the Spark Thrift Server refuses to start when `hive.server2.enable.doAs` is `true` and `hive.server2.authentication` verifies user identities (any recognized type other than `NONE`/`NOSASL`). It impersonates the connecting user for Hive metastore calls and driver-side file system access, but executor-side data access still runs as the server's own service identity, so storage-level permissions are checked against the service principal rather than the connecting user (SPARK-5159). Spark 4.4 logs a warning for the same configuration. To restore the previous behavior, set `spark.sql.hive.thriftServer.allowIneffectiveDoAs` to `true`. Setting `hive.server2.enable.doAs` to `false` also starts the server, but note that it stops impersonating metastore calls as well, so it is not a no-op.
28
28
- Since Spark 5.0, Spark requires Java 21 or later. Users who ran Spark 4.x on Java 17 can see two differences that come from Java. The datetime pattern letter `F` is the aligned week of month instead of the aligned day of week in month (JDK-8282081). For example, `date_format(date '2019-12-31', 'F')` returns `5` instead of `3`. Converting a `FLOAT` or `DOUBLE` value to a string gives the shortest decimal that represents the value (JDK-4511638). For example, `CAST(CAST(4567890123456789 AS FLOAT) AS STRING)` returns `4.56789E15` instead of `4.5678899E15`. A cast from `FLOAT` or `DOUBLE` to `DECIMAL` goes through this string, so its result can change as well. For example, `CAST(1.0751133388005187E17D AS DECIMAL(38, 0))` returns `107511333880051870` instead of `107511333880051872`.
29
+
- Since Spark 5.0, `try_add` and `try_multiply` combined with regular `+` or `*` follow each operator's own overflow behavior, and a chain of `try_add` or `try_multiply` returns `NULL` exactly when one of its written steps overflows. Previously, the optimizer could regroup such an expression and evaluate its operations in the mode of the outermost one. For example, with `spark.sql.ansi.enabled` set to `true`, `try_add(x + y, 1)` fails with `ARITHMETIC_OVERFLOW` when `x + y` overflows instead of returning `NULL`, `try_add(x, y) + 1` returns `NULL` when `x + y` overflows instead of failing, and `try_add(try_add(x, -5), y)` returns `x - 5 + y` when only `x + y` overflows instead of `NULL`. The previous results were wrong, and only expressions over non-constant inputs were affected. To get `NULL` when an inner `+` or `*` overflows, use `try_add` or `try_multiply` for it as well, for example `try_add(try_add(x, y), 1)`.
30
+
- Since Spark 5.0, when `spark.sql.ansi.enabled` is `false`, differently grouped chains of decimal `+` or `*` are no longer the same expression if one of their steps can overflow or round, because such a step returns `NULL` on its own overflow or rounds its own result. For example, for `DECIMAL(38, 0)` columns `a`, `b` and `c`, `SELECT (a + c) + b FROM t GROUP BY (a + b) + c` fails with `MISSING_AGGREGATION` instead of returning the value of `(a + b) + c`, which is `NULL` while `(a + c) + b` is not when `a + b` overflows. To keep such a query, write the chain in the select list as it is written in the `GROUP BY` clause.
0 commit comments