You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit f918b8e
Browse filesBrowse the repository at this point in the historyBrowse files
[SPARK-59118][SQL] Target the doAs warning at Spark 4.4
This change missed 4.3.0: neither 4.3.0-rc2 nor 4.3.0-rc3 includes it, and
branch-4.x is the 4.4.0 line. Version
spark.sql.hive.thriftServer.allowIneffectiveDoAs 4.4.0 and move the migration
note to the 4.3 to 4.4 section, matching the master guide's "Spark 4.4 logs a
warning for the same configuration".
Co-authored-by: Isaac <[email protected]>
Copy file name to clipboardExpand all lines: docs/sql-migration-guide.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -57,6 +57,7 @@ license: |
57
57
- Since Spark 4.4, with whole-stage codegen enabled, a filter condition, including the `FILTER` clause of an aggregate function, evaluates the remaining top-level conjuncts of the condition on a row after one of them evaluates to `NULL`, as the SQL standard requires, and does so in the order they appear in the condition, as Spark already did with whole-stage codegen disabled, which also evaluates a subexpression shared by several conjuncts before the condition. Only a `FALSE` conjunct stops the evaluation. One exception: an `IS NOT NULL` check on a column, or on an expression over it, that a later conjunct references is still evaluated before that conjunct and the conjuncts up to the check's own position, even when it is written after them, and can drop the row before an error one of them raises. Once a condition has a conjunct that can be `NULL`, the `IS NOT NULL` checks after that conjunct are evaluated in the order of the condition on every row, not only on rows where a conjunct is `NULL`. A row whose condition is `NULL` is still dropped, but an error raised by a later conjunct on that row, such as `ARITHMETIC_OVERFLOW` in ANSI mode, now fails the query. For example, `SELECT a FROM t WHERE NOT (a > 0 AND b > 0) AND y + 1 > 0` can fail with `ARITHMETIC_OVERFLOW` on a row where `a` is `NULL`, `b` is `1`, and `y` is `2147483647`, unless the row is dropped earlier by a `FALSE` conjunct or `IS NOT NULL` check that precedes `y + 1 > 0` in the condition, or skipped before the filter, for example by data source filter pushdown such as Parquet row group statistics. Previously the query skipped such a row with whole-stage codegen enabled. The later conjuncts also run on more rows than before, so a nondeterministic conjunct such as `rand(seed)` can return different values on the remaining rows, and a side-effecting user-defined function is called on rows that are dropped. To restore the previous behavior, set `spark.sql.legacy.filterShortCircuitOnNullConjunct` to `true`.
58
58
59
59
- Since Spark 4.4, when a format is specified, `to_timestamp`, `try_to_timestamp`, and `to_timestamp_ltz` cast a numeric first argument to a string and parse it with the format, for example `to_timestamp(20201231, 'yyyyMMdd')` returns `2020-12-31 00:00:00`. This matches the existing behavior when `spark.sql.timestampType` is `TIMESTAMP_NTZ`. Previously, with the default `TIMESTAMP_LTZ` timestamp type, such calls failed with `INTERNAL_ERROR`. Note that the string form of the number is parsed, so a DECIMAL `1.5` is parsed as `"1.5"`.
60
+
- Since Spark 4.4, the Spark Thrift Server logs a warning at startup when `hive.server2.enable.doAs` is `true` and `hive.server2.authentication` verifies user identities (any recognized type other than `NONE`/`NOSASL`), because executor-side data access still runs as the server's own service identity rather than the connecting user (SPARK-5159). Set `spark.sql.hive.thriftServer.allowIneffectiveDoAs` to `true` to acknowledge the limitation and silence the warning. Spark 5.0 is expected to refuse to start on this configuration unless that option is set.
60
61
61
62
## Upgrading from Spark SQL 4.2 to 4.3
62
63
@@ -65,7 +66,6 @@ license: |
65
66
- Since Spark 4.3, metrics produced by `Dataset.observe` include values from only the last successful task attempts instead of aggregating values from all attempts. To restore the previous behavior, set `spark.sql.legacy.observeMetricsAggregateAllAttempts` to `true`.
66
67
- Since Spark 4.3, the configuration key `spark.sql.sources.v2.bucketing.allowJoinKeysSubsetOfPartitionKeys.enabled` has been renamed to `spark.sql.sources.v2.bucketing.allowKeysSubsetOfPartitionKeys.enabled` to reflect that it now applies to storage-partitioned joins, aggregates, and windows. The old key continues to work as an alias.
67
68
- Since Spark 4.3, the Spark Thrift Server rejects setting JVM system properties through the `set:system:` session configuration overlay (for example, in a JDBC connection string). To restore the previous behavior, set `spark.sql.legacy.hive.thriftServer.allowSettingSystemProperties` to `true`.
68
-
- Since Spark 4.3, the Spark Thrift Server logs a warning at startup when `hive.server2.enable.doAs` is `true` and `hive.server2.authentication` verifies user identities (any recognized type other than `NONE`/`NOSASL`), because executor-side data access still runs as the server's own service identity rather than the connecting user (SPARK-5159). Set `spark.sql.hive.thriftServer.allowIneffectiveDoAs` to `true` to acknowledge the limitation and silence the warning. Spark 5.0 is expected to refuse to start on this configuration unless that option is set.
69
69
- Since Spark 4.3, the adaptive execution rule `org.apache.spark.sql.execution.adaptive.DynamicJoinSelection` has been renamed to `DemoteBroadcastHashJoin`, which now only demotes broadcast hash joins (emitting `NO_BROADCAST_HASH`). Its selection of shuffled hash join over sort merge join has moved to a new physical rule gated by `spark.sql.adaptive.convertSortMergeJoinToShuffledHashJoin.enabled` (default `true`). If you previously disabled the shuffled-hash-join preference by listing `org.apache.spark.sql.execution.adaptive.DynamicJoinSelection` in `spark.sql.adaptive.optimizer.excludedRules`, that name no longer matches any rule (unknown names are silently ignored); set `spark.sql.adaptive.convertSortMergeJoinToShuffledHashJoin.enabled` to `false` instead.
70
70
- Since Spark 4.3, the optimizer rule `MergeSubplans` has moved from package `org.apache.spark.sql.catalyst.optimizer` to `org.apache.spark.sql.execution.planmerging`, so its fully qualified name is now `org.apache.spark.sql.execution.planmerging.MergeSubplans` (the rule was itself renamed from `MergeScalarSubqueries` to `MergeSubplans` in Spark 4.2). If you previously disabled it by listing `org.apache.spark.sql.catalyst.optimizer.MergeSubplans`, or the older `org.apache.spark.sql.catalyst.optimizer.MergeScalarSubqueries`, in `spark.sql.optimizer.excludedRules`, that name no longer matches any rule (unknown names are silently ignored); use `org.apache.spark.sql.execution.planmerging.MergeSubplans` instead.
71
71
- Since Spark 4.3, `spark.sql.execution.replaceHashWithSortAgg` defaults to `true`. Spark now replaces a hash-based aggregate with a sort aggregate when the aggregate's child is already sorted on the grouping keys. To restore the previous behavior, set `spark.sql.execution.replaceHashWithSortAgg` to `false`.
0 commit comments