[SPARK-59118][SQL][4.x] Warn when doAs is enabled but not enforced - #58438

hive.server2.enable.doAs is accepted and HiveSessionProxy does impersonate
the connecting user for metastore calls, but queries and the storage access
they perform run as the service identity (SPARK-5159), so storage ACLs are
checked against the wrong principal. Warn at startup rather than look like
we enforce something we do not.

Warn rather than refuse, because refusing breaks deployments that have been
accepted for years and this is a minor release. Spark 5.0 refuses instead
and adds an escape-hatch conf, which the message points at.

The warning does not tell operators to just set doAs=false: that also drops
metastore impersonation, so it is not a no-op.

Auth types that never establish a user identity have nothing to impersonate,
so NONE and NOSASL stay quiet -- which is also the stock Hive config, since
Hive defaults doAs to true.

Co-Authored-By: Claude Sonnet 5 <[email protected]>


Co-authored-by: Holden Karau <[email protected]>

…de wording

Co-authored-by: Cursor <[email protected]>

…e silenced

The warning told operators to set
spark.sql.hive.thriftServer.allowIneffectiveDoAs=true, but nothing defined
that conf: an unknown spark.sql.* key is accepted silently, so following the
advice did nothing and the operator got no feedback either way. Worse, the
name ships in a released binary, so if 5.0 lands the gate under a different
name the log line points at a key that never existed.

Define it here and honor it today. Setting it acknowledges the limitation and
silences the warning; 5.0 is expected to refuse to start on the same
configuration unless it is set, so the key an operator sets now carries
forward. Versioned 4.3.0 -- it ships in 4.3.0 via RC2, not in the 4.3.1
that branch-4.3's pom currently reads between RCs.

Also test that init() actually emits the warning, since deleting the call
left every unit test green, and that the warning names the conf that
silences it -- which is what would have caught the missing conf.

Operation logging is off in that test: init() boots Hive's service graph,
and OperationManager.init attaches an appender to the root logger that
nothing ever removes.

Co-authored-by: Holden Karau <[email protected]>

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

…JVM logging state

CI runs the hive-thriftserver module in a single forked JVM
(SERIAL_SBT_TESTS=1 disables the dedicated-JVM test grouping), so
SparkExecuteStatementOperationSuite's init test ran in the same JVM as
HiveThriftServer2DoAsSuite. The init test attached an appender to the
HiveThriftServer2 logger by name; log4j2's addLoggerAppender permanently
registers an exact-name LoggerConfig for it, copying the root config's
additivity (false). Cleanup removes the appender but not the LoggerConfig,
so every later event from that logger was swallowed - never reaching the
console or any root-attached appender - and the doAs suite's log-capture
assertions failed.

Two changes:

- Split warnIfIneffectiveDoAs into ineffectiveDoAsWarning, which returns
  the warning as an Option[LogEntry], and a one-line wrapper that logs it.
  The doAs suite now asserts on the returned entry instead of capturing
  log output, mirroring how the master version asserts on the thrown
  exception.
- The init test attaches its appender to the root logger instead of the
  named logger, so it no longer leaves the non-additive LoggerConfig
  behind. It still covers end-to-end emission through init().

Co-authored-by: Cursor <[email protected]>

Co-Authored-By: Holden Karau <[email protected]>

Co-authored-by: Isaac <[email protected]>