Broadcast/hash join fallback reasons are lost from the AQE-final plan #6442

Describe the bug

When a broadcast (or shuffled) hash join declines Comet conversion and falls back to Spark, the fallback reason recorded via withFallbackReason is present on the first CometExecRule pass but is missing from the AQE-final executed plan. As a result the [COMET: <reason>] annotation does not appear in EXPLAIN output (nor in df.queryExecution.executedPlan) when AQE is enabled (the default).

Correctness is unaffected — the join still falls back and produces correct results. This is purely about the explain-time fallback-reason annotation.

Root cause

CometExecRule tags a broadcast join's fallback reason only via the broadcast-child branch:

case plan if plan.children.exists(_.isInstanceOf[BroadcastExchangeExec]) =>

On the first pass the child is a raw BroadcastExchangeExec, so the join serde runs and the reason is copied onto the returned (declined) join node. But AQE's reOptimize re-plans from the logical plan and produces a new join instance whose build side is now a BroadcastQueryStageExec. TreeNode tags do not travel to the new instance, and on the re-optimize pass the branch above no longer matches (the child is a query stage, not a BroadcastExchangeExec), so the serde never re-runs and the new join is never re-tagged.

A convertible broadcast join survives because its child becomes a CometBroadcastExchangeExec → BroadcastQueryStageExec (matched at the BroadcastQueryStageExec(_, _: CometBroadcastExchangeExec, _) case), giving the join native children so it re-converts (and re-tags) through the allExecs handler. Only the declined case loses its reason.

Scope

• Not specific to existence joins — any broadcast/hash join that declines conversion loses its reason under AQE. Surfaced by the existence-join fallback tests (CometJoinSuite), which had to disable AQE to observe the tagged reason.
• Explain-only, AQE-only. With COMET_EXPLAIN_FALLBACK_ENABLED the reason still reaches the warning log from the first pass; it is just absent from the final plan tree.

Suggested fix

Re-run the join serde for its tagging side effect (or copy the reason forward) when a broadcast/hash join declined conversion and its build side is now a BroadcastQueryStageExec, so the reason lands on the AQE-final plan node.

Context

Split out from PR #4587 (native ExistenceJoin), where CometJoinSuite tests for residual-condition and computed-key fallbacks disable AQE as a workaround pending this fix.