AQE: dynamically coalesce shuffle partitions based on runtime statistics #1987

Is your feature request related to a problem or challenge?

The adaptive execution path currently does not dynamically coalesce shuffle partitions. When a shuffle stage produces very little data spread across many partitions, downstream stages still schedule one task per partition, adding scheduler overhead and small/empty tasks. This is the AQE analog of Spark's coalesce-shuffle-partitions (elastic shuffle parallelism).

Describe the solution you'd like

After a shuffle stage completes, use its per-partition output statistics to coalesce partitions in the downstream ShuffleReaderExec partitioning, reducing the number of tasks for the next stage. The EPIC design describes this under "Coalesce Shuffle Partitions (Elastic Shuffle Parallelism)".

Additional context

A coalesce module stub exists at ballista/scheduler/src/state/aqe/coalesce.rs but runtime coalescing is not yet wired in. Older closed issue #813 covered making the static shuffle partition count configurable, which is a different concern.

Context

This is one of the documented limitations of the experimental Adaptive Query Execution (AQE) path, listed in docs/source/user-guide/tuning-guide.md under "Current limitations" and in the doc comment on AdaptiveExecutionGraph (ballista/scheduler/src/state/aqe/mod.rs). Filing a discrete tracking issue so the gap is visible outside the doc.

Parent epic: #1359