Report native child-operator spill metrics in Spark task metrics for unified shuffle plans #5382

What / Why

Native shuffle execution can inline a spill-capable operator, such as a native sort or sort-merge join, beneath ShuffleWriterExec in one native plan. The writer and its child operators have separate CometMetricNodes and separate SQL spill metrics, but the shuffle writer's task-metric bridge currently publishes only spill counters from the root writer node.

As a result, the Spark SQL operator view can show a child operator spilling while the Spark Stages/task view reports only the shuffle writer's own spill volume. This discrepancy already exists independently of #5370, which deliberately limits its fix to native shuffle writer metrics.

For example, suppose the shuffle writer spills 32 MiB to disk and a native sort running beneath it spills another 128 MiB. The Spark task should report 160 MiB of disk spill, but currently reports only the writer's 32 MiB. In-memory spill bytes must also be included when a child operator exposes a trustworthy in-memory counter; compressed disk bytes must not be reused as a substitute for memory spill bytes.

Proposed scope

• Account for spill-capable descendant operators in the native metric tree when reporting task-level spill metrics for unified shuffle plans.
• Preserve separate on-disk and in-memory spill semantics. If a child lacks an in-memory spill metric, provide an accurate source rather than copying compressed disk bytes.
• Avoid counting the same writer, child metric, or shared/reused accumulator more than once.
• Preserve completion-listener ordering so final native metrics are available for successful, failed, and canceled task attempts.
• Add focused native sort or sort-merge-join coverage that compares Spark SQL operator spill metrics with Spark task metrics on Spark 3.5 and 4.0.

Related: #3996 and #5370.