IcebergCommitExec rethrows the raw cause when the abort fails, where Spark wraps it in writingJobFailedError #6143

Corrected 2026-09-23. This issue originally said Spark 3.x wraps every non-fatal write failure in a "Writing job aborted" SparkException. That is Spark 3.3 and earlier behaviour, which Comet does not support. The description below is corrected. See #6153 for the source references.

Describe the bug

When an Iceberg write fails under the split-operator plan, IcebergCommitExec aborts the write and rethrows the original cause, on both the job-failure path and the commit-failure path.

On every Spark version Comet supports (3.4 through 4.2), Spark's own V2TableWriteExec.writeWithV2 behaves as follows:

• if batchWrite.abort succeeds, it rethrows the original cause unchanged
• if batchWrite.abort itself throws, it attaches the abort failure to the cause as suppressed and throws QueryExecutionErrors.writingJobFailedError(cause), a SparkException ("Writing job failed.") with the original failure as its cause

IcebergCommitExec matches the first case. In the second it attaches the abort failure as suppressed but still throws the raw cause, so when the abort fails the user sees a different exception type depending on whether the split plan is on.

Steps to reproduce

Make an Iceberg write fail (at commit, or in a task) with a BatchWrite whose abort also throws, with spark.comet.write.iceberg.splitOperator.enabled off and then on, and compare the thrown exception.

Expected behavior

The split plan throws the same exception as Spark's own path: the original cause when the abort succeeds, and writingJobFailedError(cause) when the abort fails. No version shim is needed, since the behaviour is the same on 3.4 through 4.2.

Additional context

Found in an audit of the native Iceberg write path. Part of #5649.