Support native MergeRowsExec on Spark 4.1+ with MergeSummary #6606

What is the problem?

Native MergeRowsExec support introduced in #5318 is currently limited to Spark 3.5.x and 4.0.x.

Spark 4.1 introduced a writer-side MergeSummary contract. The V2 write path derives semantic MERGE action counts from MergeRowsExec and passes them through the summary-aware:

BatchWrite.commit(messages, summary)

Replacing Spark's MergeRowsExec with a native operator without preserving those metrics would change the writer commit contract even when the produced rows are otherwise identical.

For that reason, Spark 4.1+ currently keeps MergeRowsExec on the JVM.

Proposed solution

Preserve the Spark 4.1+ MERGE summary contract end-to-end while allowing MergeRowsExec itself to execute natively.

This requires:

• carrying MERGE action context through native instructions
  • copy
  • insert
  • update
  • delete
• exposing Spark-compatible semantic counters:
  • numTargetRowsCopied
  • numTargetRowsInserted
  • numTargetRowsDeleted
  • numTargetRowsUpdated
  • numTargetRowsMatchedUpdated
  • numTargetRowsMatchedDeleted
  • numTargetRowsNotMatchedBySourceUpdated
  • numTargetRowsNotMatchedBySourceDeleted
• preserving the Spark-version-specific metric semantics used by MergeSummary
• making the V2 write path consume the native MergeRows metrics when constructing the summary
• forwarding the resulting MergeSummary through the summary-aware BatchWrite.commit path
• enabling native MergeRowsExec registration on Spark 4.1+ once the contract is preserved

Validation

Add differential coverage against Spark for:

• matched update
• matched delete
• not-matched insert
• not-matched-by-source update
• not-matched-by-source delete
• the eight semantic action counters
• final table contents
• the MergeSummary observed by the V2 writer

Spark 4.2 should also preserve its retry / last-attempt metric behavior.

Scope

This issue is specifically about the Spark 4.1+ MergeSummary compatibility boundary for native MergeRowsExec.

Native Iceberg merge-on-read / WriteDeltaExec support is separate and does not need to block this work.

Related

• feat: add native support for MergeRowsExec (row-level MERGE INTO) #5318 — native MergeRowsExec
• Accelerate row-level MERGE / UPDATE / DELETE plans (MergeRowsExec, ReplaceDataExec, WriteDeltaExec) #5122 — row-level MERGE / UPDATE / DELETE acceleration