Native Iceberg writes: write row lineage for copy-on-write DML and compaction on format-version 3 tables #6669

What is the problem the feature request solves?

With #6668, the native Iceberg writer takes appends and overwrites to format-version 3 tables, but declines any write whose write schema includes the row lineage columns _row_id and _last_updated_sequence_number. From Iceberg 1.10 that is every copy-on-write DELETE, UPDATE and MERGE and every rewrite_data_files on a v3 table, so all of them stay on iceberg-java.

Describe the potential solution

Write the two lineage columns natively, with the values iceberg-java writes:

• Copy-on-write DML (Spark 4.x). dropNonDataColumns projects the ReplaceData row down to the write schema's data columns. It would instead carry _row_id and _last_updated_sequence_number from the row's metadata columns. iceberg-java's ExtractRowLineage copies both from the metadata row and writes nulls for a row that arrives without one (an inserted row). Spark's metadata column flags already null _last_updated_sequence_number on updated rows and keep their _row_id.
• rewrite_data_files. The rewrite table's schema includes the two columns, so their values arrive as ordinary input columns.
• Field ids. The columns carry reserved field ids (Integer.MAX_VALUE - 107 and - 108), which the field-id decoration and the iceberg-rust schema need to accept.
• Manifest metrics. The DataFile metrics for the two columns must match iceberg-java's after rebuildDataFilesWithJavaMetrics.
• Gate. Relax requireNoMetadataColumns for exactly these two columns, and keep declining any other metadata column.

Tests should compare _row_id and _last_updated_sequence_number read back after native and iceberg-java writes on Iceberg 1.10 and 1.11, including a MERGE that inserts rows and a compaction that follows an update.

Additional context

Part of #5649. Follows #6668.