Native Iceberg scan returns wrong values for a column renamed after a VERSION AS OF snapshot #6822

Describe the bug

A VERSION AS OF read of a snapshot taken before a column was renamed returns wrong results from the native Iceberg scan. The renamed column comes back as NULL. When two columns swap names, each one returns the other's values. Spark returns the snapshot's data, and reads of the current snapshot are correct.

CometIcebergNativeScan.serializePartitions gives a task with no deletes the current table schema, and a task with deletes FileScanTask.schema(), which is also the current schema. It switches to the scan schema only when the scan reads a column the current schema no longer has (hasHistoricalColumns). A renamed column keeps its field id, so that check doesn't fire. iceberg-rust then names each output column after the current schema, and Comet matches those columns to the snapshot names Spark expects by name. A renamed column finds no match and reads as NULL. After a swap, projectFieldIds also resolves each output name in the task schema first, where the name now belongs to the other column.

Steps to reproduce

CREATE TABLE cat.db.t (id INT, a STRING, b STRING) USING iceberg;
INSERT INTO cat.db.t SELECT CAST(id AS INT), concat('A', id), concat('B', id) FROM range(20);
-- <s> is the snapshot id after the insert
ALTER TABLE cat.db.t RENAME COLUMN a TO z;
SELECT id, a FROM cat.db.t VERSION AS OF <s> ORDER BY id;

On main at 00a4b42 and on branch-1.1 at e9efd9f (Spark 4.1, Iceberg 1.11):

The results are the same when the snapshot has position deletes. Spark 3.4 with Iceberg 1.5.2 gives the same results too. I checked that through #6725 with its pruning turned off, which takes the same path as main for these queries.

Expected behavior

The native scan returns the snapshot's data, as Spark does. A VERSION AS OF task needs a schema with the snapshot's column names whenever they differ from the current schema's, not only when a column was dropped.

Additional context

Found while reviewing #6725. That PR reads with Spark's pruned scan schema by default, which carries the snapshot's names, so these queries match Spark with it. The bug stays on branch-1.1, with spark.comet.scan.icebergNative.nestedSchemaPruning.enabled=false, and for tasks that keep the full schema because the query pruned away a nested partition source or equality-delete key. A swap where the scan also has to append a partition source on one of the swapped columns fails at planning with #6725 (Invalid schema: multiple fields for name b), which is discussed on that PR.