Equality deletes ignore spec projection rules for dropped and newly added delete columns #3369

Apache Iceberg Rust version

0.10.1

Describe the bug

This if a follow up for #2961, but it also applies if it is not merged

The spec says (format/spec.md):

If a delete column in an equality delete file is later dropped from the table, it must
still be used when applying the equality deletes. If a column was added to a table and
later used as a delete column in an equality delete file, the column value is read for
older data files using normal projection rules (defaults to null).

Case A: delete column later dropped from the table.
Equality-delete files are evolved against task.schema, the current table schema. A
dropped column is not in that schema. The scan fails in
RecordBatchTransformer::generate_batch_transform with "field not found"
(ErrorKind::Unexpected), instead of using the dropped column for matching. Java's
DeleteFilter seems to fail the same way
(DeleteFilter.java:76,
DeleteFilter.java:318).

Case B: delete column added after the data file was written.
"Normal projection rules" means the reader should use the field's initial_default.
The equality-delete probe runs inside the parquet RowFilter, on raw decoded batches,
before RecordBatchTransformer resolves defaults. So a key column that is missing from
the data file is always read as null, never as initial_default.

Example: a required column added with initial_default = 7 is added to the table. An
equality-delete file keyed on added deletes added = 7. Rows in data files written
before added existed should be deleted. They are not, because they probe as null.

This is a silent wrong result, not an error. It is pinned by
test_eq_delete_on_column_absent_from_data_file (crates/iceberg/src/arrow/reader/row_filter.rs).

Before #2961, the code was also wrong, in a different way: a missing required key
column deleted every row of the data file, and a missing optional key column deleted
nothing.

To Reproduce

Context:

Found during review of #2961, which does not change these paths. See
#2961 (comment) and
#2961 (comment).

Expected behavior

• Case A: a dropped column that an equality-delete file references is still used for
  matching.
• Case B: a key column missing from an older data file is read with normal projection
  rules (its initial_default, else null) before the delete-set lookup.

Willingness to contribute

I cannot contribute a fix for this bug at this time