After promoting a float column to double, > and != filters drop rows from older files #4203

Apache Iceberg version

0.12.0 (latest release)

Please describe the bug 🐞

Currently, _task_to_record_batches (pyiceberg/io/pyarrow.py) re-binds the row filter to each data file's own schema (bind(file_schema, translated_row_filter, ...)), so for a file written before update_column("f", DoubleType()) the literal is converted to the file's float type. 0.1 becomes the 32-bit 0.1, which equals the stored value, so f > 0.1 and f != 0.1 are false for it. The scan itself returns that value promoted to double, 0.10000000149011612, which is greater than and not equal to 0.1. polars scan_iceberg, Daft read_iceberg and DuckDB iceberg_scan return the row for both filters.

import tempfile
import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.types import DoubleType

wh = tempfile.mkdtemp()
cat = SqlCatalog("c", uri=f"sqlite:///{wh}/cat.db", warehouse=f"file://{wh}")
cat.create_namespace("db")
t = cat.create_table("db.t", schema=pa.schema([("id", pa.int64()), ("f", pa.float32())]))
t.append(pa.table({"id": [1, 2], "f": pa.array([0.1, 2.5], pa.float32())}))
with t.update_schema() as u:
    u.update_column("f", DoubleType())

print(t.scan().to_arrow().to_pylist())
# [{'id': 1, 'f': 0.10000000149011612}, {'id': 2, 'f': 2.5}]
print(t.scan(row_filter="f > 0.1").to_arrow()["id"].to_pylist())
# [2]
print(t.scan(row_filter="f != 0.1").to_arrow()["id"].to_pylist())
# [2]

Both filters should return [1, 2]. Without the update_column (the column still float), f > 0.1 returns [2], which is what the file's 32-bit value gives; once the table type is double, the filter should compare the promoted value.

The same re-binding breaks a widened decimal: after update_column("d", DecimalType(12, 2)) on a decimal(5, 2) column, row_filter="d < '1000.00'" raises ArrowInvalid: Decimal type with precision 6 does not fit into precision inferred from first array element: 5 on the older file.

pyiceberg 0.12.0 (_task_to_record_batches binds to file_schema the same way on main), pyarrow 25.0.1, Python 3.12.13, macOS 26.6.2, arm64 (Darwin 25.6.0).

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time