Skip to content

Commit 2364c5b

Browse files
TRO-Wolfclaude
andcommitted
fix(datafusion): bind the scan by FIELD ID, freeze the advertised schema, allow legal promotions
Critic remediation on top of e2a213f (2xS1 + S2 + S3 + LOW). * S1-1 (silent wrong data on RENAME) — `resolve_projection` matched advertised columns against the scanned snapshot's schema BY NAME. A rename keeps the field id, rewrites the name and creates no snapshot, so `SELECT *` null-filled the renamed column over live data. Binding is now by FIELD ID (`PARQUET:field_id` on the advertised Arrow field -> `Schema::name_by_field_id`): the snapshot's name for that id is what gets selected, and `conform_batch` re-labels it to the advertised name. An advertised field with no id metadata is a loud error, never a name-based guess. * S1-2 (DataFusion panic when the advertised schema shrank under a live plan) — the provider's schema is now IMMUTABLE for the life of the instance, because DataFusion stores projection ORDINALS resolved against it: a view or deferred DataFrame re-planned after the schema moved panics inside `datafusion-expr`'s `TableScan::try_new`. Freshness moves to a new INSTANCE, exactly as Java does it (`SparkCatalog.loadTable` per query): `IcebergSchemaProvider::table` now builds a provider from freshly loaded metadata on every resolution (it used to cache one per session — BUG-005 on the shipping path), and `IcebergTableProvider::refresh` becomes `refreshed()`, returning a NEW provider and leaving plans built on the old one valid. * S2-3 (legal promotion rejected) — `conform_batch` applies Iceberg's legal type promotions (int->long, float->double, decimal precision widening) via an Arrow mirror of `iceberg::spec::is_promotion_allowed`, pinned against that authority over a 14x14 primitive matrix. Illegal changes still error, naming the column and both types. * S3-4 — the conform errors said "re-plan the query", which is a no-op for a snapshot-vs-current divergence; they now name the actual cause. * LOW-5 — measured: in the steady state the reader's batch schema is IDENTICAL to the advertised one (fields AND `PARQUET:field_id` metadata), so the equality fast path hits and nothing is rebuilt. Kept, and pinned by a test so a reader change cannot silently move every scan onto the rebuild path. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01MRhAQnro5233xoVDLfib1X
1 parent e2a213f commit 2364c5b

3 files changed

Lines changed: 939 additions & 260 deletions

File tree

0 commit comments

Comments
 (0)