Repository navigation
Commit 2364c5b
fix(datafusion): bind the scan by FIELD ID, freeze the advertised schema, allow legal promotions
Critic remediation on top of e2a213f (2xS1 + S2 + S3 + LOW).
* S1-1 (silent wrong data on RENAME) — `resolve_projection` matched advertised columns
against the scanned snapshot's schema BY NAME. A rename keeps the field id, rewrites the
name and creates no snapshot, so `SELECT *` null-filled the renamed column over live
data. Binding is now by FIELD ID (`PARQUET:field_id` on the advertised Arrow field ->
`Schema::name_by_field_id`): the snapshot's name for that id is what gets selected, and
`conform_batch` re-labels it to the advertised name. An advertised field with no id
metadata is a loud error, never a name-based guess.
* S1-2 (DataFusion panic when the advertised schema shrank under a live plan) — the
provider's schema is now IMMUTABLE for the life of the instance, because DataFusion
stores projection ORDINALS resolved against it: a view or deferred DataFrame re-planned
after the schema moved panics inside `datafusion-expr`'s `TableScan::try_new`. Freshness
moves to a new INSTANCE, exactly as Java does it (`SparkCatalog.loadTable` per query):
`IcebergSchemaProvider::table` now builds a provider from freshly loaded metadata on
every resolution (it used to cache one per session — BUG-005 on the shipping path), and
`IcebergTableProvider::refresh` becomes `refreshed()`, returning a NEW provider and
leaving plans built on the old one valid.
* S2-3 (legal promotion rejected) — `conform_batch` applies Iceberg's legal type
promotions (int->long, float->double, decimal precision widening) via an Arrow mirror of
`iceberg::spec::is_promotion_allowed`, pinned against that authority over a 14x14
primitive matrix. Illegal changes still error, naming the column and both types.
* S3-4 — the conform errors said "re-plan the query", which is a no-op for a
snapshot-vs-current divergence; they now name the actual cause.
* LOW-5 — measured: in the steady state the reader's batch schema is IDENTICAL to the
advertised one (fields AND `PARQUET:field_id` metadata), so the equality fast path hits
and nothing is rebuilt. Kept, and pinned by a test so a reader change cannot silently
move every scan onto the rebuild path.
Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01MRhAQnro5233xoVDLfib1X1 parent e2a213f commit 2364c5b
3 files changed
Lines changed: 939 additions & 260 deletions
0 commit comments