You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit f82a366
Browse filesBrowse the repository at this point in the historyBrowse files
fix(delta): feed kernel the analysis-time schema for with_schema (apache#78 schema-change-since-analysis)
Regression from the kernel-schema-shipping work: DeltaColumnMappingSuite "column
mapping batch scan should detect physical name changes" read the current data
instead of null-filling, because the driver fed `ScanBuilder::with_schema` the
LIVE snapshot schema. Kernel resolves physical names from the schema passed to
`with_schema` (StateInfo::try_new -> StructField::make_physical), so it must be
fed the schema the query was PLANNED with -- then kernel's field-id matching
null-fills any column whose id changed since analysis (Delta's schema-on-read
escape hatch), which a pure-kernel engine handles itself with no fallback.
- The JVM ships the analysis-time read schema as Delta schema JSON
(`StructType.json` from DeltaScanRule's stashed reference schema, carrying
`delta.columnMapping.physicalName` + id at every level). The driver parses it
(`serde_json` -> kernel `StructType`, the same format kernel reads from the log)
and feeds it straight to `with_schema`; it falls back to projecting the live
snapshot by column name only when no analysis-time schema is available.
- The analysis-time JSON and the Arrow-IPC names are mutually exclusive on the
wire (ship the JSON when present, else the names) -- no redundant double-ship.
- Fix `planDeltaReadSchemas` to build kernel schemas when EITHER carrier is
present (it previously gated on the IPC only, so a JSON-only payload silently
produced no schemas).
Red->green guard: CometDeltaSchemaChangeReproSuite (Comet returned data, Spark
null-filled; now both null-fill). Own-suite "physical name changes" goes green;
all contrib Delta suites stay green. (DeltaColumnMappingSuite "explicit id
matching" -- a contrived manual field-id repoint -- is a separate kernel
id-vs-name matching nuance, tracked in apache#79.)
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
0 commit comments