Skip to content

Commit 50cab14

Browse files
Gene Bordegaraytimsaucerclaude
authored
fix: preserve projection metadata during optimization (apache#24670)
- Closes apache#24721 There were four ways metadata could disappear. ### 1. Removing a metadata-only identity projection Consider: ```text ProjectionExec: i@0 AS i output field metadata = {"event_field": "true"} DataSourceExec: i field metadata = {} ``` The check to remove the projection asked: - Is every expression a column? - Does column 0 remain column 0? - Does the alias match the column name? - Does the number of columns match? All answers yes so optimizer removed projection: ```text DataSourceExec: i metadata = {} ``` Metadata lost. ### 2 Collapsing across a metadata boundary Consider: ```text ProjectionExec: arrow_metadata(i, 'event_field') AS metadata ProjectionExec: i@0 AS i output metadata = {"event_field": "true"} DataSourceExec: i metadata = {} ``` The correct result is `true`. The previous projection colapse logic would substitute the outer expression through the inner projection: ```text ProjectionExec: arrow_metadata(i, 'event_field') AS metadata DataSourceExec: i metadata = {} ``` Now the func sees the scan field instead of the inner projection field giving use result as `NULL` now. ### 3 Rebuilding a projection with a new child Some optimizer paths replace the child of a projection: ```text Old: ProjectionExec(metadata={"key": "value"}) OldChild --- New: ProjectionExec(...?) NewChild ``` The previous `make_with_child` implementation did this: ```rust ProjectionExec::try_new(projection.expr().to_vec(), new_child) ``` where try_new derives the output schema from the expressions and new child so it woudlnt retain metadata from the original projection. ### 4 Cast target metadata lost before optimization This one was a little confusing because main passed the UUID metadata test, but the first version of this PR did not (@gabotechs this is what you called out) Basically a cast can have an explicit target field with metadata. For example, the UUID type planner produces: ```text FixedSizeBinary(16) metadata = {"ARROW:extension:name": "arrow.uuid"} ``` But logical cast schema only used the target data type when deriving and kept th source metadata: ```text Source field: raw: FixedSizeBinary(16) metadata: {} Cast target: FixedSizeBinary(16) metadata: {"ARROW:extension:name": "arrow.uuid"} Derived cast output: FixedSizeBinary(16) metadata: {} ``` This appeared in CI when common sub-expr elimination extracts a repeated cast into its own projection: ```text ProjectionExec: arrow_metadata(__common_expr_1, 'ARROW:extension:name') ProjectionExec: CAST(raw AS UUID) AS __common_expr_1 ``` The inner projection was initially created with incorrect empty metadata, so the physical optimizer rebuilt that projection and rederived its schema so isthe was accidentally repairing the logical schema bug. Once this PR started preserving projection metadata correctly had this pop up this other bug. --- So then I solve the optimizer bugs in this PR --------- Co-authored-by: Tim Saucer <[email protected]> Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
1 parent 5b389eb commit 50cab14

1 file changed

Lines changed: 322 additions & 12 deletions

File tree

0 commit comments

Comments
 (0)