Repository navigation
Commit 50cab14
fix: preserve projection metadata during optimization (apache#24670)
- Closes apache#24721
There were four ways metadata could disappear.
### 1. Removing a metadata-only identity projection
Consider:
```text
ProjectionExec: i@0 AS i
output field metadata = {"event_field": "true"}
DataSourceExec: i
field metadata = {}
```
The check to remove the projection asked:
- Is every expression a column?
- Does column 0 remain column 0?
- Does the alias match the column name?
- Does the number of columns match?
All answers yes so optimizer removed projection:
```text
DataSourceExec: i
metadata = {}
```
Metadata lost.
### 2 Collapsing across a metadata boundary
Consider:
```text
ProjectionExec: arrow_metadata(i, 'event_field') AS metadata
ProjectionExec: i@0 AS i
output metadata = {"event_field": "true"}
DataSourceExec: i
metadata = {}
```
The correct result is `true`.
The previous projection colapse logic would substitute the outer expression through the inner projection:
```text
ProjectionExec: arrow_metadata(i, 'event_field') AS metadata
DataSourceExec: i
metadata = {}
```
Now the func sees the scan field instead of the inner projection field giving use result as `NULL` now.
### 3 Rebuilding a projection with a new child
Some optimizer paths replace the child of a projection:
```text
Old:
ProjectionExec(metadata={"key": "value"})
OldChild
---
New:
ProjectionExec(...?)
NewChild
```
The previous `make_with_child` implementation did this:
```rust
ProjectionExec::try_new(projection.expr().to_vec(), new_child)
```
where try_new derives the output schema from the expressions and new
child so it woudlnt retain metadata from the original projection.
### 4 Cast target metadata lost before optimization
This one was a little confusing because main passed the UUID metadata
test, but the first version of this PR did not (@gabotechs this is what
you called out)
Basically a cast can have an explicit target field with metadata. For
example, the UUID type planner produces:
```text
FixedSizeBinary(16)
metadata = {"ARROW:extension:name": "arrow.uuid"}
```
But logical cast schema only used the target data type when deriving and
kept th source metadata:
```text
Source field:
raw: FixedSizeBinary(16)
metadata: {}
Cast target:
FixedSizeBinary(16)
metadata: {"ARROW:extension:name": "arrow.uuid"}
Derived cast output:
FixedSizeBinary(16)
metadata: {}
```
This appeared in CI when common sub-expr elimination extracts a repeated cast into
its own projection:
```text
ProjectionExec: arrow_metadata(__common_expr_1, 'ARROW:extension:name')
ProjectionExec: CAST(raw AS UUID) AS __common_expr_1
```
The inner projection was initially created with incorrect empty metadata, so the physical optimizer rebuilt that projection and rederived its schema so isthe was accidentally repairing the logical schema bug.
Once this PR started preserving projection metadata correctly had this pop up this other bug.
---
So then I solve the optimizer bugs in this PR
---------
Co-authored-by: Tim Saucer <[email protected]>
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>1 parent 5b389eb commit 50cab14
1 file changed
Lines changed: 322 additions & 12 deletions
0 commit comments