You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 8ce9f60
Browse filesBrowse the repository at this point in the historyBrowse files
fix: PushDownLeafProjections yields when the extraction cannot move (apache#26085)
## Which issue does this PR close?
- Part of apache#25459. This is a
follow-up to apache#25455
(apache#14540).
## Rationale for this change
`PushDownLeafProjections` and `OptimizeProjections` undo each other in
every optimizer pass for a common plan shape: a projection of a struct
field (or another `MoveTowardsLeafNodes` expression) directly over a
node that the extraction cannot go through, such as a `TableScan` or an
`Aggregate`.
```text
Projection: leaf_udf(test.a)
TableScan: test
```
In each pass, `push_down_leaf_projections` splits this into a recovery
projection over an in-place extraction projection:
```text
Projection: __datafusion_extracted_1 AS leaf_udf(test.a)
Projection: leaf_udf(test.a) AS __datafusion_extracted_1, test.a
TableScan: test
```
Then `optimize_projections` merges the two back into the original plan.
The loop stops only because the plan at the end of a pass is the same as
at the end of the previous pass. The final plan is correct, but each
pass does the work of both rules again.
The test that apache#25455 added
found this shape. That test records which rules change the plan in each
pass. On `main` it prints, for the plan above:
```text
pass 0: push_down_leaf_projections, optimize_projections
pass 1: push_down_leaf_projections, optimize_projections
```
With this PR:
```text
pass 0: optimize_projections
```
### Which rule yields
`PushDownLeafProjections` yields. The split has no use when the
extraction projection cannot move: a plain `Projection(get_field(...))`
directly over a Parquet scan already reaches `DataSourceExec
projection=[..., get_field(s@1, value) ...]`. The existing `EXPLAIN`
checks in `projection_pushdown.slt` show this and do not change. If
`OptimizeProjections` yielded instead, the plan would keep a projection
node that does no work.
## What changes are included in this PR?
In `split_and_push_projection` (`extract_leaf_expressions.rs`), the rule
still builds the in-place extraction projection, because a mixed
projection over a `Filter` must become a pure extraction projection
before it can go through the filter. The rule now immediately tries to
push that extraction projection into its input. If it cannot move, the
rule returns the original projection unchanged. The early return for a
projection without new extractions stays: it also ends the recursion of
that push attempt.
- The decision is recorded in the module documentation of
`extract_leaf_expressions.rs` and `optimize_projections`, and as a
second entry in the "Rule Precedence" section of
`docs/source/library-user-guide/query-optimizer.md`.
## What is the testing strategy for this PR?
- `filter_and_extraction_projection_reach_fixed_point` in
`push_down_filter.rs` now checks that no rule at all changes the plan in
the last optimizer pass (before, it checked only `PushDownFilter` and
`PushDownLeafProjections`). It also covers a projection over a scan with
no filter. Without the fix it fails with
`[["push_down_leaf_projections","optimize_projections"],["push_down_leaf_projections","optimize_projections"]]`.
- Six stage snapshots in `extract_leaf_expressions.rs` change. In each
one, only the "After Pushdown" stage changes, to "(same as after
extraction)". The "Optimized" plan is the same in all six.
- One line in `unnest.slt` changes (logical plan line 03). A nested
alias is now kept: `get_field(__unnest_placeholder(...,depth=1) AS
UNNEST(recursive_unnest_table.column3), Utf8("c1"))`. The split and
merge used to remove that alias as a side effect. The physical plan is
the same. This is the same display-only nested alias that
apache#25455 showed for
`length(...)`.
Commands run:
- `cargo test --profile ci -p datafusion-optimizer --lib`: 932 passed.
- `cargo test --profile ci -p datafusion-sqllogictest --test
sqllogictests`: 526 of 526 files pass.
- `cargo fmt --all` and `cargo clippy --profile ci -p
datafusion-optimizer --all-targets -- -D warnings`: clean.
## Are there any user-facing changes?
No. Final plans and results do not change, except for the alias display
in the `unnest` plan above. Planning does less work for queries that
read struct fields.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>
Copy file name to clipboardExpand all lines: datafusion/sqllogictest/test_files/unnest.slt
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -668,7 +668,7 @@ explain select unnest(unnest(unnest(column3)['c1'])), column3 from recursive_unn
668
668
logical_plan
669
669
01)Projection: __unnest_placeholder(UNNEST(recursive_unnest_table.column3)[c1],depth=2) AS UNNEST(UNNEST(UNNEST(recursive_unnest_table.column3)[c1])), recursive_unnest_table.column3
03)----Projection: get_field(__unnest_placeholder(recursive_unnest_table.column3,depth=1), Utf8("c1")) AS __unnest_placeholder(UNNEST(recursive_unnest_table.column3)[c1]), recursive_unnest_table.column3
671
+
03)----Projection: get_field(__unnest_placeholder(recursive_unnest_table.column3,depth=1) AS UNNEST(recursive_unnest_table.column3), Utf8("c1")) AS __unnest_placeholder(UNNEST(recursive_unnest_table.column3)[c1]), recursive_unnest_table.column3
Copy file name to clipboardExpand all lines: docs/source/library-user-guide/query-optimizer.md
+2-1Lines changed: 2 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -168,9 +168,10 @@ Two rules can want the opposite order for the same pair of adjacent plan nodes.
168
168
169
169
Do not let two rules compete. Give one rule precedence, make the competing rule yield, and record the decision in the module documentation of both rules.
170
170
171
-
There is one such decision today:
171
+
There are two such decisions today:
172
172
173
173
-`PushDownFilter` yields to a _pure extraction projection_. A pure extraction projection is a projection whose expressions are only `__datafusion_extracted_N` aliases and pass-through columns. `ExtractLeafExpressions` creates it, and `PushDownLeafProjections` moves it towards the leaves. `PushDownFilter` does not move a filter below such a projection, so the projection stays next to the scan. A Parquet scan then merges the projection into the file projection and reads only the struct leaf. The filter loses nothing, because `PushDownFilter` records the predicate in `TableScan::filters` in the pass that runs before the extraction projection exists. See [issue #14540](https://github.com/apache/datafusion/issues/14540).
174
+
-`PushDownLeafProjections` yields to `OptimizeProjections`. When the extraction projection cannot move below the input of a projection, for example a `TableScan` or an `Aggregate`, `PushDownLeafProjections` leaves the projection unchanged. A split in place would give a recovery projection over an extraction projection on the same input, which `OptimizeProjections` merges back. The source absorbs the leaf expressions of the original projection just as well. See [PR #26085](https://github.com/apache/datafusion/pull/26085).
0 commit comments