Skip to content

fix: qualify extracted columns pushed under a subquery alias - #1

Open
batmnnn wants to merge 2 commits into
mainfrom
cursor/fix-extracted-column-qualification-55e3
Open

batmnnn wants to merge 2 commits into
mainfrom
cursor/fix-extracted-column-qualification-55e3

Conversation

@batmnnn

@batmnnn batmnnn commented Oct 23, 2026 •

Copy link
Copy Markdown
Owner

Important

Fork CI setup: For pull requests from a fork to main, enable workflows in your fork's Actions tab before you push.
See the fork CI setup guide.

Which issue does this PR close?

Rationale for this change

Leaf expression pushdown is on by default. The optimizer takes a struct field access out of an aggregate. The aggregate then reads an unqualified __datafusion_extracted_N column.

The optimized plan runs. A later rebuild qualifies that column. LogicalPlanBuilder::aggregate does this when it normalizes columns. Older proto decoders did the same rebuild. The aggregate output name changes. The projection above the aggregate no longer finds the field.

The column is unqualified at extraction time. The defining projection still sits above the SubqueryAlias. Pass 2 moves that projection under the alias. Only then does the input schema spell the column as alias.__datafusion_extracted_N.

What changes are included in this PR?

PushDownLeafProjections walks the plan itself. It pushes an extraction projection before that projection's inputs. After the inputs are pushed, it spells each unqualified __datafusion_extracted_N column.

The spelling comes from the one input field with that name. Join and ASOF ON keys use one side. The same name on both sides becomes left.col = right.col. A filter that can see both sides stays unqualified. A choice of side would change the predicate. normalize_cols rejects that reference.

An output name that changes is copied to the parent. An aggregate field sum(__datafusion_extracted_1) becomes sum(alias.__datafusion_extracted_1). The aggregate expression keeps that qualified name as its field name. Pass 1 already keeps the user-facing name with a recovery projection. A root Window has no such wrapper. This pass restores the root schema only.

A column reorder is not a rename. The old name is still in the new schema. The rule matches a rename by the name string. Grouping sets, mixed grouping sets, and window arguments use this same path.

What is the testing strategy for this PR?

roundtrip_aggregate_extracted_column_under_subquery_alias runs the issue SQL through logical_plan_to_bytes and logical_plan_from_bytes. The schema and the debug plan stay the same.

roundtrip_join_same_extracted_name_on_both_sides checks the per-side join keys and the unqualified filter after a proto round trip.

test_grouping_set_qualifies_extracted_column and test_mixed_grouping_set_qualifies_extracted_column check single and mixed grouping sets. test_window_qualifies_extracted_column_under_subquery_alias checks a window above a subquery alias. test_join_qualifies_same_extracted_name_per_side checks the join. Each aggregate and window test rebuilds the node with normalize_cols and requires the same output names.

projection_pushdown.slt runs the changed IN-subquery filters as a derived table. The count is 3. ORDER BY value DESC LIMIT 1 returns id 5.

cargo test -p datafusion-optimizer --lib passed (953 tests). cargo test -p datafusion-sqllogictest --test sqllogictests passed (537 files). cargo test -p datafusion-proto --all-targets passed (18 library tests and 305 integration tests). cargo test -p datafusion-proto --doc passed (4 tests). ci/scripts/rust_clippy.sh passed. cargo fmt -p datafusion-optimizer -p datafusion-proto -- --check passed.

Are there any user-facing changes?

Explain text shows the qualifier on an extracted column above a subquery alias. An aggregate field uses that qualified name. There is no public API change.

ExtractLeafExpressions still leaves window arguments in place. A window that already reads an extracted column is qualified by this pass.

Open in Web Open in Cursor 

PushDownLeafProjections moved the extraction projection under a
SubqueryAlias and left parent references unqualified. An aggregate
output name then changed when a rebuild normalized those columns.

Qualify each unambiguous extracted column to the input schema. Alias
schema-defining expressions back to the previous output field name.

Co-authored-by: Shubham Yadav <[email protected]>
The column is unqualified until the defining projection moves under a
subquery alias. After that push, spell each extracted column from the
one input field of the same name. Follow output names that change, such
as an aggregate field. Do not wrap the aggregate or window expression
in a second alias.

A reorder that also adds a qualifier is not a rename. Join keys are
spelled per side. A filter that matches both sides stays unqualified.

Co-authored-by: Shubham Yadav <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Leaf expression pushdown leaves an unqualified column in an Aggregate's output name, so the plan fails a datafusion-proto round trip

2 participants