Repository navigation
Conversation
PushDownLeafProjections moved the extraction projection under a SubqueryAlias and left parent references unqualified. An aggregate output name then changed when a rebuild normalized those columns. Qualify each unambiguous extracted column to the input schema. Alias schema-defining expressions back to the previous output field name. Co-authored-by: Shubham Yadav <[email protected]>
The column is unqualified until the defining projection moves under a subquery alias. After that push, spell each extracted column from the one input field of the same name. Follow output names that change, such as an aggregate field. Do not wrap the aggregate or window expression in a second alias. A reorder that also adds a qualifier is not a rename. Join keys are spelled per side. A filter that matches both sides stays unqualified. Co-authored-by: Shubham Yadav <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Important
Fork CI setup: For pull requests from a fork to
main, enable workflows in your fork's Actions tab before you push.See the fork CI setup guide.
Which issue does this PR close?
Rationale for this change
Leaf expression pushdown is on by default. The optimizer takes a struct field access out of an aggregate. The aggregate then reads an unqualified
__datafusion_extracted_Ncolumn.The optimized plan runs. A later rebuild qualifies that column.
LogicalPlanBuilder::aggregatedoes this when it normalizes columns. Older proto decoders did the same rebuild. The aggregate output name changes. The projection above the aggregate no longer finds the field.The column is unqualified at extraction time. The defining projection still sits above the
SubqueryAlias. Pass 2 moves that projection under the alias. Only then does the input schema spell the column asalias.__datafusion_extracted_N.What changes are included in this PR?
PushDownLeafProjectionswalks the plan itself. It pushes an extraction projection before that projection's inputs. After the inputs are pushed, it spells each unqualified__datafusion_extracted_Ncolumn.The spelling comes from the one input field with that name. Join and ASOF
ONkeys use one side. The same name on both sides becomesleft.col = right.col. A filter that can see both sides stays unqualified. A choice of side would change the predicate.normalize_colsrejects that reference.An output name that changes is copied to the parent. An aggregate field
sum(__datafusion_extracted_1)becomessum(alias.__datafusion_extracted_1). The aggregate expression keeps that qualified name as its field name. Pass 1 already keeps the user-facing name with a recovery projection. A rootWindowhas no such wrapper. This pass restores the root schema only.A column reorder is not a rename. The old name is still in the new schema. The rule matches a rename by the name string. Grouping sets, mixed grouping sets, and window arguments use this same path.
What is the testing strategy for this PR?
roundtrip_aggregate_extracted_column_under_subquery_aliasruns the issue SQL throughlogical_plan_to_bytesandlogical_plan_from_bytes. The schema and the debug plan stay the same.roundtrip_join_same_extracted_name_on_both_sideschecks the per-side join keys and the unqualified filter after a proto round trip.test_grouping_set_qualifies_extracted_columnandtest_mixed_grouping_set_qualifies_extracted_columncheck single and mixed grouping sets.test_window_qualifies_extracted_column_under_subquery_aliaschecks a window above a subquery alias.test_join_qualifies_same_extracted_name_per_sidechecks the join. Each aggregate and window test rebuilds the node withnormalize_colsand requires the same output names.projection_pushdown.sltruns the changed IN-subquery filters as a derived table. The count is 3.ORDER BY value DESC LIMIT 1returns id 5.cargo test -p datafusion-optimizer --libpassed (953 tests).cargo test -p datafusion-sqllogictest --test sqllogictestspassed (537 files).cargo test -p datafusion-proto --all-targetspassed (18 library tests and 305 integration tests).cargo test -p datafusion-proto --docpassed (4 tests).ci/scripts/rust_clippy.shpassed.cargo fmt -p datafusion-optimizer -p datafusion-proto -- --checkpassed.Are there any user-facing changes?
Explain text shows the qualifier on an extracted column above a subquery alias. An aggregate field uses that qualified name. There is no public API change.
ExtractLeafExpressionsstill leaves window arguments in place. A window that already reads an extracted column is qualified by this pass.