Repository navigation
fix: keep a projection's field names when decorrelating a predicate s… - #26657
jayzhan211 merged 6 commits into
Conversation
…ubquery
A set comparison in a projection without an alias failed with
Internal error: Assertion failed: compatible: Failed due to a difference in schemas:
original schema: ... Field { name: "Int64(2) > ALL", data_type: Boolean, nullable: true }
new schema: ... Field { name: "CASE WHEN EXISTS THEN Boolean(false) ...", ... }
This rule rebuilds a projection from its expressions, and it preserved the output field
names from `Expr::schema_name()` of the expression it was given. That is not the name the
rebuilt projection uses, for two reasons: `schema_name()` keeps a column's qualifier
(`t.k`) while the field is named after the bare column, and earlier rules rewrite
expressions with `map_expressions`, which keeps the schema but can leave an expression
whose display name no longer matches its field name. `RewriteSetComparison` does exactly
that -- it replaces `x > ALL (..)` with a `CASE WHEN EXISTS ..` -- so this rule rebuilt the
projection under the stale name and the optimizer's "the schema is invariant under
optimization" check failed. `datafusion.optimizer.skip_failed_rules` defaults to false, so
an ordinary query returned an internal error asking the user to file a bug report.
Take the field names from the projection's own schema, and compare them against the name
the rebuilt projection would give the expression -- the bare column name for a column, the
alias name for an alias, the display name otherwise -- so an alias is only added when the
rebuilt name would really differ. Comparing against the schema name alone would alias a
qualified column and lose its qualifier, which breaks the same check from the other side.
Tested with three cases in datafusion/sqllogictest/test_files/subquery.slt, and with
subquery.slt, null_aware_mark_join.slt, prepare.slt and cargo test -p datafusion-optimizer.
…projection-field-name Only subquery.slt conflicted: upstream and this branch both appended tests at the end of the file. Kept upstream's new cases and appended this branch's three cases after them.
|
Verified: fails on main exactly as you describe, returns Taking the name from the projection's schema is right — Also checked the subquery.slt + 948 optimizer tests green locally. |
…projection-field-name Only subquery.slt conflicted: upstream (apache#25258) and this branch both appended tests at the end of the file. Kept upstream's new cases and left this branch's three cases above them.
namanjain24-sudo
left a comment
There was a problem hiding this comment.
Checked this out and reverted decorrelate_predicate_subquery.rs to main: the two unaliased cases then fail with Optimizer rule 'decorrelate_predicate_subquery' failed, and pass with the change. The as aliased case passes either way, so it reads as a control rather than a regression test — worth a word in the comment saying so.
One suggestion: projected_field_name duplicates Expr::qualified_name() in datafusion/expr/src/expr.rs, which has the same three arms (Column → bare name, Alias → alias name, otherwise schema_name()) and is documented for exactly this case — "Used when the expression forms the output field of a certain plan. The result is the field's qualifier and field name in the plan's output schema."
Dropping the helper and using rewritten_expr.qualified_name().1 instead keeps 951 datafusion-optimizer tests and subquery.slt green locally, and avoids a second copy of the field-naming rule that can drift from the first.
Minor: projection.expr is paired with field_names[index] positionally; a .zip() over the two would make the pairing structural.
Also the description still carries the authoring scaffolding ("Post the issue first…", "Paste from the horizontal rule down", and a bare Closes #) — worth tidying before a maintainer reads it.
`Expr::qualified_name` implements and documents exactly the three arms the local helper had, so the helper was a second copy of the field-naming rule that a later change to either one could drift from. Zip the projection's expressions with its schema's field names so the pairing is structural instead of positional, and say in subquery.slt that the aliased case is a control rather than a regression test. Verified on the branch merged with main: subquery.slt, subquery_projection.slt, null_aware_anti_join.slt, null_aware_mark_join.slt and prepare.slt all exit 0, and cargo test -p datafusion-optimizer is 951 + 29 + 5 passed, 0 failed.
raghavyadav01
left a comment
There was a problem hiding this comment.
Looks right now. Taking the name from projection.schema and comparing against qualified_name().1 is the correct pair — schema_name() was never the field name.
The comment is the part worth keeping: it explains why naming from the expression is the tempting wrong answer, which isn't obvious from the diff. LGTM.
|
Fixed in 12a42d9, verified — thanks, anything else? |
|
All three are in — One thing left: the description still carries the authoring scaffolding at the top ("Post the issue first…", "Closes # — if no issue was filed"). Worth tidying before a maintainer reads it. |
raghavyadav01
left a comment
There was a problem hiding this comment.
Nothing else from me, all three points are in 12a42d9.
One thing left: the description still has the authoring scaffolding in it ("Post the issue first…", "Paste from the horizontal rule down", and a bare Closes #). Worth tidying, it is the first thing a committer reads.
Needs a committer from here.
|
Description tidied as you both asked — thanks. |
|
@jayzhan211 would you take a look? Review points are in |
jayzhan211
left a comment
There was a problem hiding this comment.
@SingletonYe thanks for tracking this down and for the clear repro! The stale name comes from RewriteSetComparison, so could you move the fix there? Details and a tested diff are inline.
| // The projection's own schema is the authority for its output field names. | ||
| // Earlier rules rewrite expressions with `map_expressions`, which keeps the | ||
| // schema but can leave an expression whose display name no longer matches its | ||
| // field name (`RewriteSetComparison` turns `x > ALL (…)` into a `CASE` and does |
There was a problem hiding this comment.
Since RewriteSetComparison is what leaves the stale name, could you fix it there with NamePreserver, the same way simplify_exprs and type_coercion wrap map_expressions? With this, the change to this file isn't needed: on main's version of this file plus the diff below, your new slt cases, subquery*.slt, merge_into.slt, null_aware_mark_join.slt and the optimizer unit tests all pass. It also stops this rule leaving mismatched names in Aggregate/Window nodes.
+use datafusion_expr::expr_rewriter::NamePreserver;
...
+ let name_preserver = NamePreserver::new(&plan);
plan.map_expressions(|expr| {
+ let original_name = name_preserver.save(&expr);
expr.transform_up(|expr| rewrite_set_comparison(expr, &schema))
+ .map(|t| t.update_data(|expr| original_name.restore(expr)))
})| true | ||
|
|
||
| query BI | ||
| select 2 > all (select v from (values (1), (0)) t(v)), 9 as k; |
There was a problem hiding this comment.
Optional: could you add a case with a qualified column next to the set comparison? It fails on main with the same internal error and passes here:
statement ok
create table t_qual(k int) as values (1), (2);
query IB
select t_qual.k, t_qual.k > all (select v from (values (1), (0)) s(v)) from t_qual order by t_qual.k;
----
1 false
2 true
statement ok
drop table t_qual;`RewriteSetComparison` replaces `x > ALL (…)` with a `CASE WHEN EXISTS …`, whose display name is not the name of the field it produces. Rewriting an expression without preserving its name therefore changes the output schema, and the optimizer's "the schema is invariant under optimization" check failed with an internal error instead of returning the answer. Wrapping `map_expressions` in a `NamePreserver`, the same way `simplify_exprs` and `type_coercion` do, fixes it at the rule that introduces the stale name. That replaces the earlier fix in `DecorrelatePredicateSubquery`, which only covered projections; this also stops the rule leaving mismatched names inside `Aggregate` and `Window` nodes. Adds a fourth sqllogictest case with a qualified column next to the set comparison, which is the other half of the same check. Verified: subquery.slt (all four cases), subquery_projection.slt, subquery_sort.slt, merge_into.slt and null_aware_mark_join.slt pass, and `cargo test -p datafusion-optimizer` is 951 + 29 + 5 passed, 0 failed.
|
Thanks — moved it to the rule that leaves the name.
Added your qualified-column case as a fourth Verified on the branch: |
|
Thanks @SingletonYe @namanjain24-sudo and @raghavyadav01 ! |
Which issue does this PR close?
No issue was filed; this body carries the report.
Rationale for this change
A set comparison in a projection that has no alias fails with an internal error:
DecorrelatePredicateSubqueryrebuilds a projection from its expressions, and it preserved the outputfield names from
Expr::schema_name(). That is not the name the rebuilt projection uses, for tworeasons:
schema_name()keeps a column's qualifier (t.k) while the field is named after the barecolumn, and earlier rules rewrite expressions with
map_expressions, which keeps the schema but canleave an expression whose display name no longer matches its field name.
RewriteSetComparisondoesexactly that — it replaces
x > ALL (…)with aCASE WHEN EXISTS …— so this rule rebuilt theprojection under the stale name and the optimizer's schema-invariance check failed.
datafusion.optimizer.skip_failed_rulesdefaults to false, so the user sees the internal error.Not on the release line: 55.2.0 does not contain #24972, so the same query still reports
Physical plan does not support logical expression Exists(…)there.What changes are included in this PR?
datafusion/optimizer/src/decorrelate_predicate_subquery.rs: take the field names from theprojection's own schema, and compare them against the name the rebuilt projection would use
(
Expr::qualified_name, i.e. the bare column name for a column, the alias name for an alias, thedisplay name otherwise), so an alias is added only when it would really differ. Comparing against
the schema name alone would alias a qualified column and lose its qualifier — the same check
failing from the other side.
datafusion/sqllogictest/test_files/subquery.slt: three cases — unaliased, unaliased with a secondprojected column, and aliased. The aliased one is a control rather than a regression test: it
passes on
maintoo, and pins the alias arm of the name comparison.What is the testing strategy for this PR?
The two unaliased
.sltcases fail onmainwithOptimizer rule 'decorrelate_predicate_subquery' failedand pass with the change; the aliased case is the control described above. On the branchmerged with
main,subquery.slt,subquery_projection.slt,null_aware_anti_join.slt,null_aware_mark_join.sltandprepare.sltall pass with 0 errors,cargo test -p datafusion-optimizeris 951 + 29 + 5 passed and 0 failed, andcargo fmt --all --checkis clean.Are there any user-facing changes?
No.