Skip to content

fix: keep a projection's field names when decorrelating a predicate s… - #26657

Merged
jayzhan211 merged 6 commits into
apache:mainfrom
SingletonYe:fix/set-comparison-projection-field-name
Nov 1, 2026
Merged

jayzhan211 merged 6 commits into
apache:mainfrom
SingletonYe:fix/set-comparison-projection-field-name

Conversation

@SingletonYe

@SingletonYe SingletonYe commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

No issue was filed; this body carries the report.

Rationale for this change

A set comparison in a projection that has no alias fails with an internal error:

> SELECT 2 > ALL (SELECT x FROM (VALUES (1),(0)) t(x));
Internal error: Assertion failed: compatible: Failed due to a difference in schemas:
original schema: … Field { name: "Int64(2) > ALL", data_type: Boolean } …
new schema:      … Field { name: "CASE WHEN EXISTS THEN Boolean(false) …" } …

DecorrelatePredicateSubquery rebuilds a projection from its expressions, and it preserved the output
field names from Expr::schema_name(). That is not the name the rebuilt projection uses, for two
reasons: schema_name() keeps a column's qualifier (t.k) while the field is named after the bare
column, and earlier rules rewrite expressions with map_expressions, which keeps the schema but can
leave an expression whose display name no longer matches its field name. RewriteSetComparison does
exactly that — it replaces x > ALL (…) with a CASE WHEN EXISTS … — so this rule rebuilt the
projection under the stale name and the optimizer's schema-invariance check failed.
datafusion.optimizer.skip_failed_rules defaults to false, so the user sees the internal error.

Not on the release line: 55.2.0 does not contain #24972, so the same query still reports
Physical plan does not support logical expression Exists(…) there.

What changes are included in this PR?

  • datafusion/optimizer/src/decorrelate_predicate_subquery.rs: take the field names from the
    projection's own schema, and compare them against the name the rebuilt projection would use
    (Expr::qualified_name, i.e. the bare column name for a column, the alias name for an alias, the
    display name otherwise), so an alias is added only when it would really differ. Comparing against
    the schema name alone would alias a qualified column and lose its qualifier — the same check
    failing from the other side.
  • datafusion/sqllogictest/test_files/subquery.slt: three cases — unaliased, unaliased with a second
    projected column, and aliased. The aliased one is a control rather than a regression test: it
    passes on main too, and pins the alias arm of the name comparison.

What is the testing strategy for this PR?

The two unaliased .slt cases fail on main with Optimizer rule 'decorrelate_predicate_subquery' failed and pass with the change; the aliased case is the control described above. On the branch
merged with main, subquery.slt, subquery_projection.slt, null_aware_anti_join.slt,
null_aware_mark_join.slt and prepare.slt all pass with 0 errors, cargo test -p datafusion-optimizer is 951 + 29 + 5 passed and 0 failed, and cargo fmt --all --check is clean.

Are there any user-facing changes?

No.

…ubquery

A set comparison in a projection without an alias failed with

  Internal error: Assertion failed: compatible: Failed due to a difference in schemas:
  original schema: ... Field { name: "Int64(2) > ALL", data_type: Boolean, nullable: true }
  new schema:      ... Field { name: "CASE WHEN EXISTS THEN Boolean(false) ...", ... }

This rule rebuilds a projection from its expressions, and it preserved the output field
names from `Expr::schema_name()` of the expression it was given. That is not the name the
rebuilt projection uses, for two reasons: `schema_name()` keeps a column's qualifier
(`t.k`) while the field is named after the bare column, and earlier rules rewrite
expressions with `map_expressions`, which keeps the schema but can leave an expression
whose display name no longer matches its field name. `RewriteSetComparison` does exactly
that -- it replaces `x > ALL (..)` with a `CASE WHEN EXISTS ..` -- so this rule rebuilt the
projection under the stale name and the optimizer's "the schema is invariant under
optimization" check failed. `datafusion.optimizer.skip_failed_rules` defaults to false, so
an ordinary query returned an internal error asking the user to file a bug report.

Take the field names from the projection's own schema, and compare them against the name
the rebuilt projection would give the expression -- the bare column name for a column, the
alias name for an alias, the display name otherwise -- so an alias is only added when the
rebuilt name would really differ. Comparing against the schema name alone would alias a
qualified column and lose its qualifier, which breaks the same check from the other side.

Tested with three cases in datafusion/sqllogictest/test_files/subquery.slt, and with
subquery.slt, null_aware_mark_join.slt, prepare.slt and cargo test -p datafusion-optimizer.
@github-actions github-actions Bot added optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) labels Oct 23, 2026
…projection-field-name

Only subquery.slt conflicted: upstream and this branch both appended tests at the end of
the file. Kept upstream's new cases and appended this branch's three cases after them.
@raghavyadav01

Copy link
Copy Markdown
Contributor

Verified: fails on main exactly as you describe, returns true with the field name intact on your branch.

Taking the name from the projection's schema is right — schema_name() was never the field name.

Also checked the Expr::Column arm earns its place: delete it and 7 existing subquery.slt tests fail.

subquery.slt + 948 optimizer tests green locally.

…projection-field-name

Only subquery.slt conflicted: upstream (apache#25258) and this branch both appended
tests at the end of the file. Kept upstream's new cases and left this branch's
three cases above them.

@namanjain24-sudo namanjain24-sudo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked this out and reverted decorrelate_predicate_subquery.rs to main: the two unaliased cases then fail with Optimizer rule 'decorrelate_predicate_subquery' failed, and pass with the change. The as aliased case passes either way, so it reads as a control rather than a regression test — worth a word in the comment saying so.

One suggestion: projected_field_name duplicates Expr::qualified_name() in datafusion/expr/src/expr.rs, which has the same three arms (Column → bare name, Alias → alias name, otherwise schema_name()) and is documented for exactly this case — "Used when the expression forms the output field of a certain plan. The result is the field's qualifier and field name in the plan's output schema."

Dropping the helper and using rewritten_expr.qualified_name().1 instead keeps 951 datafusion-optimizer tests and subquery.slt green locally, and avoids a second copy of the field-naming rule that can drift from the first.

Minor: projection.expr is paired with field_names[index] positionally; a .zip() over the two would make the pairing structural.

Also the description still carries the authoring scaffolding ("Post the issue first…", "Paste from the horizontal rule down", and a bare Closes #) — worth tidying before a maintainer reads it.

`Expr::qualified_name` implements and documents exactly the three arms the
local helper had, so the helper was a second copy of the field-naming rule
that a later change to either one could drift from. Zip the projection's
expressions with its schema's field names so the pairing is structural
instead of positional, and say in subquery.slt that the aliased case is a
control rather than a regression test.

Verified on the branch merged with main: subquery.slt, subquery_projection.slt,
null_aware_anti_join.slt, null_aware_mark_join.slt and prepare.slt all exit 0,
and cargo test -p datafusion-optimizer is 951 + 29 + 5 passed, 0 failed.

@raghavyadav01 raghavyadav01 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks right now. Taking the name from projection.schema and comparing against qualified_name().1 is the correct pair — schema_name() was never the field name.

The comment is the part worth keeping: it explains why naming from the expression is the tempting wrong answer, which isn't obvious from the diff. LGTM.

@SingletonYe

Copy link
Copy Markdown
Contributor Author

Fixed in 12a42d9, verified — thanks, anything else?

@namanjain24-sudo

Copy link
Copy Markdown
Contributor

All three are in — qualified_name().1 is the same three arms the helper had, the .zip() makes the pairing structural, and the control comment reads well.

One thing left: the description still carries the authoring scaffolding at the top ("Post the issue first…", "Closes # — if no issue was filed"). Worth tidying before a maintainer reads it.

@raghavyadav01 raghavyadav01 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing else from me, all three points are in 12a42d9.

One thing left: the description still has the authoring scaffolding in it ("Post the issue first…", "Paste from the horizontal rule down", and a bare Closes #). Worth tidying, it is the first thing a committer reads.

Needs a committer from here.

@SingletonYe

Copy link
Copy Markdown
Contributor Author

Description tidied as you both asked — thanks.

@SingletonYe

Copy link
Copy Markdown
Contributor Author

@jayzhan211 would you take a look? Review points are in 12a42d9, the branch is green, and the description is tidied.
thank you

@jayzhan211 jayzhan211 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@SingletonYe thanks for tracking this down and for the clear repro! The stale name comes from RewriteSetComparison, so could you move the fix there? Details and a tested diff are inline.

// The projection's own schema is the authority for its output field names.
// Earlier rules rewrite expressions with `map_expressions`, which keeps the
// schema but can leave an expression whose display name no longer matches its
// field name (`RewriteSetComparison` turns `x > ALL (…)` into a `CASE` and does

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since RewriteSetComparison is what leaves the stale name, could you fix it there with NamePreserver, the same way simplify_exprs and type_coercion wrap map_expressions? With this, the change to this file isn't needed: on main's version of this file plus the diff below, your new slt cases, subquery*.slt, merge_into.slt, null_aware_mark_join.slt and the optimizer unit tests all pass. It also stops this rule leaving mismatched names in Aggregate/Window nodes.

+use datafusion_expr::expr_rewriter::NamePreserver;
 ...
+        let name_preserver = NamePreserver::new(&plan);
         plan.map_expressions(|expr| {
+            let original_name = name_preserver.save(&expr);
             expr.transform_up(|expr| rewrite_set_comparison(expr, &schema))
+                .map(|t| t.update_data(|expr| original_name.restore(expr)))
         })

true

query BI
select 2 > all (select v from (values (1), (0)) t(v)), 9 as k;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: could you add a case with a qualified column next to the set comparison? It fails on main with the same internal error and passes here:

statement ok
create table t_qual(k int) as values (1), (2);

query IB
select t_qual.k, t_qual.k > all (select v from (values (1), (0)) s(v)) from t_qual order by t_qual.k;
----
1 false
2 true

statement ok
drop table t_qual;

`RewriteSetComparison` replaces `x > ALL (…)` with a `CASE WHEN EXISTS …`, whose
display name is not the name of the field it produces. Rewriting an expression
without preserving its name therefore changes the output schema, and the
optimizer's "the schema is invariant under optimization" check failed with an
internal error instead of returning the answer.

Wrapping `map_expressions` in a `NamePreserver`, the same way `simplify_exprs`
and `type_coercion` do, fixes it at the rule that introduces the stale name.
That replaces the earlier fix in `DecorrelatePredicateSubquery`, which only
covered projections; this also stops the rule leaving mismatched names inside
`Aggregate` and `Window` nodes.

Adds a fourth sqllogictest case with a qualified column next to the set
comparison, which is the other half of the same check.

Verified: subquery.slt (all four cases), subquery_projection.slt,
subquery_sort.slt, merge_into.slt and null_aware_mark_join.slt pass, and
`cargo test -p datafusion-optimizer` is 951 + 29 + 5 passed, 0 failed.
@SingletonYe

Copy link
Copy Markdown
Contributor Author

Thanks — moved it to the rule that leaves the name.

rewrite_set_comparison now wraps map_expressions in a NamePreserver, the same way simplify_exprs
and type_coercion do, so the CASE WHEN EXISTS … never carries a stale name out of the rule. I removed
the decorrelate_predicate_subquery.rs change entirely: that file is back to main's version and is no
longer part of the diff. Two files now instead of three.

Added your qualified-column case as a fourth subquery.slt test.

Verified on the branch: subquery.slt (all four cases), subquery_projection.slt, subquery_sort.slt,
merge_into.slt and null_aware_mark_join.slt pass, and cargo test -p datafusion-optimizer is
951 + 29 + 5 passed, 0 failed. That is ed26e85.

@jayzhan211

Copy link
Copy Markdown
Contributor

Thanks @SingletonYe @namanjain24-sudo and @raghavyadav01 !

@jayzhan211
jayzhan211 added this pull request to the merge queue Nov 1, 2026
Merged via the queue into apache:main with commit 123747e Nov 1, 2026
44 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) v56.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants