You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 290ad15
Browse filesBrowse the repository at this point in the historyBrowse files
fix: decorrelate grouping sets that leave out the correlated column when safe (apache#25781)
## Which issue does this PR close?
- Closesapache#25708.
## Rationale for this change
apache#25529 fixed a wrong-results bug where a correlated filter under a
grouping-set `Aggregate` was pulled up into the aggregate's grouping
sets, but the guard it added is more conservative than it needs to be:
it rejects the pull up whenever *any* grouping set omits the correlated
column, even in cases like
```sql
SELECT o.k FROM o
WHERE EXISTS (
SELECT 1 FROM i WHERE i.k = o.k
GROUP BY GROUPING SETS ((i.k), (i.j))
)
```
where nothing above the aggregate ever reads `i.k`, so filling it in
changes nothing the query can observe. These queries now fail to plan
(`This feature is not implemented: ... does not support logical
expression Exists`) instead of decorrelating.
## What changes are included in this PR?
- `datafusion/optimizer/src/decorrelate.rs`:
`grouping_sets_cover_pull_up_cols` becomes
`grouping_sets_pull_up_outcome`, returning a three-way
`GroupingSetsPullUp` (`NoColumnsToAdd` / `SafeToExtend` / `Unsafe`)
instead of a bool. A set that omits a required column is now only
`Unsafe` when:
- some grouping set is empty (`ROLLUP`/`CUBE` always hold one), or
- the aggregate itself computes `GROUPING`/`GROUPING_ID`, or
- a node above the aggregate reads the column, per a new
`column_refs_above_aggregate` set.
- Since `PullUpCorrelatedExpr::f_up` runs bottom-up, it hasn't visited
whatever sits above the `Aggregate` (a `HAVING`, a `Projection`) by the
time it needs to decide. `column_refs_above_aggregate` is instead
computed once, top-down, over the original (not yet rewritten) subquery
plan by a new `columns_read_above_aggregate` helper, and threaded in via
a new `with_column_refs_above_aggregate` builder call at all three call
sites (`scalar_subquery_to_join.rs`,
`decorrelate_predicate_subquery.rs`, `decorrelate_lateral_join.rs`).
- `datafusion/sqllogictest/test_files/subquery.slt`: the `EXISTS` case
above now decorrelates and asserts correct results; added a case showing
a `GROUPING(k)` read still keeps the subquery correlated (correctly),
alongside the existing `HAVING`/empty-set cases that stay correlated.
## What is the testing strategy for this PR?
Extended `datafusion/sqllogictest/test_files/subquery.slt`'s existing
`apache#25519` regression block:
- The previously-rejected `EXISTS ... GROUPING SETS ((i.k), (i.j))` case
is now a `query IB` asserting the correct results.
- Added a case with `GROUPING(k)` read out of the aggregate, asserting
it still correctly stays correlated.
- The `HAVING i.k IS NULL` and every empty-set (`ROLLUP`/`CUBE`/explicit
`()`) case are unchanged and still assert they stay correlated.
Ran the full `datafusion-optimizer` unit + integration suite (905 + 26
tests) and the full sqllogictest suite (523 files), all green.
## Are there any user-facing changes?
More `EXISTS`/`IN`/lateral-join subqueries with a correlated filter
under a grouping-set aggregate now decorrelate into a join instead of
failing to plan. No public API changes.
---------
Co-authored-by: namanjain24-sudo <[email protected]>
0 commit comments