Skip to content

fix: WITH ... AS MATERIALIZED returns wrong rows under ORDER BY ... LIMIT - #27416

Merged
adriangb merged 2 commits into
apache:mainfrom
jayzhan211:fix/27219-topk-agg-multi-child
Nov 14, 2026
Merged

adriangb merged 2 commits into
apache:mainfrom
jayzhan211:fix/27219-topk-agg-multi-child

Conversation

@jayzhan211

@jayzhan211 jayzhan211 commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

When a query declares a MATERIALIZED CTE whose body groups by the sort column, ORDER BY ... LIMIT n above that query returns wrong rows. The query below returns no rows instead of 3:

SET datafusion.sql_parser.dialect = 'PostgreSQL';  -- needed to parse MATERIALIZED
CREATE TABLE t AS SELECT value AS x FROM generate_series(1, 200);

SELECT * FROM (
  WITH c AS MATERIALIZED (SELECT x FROM t GROUP BY x)
  SELECT x FROM c WHERE x > 100
) s ORDER BY x LIMIT 3;
Query main This PR
Above no rows 101, 102, 103
Same CTE, SELECT c.x, m.n FROM c CROSS JOIN (SELECT count(*) AS n FROM c) m n = 3 n = 200

The physical optimizer rule TopKAggregation starts at SortExec: TopK(fetch=3) and walks down through nodes that keep the row count. MaterializedCteExec says it keeps the row count, and its first child is the CTE body. So the walk goes into the body and sets lim=[3] on GROUP BY x. The body runs once, and every reference to the CTE reads its output, so every reference sees only the groups that the limit kept. MaterializedCteExec outputs only its second child, the continuation; the body is never its output. 56.0.0-rc1 has the same bug; 55.2.0 returned the right rows because it ignored MATERIALIZED. This PR targets main only and doesn't propose a backport.

This PR stops the walk at MaterializedCteExec.

What changes are included in this PR?

datafusion/physical-optimizer/src/topk_aggregation.rs: one new branch stops the walk at MaterializedCteExec. cte_materialized.slt: the two queries above.

The decision I'd most like reviewed: the rule names MaterializedCteExec instead of stopping at every node with more than one child. The general rule would also remove a lim that main sets today, on an aggregate that is the first input of a UnionExec, which is a valid limit:

SortExec: TopK(fetch=3), expr=[x@0 ASC NULLS LAST]
  UnionExec
    AggregateExec: mode=FinalPartitioned, gby=[x@0 as x], aggr=[], lim=[3]
    ...
Why the walk doesn't go into the continuation instead

Limiting an aggregate in the continuation would be valid, but it is a new optimization, and it only matters when ORDER BY ... LIMIT is outside the WITH query. When it is inside, the sort is already in the continuation and the rule applies there as before:

MaterializedCteExec: name=c
  DataSourceExec: ...
  SortPreservingMergeExec: [y@0 ASC NULLS LAST], fetch=3
    SortExec: TopK(fetch=3), expr=[y@0 ASC NULLS LAST]
      AggregateExec: mode=FinalPartitioned, gby=[y@0 as y], aggr=[], lim=[3]

Who is affected: queries that use AS MATERIALIZED (new in 56, PostgreSQL dialect only) with ORDER BY + LIMIT above the WITH query and a GROUP BY on the sort column in the CTE body. No other plan changes. No API change.

What is the testing strategy for this PR?

  • The two queries above are added to cte_materialized.slt. Both fail on main and pass with this PR.
  • The full sqllogictest suite passes (306 files) with no other plan changes, and so do the datafusion-physical-optimizer unit tests.
  • I compared the plans of several UNION ALL, scalar subquery, and MATERIALIZED CTE queries before and after this change. Only the plan of the query above changed: its body aggregates no longer have lim=[3].

Are there any user-facing changes?

Yes: these queries now return correct results. There are no API changes.

@github-actions github-actions Bot added optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) labels Nov 14, 2026
@jayzhan211
jayzhan211 marked this pull request as ready for review November 14, 2026 05:29
@jayzhan211
jayzhan211 requested review from adriangb and alamb November 14, 2026 05:29

@adriangb adriangb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you @jayzhan211

@adriangb
adriangb added this pull request to the merge queue Nov 14, 2026
Merged via the queue into apache:main with commit d5e4e70 Nov 14, 2026
44 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) v57.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

WITH ... AS MATERIALIZED returns wrong rows under ORDER BY ... LIMIT: top-k aggregation limits the shared CTE body

2 participants