Repository navigation
fix: WITH ... AS MATERIALIZED returns wrong rows under ORDER BY ... LIMIT - #27416
Merged
Merged
Conversation
jayzhan211
marked this pull request as ready for review
November 14, 2026 05:29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
WITH ... AS MATERIALIZEDreturns wrong rows underORDER BY ... LIMIT: top-k aggregation limits the shared CTE body #27219.Rationale for this change
When a query declares a
MATERIALIZEDCTE whose body groups by the sort column,ORDER BY ... LIMIT nabove that query returns wrong rows. The query below returns no rows instead of 3:SELECT c.x, m.n FROM c CROSS JOIN (SELECT count(*) AS n FROM c) mn= 3n= 200The physical optimizer rule
TopKAggregationstarts atSortExec: TopK(fetch=3)and walks down through nodes that keep the row count.MaterializedCteExecsays it keeps the row count, and its first child is the CTE body. So the walk goes into the body and setslim=[3]onGROUP BY x. The body runs once, and every reference to the CTE reads its output, so every reference sees only the groups that the limit kept.MaterializedCteExecoutputs only its second child, the continuation; the body is never its output. 56.0.0-rc1 has the same bug; 55.2.0 returned the right rows because it ignoredMATERIALIZED. This PR targets main only and doesn't propose a backport.This PR stops the walk at
MaterializedCteExec.What changes are included in this PR?
datafusion/physical-optimizer/src/topk_aggregation.rs: one new branch stops the walk atMaterializedCteExec.cte_materialized.slt: the two queries above.The decision I'd most like reviewed: the rule names
MaterializedCteExecinstead of stopping at every node with more than one child. The general rule would also remove alimthat main sets today, on an aggregate that is the first input of aUnionExec, which is a valid limit:Why the walk doesn't go into the continuation instead
Limiting an aggregate in the continuation would be valid, but it is a new optimization, and it only matters when
ORDER BY ... LIMITis outside theWITHquery. When it is inside, the sort is already in the continuation and the rule applies there as before:Who is affected: queries that use
AS MATERIALIZED(new in 56, PostgreSQL dialect only) withORDER BY+LIMITabove theWITHquery and aGROUP BYon the sort column in the CTE body. No other plan changes. No API change.What is the testing strategy for this PR?
cte_materialized.slt. Both fail on main and pass with this PR.datafusion-physical-optimizerunit tests.UNION ALL, scalar subquery, andMATERIALIZEDCTE queries before and after this change. Only the plan of the query above changed: its body aggregates no longer havelim=[3].Are there any user-facing changes?
Yes: these queries now return correct results. There are no API changes.