You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit a218140
Browse filesBrowse the repository at this point in the historyBrowse files
refactor: moving WindowTopN before EnsureRequirements (apache#24191)
## Which issue does this PR close?
- Closesapache#21594
## Rationale for this change
The WindowTopN physical optimizer rule currently runs after
EnsureRequirements, which means it must pattern-match through SortExec
nodes that EnsureRequirements inserts:
```
FilterExec(rn <= K)
[optional ProjectionExec]
BoundedWindowAggExec(ROW_NUMBER PARTITION BY ... ORDER BY ...)
SortExec(partition_keys, order_keys) ← inserted by EnforceSorting
```
By moving WindowTopN to run before EnsureRequirements, we can simplify
the logic to avoid pattern-matching through `SortExec` node and instead
use the `ORDER BY` and `PARTITION BY` on the `BoundedWindowAggExec`
## What changes are included in this PR?
Moved `WindowTopN` to run before `EnsureRequirements`, and updated its
logic to not expect a `SortExec` and instead use the `ORDER BY` and
`PARTITION BY` on the `BoundedWindowAggExec`
## Are these changes tested?
Yes
## Are there any user-facing changes?
These are just optimizer changes (order of optimizations and internal
logic of `WindowTopN`), no API changes
Copy file name to clipboardExpand all lines: datafusion/core/src/optimizer_rule_reference.md
+4-4Lines changed: 4 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -75,10 +75,10 @@ in multiple phases.
75
75
| 3 |`join_selection`| - | Chooses join implementation, build side, and partition mode from statistics and stream properties. |
76
76
| 4 |`LimitedDistinctAggregation`| - | Pushes limit hints into grouped distinct-style aggregations when only a small result is needed. |
77
77
| 5 |`FilterPushdown`| pre-optimization phase | Pushes supported physical filters down toward data sources before distribution and sorting are enforced. |
78
-
| 6 |`EnsureRequirements`| -| Enforces both distribution and sorting requirements in a single idempotent rule. |
79
-
| 7 |`CombinePartialFinalAggregate`| - |Collapses adjacent partial and final aggregates when the distributed shape makes them redundant.|
80
-
| 8 |`OptimizeAggregateOrder`| - |Updates aggregate expressions to use the best ordering once sort requirements are known. |
81
-
| 9 |`WindowTopN`| - |Replaces eligible row-number window and filter patterns with per-partition TopK execution.|
78
+
| 6 |`WindowTopN`| -| Replaces eligible row-number window and filter patterns with per-partition TopK execution.|
79
+
| 7 |`EnsureRequirements`| - |Enforces both distribution and sorting requirements in a single idempotent rule. |
80
+
| 8 |`CombinePartialFinalAggregate`| - |Collapses adjacent partial and final aggregates when the distributed shape makes them redundant.|
81
+
| 9 |`OptimizeAggregateOrder`| - |Updates aggregate expressions to use the best ordering once sort requirements are known. |
82
82
| 10 |`ProjectionPushdown`| early pass | Pushes projections toward inputs before later physical rewrites add more limit and TopK structure. |
83
83
| 11 |`OutputRequirements`| remove phase | Removes the temporary output-requirement helper nodes after requirement-sensitive planning is done. |
84
84
| 12 |`LimitAggregation`| - | Passes a limit hint into eligible aggregations so they can keep fewer accumulator buckets. |
// with Window(ROW_NUMBER) → PartitionedTopKExec(fetch=K).
137
-
// Must run after EnsureRequirements (which inserts SortExec) and before
138
-
// ProjectionPushdown (which embeds projections into FilterExec).
139
-
Arc::new(WindowTopN::new()),
140
141
// TODO: `try_embed_to_hash_join` in the ProjectionPushdown rule would be block by the CoalesceBatches, so add it before CoalesceBatches. Maybe optimize it in the future.
141
142
Arc::new(ProjectionPushdown::new()),
142
143
// Remove the ancillary output requirement operator since we are done with the planning
0 commit comments