Skip to content

Commit 6e69fe9

Browse files
zhuqi-lucasalambclaude
authored
fix(physical-optimizer): decline the WindowTopN rewrite when the FilterExec carries a fetch (apache#23599)
# Which issue does this PR close? - Follow-up to apache#21479 (WindowTopN optimizer rule). - Leaves apache#21596 (embedded projections) open — see the note at the bottom. # Rationale for this change `WindowTopN` replaces the matched `FilterExec` outright, so anything the filter carries beyond its predicate has to be reproduced in the new plan, or the rule has to decline. `FilterExec::fetch` is applied by `FilterExec::execute` *after* the predicate. The rewritten plan has nowhere to put it: a `PartitionedTopKExec` bounds rows **per partition**, which is not the same as a row limit over the filtered output. Today the fetch is dropped on the floor: ``` FilterExec: row_number@2 <= 3, fetch=1 BoundedWindowAggExec: ... PlaceholderRowExec ``` is rewritten to ``` BoundedWindowAggExec: ... PartitionedTopKExec: fn=row_number, fetch=3, partition=[pk@0], order=[val@1 ASC] PlaceholderRowExec ``` — up to 3 rows *per partition* instead of 1 row overall. **This is not reachable from the default rule list**, which runs `LimitPushdown` (position 162) well after `WindowTopN` (position 117), so no stock plan reaches the rule with a fetch set. The guard is cheap insurance for pipelines that reorder the two, and it costs one `is_some()` check rather than a new code path. # What changes are included in this PR? - `WindowTopN::try_transform` declines when `filter.fetch().is_some()`, with a comment recording why the fetch cannot be carried and that the default order makes this unreachable today. # Are these changes tested? Yes — `filter_with_fetch_is_declined` asserts the plan is left alone, and first asserts that the *same fixture without the fetch* is rewritten, so the test cannot pass because the shape was wrong. Removing the guard fails it with the plan diff shown above. # Are there any user-facing changes? No. No stock plan reaches this path; the rule declines in one more case for pipelines that do. --- ### Note on the original scope This PR previously also taught `WindowTopN` to carry an embedded `FilterExec` projection through the rewrite (apache#21596). Per the discussion below with @2010YOUY01, that part is dropped: it is real logic rather than a bail-out, and under the default rule order `ProjectionPushdown` (142) runs *after* `WindowTopN` (117), so no e2e test can keep it honest. apache#21596's premise — "this happens when `ProjectionPushdown` runs before `WindowTopN`" — does not hold for the default list, and I have noted that on the issue so the next attempt starts from the right assumption. Co-authored-by: Andrew Lamb <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]>
1 parent bcd205e commit 6e69fe9

2 files changed

Lines changed: 45 additions & 0 deletions

File tree

‎datafusion/core/tests/physical_optimizer/window_topn.rs‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -775,6 +775,36 @@ async fn partitioned_topk_exec_exposes_metrics() -> Result<()> {
775775
Ok(())
776776
}
777777

778+
/// `FilterExec::fetch` is applied after the predicate, and the rewritten plan
779+
/// has nowhere to carry it — a `PartitionedTopKExec` bounds rows *per
780+
/// partition*, which is a different thing. The rule must decline rather than
781+
/// return more rows than were asked for.
782+
///
783+
/// Not reachable from the default rule list, which runs `LimitPushdown` after
784+
/// `WindowTopN`; this pins the behaviour for pipelines that reorder the two.
785+
#[test]
786+
fn filter_with_fetch_is_declined() -> Result<()> {
787+
let filter = build_window_topn_plan(3, Operator::LtEq)?;
788+
789+
// Non-vacuity: without the fetch, this very shape is rewritten.
790+
assert!(
791+
find_partitioned_topk(&optimize(Arc::clone(&filter))?).is_some(),
792+
"the fixture must be a shape the rule rewrites, or the assertion below \
793+
would hold for the wrong reason"
794+
);
795+
796+
let with_fetch = filter
797+
.with_fetch(Some(1))
798+
.expect("FilterExec accepts a fetch");
799+
let optimized = optimize(Arc::clone(&with_fetch))?;
800+
assert_eq!(
801+
plan_str(optimized.as_ref()),
802+
plan_str(with_fetch.as_ref()),
803+
"a FilterExec carrying a fetch must be left alone"
804+
);
805+
Ok(())
806+
}
807+
778808
/// The operator emits the `ROW_NUMBER()` column itself, so the plan holds no
779809
/// `BoundedWindowAggExec` at all — and the column's values still restart at 1
780810
/// in every partition, across several execution partitions and with ties in

‎datafusion/physical-optimizer/src/window_topn.rs‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -162,6 +162,21 @@ impl WindowTopN {
162162
return None;
163163
}
164164

165+
// The rewrite replaces the `FilterExec` entirely, so anything it
166+
// carries beyond the predicate has to be reproduced or declined.
167+
// `fetch` is applied by `FilterExec::execute` *after* the predicate,
168+
// and the rewritten plan has nowhere to put it: a
169+
// `PartitionedTopKExec` bounds rows per partition, which is not the
170+
// same as a row limit over the filtered output. Declining keeps the
171+
// rule from silently returning more rows than were asked for.
172+
//
173+
// The default rule list runs `LimitPushdown` after this rule, so no
174+
// plan reaches here with a fetch today. The guard is cheap insurance
175+
// for pipelines that reorder the two.
176+
if filter.fetch().is_some() {
177+
return None;
178+
}
179+
165180
// Step 2: Extract limit from predicate (rn <= K, rn < K, etc.)
166181
let (col_idx, limit_n) = extract_window_limit(filter.predicate())?;
167182

0 commit comments

Comments
 (0)