You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 32bd81f
Browse filesBrowse the repository at this point in the historyBrowse files
fix: preserve window inputs when replacing TopK sorts (apache#26719)
## Which issue does this PR close?
Closesapache#26617.
## Rationale for this change
With multiple execution partitions, replacing an outer TopK sort with an
ordered merge can push its limit below a window that needs all input
rows.
The reproducer returns `count(*) OVER () = 1` instead of 3.
## What changes are included in this PR?
- Only propagate fetch through operators that support limit pushdown.
- Retain the original limit above the replacement plan when removing the
sort.
- Add SQL regressions for `LIMIT 1` and `LIMIT 1 OFFSET 1` with four
target
partitions.
## What is the testing strategy for this PR?
The new cases in `window_limits.slt` expect `(3, 1)` and `(3, 2)`.
Before the fix, they return `(1, 1)` and `(2, 2)`.
## Are there any user-facing changes?
None other than user gets correct window results when an outer TopK sort
is replaced with an
ordered merge.
Copy file name to clipboardExpand all lines: datafusion/physical-optimizer/src/ensure_requirements/enforce_sorting/replace_with_order_preserving_variants.rs
Copy file name to clipboardExpand all lines: datafusion/sqllogictest/test_files/window_limits.slt
+89Lines changed: 89 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -898,6 +898,47 @@ physical_plan
898
898
statement ok
899
899
DROP TABLE huge_frame_t;
900
900
901
+
# Regression for #26617: count all rows before applying LIMIT.
902
+
statement ok
903
+
SET datafusion.execution.target_partitions = 4;
904
+
905
+
query II
906
+
SELECT count(*) OVER () AS n,
907
+
row_number() OVER (PARTITION BY length('x') ORDER BY a) AS r
908
+
FROM (SELECT 1 AS a UNION ALL SELECT 2 UNION ALL SELECT 3)
909
+
ORDER BY r LIMIT 1;
910
+
----
911
+
3 1
912
+
913
+
query II
914
+
SELECT count(*) OVER () AS n,
915
+
row_number() OVER (PARTITION BY length('x') ORDER BY a) AS r
916
+
FROM (SELECT 1 AS a UNION ALL SELECT 2 UNION ALL SELECT 3)
917
+
ORDER BY r LIMIT 1 OFFSET 1;
918
+
----
919
+
3 2
920
+
921
+
# When replacing the outer TopK is safe, LimitPushdown must absorb the
922
+
# redundant GlobalLimitExec into SortPreservingMergeExec.
923
+
query TT
924
+
EXPLAIN
925
+
SELECT depname, row_number() OVER (PARTITION BY depname ORDER BY empno) AS r
926
+
FROM employees
927
+
ORDER BY depname LIMIT 1;
928
+
----
929
+
logical_plan
930
+
01)Sort: employees.depname ASC NULLS LAST, fetch=1
931
+
02)--Projection: employees.depname, row_number() PARTITION BY [employees.depname] ORDER BY [employees.empno ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW AS r
932
+
03)----WindowAggr: windowExpr=[[row_number() PARTITION BY [employees.depname] ORDER BY [employees.empno ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW]]
02)--ProjectionExec: expr=[depname@0 as depname, row_number() PARTITION BY [employees.depname] ORDER BY [employees.empno ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW@2 as r]
937
+
03)----BoundedWindowAggExec: wdw=[row_number() PARTITION BY [employees.depname] ORDER BY [employees.empno ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW: Field { "row_number() PARTITION BY [employees.depname] ORDER BY [employees.empno ASC NULLS LAST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW": UInt64 }, frame: RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW], mode=[Sorted]
938
+
04)------SortExec: expr=[depname@0 ASC NULLS LAST, empno@1 ASC NULLS LAST], preserve_partitioning=[true]
0 commit comments