You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit e123e6f
Browse filesBrowse the repository at this point in the historyBrowse files
Allow Range partitioned inputs to PartitionedTopK (apache#23355)
## Which issue does this PR close?
- Closesapache#23290.
## What changes are included in this PR?
In two commits:
* Improve the robustness of `WindowTopN` to intermediate
`RepartitionExec` nodes, to ensure that `PartitionedTopK` can still be
used when `target_partitions` mismatches physical partitions.
* Use `required_input_distributions` to declare required partitioning
for `PartitionedTopKExec`, `BoundedWindowAggExec`, and `WindowAggExec`.
## Are these changes tested?
Yes, new `slt` tests are added for both the `WindowTopN` robustness fix,
and for the use of `required_input_distributions` in `PartitionedTopK`.
## Are there any user-facing changes?
No: only plan changes.
Copy file name to clipboardExpand all lines: datafusion/sqllogictest/test_files/range_partitioning.slt
+157Lines changed: 157 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -907,5 +907,162 @@ SELECT range_key, SUM(value) OVER (ORDER BY value) FROM range_partitioned ORDER
907
907
30 1060
908
908
35 1410
909
909
910
+
911
+
912
+
##########
913
+
# TEST 26: PartitionedTopK on Range Partition Column
914
+
# Exact Range([range_key]) satisfies the TopK partition key and avoids repartitioning.
915
+
##########
916
+
917
+
statement ok
918
+
set datafusion.optimizer.enable_window_topn = true;
919
+
920
+
statement ok
921
+
set datafusion.optimizer.subset_repartition_threshold = 4;
922
+
923
+
statement ok
924
+
set datafusion.optimizer.preserve_file_partitions = 0;
925
+
926
+
query TT
927
+
EXPLAIN SELECT * FROM (
928
+
SELECT range_key, value, ROW_NUMBER() OVER (PARTITION BY range_key ORDER BY value DESC) as rn
929
+
FROM range_partitioned
930
+
) WHERE rn <= 1;
931
+
----
932
+
physical_plan
933
+
01)ProjectionExec: expr=[range_key@0 as range_key, value@1 as value, row_number() PARTITION BY [range_partitioned.range_key] ORDER BY [range_partitioned.value DESC NULLS FIRST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW@2 as rn]
934
+
02)--BoundedWindowAggExec: wdw=[row_number() PARTITION BY [range_partitioned.range_key] ORDER BY [range_partitioned.value DESC NULLS FIRST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW: Field { "row_number() PARTITION BY [range_partitioned.range_key] ORDER BY [range_partitioned.value DESC NULLS FIRST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW": UInt64 }, frame: RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW], mode=[Sorted]
SELECT range_key, value, ROW_NUMBER() OVER (PARTITION BY range_key ORDER BY value DESC) as rn
941
+
FROM range_partitioned
942
+
) WHERE rn <= 1
943
+
ORDER BY range_key;
944
+
----
945
+
1 10 1
946
+
5 50 1
947
+
10 100 1
948
+
15 150 1
949
+
20 200 1
950
+
25 250 1
951
+
30 300 1
952
+
35 350 1
953
+
954
+
955
+
##########
956
+
# TEST 27: PartitionedTopK on Non-Range Column
957
+
# Partitioning on a non-range key cannot reuse Range([range_key]) and
958
+
# requires hash repartitioning.
959
+
##########
960
+
961
+
statement ok
962
+
set datafusion.optimizer.subset_repartition_threshold = 4;
963
+
964
+
statement ok
965
+
set datafusion.optimizer.preserve_file_partitions = 0;
966
+
967
+
query TT
968
+
EXPLAIN SELECT * FROM (
969
+
SELECT non_range_key, value, ROW_NUMBER() OVER (PARTITION BY non_range_key ORDER BY value DESC) as rn
970
+
FROM range_partitioned
971
+
) WHERE rn <= 1;
972
+
----
973
+
physical_plan
974
+
01)ProjectionExec: expr=[non_range_key@0 as non_range_key, value@1 as value, row_number() PARTITION BY [range_partitioned.non_range_key] ORDER BY [range_partitioned.value DESC NULLS FIRST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW@2 as rn]
975
+
02)--BoundedWindowAggExec: wdw=[row_number() PARTITION BY [range_partitioned.non_range_key] ORDER BY [range_partitioned.value DESC NULLS FIRST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW: Field { "row_number() PARTITION BY [range_partitioned.non_range_key] ORDER BY [range_partitioned.value DESC NULLS FIRST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW": UInt64 }, frame: RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW], mode=[Sorted]
SELECT non_range_key, value, ROW_NUMBER() OVER (PARTITION BY non_range_key ORDER BY value DESC) as rn
983
+
FROM range_partitioned
984
+
) WHERE rn <= 1
985
+
ORDER BY non_range_key;
986
+
----
987
+
1 300 1
988
+
2 350 1
989
+
990
+
991
+
##########
992
+
# TEST 28: PartitionedTopK Reuses Range Subset Partitioning
993
+
# With subset threshold met and preserve-file disabled, Range([range_key])
994
+
# satisfies partitioning by (range_key, non_range_key).
995
+
##########
996
+
997
+
statement ok
998
+
set datafusion.optimizer.subset_repartition_threshold = 4;
999
+
1000
+
statement ok
1001
+
set datafusion.optimizer.preserve_file_partitions = 0;
1002
+
1003
+
query TT
1004
+
EXPLAIN SELECT * FROM (
1005
+
SELECT range_key, non_range_key, value, ROW_NUMBER() OVER (PARTITION BY range_key, non_range_key ORDER BY value DESC) as rn
1006
+
FROM range_partitioned
1007
+
) WHERE rn <= 1;
1008
+
----
1009
+
physical_plan
1010
+
01)ProjectionExec: expr=[range_key@0 as range_key, non_range_key@1 as non_range_key, value@2 as value, row_number() PARTITION BY [range_partitioned.range_key, range_partitioned.non_range_key] ORDER BY [range_partitioned.value DESC NULLS FIRST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW@3 as rn]
1011
+
02)--BoundedWindowAggExec: wdw=[row_number() PARTITION BY [range_partitioned.range_key, range_partitioned.non_range_key] ORDER BY [range_partitioned.value DESC NULLS FIRST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW: Field { "row_number() PARTITION BY [range_partitioned.range_key, range_partitioned.non_range_key] ORDER BY [range_partitioned.value DESC NULLS FIRST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW": UInt64 }, frame: RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW], mode=[Sorted]
SELECT range_key, non_range_key, value, ROW_NUMBER() OVER (PARTITION BY range_key, non_range_key ORDER BY value DESC) as rn
1018
+
FROM range_partitioned
1019
+
) WHERE rn <= 1
1020
+
ORDER BY range_key, non_range_key;
1021
+
----
1022
+
1 1 10 1
1023
+
5 2 50 1
1024
+
10 1 100 1
1025
+
15 2 150 1
1026
+
20 1 200 1
1027
+
25 2 250 1
1028
+
30 1 300 1
1029
+
35 2 350 1
1030
+
1031
+
1032
+
##########
1033
+
# TEST 29: Range Subset PartitionedTopK Rehashes Below Subset Threshold
1034
+
# Range([range_key]) is only a subset of PARTITION BY (range_key, non_range_key),
1035
+
# so it should not satisfy the TopK partition key when subset satisfaction is
1036
+
# disabled.
1037
+
##########
1038
+
1039
+
statement ok
1040
+
set datafusion.execution.target_partitions = 4;
1041
+
1042
+
statement ok
1043
+
set datafusion.optimizer.subset_repartition_threshold = 5;
1044
+
1045
+
statement ok
1046
+
set datafusion.optimizer.preserve_file_partitions = 0;
1047
+
1048
+
query TT
1049
+
EXPLAIN SELECT * FROM (
1050
+
SELECT range_key, non_range_key, value, ROW_NUMBER() OVER (PARTITION BY range_key, non_range_key ORDER BY value DESC) as rn
1051
+
FROM range_partitioned
1052
+
) WHERE rn <= 1;
1053
+
----
1054
+
physical_plan
1055
+
01)ProjectionExec: expr=[range_key@0 as range_key, non_range_key@1 as non_range_key, value@2 as value, row_number() PARTITION BY [range_partitioned.range_key, range_partitioned.non_range_key] ORDER BY [range_partitioned.value DESC NULLS FIRST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW@3 as rn]
1056
+
02)--BoundedWindowAggExec: wdw=[row_number() PARTITION BY [range_partitioned.range_key, range_partitioned.non_range_key] ORDER BY [range_partitioned.value DESC NULLS FIRST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW: Field { "row_number() PARTITION BY [range_partitioned.range_key, range_partitioned.non_range_key] ORDER BY [range_partitioned.value DESC NULLS FIRST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW": UInt64 }, frame: RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW], mode=[Sorted]
set datafusion.optimizer.repartition_windows = false;
1056
+
1057
+
statement ok
1058
+
set datafusion.execution.batch_size = 1;
1059
+
1060
+
statement ok
1061
+
set datafusion.optimizer.enable_window_topn = true;
1062
+
1063
+
query TT
1064
+
EXPLAIN SELECT * FROM (
1065
+
SELECT c1, c2, ROW_NUMBER() OVER (PARTITION BY c1 ORDER BY c2 DESC) as rn
1066
+
FROM t
1067
+
) WHERE rn <= 1;
1068
+
----
1069
+
logical_plan
1070
+
01)Projection: t.c1, t.c2, row_number() PARTITION BY [t.c1] ORDER BY [t.c2 DESC NULLS FIRST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW AS rn
1071
+
02)--Filter: row_number() PARTITION BY [t.c1] ORDER BY [t.c2 DESC NULLS FIRST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW <= UInt64(1)
1072
+
03)----WindowAggr: windowExpr=[[row_number() PARTITION BY [t.c1] ORDER BY [t.c2 DESC NULLS FIRST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW]]
1073
+
04)------TableScan: t projection=[c1, c2]
1074
+
physical_plan
1075
+
01)ProjectionExec: expr=[c1@0 as c1, c2@1 as c2, row_number() PARTITION BY [t.c1] ORDER BY [t.c2 DESC NULLS FIRST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW@2 as rn]
03)----BoundedWindowAggExec: wdw=[row_number() PARTITION BY [t.c1] ORDER BY [t.c2 DESC NULLS FIRST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW: Field { "row_number() PARTITION BY [t.c1] ORDER BY [t.c2 DESC NULLS FIRST] RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW": UInt64 }, frame: RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW], mode=[Sorted]
0 commit comments