You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 4ca40d9
Browse filesBrowse the repository at this point in the historyBrowse files
feat: enable datafusion.optimizer.enable_window_topn by default (apache#27197)
## Which issue does this PR close?
- Closesapache#24608.
## Rationale for this change
`enable_window_topn` has been off since apache#21479 because of
high-cardinality regressions against the sort plan. The work items in
apache#24608 are now done:
- apache#23096,
- apache#25730,
- apache#25968,
- apache#26133
The h2o window benchmark's Top-N queries
(`benchmarks/queries/h2o/window.sql`, Q13–Q29, small dataset,
`J1_1e7_1e7_NA.parquet`) are now faster with the flag on in every case,
up to 100K partitions and including the heavy-ties variants. `dfbench
h2o --iterations 5`, median,
`DATAFUSION_OPTIMIZER_ENABLE_WINDOW_TOPN=false` for the baseline, same
binary, Apple M4 Max:
| Query | function, partitions | off (ms) | on (ms) | change |
|---|---|---:|---:|---|
| Q13 | ROW_NUMBER, id2 | 127.4 | 25.8 | 4.93x faster |
| Q14 | ROW_NUMBER, 100 | 190.8 | 35.9 | 5.32x faster |
| Q15 | ROW_NUMBER, 1K | 173.6 | 35.7 | 4.86x faster |
| Q16 | ROW_NUMBER, 10K | 149.7 | 33.0 | 4.54x faster |
| Q17 | ROW_NUMBER, 100K | 151.1 | 49.1 | 3.08x faster |
| Q18 | RANK, 100 | 226.3 | 28.8 | 7.86x faster |
| Q19 | RANK, 1K | 186.0 | 29.6 | 6.28x faster |
| Q20 | RANK, 1K, heavy ties | 209.8 | 57.1 | 3.67x faster |
| Q21 | RANK, id2 | 174.2 | 25.4 | 6.87x faster |
| Q22 | RANK, id2, heavy ties | 190.5 | 37.2 | 5.13x faster |
| Q23 | RANK, 100K | 188.1 | 63.5 | 2.96x faster |
| Q24 | DENSE_RANK, 100 | 248.4 | 38.6 | 6.43x faster |
| Q25 | DENSE_RANK, 1K | 199.6 | 36.9 | 5.40x faster |
| Q26 | DENSE_RANK, 1K, heavy ties | 232.2 | 43.3 | 5.37x faster |
| Q27 | DENSE_RANK, id2 | 211.3 | 45.7 | 4.63x faster |
| Q28 | DENSE_RANK, id2, heavy ties | 208.7 | 53.6 | 3.89x faster |
| Q29 | DENSE_RANK, 100K | 205.8 | 123.3 | 1.67x faster |
## What changes are included in this PR?
- `enable_window_topn` defaults to `true`; its doc comment now lists
ROW_NUMBER, RANK and DENSE_RANK and drops the high-cardinality warning.
- `window_topn.slt` restores the setting with `RESET` instead of `SET
... = false`.
- `configs.md` and `information_schema.slt` regenerated.
- The h2o `window_sorted` settings comment no longer says the flag
defaults to false (it still sets it explicitly).
## What is the testing strategy for this PR?
Existing coverage. With the flag on by default, the whole sqllogictest
suite and `core_integration` pass with only the `information_schema.slt`
default change; `qualify.slt`'s `rank <= 3` / `dense_rank <= 2` cases
now run through `PartitionedTopKExec` and return the same results.
`window_topn.slt` still covers the disabled path explicitly.
## Are there any user-facing changes?
Yes. Top-K-per-partition queries (`rn <= K`, `rn < K` over ROW_NUMBER /
RANK / DENSE_RANK) plan with `PartitionedTopKExec` instead of a full
sort, so `EXPLAIN` output changes; results don't. Set
`datafusion.optimizer.enable_window_topn = false` to get the old plan.
---------
Co-authored-by: Andrew Lamb <[email protected]>
@@ -488,7 +488,7 @@ datafusion.optimizer.enable_topk_dynamic_filter_pushdown true When set to true,
488
488
datafusion.optimizer.enable_topk_repartition true When set to true, the optimizer will push TopK (Sort with fetch) below hash repartition when the partition key is a prefix of the sort key, reducing data volume before the shuffle.
489
489
datafusion.optimizer.enable_unions_to_filter false When set to true, the logical optimizer will rewrite `UNION DISTINCT` branches that read from the same source and differ only by filter predicates into a single branch with a combined filter. This optimization is conservative and only applies when the branches share the same source and compatible wrapper nodes such as identical projections or aliases.
490
490
datafusion.optimizer.enable_window_limits true When set to true, the optimizer will attempt to push limit operations past window functions, if possible
491
-
datafusion.optimizer.enable_window_topn false When set to true, the optimizer will replace Filter(rn<=K) → Window(ROW_NUMBER) → Sort patterns with a PartitionedTopKExec that maintains per-partition heaps, avoiding a full sort of the input. When the window partition key has low cardinality, enabling this optimization can improve performance. However, for high cardinality keys, it may cause regressions in both memory usage and runtime.
491
+
datafusion.optimizer.enable_window_topn true When set to true, the optimizer will replace Filter(rn<=K) → Window(ROW_NUMBER, RANK or DENSE_RANK) → Sort patterns with a PartitionedTopKExec that keeps only the top K rows of each window partition, avoiding a full sort of the input.
492
492
datafusion.optimizer.expand_views_at_output false When set to true, if the returned type is a view type then the output will be coerced to a non-view. Coerces `Utf8View` to `LargeUtf8`, and `BinaryView` to `LargeBinary`.
493
493
datafusion.optimizer.filter_null_join_keys false When set to true, the optimizer will insert filters before a join between a nullable and non-nullable column to filter out nulls on the nullable side. This filter can add additional overhead when the file format does not fully support predicate push down.
494
494
datafusion.optimizer.hash_join_inlist_pushdown_max_distinct_values 150 Maximum number of distinct values (rows) in the build side of a hash join to be pushed down as an InList expression for dynamic filtering. Build sides with more rows than this will use hash table lookups instead. Set to 0 to always use hash table lookups. This provides an additional limit beyond `hash_join_inlist_pushdown_max_size` to prevent very large IN lists that might not provide much benefit over hash table lookups. This uses the deduplicated row count once the build side has been evaluated. The default is 150 values per partition. This is inspired by Trino's `max-filter-keys-per-column` setting. See: <https://trino.io/docs/current/admin/dynamic-filtering.html#dynamic-filter-collection-thresholds>
Copy file name to clipboardExpand all lines: docs/source/user-guide/configs.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -143,7 +143,7 @@ The following configuration settings are available:
143
143
| datafusion.optimizer.enable_round_robin_repartition | true | When set to true, the physical plan optimizer will try to add round robin repartitioning to increase parallelism to leverage more CPU cores |
144
144
| datafusion.optimizer.enable_topk_aggregation | true | When set to true, the optimizer will attempt to perform limit operations during aggregations, if possible |
145
145
| datafusion.optimizer.enable_window_limits | true | When set to true, the optimizer will attempt to push limit operations past window functions, if possible |
146
-
| datafusion.optimizer.enable_window_topn | false | When set to true, the optimizer will replace Filter(rn<=K) → Window(ROW_NUMBER) → Sort patterns with a PartitionedTopKExec that maintains per-partition heaps, avoiding a full sort of the input. When the window partition key has low cardinality, enabling this optimization can improve performance. However, for high cardinality keys, it may cause regressions in both memory usage and runtime. |
146
+
| datafusion.optimizer.enable_window_topn | true | When set to true, the optimizer will replace Filter(rn<=K) → Window(ROW_NUMBER, RANK or DENSE_RANK) → Sort patterns with a PartitionedTopKExec that keeps only the top K rows of each window partition, avoiding a full sort of the input. |
147
147
| datafusion.optimizer.enable_row_number_to_aggregate | false | When set to true, the optimizer will rewrite a "top-1 per group" pattern of the form `Filter(row_number() = 1)` over a `PARTITION BY` window into an `Aggregate(first_value(... ORDER BY ...) GROUP BY partition)`. This avoids buffering / fully sorting the input, which is especially beneficial on wide payloads. Disabled by default because on nested/wide value types the aggregate only becomes memory efficient once the columnar nested-type `GroupsAccumulator` fast path is available; enabling it without that support can route wide payloads onto the slow per-group accumulator path. |
148
148
| datafusion.optimizer.enable_topk_repartition | true | When set to true, the optimizer will push TopK (Sort with fetch) below hash repartition when the partition key is a prefix of the sort key, reducing data volume before the shuffle. |
149
149
| datafusion.optimizer.enable_topk_dynamic_filter_pushdown | true | When set to true, the optimizer will attempt to push down TopK dynamic filters into the file scan phase. |
0 commit comments