Skip to content

Commit 7ef79eb

Browse files
compheadcomphead
committed
feat: support WindowGroupLimit 
1 parent add9758 commit 7ef79eb

15 files changed

Lines changed: 406 additions & 123 deletions

File tree

‎docs/source/contributor-guide/roadmap.md‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -29,8 +29,8 @@ Native window execution runs by default (`spark.comet.exec.window.enabled`). The
2929
`first_value`, `last_value`), and the `count`, `min`, `max`, `sum`, and `avg` aggregates are accelerated.
3030
Remaining work is to close the gaps that still fall back to Spark: statistical aggregates (`stddev`, `variance`,
3131
`corr`, `covar`) and `collect_list` / `collect_set` as window functions ([#4766]), `GROUPS` frames ([#4836]), `RANGE` frames with explicit date or
32-
decimal offsets ([#4834]), `first_value` / `last_value` on `RANGE` frames with a literal offset ([#4835]),
33-
non-literal `lag` / `lead` default values ([#4268]), and `WindowGroupLimitExec` ([#4837]). See the
32+
decimal offsets ([#4834]), `first_value` / `last_value` on `RANGE` frames with a literal offset ([#4835]), and
33+
non-literal `lag` / `lead` default values ([#4268]). See the
3434
[window function compatibility guide](../user-guide/latest/compatibility/operators.md) for the complete list of
3535
supported functions, frames, and fallback cases.
3636

@@ -39,7 +39,6 @@ supported functions, frames, and fallback cases.
3939
[#4834]: https://github.com/apache/datafusion-comet/issues/4834
4040
[#4835]: https://github.com/apache/datafusion-comet/issues/4835
4141
[#4836]: https://github.com/apache/datafusion-comet/issues/4836
42-
[#4837]: https://github.com/apache/datafusion-comet/issues/4837
4342

4443
## Native Lambda Evaluation
4544

‎docs/source/user-guide/latest/compatibility/floating-point.md‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,3 +27,16 @@ So Comet adds additional normalization expression of NaN and zero for comparison
2727
to Spark in some cases, especially when the data contains both positive and negative zero. This is likely an edge
2828
case that is not of concern for many users. If it is a concern, setting `spark.comet.exec.strictFloatingPoint=true`
2929
will make relevant operations fall back to Spark.
30+
31+
## Ordering: signed zero (`-0.0` vs `+0.0`)
32+
33+
Spark's `ORDER BY`, `RANK`, `DENSE_RANK`, and window frame comparisons route through
34+
`SQLOrderingUtil.compareDoubles` / `compareFloats`, which explicitly define `-0.0 == 0.0`. Comet's
35+
native sort and `WindowGroupLimitExec` use the `arrow-row` row-format encoder for `ORDER BY` keys,
36+
which applies Rust's total-ordering transform to the raw IEEE-754 bits. Under that encoding `-0.0`
37+
sorts strictly less than `+0.0`, so a partition that mixes the two zeros can produce a rank
38+
distribution that differs from Spark. For example, `RANK() OVER (ORDER BY v ASC)` over
39+
`[-0.0, 0.0, 1.0]` filtered to `rk <= 1` returns two rows in Spark (both zeros tied at rank 1) but
40+
one row in Comet (`-0.0` at rank 1, `+0.0` at rank 2). If your workload materially mixes `-0.0`
41+
and `+0.0` in a ranked column, prefer Spark for that stage or normalize the column to `+0.0`
42+
upstream.

‎docs/source/user-guide/latest/compatibility/operators.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -71,8 +71,8 @@ incorrect result. When any single window expression in a `WindowExec` falls back
7171
window are not supported by Spark either.
7272
- Any `PARTITION BY` or `ORDER BY` expression that Comet cannot serialize.
7373

74-
`WindowGroupLimitExec` (window-based limit pushdown) is not yet supported and falls back to Spark
75-
([#4837](https://github.com/apache/datafusion-comet/issues/4837)).
74+
`WindowGroupLimitExec` (window-based limit pushdown for `ROW_NUMBER`, `RANK`, and `DENSE_RANK`)
75+
runs natively; it is controlled by `spark.comet.exec.windowGroupLimit.enabled` (default: true).
7676

7777
## Round-Robin Partitioning
7878

‎docs/source/user-guide/latest/operators.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -103,7 +103,7 @@ omitted from the tables below and may be reconsidered based on demand:
103103
| Operator | Status | Notes |
104104
| ---------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
105105
| `WindowExec` | ⚠️ | Runs natively and is enabled by default. A broad set of window functions is accelerated; unsupported shapes fall back to Spark. See [window function compatibility](compatibility/operators.md). |
106-
| `WindowGroupLimitExec` | 🔜 | Window-based limit pushdown falls back today ([#4837](https://github.com/apache/datafusion-comet/issues/4837)). |
106+
| `WindowGroupLimitExec` | ✅ | Streaming per-partition top-K pushdown for `ROW_NUMBER`, `RANK`, and `DENSE_RANK`. |
107107

108108
## Generators and set operations
109109

0 commit comments

Comments
 (0)