Repository navigation
Commit 18cbb87
fix: do not use the top-k aggregate stream for a DISTINCT limit with no ordering direction (apache#25349)
## Which issue does this PR close?
- Closes apache#25350.
## Rationale for this change
`SELECT DISTINCT ... LIMIT n` with no `ORDER BY` fails at execution with
`Internal error: Ordering direction required for DISTINCT with limit`
when one of the projected columns is a window function:
```sql
CREATE TABLE t (env VARCHAR, region VARCHAR, deployment VARCHAR);
INSERT INTO t VALUES ('prod','a','x'), ('prod','a','y'), ('staging','b','x');
SELECT DISTINCT env, region, count(*) OVER (PARTITION BY env, deployment) AS n FROM t LIMIT 30;
```
The mechanism:
1. `LimitedDistinctAggregation` pushes `LimitOptions::new(limit)` (no
direction, `descending: None`) into the `AggregateExec` while
`is_unordered_unfiltered_group_by_distinct()` is true.
2. `EnsureRequirements` then inserts the sort that the window function
needs below the aggregation. `replace_children_if_necessary` rebuilds
the aggregation through `replace_children` with
`ChildrenPropertiesMode::Recompute`. This derives the properties again
from the sorted child but keeps `limit_options`. The rebuilt node has
`ordering_mode=PartiallySorted(..)` and an output ordering, so the
predicate is now false.
3. `execute_typed` selects `GroupedTopKAggregateStream`. Its constructor
needs an ordering direction, and no rule supplied one, so it returns the
error.
The top-k stream also reads only the first group key
(`aggr.group_expr().expr()[0]`). Thus it would give wrong results for a
multi-column `DISTINCT` even with a direction. The correct behavior is
to not select that stream in this case.
## What changes are included in this PR?
`AggregateExec::execute_typed` selects the top-k path only when an
ordering direction is available, which is what
`GroupedTopKAggregateStream::new` requires:
- `limit_options().descending` is set. `TopKAggregation` sets it, and
that rule accepts only a single group key.
- Or `get_minmax_desc()` supplies a direction.
Otherwise, execution uses the regular grouped streams. This is correct
because they treat the pushed limit as a soft limit, and the `LIMIT`
above the aggregation still truncates the result.
The limit pushdown and the plans do not change. Only the stream
selection at execution changes.
## What is the testing strategy for this PR?
A new case in `datafusion/sqllogictest/test_files/aggregate.slt` runs
the multi-column and single-column forms of the query and checks the
rows (the same rows as PostgreSQL 16.15 and DuckDB 1.5.2). Without the
`mod.rs` change, both queries fail with the internal error. With the
change, they pass.
Also run locally, on this branch rebased onto current `main`:
- `cargo test -p datafusion-physical-plan -p
datafusion-physical-optimizer`
- `cargo test --profile ci -p datafusion-sqllogictest --test
sqllogictests` (all files)
## Are there any user-facing changes?
Queries that failed with `Internal error: Ordering direction required
for DISTINCT with limit` now return the correct rows. There are no API
changes and no plan changes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude <[email protected]>1 parent be5a96e commit 18cbb87
2 files changed
Lines changed: 47 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1219 | 1219 | | |
1220 | 1220 | | |
1221 | 1221 | | |
1222 | | - | |
| 1222 | + | |
| 1223 | + | |
| 1224 | + | |
| 1225 | + | |
| 1226 | + | |
| 1227 | + | |
| 1228 | + | |
| 1229 | + | |
| 1230 | + | |
| 1231 | + | |
| 1232 | + | |
| 1233 | + | |
| 1234 | + | |
| 1235 | + | |
| 1236 | + | |
| 1237 | + | |
| 1238 | + | |
1223 | 1239 | | |
1224 | 1240 | | |
| 1241 | + | |
1225 | 1242 | | |
1226 | 1243 | | |
1227 | 1244 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7522 | 7522 | | |
7523 | 7523 | | |
7524 | 7524 | | |
| 7525 | + | |
| 7526 | + | |
| 7527 | + | |
| 7528 | + | |
| 7529 | + | |
| 7530 | + | |
| 7531 | + | |
| 7532 | + | |
| 7533 | + | |
| 7534 | + | |
| 7535 | + | |
| 7536 | + | |
| 7537 | + | |
| 7538 | + | |
| 7539 | + | |
| 7540 | + | |
| 7541 | + | |
| 7542 | + | |
| 7543 | + | |
| 7544 | + | |
| 7545 | + | |
| 7546 | + | |
| 7547 | + | |
| 7548 | + | |
| 7549 | + | |
| 7550 | + | |
| 7551 | + | |
| 7552 | + | |
| 7553 | + | |
7525 | 7554 | | |
7526 | 7555 | | |
7527 | 7556 | | |
| |||
0 commit comments