Repository navigation
fix: do not use the top-k aggregate stream for a DISTINCT limit with no ordering direction - #25349
Merged
Merged
Conversation
…no ordering direction `LimitedDistinctAggregation` pushes a direction-less `LimitOptions` into an `AggregateExec` while `is_unordered_unfiltered_group_by_distinct()` holds. A later rule can insert the sort a window function requires below the aggregate, which gives the rebuilt aggregate an output ordering and makes the predicate false while the limit stays attached. `execute_typed` then selected `GroupedTopKAggregateStream`, whose constructor needs a direction that was never supplied, and failed with `Internal error: Ordering direction required for DISTINCT with limit`. Take the top-k path only when a direction is available, which is what the constructor requires, and otherwise fall back to the regular grouped streams: they treat the limit as a soft limit and the `LIMIT` above the aggregate still truncates the result.
adriangb
requested review from
2010YOUY01 and
xudong963
and
a balanced review from Copilot
September 15, 2026 22:03
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The execution guard matches the top-k stream contract and the regression tests cover the reported failure.
Pull request overview
Prevents directionless DISTINCT ... LIMIT queries from incorrectly using the ordered top-k aggregation stream.
Changes:
- Requires an ordering direction before selecting the top-k stream.
- Adds SQL regression coverage for single- and multi-column distinct queries with window functions.
File summaries
| File | Description |
|---|---|
datafusion/physical-plan/src/aggregates/mod.rs |
Guards top-k stream selection. |
datafusion/sqllogictest/test_files/aggregate.slt |
Adds regression queries and expected results. |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 0
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #25349 +/- ##
==========================================
- Coverage 81.92% 81.92% -0.01%
==========================================
Files 1135 1135
Lines 427772 427773 +1
Branches 427772 427773 +1
==========================================
- Hits 350456 350449 -7
- Misses 56373 56377 +4
- Partials 20943 20947 +4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
`EnforceSorting` was retired in favour of `EnsureRequirements`, which is the rule that inserts the sort below the aggregation in this case. Co-Authored-By: Claude Opus 5 <[email protected]>
2010YOUY01
approved these changes
Sep 16, 2026
2010YOUY01
left a comment
Member
There was a problem hiding this comment.
Thank you, this makes sense to me.
Omega359
pushed a commit
to Omega359/arrow-datafusion
that referenced
this pull request
Oct 11, 2026
…no ordering direction (apache#25349) ## Which issue does this PR close? - Closes apache#25350. ## Rationale for this change `SELECT DISTINCT ... LIMIT n` with no `ORDER BY` fails at execution with `Internal error: Ordering direction required for DISTINCT with limit` when one of the projected columns is a window function: ```sql CREATE TABLE t (env VARCHAR, region VARCHAR, deployment VARCHAR); INSERT INTO t VALUES ('prod','a','x'), ('prod','a','y'), ('staging','b','x'); SELECT DISTINCT env, region, count(*) OVER (PARTITION BY env, deployment) AS n FROM t LIMIT 30; ``` The mechanism: 1. `LimitedDistinctAggregation` pushes `LimitOptions::new(limit)` (no direction, `descending: None`) into the `AggregateExec` while `is_unordered_unfiltered_group_by_distinct()` is true. 2. `EnsureRequirements` then inserts the sort that the window function needs below the aggregation. `replace_children_if_necessary` rebuilds the aggregation through `replace_children` with `ChildrenPropertiesMode::Recompute`. This derives the properties again from the sorted child but keeps `limit_options`. The rebuilt node has `ordering_mode=PartiallySorted(..)` and an output ordering, so the predicate is now false. 3. `execute_typed` selects `GroupedTopKAggregateStream`. Its constructor needs an ordering direction, and no rule supplied one, so it returns the error. The top-k stream also reads only the first group key (`aggr.group_expr().expr()[0]`). Thus it would give wrong results for a multi-column `DISTINCT` even with a direction. The correct behavior is to not select that stream in this case. ## What changes are included in this PR? `AggregateExec::execute_typed` selects the top-k path only when an ordering direction is available, which is what `GroupedTopKAggregateStream::new` requires: - `limit_options().descending` is set. `TopKAggregation` sets it, and that rule accepts only a single group key. - Or `get_minmax_desc()` supplies a direction. Otherwise, execution uses the regular grouped streams. This is correct because they treat the pushed limit as a soft limit, and the `LIMIT` above the aggregation still truncates the result. The limit pushdown and the plans do not change. Only the stream selection at execution changes. ## What is the testing strategy for this PR? A new case in `datafusion/sqllogictest/test_files/aggregate.slt` runs the multi-column and single-column forms of the query and checks the rows (the same rows as PostgreSQL 16.15 and DuckDB 1.5.2). Without the `mod.rs` change, both queries fail with the internal error. With the change, they pass. Also run locally, on this branch rebased onto current `main`: - `cargo test -p datafusion-physical-plan -p datafusion-physical-optimizer` - `cargo test --profile ci -p datafusion-sqllogictest --test sqllogictests` (all files) ## Are there any user-facing changes? Queries that failed with `Internal error: Ordering direction required for DISTINCT with limit` now return the correct rows. There are no API changes and no plan changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Internal error: Ordering direction required for DISTINCT with limit#25350.Rationale for this change
SELECT DISTINCT ... LIMIT nwith noORDER BYfails at execution withInternal error: Ordering direction required for DISTINCT with limitwhen one of the projected columns is a window function:The mechanism:
LimitedDistinctAggregationpushesLimitOptions::new(limit)(no direction,descending: None) into theAggregateExecwhileis_unordered_unfiltered_group_by_distinct()is true.EnsureRequirementsthen inserts the sort that the window function needs below the aggregation.replace_children_if_necessaryrebuilds the aggregation throughreplace_childrenwithChildrenPropertiesMode::Recompute. This derives the properties again from the sorted child but keepslimit_options. The rebuilt node hasordering_mode=PartiallySorted(..)and an output ordering, so the predicate is now false.execute_typedselectsGroupedTopKAggregateStream. Its constructor needs an ordering direction, and no rule supplied one, so it returns the error.The top-k stream also reads only the first group key (
aggr.group_expr().expr()[0]). Thus it would give wrong results for a multi-columnDISTINCTeven with a direction. The correct behavior is to not select that stream in this case.What changes are included in this PR?
AggregateExec::execute_typedselects the top-k path only when an ordering direction is available, which is whatGroupedTopKAggregateStream::newrequires:limit_options().descendingis set.TopKAggregationsets it, and that rule accepts only a single group key.get_minmax_desc()supplies a direction.Otherwise, execution uses the regular grouped streams. This is correct because they treat the pushed limit as a soft limit, and the
LIMITabove the aggregation still truncates the result.The limit pushdown and the plans do not change. Only the stream selection at execution changes.
What is the testing strategy for this PR?
A new case in
datafusion/sqllogictest/test_files/aggregate.sltruns the multi-column and single-column forms of the query and checks the rows (the same rows as PostgreSQL 16.15 and DuckDB 1.5.2). Without themod.rschange, both queries fail with the internal error. With the change, they pass.Also run locally, on this branch rebased onto current
main:cargo test -p datafusion-physical-plan -p datafusion-physical-optimizercargo test --profile ci -p datafusion-sqllogictest --test sqllogictests(all files)Are there any user-facing changes?
Queries that failed with
Internal error: Ordering direction required for DISTINCT with limitnow return the correct rows. There are no API changes and no plan changes.🤖 Generated with Claude Code