Skip to content

fix: do not use the top-k aggregate stream for a DISTINCT limit with no ordering direction - #25349

Merged
adriangb merged 2 commits into
apache:mainfrom
pydantic:distinct-limit-without-direction
Sep 16, 2026
Merged

adriangb merged 2 commits into
apache:mainfrom
pydantic:distinct-limit-without-direction

Conversation

@adriangb

@adriangb adriangb commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

SELECT DISTINCT ... LIMIT n with no ORDER BY fails at execution with Internal error: Ordering direction required for DISTINCT with limit when one of the projected columns is a window function:

CREATE TABLE t (env VARCHAR, region VARCHAR, deployment VARCHAR);
INSERT INTO t VALUES ('prod','a','x'), ('prod','a','y'), ('staging','b','x');
SELECT DISTINCT env, region, count(*) OVER (PARTITION BY env, deployment) AS n FROM t LIMIT 30;

The mechanism:

  1. LimitedDistinctAggregation pushes LimitOptions::new(limit) (no direction, descending: None) into the AggregateExec while is_unordered_unfiltered_group_by_distinct() is true.
  2. EnsureRequirements then inserts the sort that the window function needs below the aggregation. replace_children_if_necessary rebuilds the aggregation through replace_children with ChildrenPropertiesMode::Recompute. This derives the properties again from the sorted child but keeps limit_options. The rebuilt node has ordering_mode=PartiallySorted(..) and an output ordering, so the predicate is now false.
  3. execute_typed selects GroupedTopKAggregateStream. Its constructor needs an ordering direction, and no rule supplied one, so it returns the error.

The top-k stream also reads only the first group key (aggr.group_expr().expr()[0]). Thus it would give wrong results for a multi-column DISTINCT even with a direction. The correct behavior is to not select that stream in this case.

What changes are included in this PR?

AggregateExec::execute_typed selects the top-k path only when an ordering direction is available, which is what GroupedTopKAggregateStream::new requires:

  • limit_options().descending is set. TopKAggregation sets it, and that rule accepts only a single group key.
  • Or get_minmax_desc() supplies a direction.

Otherwise, execution uses the regular grouped streams. This is correct because they treat the pushed limit as a soft limit, and the LIMIT above the aggregation still truncates the result.

The limit pushdown and the plans do not change. Only the stream selection at execution changes.

What is the testing strategy for this PR?

A new case in datafusion/sqllogictest/test_files/aggregate.slt runs the multi-column and single-column forms of the query and checks the rows (the same rows as PostgreSQL 16.15 and DuckDB 1.5.2). Without the mod.rs change, both queries fail with the internal error. With the change, they pass.

Also run locally, on this branch rebased onto current main:

  • cargo test -p datafusion-physical-plan -p datafusion-physical-optimizer
  • cargo test --profile ci -p datafusion-sqllogictest --test sqllogictests (all files)

Are there any user-facing changes?

Queries that failed with Internal error: Ordering direction required for DISTINCT with limit now return the correct rows. There are no API changes and no plan changes.

🤖 Generated with Claude Code

…no ordering direction

`LimitedDistinctAggregation` pushes a direction-less `LimitOptions` into an
`AggregateExec` while `is_unordered_unfiltered_group_by_distinct()` holds. A
later rule can insert the sort a window function requires below the
aggregate, which gives the rebuilt aggregate an output ordering and makes the
predicate false while the limit stays attached. `execute_typed` then selected
`GroupedTopKAggregateStream`, whose constructor needs a direction that was
never supplied, and failed with `Internal error: Ordering direction required
for DISTINCT with limit`.

Take the top-k path only when a direction is available, which is what the
constructor requires, and otherwise fall back to the regular grouped streams:
they treat the limit as a soft limit and the `LIMIT` above the aggregate
still truncates the result.
@adriangb
adriangb requested review from 2010YOUY01 and xudong963 and a balanced review from Copilot September 15, 2026 22:03
@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) physical-plan Changes to the physical-plan crate labels Sep 15, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The execution guard matches the top-k stream contract and the regression tests cover the reported failure.

Pull request overview

Prevents directionless DISTINCT ... LIMIT queries from incorrectly using the ordered top-k aggregation stream.

Changes:

  • Requires an ordering direction before selecting the top-k stream.
  • Adds SQL regression coverage for single- and multi-column distinct queries with window functions.
File summaries
File Description
datafusion/physical-plan/src/aggregates/mod.rs Guards top-k stream selection.
datafusion/sqllogictest/test_files/aggregate.slt Adds regression queries and expected results.
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 0
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@codecov-commenter

codecov-commenter commented Sep 15, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.92%. Comparing base (22651d2) to head (00c757f).
⚠️ Report is 6 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #25349      +/-   ##
==========================================
- Coverage   81.92%   81.92%   -0.01%     
==========================================
  Files        1135     1135              
  Lines      427772   427773       +1     
  Branches   427772   427773       +1     
==========================================
- Hits       350456   350449       -7     
- Misses      56373    56377       +4     
- Partials    20943    20947       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

`EnforceSorting` was retired in favour of `EnsureRequirements`, which is the
rule that inserts the sort below the aggregation in this case.

Co-Authored-By: Claude Opus 5 <[email protected]>

@2010YOUY01 2010YOUY01 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, this makes sense to me.

@adriangb
adriangb added this pull request to the merge queue Sep 16, 2026
Merged via the queue into apache:main with commit 18cbb87 Sep 16, 2026
42 of 43 checks passed
@adriangb
adriangb deleted the distinct-limit-without-direction branch September 16, 2026 02:25
Omega359 pushed a commit to Omega359/arrow-datafusion that referenced this pull request Oct 11, 2026
…no ordering direction (apache#25349)

## Which issue does this PR close?

- Closes apache#25350.

## Rationale for this change

`SELECT DISTINCT ... LIMIT n` with no `ORDER BY` fails at execution with
`Internal error: Ordering direction required for DISTINCT with limit`
when one of the projected columns is a window function:

```sql
CREATE TABLE t (env VARCHAR, region VARCHAR, deployment VARCHAR);
INSERT INTO t VALUES ('prod','a','x'), ('prod','a','y'), ('staging','b','x');
SELECT DISTINCT env, region, count(*) OVER (PARTITION BY env, deployment) AS n FROM t LIMIT 30;
```

The mechanism:

1. `LimitedDistinctAggregation` pushes `LimitOptions::new(limit)` (no
direction, `descending: None`) into the `AggregateExec` while
`is_unordered_unfiltered_group_by_distinct()` is true.
2. `EnsureRequirements` then inserts the sort that the window function
needs below the aggregation. `replace_children_if_necessary` rebuilds
the aggregation through `replace_children` with
`ChildrenPropertiesMode::Recompute`. This derives the properties again
from the sorted child but keeps `limit_options`. The rebuilt node has
`ordering_mode=PartiallySorted(..)` and an output ordering, so the
predicate is now false.
3. `execute_typed` selects `GroupedTopKAggregateStream`. Its constructor
needs an ordering direction, and no rule supplied one, so it returns the
error.

The top-k stream also reads only the first group key
(`aggr.group_expr().expr()[0]`). Thus it would give wrong results for a
multi-column `DISTINCT` even with a direction. The correct behavior is
to not select that stream in this case.

## What changes are included in this PR?

`AggregateExec::execute_typed` selects the top-k path only when an
ordering direction is available, which is what
`GroupedTopKAggregateStream::new` requires:

- `limit_options().descending` is set. `TopKAggregation` sets it, and
that rule accepts only a single group key.
- Or `get_minmax_desc()` supplies a direction.

Otherwise, execution uses the regular grouped streams. This is correct
because they treat the pushed limit as a soft limit, and the `LIMIT`
above the aggregation still truncates the result.

The limit pushdown and the plans do not change. Only the stream
selection at execution changes.

## What is the testing strategy for this PR?

A new case in `datafusion/sqllogictest/test_files/aggregate.slt` runs
the multi-column and single-column forms of the query and checks the
rows (the same rows as PostgreSQL 16.15 and DuckDB 1.5.2). Without the
`mod.rs` change, both queries fail with the internal error. With the
change, they pass.

Also run locally, on this branch rebased onto current `main`:

- `cargo test -p datafusion-physical-plan -p
datafusion-physical-optimizer`
- `cargo test --profile ci -p datafusion-sqllogictest --test
sqllogictests` (all files)

## Are there any user-facing changes?

Queries that failed with `Internal error: Ordering direction required
for DISTINCT with limit` now return the correct rows. There are no API
changes and no plan changes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-plan Changes to the physical-plan crate sqllogictest SQL Logic Tests (.slt) v56.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SELECT DISTINCT ... LIMIT n with a window function fails with Internal error: Ordering direction required for DISTINCT with limit

6 participants