Skip to content

AQE: switch from streaming aggregation to hash aggregation based on runtime statistics #1989

Open

Description

@andygrove

Is your feature request related to a problem or challenge?

The adaptive execution path does not switch from streaming aggregation to hash aggregation (or vice versa) based on runtime statistics. Streaming aggregation depends on sorted input and assumptions about group cardinality; when runtime statistics contradict the plan-time estimate, the chosen aggregation strategy can be suboptimal.

Describe the solution you'd like

Add an adaptive physical optimizer rule that reconsiders the aggregation strategy for not-yet-resolved aggregate stages using exact runtime statistics (input ordering, group cardinality), switching between streaming and hash aggregation as appropriate.

Context

This is one of the documented limitations of the experimental Adaptive Query Execution (AQE) path, listed in docs/source/user-guide/tuning-guide.md under "Current limitations" and in the doc comment on AdaptiveExecutionGraph (ballista/scheduler/src/state/aqe/mod.rs). Filing a discrete tracking issue so the gap is visible outside the doc.

Parent epic: #1359

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions