Skip to content

Commit 957c8d6

Browse files
mkleenclaude
andcommitted
fix: keep DISTINCT on mixed aggregate nodes, and correct three mis-tagged functions
Review of the previous commit turned up two problems. `EliminateAggregateDistinct` stripped a no-op `DISTINCT` from one aggregate without regard for the others on the same node, which changed which plans `SingleDistinctToGroupBy` rewrites. That rule keys off how many distinct aggregates a node has and whether they share an argument, so removing one can push a node either way: `sum(DISTINCT cast(x)), max(DISTINCT x)` newly qualified once the max lost its flag, gaining an inner group by it never had, and where a rewrite happened regardless a shared inner group key turned into a separate accumulator at that finer grain. Gate the rule on every `DISTINCT` in the node being `Ignored`. This is conservative - `min(DISTINCT x), count(DISTINCT y)` now keeps the min flag though no rewrite is possible either way - but it cannot regress a plan. `group_by.slt` returns to its original expectations. `stddev`, `stddev_pop`, `var_samp`, `var_pop` and `approx_median` were tagged `Unsupported` on the grounds that their accumulators return `not_impl_err!`. That path is only reached when `SingleDistinctToGroupBy` cannot fire. All five take a single argument, so the rewrite deduplicates the input for them and `f(DISTINCT x)` returns the right answer today, as `aggregate.slt:749` already asserts for `approx_median`. Leave them at the default `Honored` with a comment, since enforcing `Unsupported` later would have broken those queries. Verified the remaining `Unsupported` tags: `approx_percentile_cont`, `nth_value` and `covar_samp` take more than one argument, so no rewrite fires and they silently return the non-distinct answer, while `approx_percentile_cont_with_weight` errors. Also widen the `Honored` and `Unsupported` doc comments, which described only the accumulator and promised a planning-time rejection that does not exist yet. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_015YwFpxheiiUNMadTNyeHiu
1 parent d96fa56 commit 957c8d6

7 files changed

Lines changed: 185 additions & 60 deletions

File tree

‎datafusion/expr/src/udaf.rs‎

Lines changed: 15 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -948,10 +948,13 @@ pub trait AggregateUDFImpl: Debug + DynEq + DynHash + Send + Sync + Any {
948948
/// How this function treats the `DISTINCT` modifier.
949949
///
950950
/// Return [`DistinctHandling::Ignored`] for duplicate-insensitive
951-
/// functions so that `f(DISTINCT x)` is planned as `f(x)`, and
952-
/// [`DistinctHandling::Unsupported`] if the accumulator does not
953-
/// read `is_distinct`, so that `DISTINCT` is rejected at planning
954-
/// time rather than silently ignored.
951+
/// functions so that `f(DISTINCT x)` is planned as `f(x)`.
952+
///
953+
/// Return [`DistinctHandling::Unsupported`] if the accumulator neither
954+
/// reads `is_distinct` nor is reached only after the planner has already
955+
/// deduplicated the input. Nothing reads this variant yet: rejecting such
956+
/// queries at planning time, rather than silently returning the
957+
/// non-distinct answer, is a follow-up change.
955958
fn distinct_handling(&self) -> DistinctHandling {
956959
DistinctHandling::Honored
957960
}
@@ -1743,11 +1746,15 @@ pub enum DistinctHandling {
17431746
/// The result is the same with or without `DISTINCT`, so the planner
17441747
/// is free to drop it. `min`, `max`, `bool_and`, `bit_or`, ...
17451748
Ignored,
1746-
/// The accumulator honours `AccumulatorArgs::is_distinct` and
1747-
/// deduplicates its input. `count`, `sum`, `avg`, `array_agg`, ...
1749+
/// `DISTINCT` is applied, so the planner must leave it alone. Either the
1750+
/// accumulator reads `AccumulatorArgs::is_distinct` and deduplicates its
1751+
/// input (`count`, `sum`, `avg`, `array_agg`, ...), or the planner does it
1752+
/// first by rewriting the aggregate into a group by (`stddev`, `var_samp`,
1753+
/// `approx_median`, ...). This is the default.
17481754
Honored,
1749-
/// The accumulator does not implement `DISTINCT`. Planning
1750-
/// `f(DISTINCT ...)` is an error. `corr`, `regr_*`, `nth_value`, ...
1755+
/// The accumulator does not implement `DISTINCT` and nothing deduplicates
1756+
/// the input for it, so `f(DISTINCT ...)` either errors or silently
1757+
/// returns the non-distinct answer. `corr`, `regr_*`, `nth_value`, ...
17511758
Unsupported,
17521759
}
17531760

‎datafusion/functions-aggregate/src/approx_median.rs‎

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,6 @@ use std::fmt::Debug;
2525
use std::sync::Arc;
2626

2727
use datafusion_common::{Result, not_impl_err};
28-
use datafusion_expr::DistinctHandling;
2928
use datafusion_expr::function::{AccumulatorArgs, StateFieldsArgs};
3029
use datafusion_expr::utils::format_state_name;
3130
use datafusion_expr::{
@@ -149,8 +148,8 @@ impl AggregateUDFImpl for ApproxMedian {
149148
self.doc()
150149
}
151150

152-
fn distinct_handling(&self) -> DistinctHandling {
153-
// The accumulator rejects `DISTINCT` with `not_impl_err!`.
154-
DistinctHandling::Unsupported
155-
}
151+
// Left at the default `Honored`. The accumulator rejects `DISTINCT` with
152+
// `not_impl_err!`, but this takes a single argument, so
153+
// `SingleDistinctToGroupBy` deduplicates the input before the accumulator
154+
// ever sees it and `f(DISTINCT x)` returns the right answer today.
156155
}

‎datafusion/functions-aggregate/src/stddev.rs‎

Lines changed: 8 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,6 @@ use arrow::datatypes::FieldRef;
2727
use arrow::{array::ArrayRef, datatypes::DataType, datatypes::Field};
2828
use datafusion_common::ScalarValue;
2929
use datafusion_common::{Result, internal_err, not_impl_err};
30-
use datafusion_expr::DistinctHandling;
3130
use datafusion_expr::function::{AccumulatorArgs, StateFieldsArgs};
3231
use datafusion_expr::utils::format_state_name;
3332
use datafusion_expr::{
@@ -142,10 +141,10 @@ impl AggregateUDFImpl for Stddev {
142141
self.doc()
143142
}
144143

145-
fn distinct_handling(&self) -> DistinctHandling {
146-
// The accumulator rejects `DISTINCT` with `not_impl_err!`.
147-
DistinctHandling::Unsupported
148-
}
144+
// Left at the default `Honored`. The accumulator rejects `DISTINCT` with
145+
// `not_impl_err!`, but this takes a single argument, so
146+
// `SingleDistinctToGroupBy` deduplicates the input before the accumulator
147+
// ever sees it and `f(DISTINCT x)` returns the right answer today.
149148
}
150149

151150
make_udaf_expr_and_func!(
@@ -247,10 +246,10 @@ impl AggregateUDFImpl for StddevPop {
247246
self.doc()
248247
}
249248

250-
fn distinct_handling(&self) -> DistinctHandling {
251-
// The accumulator rejects `DISTINCT` with `not_impl_err!`.
252-
DistinctHandling::Unsupported
253-
}
249+
// Left at the default `Honored`. The accumulator rejects `DISTINCT` with
250+
// `not_impl_err!`, but this takes a single argument, so
251+
// `SingleDistinctToGroupBy` deduplicates the input before the accumulator
252+
// ever sees it and `f(DISTINCT x)` returns the right answer today.
254253
}
255254

256255
/// An accumulator to compute the average

‎datafusion/functions-aggregate/src/variance.rs‎

Lines changed: 8 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,6 @@ use arrow::{
2626
};
2727
use datafusion_common::cast::{as_float64_array, as_uint64_array};
2828
use datafusion_common::{Result, ScalarValue};
29-
use datafusion_expr::DistinctHandling;
3029
use datafusion_expr::{
3130
Accumulator, AggregateUDFImpl, Documentation, GroupSelection, GroupsAccumulator,
3231
Signature, Volatility,
@@ -152,10 +151,10 @@ impl AggregateUDFImpl for VarianceSample {
152151
self.doc()
153152
}
154153

155-
fn distinct_handling(&self) -> DistinctHandling {
156-
// The accumulator rejects `DISTINCT` with `not_impl_err!`.
157-
DistinctHandling::Unsupported
158-
}
154+
// Left at the default `Honored`. The accumulator rejects `DISTINCT` with
155+
// `not_impl_err!`, but this takes a single argument, so
156+
// `SingleDistinctToGroupBy` deduplicates the input before the accumulator
157+
// ever sees it and `f(DISTINCT x)` returns the right answer today.
159158
}
160159

161160
#[user_doc(
@@ -259,10 +258,10 @@ impl AggregateUDFImpl for VariancePopulation {
259258
self.doc()
260259
}
261260

262-
fn distinct_handling(&self) -> DistinctHandling {
263-
// The accumulator rejects `DISTINCT` with `not_impl_err!`.
264-
DistinctHandling::Unsupported
265-
}
261+
// Left at the default `Honored`. The accumulator rejects `DISTINCT` with
262+
// `not_impl_err!`, but this takes a single argument, so
263+
// `SingleDistinctToGroupBy` deduplicates the input before the accumulator
264+
// ever sees it and `f(DISTINCT x)` returns the right answer today.
266265
}
267266

268267
/// An accumulator to compute variance

‎datafusion/optimizer/src/eliminate_aggregate_distinct.rs‎

Lines changed: 117 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ use crate::optimizer::ApplyOrder;
2222
use crate::{OptimizerConfig, OptimizerRule};
2323

2424
use datafusion_common::Result;
25-
use datafusion_common::tree_node::{Transformed, TreeNode};
25+
use datafusion_common::tree_node::{Transformed, TreeNode, TreeNodeRecursion};
2626
use datafusion_expr::expr::{AggregateFunction, AggregateFunctionParams};
2727
use datafusion_expr::expr_rewriter::NamePreserver;
2828
use datafusion_expr::{DistinctHandling, Expr, LogicalPlan};
@@ -80,7 +80,10 @@ impl OptimizerRule for EliminateAggregateDistinct {
8080
// Aggregate expressions only appear on Aggregate nodes, so every other
8181
// node is a cheap no-op. Window functions carry their own `distinct`
8282
// flag and are out of scope.
83-
if !matches!(plan, LogicalPlan::Aggregate(_)) {
83+
let LogicalPlan::Aggregate(aggregate) = &plan else {
84+
return Ok(Transformed::no(plan));
85+
};
86+
if !every_distinct_is_ignored(&aggregate.aggr_expr)? {
8487
return Ok(Transformed::no(plan));
8588
}
8689

@@ -101,6 +104,43 @@ impl OptimizerRule for EliminateAggregateDistinct {
101104
}
102105
}
103106

107+
/// Whether this node has a `DISTINCT` to drop and every `DISTINCT` on it can go.
108+
///
109+
/// Stripping only some of them would change which plans
110+
/// [`crate::single_distinct_to_groupby::SingleDistinctToGroupBy`] rewrites,
111+
/// because that rule keys off how many distinct aggregates a node has and
112+
/// whether they share one argument. Removing one can push a node either way:
113+
/// it can newly qualify a node that has two distinct columns and now has one,
114+
/// or, where a rewrite happens regardless, it can turn a shared inner group
115+
/// key into a separate accumulator at that finer grain. Neither is the point
116+
/// of this rule, so a node keeps every flag unless it can lose them all.
117+
///
118+
/// This is conservative: `min(DISTINCT x), count(DISTINCT y)` keeps the `min`
119+
/// flag even though no rewrite is possible either way.
120+
fn every_distinct_is_ignored(aggr_expr: &[Expr]) -> Result<bool> {
121+
let mut found_ignored = false;
122+
for expr in aggr_expr {
123+
let mut all_ignored = true;
124+
expr.apply(|e| {
125+
if let Expr::AggregateFunction(AggregateFunction { func, params }) = e
126+
&& params.distinct
127+
{
128+
if func.distinct_handling() == DistinctHandling::Ignored {
129+
found_ignored = true;
130+
} else {
131+
all_ignored = false;
132+
return Ok(TreeNodeRecursion::Stop);
133+
}
134+
}
135+
Ok(TreeNodeRecursion::Continue)
136+
})?;
137+
if !all_ignored {
138+
return Ok(false);
139+
}
140+
}
141+
Ok(found_ignored)
142+
}
143+
104144
/// Drops `DISTINCT` from `expr` if it is an aggregate that ignores duplicates.
105145
///
106146
/// An idempotent merge is also commutative, so an `Ignored` function is
@@ -147,6 +187,7 @@ mod tests {
147187
use crate::assert_optimized_plan_eq_snapshot;
148188
use crate::test::*;
149189

190+
use crate::single_distinct_to_groupby::SingleDistinctToGroupBy;
150191
use datafusion_expr::{ExprFunctionExt, LogicalPlanBuilder, col, lit};
151192
use datafusion_functions_aggregate::expr_fn::{bit_xor, max, min, sum};
152193

@@ -225,9 +266,10 @@ mod tests {
225266
")
226267
}
227268

228-
/// A plan mixing the two strips only the duplicate-insensitive one.
269+
/// A node keeps every flag when one of them has to stay, so that this rule
270+
/// cannot change which plans `SingleDistinctToGroupBy` rewrites.
229271
#[test]
230-
fn eliminate_distinct_from_min_only() -> Result<()> {
272+
fn mixed_node_is_left_alone() -> Result<()> {
231273
let table_scan = test_table_scan()?;
232274
let plan = LogicalPlanBuilder::from(table_scan)
233275
.aggregate(
@@ -240,7 +282,44 @@ mod tests {
240282
.build()?;
241283

242284
assert_optimized_plan_equal!(plan, @r"
243-
Aggregate: groupBy=[[test.a]], aggr=[[min(test.b) AS min(DISTINCT test.b), sum(DISTINCT test.c)]]
285+
Aggregate: groupBy=[[test.a]], aggr=[[min(DISTINCT test.b), sum(DISTINCT test.c)]]
286+
TableScan: test
287+
")
288+
}
289+
290+
/// Several duplicate-insensitive aggregates all lose the flag together.
291+
#[test]
292+
fn eliminate_distinct_from_every_ignored_aggregate() -> Result<()> {
293+
let table_scan = test_table_scan()?;
294+
let plan = LogicalPlanBuilder::from(table_scan)
295+
.aggregate(
296+
vec![col("a")],
297+
vec![
298+
min(col("b")).distinct().build()?,
299+
max(col("c")).distinct().build()?,
300+
],
301+
)?
302+
.build()?;
303+
304+
assert_optimized_plan_equal!(plan, @r"
305+
Aggregate: groupBy=[[test.a]], aggr=[[min(test.b) AS min(DISTINCT test.b), max(test.c) AS max(DISTINCT test.c)]]
306+
TableScan: test
307+
")
308+
}
309+
310+
/// A non-distinct aggregate alongside is no obstacle.
311+
#[test]
312+
fn eliminate_distinct_beside_non_distinct_aggregate() -> Result<()> {
313+
let table_scan = test_table_scan()?;
314+
let plan = LogicalPlanBuilder::from(table_scan)
315+
.aggregate(
316+
vec![col("a")],
317+
vec![min(col("b")).distinct().build()?, sum(col("c"))],
318+
)?
319+
.build()?;
320+
321+
assert_optimized_plan_equal!(plan, @r"
322+
Aggregate: groupBy=[[test.a]], aggr=[[min(test.b) AS min(DISTINCT test.b), sum(test.c)]]
244323
TableScan: test
245324
")
246325
}
@@ -267,6 +346,39 @@ mod tests {
267346
")
268347
}
269348

349+
/// The conservative gate exists so `SingleDistinctToGroupBy` still sees the
350+
/// plan exactly as it did before this rule was added.
351+
#[test]
352+
fn mixed_node_still_reaches_single_distinct_to_groupby() -> Result<()> {
353+
let table_scan = test_table_scan()?;
354+
let plan = LogicalPlanBuilder::from(table_scan)
355+
.aggregate(
356+
vec![col("a")],
357+
vec![
358+
min(col("b")).distinct().build()?,
359+
sum(col("b")).distinct().build()?,
360+
],
361+
)?
362+
.build()?;
363+
364+
let optimizer_ctx = OptimizerContext::new().with_max_passes(1);
365+
let rules: Vec<Arc<dyn OptimizerRule + Send + Sync>> = vec![
366+
Arc::new(EliminateAggregateDistinct::new()),
367+
Arc::new(SingleDistinctToGroupBy::new()),
368+
];
369+
assert_optimized_plan_eq_snapshot!(
370+
optimizer_ctx,
371+
rules,
372+
plan,
373+
@r"
374+
Projection: test.a, min(alias1) AS min(DISTINCT test.b), sum(alias1) AS sum(DISTINCT test.b)
375+
Aggregate: groupBy=[[test.a]], aggr=[[min(alias1), sum(alias1)]]
376+
Aggregate: groupBy=[[test.a, test.b AS alias1]], aggr=[[]]
377+
TableScan: test
378+
",
379+
)
380+
}
381+
270382
/// A plan with no Aggregate takes the no-op path.
271383
#[test]
272384
fn non_aggregate_plan_is_unchanged() -> Result<()> {

‎datafusion/sqllogictest/test_files/aggregates_simplify.slt‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -536,7 +536,8 @@ physical_plan
536536
07)------------AggregateExec: mode=Partial, gby=[g@0 as g, v@1 as alias1], aggr=[]
537537
08)--------------DataSourceExec: partitions=1, partition_sizes=[1]
538538

539-
# Mixed: only the duplicate-insensitive aggregate loses its DISTINCT
539+
# Mixed: a node that still needs one DISTINCT keeps all of them, so this rule
540+
# cannot change which plans SingleDistinctToGroupBy rewrites
540541
query III
541542
SELECT g, min(DISTINCT v), sum(DISTINCT v) FROM distinct_simplify_t GROUP BY g ORDER BY g;
542543
----
@@ -547,12 +548,12 @@ query TT
547548
EXPLAIN SELECT g, min(DISTINCT v), sum(DISTINCT v) FROM distinct_simplify_t GROUP BY g;
548549
----
549550
logical_plan
550-
01)Aggregate: groupBy=[[distinct_simplify_t.g]], aggr=[[min(distinct_simplify_t.v) AS min(DISTINCT distinct_simplify_t.v), sum(DISTINCT CAST(distinct_simplify_t.v AS Int64))]]
551+
01)Aggregate: groupBy=[[distinct_simplify_t.g]], aggr=[[min(DISTINCT distinct_simplify_t.v), sum(DISTINCT CAST(distinct_simplify_t.v AS Int64))]]
551552
02)--TableScan: distinct_simplify_t projection=[g, v]
552553
physical_plan
553-
01)AggregateExec: mode=FinalPartitioned, gby=[g@0 as g], aggr=[min(distinct_simplify_t.v) as min(DISTINCT distinct_simplify_t.v), sum(DISTINCT distinct_simplify_t.v)]
554+
01)AggregateExec: mode=FinalPartitioned, gby=[g@0 as g], aggr=[min(DISTINCT distinct_simplify_t.v), sum(DISTINCT distinct_simplify_t.v)]
554555
02)--RepartitionExec: partitioning=Hash([g@0], 4), input_partitions=1
555-
03)----AggregateExec: mode=Partial, gby=[g@0 as g], aggr=[min(distinct_simplify_t.v) as min(DISTINCT distinct_simplify_t.v), sum(DISTINCT distinct_simplify_t.v)]
556+
03)----AggregateExec: mode=Partial, gby=[g@0 as g], aggr=[min(DISTINCT distinct_simplify_t.v), sum(DISTINCT distinct_simplify_t.v)]
556557
04)------DataSourceExec: partitions=1, partition_sizes=[1]
557558

558559
# FILTER is applied before deduplication, so it survives the rewrite

0 commit comments

Comments
 (0)