Skip to content

Commit a7cac9a

Browse files
fix: omit ordering fields from the state of order-insensitive aggregates (apache#25402)
## Which issue does this PR close? - Part of apache#25401. This fixes the `min`/`max` schema mismatch. The `avg`/`bit_*`/`stddev`/`var_*` panics have a different cause and are fixed in apache#25403; the two PRs are independent. ## Rationale for this change An order-insensitive aggregate called with an `ORDER BY` fails whenever the aggregation runs in two phases (`Partial` → `Final`/`FinalPartitioned`), grouped or not: ```sql CREATE TABLE d (g INT, k INT, v INT) AS VALUES (1, 2, 20), (1, 1, 10), (2, 4, 40), (2, 3, 30); SELECT g, min(v ORDER BY k) FROM d GROUP BY g; -- Arrow error: Invalid argument error: number of columns(2) must match number of fields(3) in schema ``` `AggregateExprBuilder::build()` derived the aggregate's `ordering_fields` from its `ORDER BY` even when the function is order-insensitive. `AggregateFunctionExpr::order_bys()` hides those expressions from the accumulator, but `state_fields()` still passed `ordering_fields`. `Min` and `Max` use the default `AggregateUDFImpl::state_fields`, which appends them, while their accumulators only emit the value. The partial state schema therefore declared fields that no column filled. `sum`, `count`, `bool_and` and `bool_or` override `state_fields` without ordering fields, so they already worked. ## What changes are included in this PR? `AggregateExprBuilder::build()` now drops `order_bys` when the function's `order_sensitivity()` is insensitive. `ordering_fields` is then empty from the start, so the aggregate carries no ordering state that later code has to ignore. Order-sensitive aggregates are unchanged. None of the built-in order-insensitive aggregates reads `order_bys` from `AccumulatorArgs`, and plans display aggregates by `name()`, so EXPLAIN output does not change. ## What is the testing strategy for this PR? Two queries in `group_by.slt`, in the section that sets `target_partitions = 8`, each selecting `MIN`, `MAX` and `SUM` with `ORDER BY ts DESC` over `sales_global`, with `SUM` as a control: - Grouped by `country`. The plan is `Partial` → `FinalPartitioned`. On `main` it fails with `number of columns(4) must match number of fields(6)`. - Ungrouped, with `WHERE amount > 0` so the input is multi-partition and the plan is `Partial` → `Final`. On `main` it fails with `number of columns(3) must match number of fields(5)`. Without the filter, the single-partition input runs in `Single` mode and never builds a partial state. Both queries pass with this change. The other `.slt` files covering aggregates still pass (`group_by`, `aggregate`, `array_agg`, `first_last_*`, `distinct_on`, `agg_func_substitute`, `window`, `order`). ## Are there any user-facing changes? Queries that failed now return results. No API changes. ## Notes for reviewers The state field *names* for ordering fields are being reworked in apache#25196. This PR only changes whether order-insensitive aggregates have ordering fields at all, so the two shouldn't interact beyond a possible textual rebase. --------- Co-authored-by: Neil Conway <[email protected]>
1 parent a2b8093 commit a7cac9a

2 files changed

Lines changed: 135 additions & 10 deletions

File tree

‎datafusion/physical-expr/src/aggregate.rs‎

Lines changed: 108 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -264,6 +264,15 @@ impl AggregateExprBuilder {
264264
} = self;
265265
assert_or_internal_err!(!args.is_empty(), "args should not be empty");
266266

267+
// An order-insensitive aggregate ignores its ORDER BY, so drop it here.
268+
// Everything derived from `order_bys` below, such as the ordering fields
269+
// in the aggregate's state, then agrees that there is no ordering.
270+
let order_bys = if fun.order_sensitivity().is_insensitive() {
271+
vec![]
272+
} else {
273+
order_bys
274+
};
275+
267276
let ordering_types = order_bys
268277
.iter()
269278
.map(|e| e.expr.data_type(&schema))
@@ -772,11 +781,7 @@ impl AggregateFunctionExpr {
772781

773782
/// Returns the ORDER BY expressions for the aggregate function.
774783
pub fn order_bys(&self) -> &[PhysicalSortExpr] {
775-
if self.order_sensitivity().is_insensitive() {
776-
&[]
777-
} else {
778-
&self.order_bys
779-
}
784+
&self.order_bys
780785
}
781786

782787
/// Indicates whether aggregator can produce the correct result with any
@@ -1052,10 +1057,7 @@ impl AggregateFunctionExpr {
10521057
args: Vec<Arc<dyn PhysicalExpr>>,
10531058
order_by_exprs: Vec<Arc<dyn PhysicalExpr>>,
10541059
) -> Option<AggregateFunctionExpr> {
1055-
if args.len() != self.args.len()
1056-
|| (self.order_sensitivity() != AggregateOrderSensitivity::Insensitive
1057-
&& order_by_exprs.len() != self.order_bys.len())
1058-
{
1060+
if args.len() != self.args.len() || order_by_exprs.len() != self.order_bys.len() {
10591061
return None;
10601062
}
10611063

@@ -1192,7 +1194,9 @@ mod tests {
11921194

11931195
use arrow::datatypes::Field;
11941196
use datafusion_common::metadata::FieldMetadata;
1195-
use datafusion_expr::{col, test::function_stub::sum};
1197+
use datafusion_expr::{
1198+
AggregateUDFImpl, Signature, Volatility, col, test::function_stub::sum,
1199+
};
11961200

11971201
fn aggregate_test_schema() -> Result<(Schema, DFSchema)> {
11981202
let schema = Schema::new(vec![Field::new("column1", DataType::Int64, true)]);
@@ -1263,4 +1267,98 @@ mod tests {
12631267

12641268
Ok(())
12651269
}
1270+
1271+
/// An aggregate that uses the default `AggregateUDFImpl::state_fields`,
1272+
/// which appends the ordering fields to the state.
1273+
#[derive(Debug, PartialEq, Eq, Hash)]
1274+
struct DefaultStateUdaf {
1275+
signature: Signature,
1276+
order_insensitive: bool,
1277+
}
1278+
1279+
impl DefaultStateUdaf {
1280+
fn new(order_insensitive: bool) -> Self {
1281+
Self {
1282+
signature: Signature::any(1, Volatility::Immutable),
1283+
order_insensitive,
1284+
}
1285+
}
1286+
}
1287+
1288+
impl AggregateUDFImpl for DefaultStateUdaf {
1289+
fn name(&self) -> &str {
1290+
"default_state_udaf"
1291+
}
1292+
1293+
fn signature(&self) -> &Signature {
1294+
&self.signature
1295+
}
1296+
1297+
fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
1298+
Ok(arg_types[0].clone())
1299+
}
1300+
1301+
/// Always fails, reporting how many ORDER BY expressions it was given.
1302+
fn accumulator(&self, acc_args: AccumulatorArgs) -> Result<Box<dyn Accumulator>> {
1303+
not_impl_err!("accumulator with {} order_bys", acc_args.order_bys.len())
1304+
}
1305+
1306+
fn order_sensitivity(&self) -> AggregateOrderSensitivity {
1307+
if self.order_insensitive {
1308+
AggregateOrderSensitivity::Insensitive
1309+
} else {
1310+
AggregateOrderSensitivity::HardRequirement
1311+
}
1312+
}
1313+
}
1314+
1315+
/// Builds `default_state_udaf(v ORDER BY k)`.
1316+
fn build_with_order_by(order_insensitive: bool) -> Result<AggregateFunctionExpr> {
1317+
let schema = Arc::new(Schema::new(vec![
1318+
Field::new("v", DataType::Int64, true),
1319+
Field::new("k", DataType::Int64, true),
1320+
]));
1321+
let fun = AggregateUDF::from(DefaultStateUdaf::new(order_insensitive));
1322+
AggregateExprBuilder::new(Arc::new(fun), vec![Arc::new(Column::new("v", 0))])
1323+
.order_by(vec![PhysicalSortExpr {
1324+
expr: Arc::new(Column::new("k", 1)),
1325+
options: SortOptions::default(),
1326+
}])
1327+
.schema(schema)
1328+
.alias("default_state_udaf(v) ORDER BY [k ASC NULLS LAST]")
1329+
.build()
1330+
}
1331+
1332+
#[test]
1333+
fn order_insensitive_aggregate_discards_order_by() -> Result<()> {
1334+
let expr = build_with_order_by(true)?;
1335+
assert!(expr.order_bys().is_empty());
1336+
assert!(expr.all_expressions().order_by_exprs.is_empty());
1337+
// Only the value: the default `state_fields` has no ordering fields to append
1338+
assert_eq!(expr.state_fields()?.len(), 1);
1339+
let err = expr.create_accumulator().unwrap_err();
1340+
assert!(err.message().contains("accumulator with 0 order_bys"));
1341+
1342+
// Rewriting the expressions does not bring the ORDER BY back
1343+
let rewritten = expr
1344+
.with_new_expressions(expr.expressions(), vec![])
1345+
.expect("rewrite is supported");
1346+
assert!(rewritten.order_bys().is_empty());
1347+
assert_eq!(rewritten.state_fields()?.len(), 1);
1348+
1349+
Ok(())
1350+
}
1351+
1352+
#[test]
1353+
fn order_sensitive_aggregate_keeps_order_by() -> Result<()> {
1354+
let expr = build_with_order_by(false)?;
1355+
assert_eq!(expr.order_bys().len(), 1);
1356+
assert_eq!(expr.all_expressions().order_by_exprs.len(), 1);
1357+
// The value, followed by the ordering field
1358+
assert_eq!(expr.state_fields()?.len(), 2);
1359+
let err = expr.create_accumulator().unwrap_err();
1360+
assert!(err.message().contains("accumulator with 1 order_bys"));
1361+
1362+
Ok(())
1363+
}
12661364
}

‎datafusion/sqllogictest/test_files/group_by.slt‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3075,6 +3075,33 @@ FRA 50 200
30753075
GRC 30 80
30763076
TUR 75 100
30773077

3078+
# An ORDER BY on an order-insensitive aggregator is ignored, so its partial
3079+
# state must not include ordering fields, with or without GROUP BY. MIN and MAX
3080+
# use the default state fields; SUM is a control that already defines its own.
3081+
query TRRR
3082+
SELECT country, MIN(amount ORDER BY ts DESC) AS min1,
3083+
MAX(amount ORDER BY ts DESC) AS max1,
3084+
SUM(amount ORDER BY ts DESC) AS sum1
3085+
FROM sales_global
3086+
GROUP BY country
3087+
ORDER BY country
3088+
----
3089+
FRA 50 200 250
3090+
GRC 30 80 110
3091+
TUR 75 100 175
3092+
3093+
# Without GROUP BY too. The filter makes the input multi-partition, so the
3094+
# aggregation runs in Partial and Final modes; over the single partition of
3095+
# `sales_global` alone it would run in Single mode and never build a state.
3096+
query RRR
3097+
SELECT MIN(amount ORDER BY ts DESC) AS min1,
3098+
MAX(amount ORDER BY ts DESC) AS max1,
3099+
SUM(amount ORDER BY ts DESC) AS sum1
3100+
FROM sales_global
3101+
WHERE amount > 0
3102+
----
3103+
30 200 535
3104+
30783105
# Conversion in between FIRST_VALUE and LAST_VALUE to resolve
30793106
# contradictory requirements should work in multi partitions.
30803107
query TT

0 commit comments

Comments
 (0)