Skip to content

Commit 46f6311

Browse files
authored
fix: keep one row per key in a native PartialMerge that runs short of memory (#6867)
* fix: keep one row per key in a native PartialMerge that runs short of memory Comet ran Spark's PartialMerge aggregate as DataFusion Partial, which emits the groups it holds when it runs short of memory and starts again. In Spark's single-distinct plan the stage above the PartialMerge counts its rows, so count(DISTINCT) counted every copy of a key. Run a PartialMerge as DataFusion Single instead, which spills. MergeAsPartial evaluates to a struct of its state, and a projection unpacks it into the state columns that the Partial plan emitted. SpillReplayPool also covers SingleHashAggregateStream, whose spill replay starves the same way as the final aggregates' do (#6254). Closes #6857. * Emit PartialMerge states a column at a time for row-only aggregates MergeAsPartial now wraps an inner aggregate that has no GroupsAccumulator (first, last and any_value without an ordering) in a GroupsAccumulatorAdapter itself. Under Single mode, DataFusion otherwise wrapped the wrapper's row accumulators and evaluated every group to a one-row struct. Also write the planner's two output wrappers the same way, check the planner test's output with assert_batches_sorted_eq and cover first there, and drop two settings that the memory-pressure tests don't read.
1 parent 71ae228 commit 46f6311

5 files changed

Lines changed: 453 additions & 70 deletions

File tree

‎native/core/src/execution/jni_api.rs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -963,7 +963,7 @@ fn configure_skip_partial_aggregation(config: &mut SessionConfig, plan: &Operato
963963
if count.children.len() == 1)
964964
})
965965
}
966-
// PartialMerge is represented as native Partial, but consumes states, not rows.
966+
// PartialMerge consumes states, not rows.
967967
_ => false,
968968
},
969969
_ => true,

‎native/core/src/execution/memory_pools/spill_replay.rs‎

Lines changed: 11 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,8 @@
2121
//! replays them through an `OrderedFinalAggregateStream` that has no way to spill, so a refused
2222
//! memory request there fails the task. `FinalHashAggregateStream` does this, and so does
2323
//! `OrderedFinalAggregateStream` itself, which DataFusion uses when the input is sorted on some of
24-
//! the grouping keys. The merge reserves read buffers for as many spill files as fit, and those
24+
//! the grouping keys. So does `SingleHashAggregateStream`, which Comet runs for a Spark
25+
//! PartialMerge (#6857). The merge reserves read buffers for as many spill files as fit, and those
2526
//! buffers belong to the same consumer, so the replay often finds the consumer's share already
2627
//! taken.
2728
//!
@@ -31,9 +32,9 @@
3132
//!
3233
//! This relies on the following in DataFusion 55.1, which a DataFusion upgrade has to re-check:
3334
//!
34-
//! - The two aggregates name their consumers `FinalHashAggregateStream[{partition}]` and
35-
//! `OrderedFinalAggregateStream[{partition}]`, and their merge and replay reserve through sibling
36-
//! reservations of that consumer.
35+
//! - The three aggregates name their consumers `FinalHashAggregateStream[{partition}]`,
36+
//! `OrderedFinalAggregateStream[{partition}]` and `SingleHashAggregateStream[{partition}]`, and
37+
//! their merge and replay reserve through sibling reservations of that consumer.
3738
//! - A final aggregate grows one reservation while another of its reservations holds memory only
3839
//! during the replay, while the merge holds its read buffers. Before that, the aggregate's table
3940
//! is its only reservation holding memory, so a refusal stands and makes it spill. The merge
@@ -45,7 +46,8 @@
4546
//!
4647
//! Remove this, as #6583 describes, once Comet's DataFusion has apache/datafusion#25383, which
4748
//! leaves the replay room when the merge picks its files. The #6254 tests in `CometAggregateSuite`
48-
//! fail without this on DataFusion 55.1, so they show whether the replay still needs it.
49+
//! fail without this on DataFusion 55.1, and its PartialMerge read-back test for #6857 fails
50+
//! without it on 55.2, so they show whether the replay still needs it.
4951
5052
use std::collections::HashMap;
5153
use std::fmt;
@@ -174,11 +176,12 @@ pub(super) fn unwrap_spill_replay(pool: &Arc<dyn MemoryPool>) -> Option<&Arc<dyn
174176
.map(|replay| &replay.inner)
175177
}
176178

177-
/// Whether `consumer` belongs to a final aggregate whose spill replay can't spill.
179+
/// Whether `consumer` belongs to a final or single aggregate, whose spill replay can't spill.
178180
fn is_final_aggregate(consumer: &MemoryConsumer) -> bool {
179181
let name = consumer.name();
180182
name.starts_with("FinalHashAggregateStream[")
181183
|| name.starts_with("OrderedFinalAggregateStream[")
184+
|| name.starts_with("SingleHashAggregateStream[")
182185
}
183186

184187
#[cfg(test)]
@@ -353,14 +356,14 @@ mod tests {
353356
for name in [
354357
"FinalHashAggregateStream[3]",
355358
"OrderedFinalAggregateStream[0]",
359+
"SingleHashAggregateStream[0]",
356360
] {
357361
assert!(is_final_aggregate(&MemoryConsumer::new(name)), "{name}");
358362
}
359-
// Partial aggregates emit early instead of spilling, and Comet never plans single ones.
363+
// Partial aggregates emit early instead of spilling.
360364
for name in [
361365
"PartialHashAggregateStream[0]",
362366
"OrderedPartialAggregateStream[0]",
363-
"SingleHashAggregateStream[0]",
364367
"ExternalSorterMerge[0]",
365368
] {
366369
assert!(!is_final_aggregate(&MemoryConsumer::new(name)), "{name}");

‎native/core/src/execution/merge_as_partial.rs‎

Lines changed: 151 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -20,46 +20,58 @@
2020
//! Spark's PartialMerge mode merges intermediate state buffers and outputs intermediate
2121
//! state (not final values). DataFusion has no equivalent mode — `Partial` calls
2222
//! `update_batch` and outputs state, while `Final` calls `merge_batch` and outputs
23-
//! evaluated results.
23+
//! evaluated results. `PartialReduce` merges state and outputs state, but from DataFusion 56
24+
//! it emits a group more than once when it runs short of memory, as `Partial` does.
2425
//!
25-
//! This wrapper bridges the gap: it operates under DataFusion's `Partial` mode (which
26-
//! outputs state) but redirects `update_batch` calls to `merge_batch`, giving merge
27-
//! semantics with state output.
26+
//! This wrapper bridges the gap: it redirects `update_batch` calls to `merge_batch`, and it
27+
//! evaluates to its state, packed into a struct. Under DataFusion's `Partial` mode, which
28+
//! outputs state, that gives merge semantics with state output. Under `Single` mode, which
29+
//! spills rather than emit a group twice, the planner unpacks the struct into the state columns.
2830
2931
use std::fmt::Debug;
3032
use std::hash::{Hash, Hasher};
33+
use std::sync::Arc;
3134

32-
use arrow::array::{ArrayRef, BooleanArray};
33-
use arrow::datatypes::{DataType, Field, FieldRef};
34-
use datafusion::common::Result;
35+
use arrow::array::{ArrayRef, BooleanArray, StructArray};
36+
use arrow::datatypes::{DataType, Field, FieldRef, Fields, Schema};
37+
use datafusion::common::{internal_err, Result};
3538
use datafusion::logical_expr::function::AccumulatorArgs;
3639
use datafusion::logical_expr::function::StateFieldsArgs;
3740
use datafusion::logical_expr::{
3841
Accumulator, AggregateUDF, AggregateUDFImpl, EmitTo, GroupsAccumulator, ReversedUDAF,
3942
Signature, Volatility,
4043
};
4144
use datafusion::physical_expr::aggregate::AggregateFunctionExpr;
45+
use datafusion::physical_expr::{
46+
expressions::Column, GroupsAccumulatorAdapter, PhysicalExpr, PhysicalSortExpr,
47+
};
48+
use datafusion::physical_plan::{projection::ProjectionExec, ExecutionPlan};
4249
use datafusion::scalar::ScalarValue;
50+
use datafusion_comet_spark_expr::GetStructField;
4351

4452
use crate::execution::spark_aggregate_state::PartialMergeStateDecoder;
4553

46-
/// An AggregateUDF wrapper that gives merge semantics in Partial mode.
54+
/// An AggregateUDF wrapper that merges state and outputs state in Partial or Single mode.
4755
///
4856
/// When DataFusion runs an AggregateExec in Partial mode, it calls `update_batch`
4957
/// on each accumulator and outputs `state()`. This wrapper intercepts `update_batch`
5058
/// and redirects it to `merge_batch` on the inner accumulator, effectively
51-
/// implementing PartialMerge: merge inputs, output state.
59+
/// implementing PartialMerge: merge inputs, output state. In Single mode, DataFusion
60+
/// outputs `evaluate()` instead, which returns the inner accumulator's state as a struct
61+
/// of the state fields.
5262
///
5363
/// We store the inner AggregateUDF (not the AggregateFunctionExpr) to avoid keeping
5464
/// references to UnboundColumn expressions that would panic if evaluated.
5565
#[derive(Debug)]
5666
pub struct MergeAsPartialUDF {
5767
/// The inner aggregate UDF, cloned from the original expression.
5868
inner_udf: AggregateUDF,
59-
/// Pre-computed return type from the original expression.
60-
return_type: DataType,
61-
/// Pre-computed state fields from the original expression.
62-
cached_state_fields: Vec<FieldRef>,
69+
/// The original expression's return field, which the inner accumulator is built with, since
70+
/// the wrapper returns its state instead.
71+
inner_return_field: FieldRef,
72+
/// Pre-computed state fields from the original expression, which are also the fields of
73+
/// the struct the wrapper evaluates to.
74+
cached_state_fields: Fields,
6375
/// Cached signature that accepts state field types.
6476
signature: Signature,
6577
/// Override argument fields used only when constructing the inner accumulator.
@@ -87,8 +99,7 @@ impl Hash for MergeAsPartialUDF {
8799
impl MergeAsPartialUDF {
88100
pub fn new(inner_expr: &AggregateFunctionExpr) -> Result<Self> {
89101
let name = format!("merge_as_partial_{}", inner_expr.name());
90-
let return_type = inner_expr.field().data_type().clone();
91-
let cached_state_fields = inner_expr.state_fields()?;
102+
let cached_state_fields = Fields::from(inner_expr.state_fields()?);
92103
let accumulator_expr_fields =
93104
Self::accumulator_expr_fields(inner_expr, &cached_state_fields);
94105
let state_decoder = PartialMergeStateDecoder::try_new(inner_expr, &cached_state_fields)?;
@@ -99,7 +110,7 @@ impl MergeAsPartialUDF {
99110

100111
Ok(Self {
101112
inner_udf: inner_expr.fun().clone(),
102-
return_type,
113+
inner_return_field: inner_expr.field(),
103114
cached_state_fields,
104115
signature,
105116
accumulator_expr_fields,
@@ -136,6 +147,7 @@ impl MergeAsPartialUDF {
136147
.as_deref()
137148
.unwrap_or(args.expr_fields);
138149
AccumulatorArgs {
150+
return_field: Arc::clone(&self.inner_return_field),
139151
expr_fields,
140152
..args
141153
}
@@ -153,14 +165,20 @@ impl AggregateUDFImpl for MergeAsPartialUDF {
153165

154166
fn return_type(&self, _arg_types: &[DataType]) -> Result<DataType> {
155167
// In Partial mode, return_type isn't used for output schema (state_fields is).
156-
// Return the inner function's return type for consistency.
157-
Ok(self.return_type.clone())
168+
// In Single mode, the wrapper evaluates to its state.
169+
Ok(DataType::Struct(self.cached_state_fields.clone()))
170+
}
171+
172+
fn is_nullable(&self) -> bool {
173+
// The state struct itself is never null, so each state field unpacked from it keeps its
174+
// own nullability.
175+
false
158176
}
159177

160178
fn state_fields(&self, _args: StateFieldsArgs) -> Result<Vec<FieldRef>> {
161179
// Cached at construction: state schema depends on the inner aggregate's
162180
// return type, not on StateFieldsArgs.
163-
Ok(self.cached_state_fields.clone())
181+
Ok(self.cached_state_fields.to_vec())
164182
}
165183

166184
fn accumulator(&self, args: AccumulatorArgs) -> Result<Box<dyn Accumulator>> {
@@ -170,24 +188,34 @@ impl AggregateUDFImpl for MergeAsPartialUDF {
170188
Ok(Box::new(MergeAsPartialAccumulator {
171189
inner: inner_acc,
172190
state_decoder: self.state_decoder.clone(),
191+
state_fields: self.cached_state_fields.clone(),
173192
}))
174193
}
175194

176-
fn groups_accumulator_supported(&self, args: AccumulatorArgs) -> bool {
177-
self.inner_udf
178-
.groups_accumulator_supported(self.inner_accumulator_args(args))
195+
fn groups_accumulator_supported(&self, _args: AccumulatorArgs) -> bool {
196+
// An inner aggregate without a GroupsAccumulator gets an adapter below, so that Single
197+
// mode emits its states a column at a time rather than evaluating each group to a struct.
198+
true
179199
}
180200

181201
fn create_groups_accumulator(
182202
&self,
183203
args: AccumulatorArgs,
184204
) -> Result<Box<dyn GroupsAccumulator>> {
185-
let inner_acc = self
205+
let inner_args = self.inner_accumulator_args(args);
206+
let inner_acc: Box<dyn GroupsAccumulator> = if self
186207
.inner_udf
187-
.create_groups_accumulator(self.inner_accumulator_args(args))?;
208+
.groups_accumulator_supported(inner_args.clone())
209+
{
210+
self.inner_udf.create_groups_accumulator(inner_args)?
211+
} else {
212+
let factory = InnerAccumulatorFactory::new(&self.inner_udf, &inner_args);
213+
Box::new(GroupsAccumulatorAdapter::new(move || factory.create()))
214+
};
188215
Ok(Box::new(MergeAsPartialGroupsAccumulator {
189216
inner: inner_acc,
190217
state_decoder: self.state_decoder.clone(),
218+
state_fields: self.cached_state_fields.clone(),
191219
}))
192220
}
193221

@@ -204,10 +232,88 @@ impl AggregateUDFImpl for MergeAsPartialUDF {
204232
}
205233
}
206234

207-
/// Accumulator wrapper that redirects update_batch to merge_batch.
235+
/// Owned copies of the arguments that build an inner row accumulator, which a
236+
/// `GroupsAccumulatorAdapter` does for each new group, after `AccumulatorArgs` has gone.
237+
struct InnerAccumulatorFactory {
238+
udf: AggregateUDF,
239+
return_field: FieldRef,
240+
schema: Schema,
241+
ignore_nulls: bool,
242+
order_bys: Vec<PhysicalSortExpr>,
243+
is_reversed: bool,
244+
name: String,
245+
is_distinct: bool,
246+
exprs: Vec<Arc<dyn PhysicalExpr>>,
247+
expr_fields: Vec<FieldRef>,
248+
}
249+
250+
impl InnerAccumulatorFactory {
251+
fn new(udf: &AggregateUDF, args: &AccumulatorArgs) -> Self {
252+
Self {
253+
udf: udf.clone(),
254+
return_field: Arc::clone(&args.return_field),
255+
schema: args.schema.clone(),
256+
ignore_nulls: args.ignore_nulls,
257+
order_bys: args.order_bys.to_vec(),
258+
is_reversed: args.is_reversed,
259+
name: args.name.to_string(),
260+
is_distinct: args.is_distinct,
261+
exprs: args.exprs.to_vec(),
262+
expr_fields: args.expr_fields.to_vec(),
263+
}
264+
}
265+
266+
fn create(&self) -> Result<Box<dyn Accumulator>> {
267+
self.udf.accumulator(AccumulatorArgs {
268+
return_field: Arc::clone(&self.return_field),
269+
schema: &self.schema,
270+
ignore_nulls: self.ignore_nulls,
271+
order_bys: &self.order_bys,
272+
is_reversed: self.is_reversed,
273+
name: &self.name,
274+
is_distinct: self.is_distinct,
275+
exprs: &self.exprs,
276+
expr_fields: &self.expr_fields,
277+
})
278+
}
279+
}
280+
281+
/// Projects the output of a Single-mode `aggregate` whose aggregates are all [`MergeAsPartialUDF`]s
282+
/// into the output of a Partial-mode one: its `num_grouping` grouping columns, then one column per
283+
/// state field, unpacked from the struct each aggregate evaluates to.
284+
pub(crate) fn unpack_state_structs(
285+
aggregate: Arc<dyn ExecutionPlan>,
286+
num_grouping: usize,
287+
) -> Result<Arc<dyn ExecutionPlan>> {
288+
let schema = aggregate.schema();
289+
let mut exprs: Vec<(Arc<dyn PhysicalExpr>, String)> = Vec::new();
290+
for (idx, field) in schema.fields().iter().enumerate() {
291+
let column: Arc<dyn PhysicalExpr> = Arc::new(Column::new(field.name(), idx));
292+
if idx < num_grouping {
293+
exprs.push((column, field.name().clone()));
294+
continue;
295+
}
296+
let DataType::Struct(state_fields) = field.data_type() else {
297+
return internal_err!(
298+
"Expected a struct of aggregate state in column {idx}, got {}",
299+
field.data_type()
300+
);
301+
};
302+
for (ordinal, state_field) in state_fields.iter().enumerate() {
303+
exprs.push((
304+
Arc::new(GetStructField::new(Arc::clone(&column), ordinal)),
305+
state_field.name().clone(),
306+
));
307+
}
308+
}
309+
Ok(Arc::new(ProjectionExec::try_new(exprs, aggregate)?))
310+
}
311+
312+
/// Accumulator wrapper that redirects update_batch to merge_batch and evaluates to its state.
208313
struct MergeAsPartialAccumulator {
209314
inner: Box<dyn Accumulator>,
210315
state_decoder: PartialMergeStateDecoder,
316+
state_fields: Fields,
211317
}
212318

213319
impl Debug for MergeAsPartialAccumulator {
@@ -229,7 +335,17 @@ impl Accumulator for MergeAsPartialAccumulator {
229335
}
230336

231337
fn evaluate(&mut self) -> Result<ScalarValue> {
232-
self.inner.evaluate()
338+
let state = self
339+
.inner
340+
.state()?
341+
.iter()
342+
.map(ScalarValue::to_array)
343+
.collect::<Result<Vec<_>>>()?;
344+
Ok(ScalarValue::Struct(Arc::new(StructArray::try_new(
345+
self.state_fields.clone(),
346+
state,
347+
None,
348+
)?)))
233349
}
234350

235351
fn state(&mut self) -> Result<Vec<ScalarValue>> {
@@ -241,10 +357,12 @@ impl Accumulator for MergeAsPartialAccumulator {
241357
}
242358
}
243359

244-
/// GroupsAccumulator wrapper that redirects update_batch to merge_batch.
360+
/// GroupsAccumulator wrapper that redirects update_batch to merge_batch and evaluates to its
361+
/// state.
245362
struct MergeAsPartialGroupsAccumulator {
246363
inner: Box<dyn GroupsAccumulator>,
247364
state_decoder: PartialMergeStateDecoder,
365+
state_fields: Fields,
248366
}
249367

250368
impl Debug for MergeAsPartialGroupsAccumulator {
@@ -290,7 +408,12 @@ impl GroupsAccumulator for MergeAsPartialGroupsAccumulator {
290408
}
291409

292410
fn evaluate(&mut self, emit_to: EmitTo) -> Result<ArrayRef> {
293-
self.inner.evaluate(emit_to)
411+
let state = self.inner.state(emit_to)?;
412+
Ok(Arc::new(StructArray::try_new(
413+
self.state_fields.clone(),
414+
state,
415+
None,
416+
)?))
294417
}
295418

296419
fn state(&mut self, emit_to: EmitTo) -> Result<Vec<ArrayRef>> {

0 commit comments

Comments
 (0)