Skip to content

Commit d8ead26

Browse files
authored
fix: spill a native PartialMerge whose input is sorted on some of its keys (#7222)
* fix: spill a native PartialMerge whose input is sorted on some of its keys Comet runs a Spark PartialMerge as DataFusion Single so that it spills rather than emit a key twice (#6857). When the input is sorted on a prefix of the grouping keys, DataFusion 55.2 runs Single with the legacy GroupedHashAggregateStream, whose spill asks for more than its share of the pool to sort the batch it writes, and the task fails. Plan that PartialMerge as DataFusion Final instead. Final runs an OrderedFinalAggregateStream there, which spills and reads its spill files back, and MergeAsPartial evaluates to its state under Final as it does under Single. Final reads each state from the columns after the keys and spills its groups in the input's types, so the planner switches only when the input is the grouping keys followed by native-typed states. A Spark partial's binary collect buffers keep the Single plan. Closes #6880. * refactor: decide the PartialMerge aggregate mode before building it Read the input order from the child's equivalence properties, as AggregateExec does, instead of building the aggregate as Single and again as Final. Simplify the state type guard, note when the switch can be rechecked, and cover input sorted on every key in the planner test. * address review: lazy sort check, document fair-limit exemption, cover changing g and a sort-merge join * address review: walk state types once, share planner test helpers * test: cover the PartialMerge inputs Final rejects and run the join case under AQE Add planner cases for swapped keys, states after one key and a column after the states, which each keep the PartialMerge as Single. Check that the join test's PartialMerge reaches the sort-merge join without an exchange, and keep AQE on by disabling the suite's runtime broadcast threshold. Name the DataFusion 56 stream as it is on main. * address review: keep Single for a PartialMerge over fully sorted input * address review: shorten the PartialMerge planning comment
1 parent 61fbbe2 commit d8ead26

7 files changed

Lines changed: 419 additions & 86 deletions

File tree

‎docs/source/contributor-guide/memory_management.md‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -396,7 +396,12 @@ over a native sort, a refusal fails the query, so the pool checks its requests o
396396
pool, as it does those of a consumer that cannot spill (`limited_to_its_share` in `fair_pool.rs`).
397397
It still counts as a consumer that can spill when the pool divides the task's share, so a sort
398398
below it stays at half of the share and leaves the aggregate the other half. The partially sorted
399-
stream still spills when Spark refuses it.
399+
stream still spills when Spark refuses it. The exemption goes by the stream's name, so it also
400+
covers a `PartialMerge` that the planner runs as a final aggregate because its input is sorted on
401+
some of its grouping keys. That one is not a `CometSortAggregateExec` and its input is only
402+
partially sorted, but it is likewise checked only against the pool and Spark's grant. A
403+
`PartialMerge` over input sorted on every grouping key stays on the hash aggregate, since the final
404+
aggregate has no spill path there and would only halve the limit of a sort below it.
400405

401406
Three details matter for tuning:
402407

‎native/core/src/execution/memory_pools/fair_pool.rs‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -163,6 +163,12 @@ impl Display for CometFairMemoryPool {
163163
/// refused request fails the query. So its requests are checked only against the pool, like those
164164
/// of a consumer that cannot spill. The partially sorted stream still spills when Spark refuses it.
165165
///
166+
/// The exemption goes by name, so it covers every `OrderedFinalAggregateStream`, including a
167+
/// pure `PartialMerge` that the planner runs as `Final` over input sorted on some of its keys
168+
/// (#6880). That one is not a `CometSortAggregateExec` and its input is only partially sorted, but
169+
/// it is checked only against the pool as well. The planner does not switch a `PartialMerge` over
170+
/// fully sorted input, which would gain no spill path and only halve the limit of a sort below it.
171+
///
166172
/// Only the check differs. The aggregate still counts as a consumer that can spill everywhere
167173
/// else, so it is one of `num_spill` and its bytes are not `unspillable`. A sort below it then
168174
/// stays limited to its fair share and leaves the aggregate the rest.

‎native/core/src/execution/memory_pools/spill_replay.rs‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -21,10 +21,10 @@
2121
//! replays them through an `OrderedFinalAggregateStream` that has no way to spill, so a refused
2222
//! memory request there fails the task. `FinalHashAggregateStream` does this, and so does
2323
//! `OrderedFinalAggregateStream` itself, which DataFusion uses when the input is sorted on some of
24-
//! the grouping keys. So does `SingleHashAggregateStream`, which Comet runs for a Spark
25-
//! PartialMerge (#6857). The merge reserves read buffers for as many spill files as fit, and those
26-
//! buffers belong to the same consumer, so the replay often finds the consumer's share already
27-
//! taken.
24+
//! the grouping keys, and which Comet also runs for a Spark PartialMerge over such input (#6880).
25+
//! So does `SingleHashAggregateStream`, which Comet runs for other Spark PartialMerges (#6857).
26+
//! The merge reserves read buffers for as many spill files as fit, and those buffers belong to the
27+
//! same consumer, so the replay often finds the consumer's share already taken.
2828
//!
2929
//! [`SpillReplayPool`] wraps a Comet pool. When the pool refuses a request from the replay, it
3030
//! records the request with the pool's `grow` instead, which ignores `CometFairMemoryPool`'s limits
@@ -182,6 +182,7 @@ pub(super) fn unwrap_spill_replay(pool: &Arc<dyn MemoryPool>) -> Option<&Arc<dyn
182182
}
183183

184184
/// The prefix of the name DataFusion gives the consumer of an `OrderedFinalAggregateStream`.
185+
/// DataFusion 56 renames the stream and its consumer `ClusteredFinalAggregateStream` (#6410).
185186
pub(super) const ORDERED_FINAL_AGGREGATE: &str = "OrderedFinalAggregateStream[";
186187

187188
/// Whether `consumer` belongs to a final or single aggregate, whose spill replay can't spill.

‎native/core/src/execution/merge_as_partial.rs‎

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,9 @@
2727
//! evaluates to its state, packed into a struct. Under DataFusion's `Partial` mode, which
2828
//! outputs state, that gives merge semantics with state output. Under `Single` mode, which
2929
//! spills rather than emit a group twice, the planner unpacks the struct into the state columns.
30+
//! It does the same under `Final` mode, which DataFusion 55 runs with an ordered aggregate that
31+
//! spills when the input is sorted on some of the grouping keys. There, `Single` falls back to a
32+
//! stream whose spill can ask for more than its share of memory (#6880).
3033
3134
use std::fmt::Debug;
3235
use std::hash::{Hash, Hasher};
@@ -51,12 +54,12 @@ use datafusion_comet_spark_expr::GetStructField;
5154

5255
use crate::execution::spark_aggregate_state::PartialMergeStateDecoder;
5356

54-
/// An AggregateUDF wrapper that merges state and outputs state in Partial or Single mode.
57+
/// An AggregateUDF wrapper that merges state and outputs state in Partial, Single or Final mode.
5558
///
5659
/// When DataFusion runs an AggregateExec in Partial mode, it calls `update_batch`
5760
/// on each accumulator and outputs `state()`. This wrapper intercepts `update_batch`
5861
/// and redirects it to `merge_batch` on the inner accumulator, effectively
59-
/// implementing PartialMerge: merge inputs, output state. In Single mode, DataFusion
62+
/// implementing PartialMerge: merge inputs, output state. In Single and Final modes, DataFusion
6063
/// outputs `evaluate()` instead, which returns the inner accumulator's state as a struct
6164
/// of the state fields.
6265
///
@@ -165,7 +168,7 @@ impl AggregateUDFImpl for MergeAsPartialUDF {
165168

166169
fn return_type(&self, _arg_types: &[DataType]) -> Result<DataType> {
167170
// In Partial mode, return_type isn't used for output schema (state_fields is).
168-
// In Single mode, the wrapper evaluates to its state.
171+
// In Single and Final modes, the wrapper evaluates to its state.
169172
Ok(DataType::Struct(self.cached_state_fields.clone()))
170173
}
171174

@@ -278,9 +281,9 @@ impl InnerAccumulatorFactory {
278281
}
279282
}
280283

281-
/// Projects the output of a Single-mode `aggregate` whose aggregates are all [`MergeAsPartialUDF`]s
282-
/// into the output of a Partial-mode one: its `num_grouping` grouping columns, then one column per
283-
/// state field, unpacked from the struct each aggregate evaluates to.
284+
/// Projects the output of a Single- or Final-mode `aggregate` whose aggregates are all
285+
/// [`MergeAsPartialUDF`]s into the output of a Partial-mode one: its `num_grouping` grouping
286+
/// columns, then one column per state field, unpacked from the struct each aggregate evaluates to.
284287
pub(crate) fn unpack_state_structs(
285288
aggregate: Arc<dyn ExecutionPlan>,
286289
num_grouping: usize,

0 commit comments

Comments
 (0)