Skip to content

Commit 766e0de

Browse files
committed
Simplify: share the grow-and-wrap helper and the test pool lookup, and link the window docs to one section
1 parent 63f51fa commit 766e0de

3 files changed

Lines changed: 22 additions & 26 deletions

File tree

‎docs/source/contributor-guide/memory_management.md‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -663,6 +663,4 @@ A checklist for triaging an executor OOM kill:
663663
4. Check whether the operators involved can spill at all. `ShuffledHashJoin` cannot, so
664664
`spark.comet.exec.forceShuffledHashJoin=true` converts a spillable sort-merge join into one that
665665
is not, for build sides under `spark.comet.exec.forceShuffledHashJoin.maxBuildSize`. Neither can
666-
`CometWindowAggExec`, which runs `PERCENT_RANK`, `CUME_DIST`, `NTILE` and the aggregates whose
667-
frame ends at `UNBOUNDED FOLLOWING`. It needs up to twice a window partition's size within its
668-
share of the pool, and a window partition that does not fit fails the task.
666+
`CometWindowAggExec`; see [Window Functions](../user-guide/latest/tuning/operators.md#window-functions).

‎docs/source/user-guide/latest/migration-guide.md‎

Lines changed: 4 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -114,13 +114,10 @@ now buffer one window partition at a time and reserve it from Comet's memory poo
114114
buffer the task's whole input without reserving anything, so a large or skewed task could grow native
115115
memory past what Comet and Spark could see. A window partition that does not fit in its share of the
116116
pool now fails the task with a memory error that names the settings to change, where Spark would
117-
spill. It needs up to twice its size within that share. Comet does not spill these functions yet.
118-
119-
A deployment that ran such a query only because its memory was untracked can fail it now. There is
120-
no key that restores the untracked behavior. Increase `spark.memory.offHeap.size`, set
121-
`spark.comet.exec.memoryPool=greedy_unified` to leave every limit to Spark, or set
122-
`spark.comet.exec.window.enabled=false` to run window functions in Spark. See
123-
[Window Functions](tuning/operators.md#window-functions).
117+
spill. A deployment that ran such a query only because its memory was untracked can fail it now.
118+
There is no key that restores the untracked behavior. See
119+
[Window Functions](tuning/operators.md#window-functions) for how much memory a window partition
120+
needs and what to change.
124121

125122
## Upgrading to Comet 1.1.0
126123

‎native/core/src/execution/operators/window_agg.rs‎

Lines changed: 17 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -34,9 +34,7 @@
3434
//! reserved against the task's memory pool. The operator still cannot spill, so a window
3535
//! partition that does not fit fails the task with a memory error instead of growing
3636
//! untracked. A window partition spread over several batches needs up to twice its size,
37-
//! because the copy is reserved while the batches it was made from still are. The consumer
38-
//! registers with the pool when the first batch is buffered, so that it does not shrink the
39-
//! share of a sort below it while the sort reads its input.
37+
//! because the copy is reserved while the batches it was made from still are.
4038
//!
4139
//! Once DataFusion's `WindowAggExec` does both, this operator can be removed.
4240
@@ -253,6 +251,11 @@ struct CometWindowAggStream {
253251
}
254252

255253
impl CometWindowAggStream {
254+
/// Reserves `bytes` more, registering the consumer with the pool on first use.
255+
fn grow(&mut self, bytes: usize) -> Result<()> {
256+
self.reservation().try_grow(bytes).map_err(with_oom_context)
257+
}
258+
256259
fn reservation(&mut self) -> &mut MemoryReservation {
257260
self.reservation.get_or_insert_with(|| {
258261
MemoryConsumer::new(self.consumer_name.clone()).register(&self.pool)
@@ -273,9 +276,7 @@ impl CometWindowAggStream {
273276
return Ok(None);
274277
}
275278
let added = self.buffered_memory.count_batch(&batch);
276-
self.reservation()
277-
.try_grow(added)
278-
.map_err(with_oom_context)?;
279+
self.grow(added)?;
279280
if self.partition_by_sort_keys.is_empty() {
280281
self.buffered.push(batch);
281282
return Ok(None);
@@ -360,9 +361,7 @@ impl CometWindowAggStream {
360361
// which is not copied. That is exact, where estimating the copy from each batch would
361362
// charge every batch sliced from one list array for all of its values.
362363
let copy = self.buffered_memory.count_batch(&batch);
363-
self.reservation()
364-
.try_grow(copy)
365-
.map_err(with_oom_context)?;
364+
self.grow(copy)?;
366365
drop(batches);
367366

368367
let mut partition_results = vec![];
@@ -478,7 +477,9 @@ mod tests {
478477
use datafusion::common::ScalarValue;
479478
use datafusion::datasource::memory::MemorySourceConfig;
480479
use datafusion::datasource::source::DataSourceExec;
481-
use datafusion::execution::memory_pool::{GreedyMemoryPool, TrackConsumersPool};
480+
use datafusion::execution::memory_pool::{
481+
GreedyMemoryPool, MemoryConsumerMetrics, TrackConsumersPool,
482+
};
482483
use datafusion::execution::runtime_env::{RuntimeEnv, RuntimeEnvBuilder};
483484
use datafusion::execution::FunctionRegistry;
484485
use datafusion::logical_expr::{
@@ -680,12 +681,16 @@ mod tests {
680681
(runtime, context)
681682
}
682683

683-
fn registered_consumers(runtime: &RuntimeEnv) -> Vec<String> {
684+
fn consumer_metrics(runtime: &RuntimeEnv) -> Vec<MemoryConsumerMetrics> {
684685
runtime
685686
.memory_pool
686687
.downcast_ref::<TrackConsumersPool<GreedyMemoryPool>>()
687688
.unwrap()
688689
.metrics()
690+
}
691+
692+
fn registered_consumers(runtime: &RuntimeEnv) -> Vec<String> {
693+
consumer_metrics(runtime)
689694
.into_iter()
690695
.map(|consumer| consumer.name)
691696
.collect()
@@ -703,11 +708,7 @@ mod tests {
703708
output.push(batch?);
704709
}
705710
// Read the consumer before dropping the stream unregisters it.
706-
let consumer = runtime
707-
.memory_pool
708-
.downcast_ref::<TrackConsumersPool<GreedyMemoryPool>>()
709-
.unwrap()
710-
.metrics()
711+
let consumer = consumer_metrics(&runtime)
711712
.into_iter()
712713
.find(|consumer| consumer.name == CONSUMER)
713714
.unwrap();

0 commit comments

Comments
 (0)