Skip to content

Commit d3406c6

Browse files
authored
fix: let Comet memory pools overcommit on grow instead of panicking (#6128)
* fix: let Comet memory pools overcommit on grow instead of panicking MemoryPool::grow must always succeed: DataFusion calls it for memory that already exists, such as a spilled batch the sort-merge join reads back. Both Comet pools implemented it as try_grow().unwrap(), so a partial grant from Spark panicked the task. Record the full amount, carry the ungranted part as overcommit, and repay it on shrink before releasing bytes to Spark, so Spark is still never handed back more than it granted (#1733). Closes #6127. * refactor: share Spark memory calls and overcommit between Comet pools Both pools duplicated the acquire/release JNI calls and wrapped them in the overcommit ledger separately, and had drifted: only one logged a failed acquire, and try_grow still used Spark's reply unclamped. Move the JNI calls, the partial-grant handback and the overcommit into one SparkMemory, generic over the Spark calls so it is tested with a fake instead of only the ledger. Skip the atomic write on release when there is no overcommit, and pass the fair pool its task attempt id for the warning. * fix: make unified try_grow repay overcommit and address review feedback - try_acquire asks Spark for outstanding overcommit on top of the request, so greedy_unified refuses try_grow while in debt instead of letting shortfalls compound unseen by Spark - document the invariant that keeps Spark from being over-released - replace the generic manager with Box<dyn SparkMemoryManager>, keep task_attempt_id in SparkMemory only, add pool test constructors - pool-level tests that fail with the old try_grow().unwrap() grow - saturating_add in both pools' grow - report overcommit in Display and try_grow errors - update the memory management contributor guide * test: drive the try_acquire repayment race and update the memory review skill Add a deterministic test that lands a concurrent release inside try_acquire's call to Spark, after it has read the debt and before it repays it, and a multi-threaded stress test of CometUnifiedMemoryPool that checks Spark is handed back exactly what it granted. SparkMemoryManager is now Send + Sync so the fake can be shared across threads, and the fake uses parking_lot so that one failed assertion does not poison the lock for every other thread. grow no longer panics when Spark grants less than asked, so the memory review skill now tells reviewers to treat a new grow call site as potential unbacked memory, and notes that only grow keeps a partial grant. * chore: drop the memory pools' redundant unsafe Send and Sync impls With SparkMemoryManager now Send + Sync, both pools are Send and Sync without them, and the compiler checks it: removing the bound makes the MemoryPool impls fail to compile. Since SparkMemory started holding a Box<dyn SparkMemoryManager>, these unsafe impls were the only thing satisfying MemoryPool's Send + Sync bound, and would have let a manager that is not thread-safe through.
1 parent 466e3fe commit d3406c6

6 files changed

Lines changed: 563 additions & 87 deletions

File tree

‎.ai/skills/review-comet-memory-pr/SKILL.md‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -68,12 +68,17 @@ Native operators reserve through DataFusion's `MemoryConsumer` and `MemoryReserv
6868
- [ ] **`try_grow` failure turns into spilling, not an error.** For a spillable operator,
6969
`ResourcesExhausted` is the signal to spill and retry. A new operator that propagates it as a
7070
query failure has converted a recoverable condition into a lost task.
71-
- [ ] **`grow` is infallible and panics if the pool refuses.** It is only correct where the caller
72-
genuinely cannot spill. A new `grow` call site needs that justification.
71+
- [ ] **Treat a new `grow` call site as potential unbacked memory.** `grow` cannot fail. When
72+
Spark grants less than asked, the Comet pools record the full amount anyway and carry the
73+
shortfall as overcommit, which Spark does not know about and can hand to another consumer
74+
or task. Nothing caps how far successive `grow` calls overcommit short of the container
75+
limit. That is acceptable only for memory that already exists and cannot be spilled, such
76+
as a spilled batch read back from disk, so a new call site needs that justification.
77+
Memory the caller is about to allocate belongs behind `try_grow`.
7378
- [ ] **Every `try_grow` has a matching `shrink`, including on the error path.** A reservation
7479
leaked on an early return is charged for the life of the task.
75-
- [ ] **A partial grant is released, not kept.** `acquired < additional` must release and report
76-
`ResourcesExhausted`.
80+
- [ ] **In `try_grow`, a partial grant is released, not kept.** `acquired < additional` must
81+
release and report `ResourcesExhausted`. Only `grow` keeps a partial grant, as overcommit.
7782
- [ ] **An operator that buffers without reserving is invisible to the pool.** If the PR adds
7883
buffering, ask where the reservation is. "It is only a few batches" is how the accounting gap
7984
grows.

‎docs/source/contributor-guide/memory_management.md‎

Lines changed: 18 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -236,8 +236,8 @@ reports the base pool's number.
236236
### The unified pools
237237

238238
`CometUnifiedMemoryPool` and `CometFairMemoryPool` (`unified_pool.rs`, `fair_pool.rs`) are the
239-
bridge to Spark. Their `try_grow` calls `CometTaskMemoryManager.acquireMemory` over JNI, which goes
240-
through Spark's ordinary `TaskMemoryManager`. That means:
239+
bridge to Spark. Their `try_grow` and `grow` both call `CometTaskMemoryManager.acquireMemory` over
240+
JNI, which goes through Spark's ordinary `TaskMemoryManager`. That means:
241241

242242
- Comet competes with Spark's own off-heap consumers (Tungsten sorters, `BytesToBytesMap`, and so
243243
on) for the same `spark.memory.offHeap.size`, and Spark's unified memory manager arbitrates.
@@ -247,8 +247,14 @@ through Spark's ordinary `TaskMemoryManager`. That means:
247247
(`CometTaskMemoryManager.java`). A Spark allocation can therefore never make a native sorter or
248248
aggregate release its reservations. Native operators spill only when their _own_ `try_grow`
249249
fails, so a JVM consumer that is blocked behind native reservations has no way to reclaim them.
250-
- A partial grant (`acquired < additional`) is released immediately and reported as
250+
- In `try_grow`, a partial grant (`acquired < additional`) is released immediately and reported as
251251
`ResourcesExhausted`, which is the signal DataFusion uses to spill.
252+
- `grow` cannot fail, so when Spark grants less than asked the pool records the full amount anyway
253+
and carries the shortfall as _overcommit_ (`spark_memory.rs`). A later `shrink` repays the
254+
overcommit before returning anything to Spark, so Spark is never handed back more than it
255+
granted. While overcommit is outstanding, `try_grow` asks Spark for it on top of the request and
256+
is refused unless Spark can cover both, so operators spill until the debt is repaid. Both pools'
257+
`Display` output and their `try_grow` errors report the current overcommit.
252258

253259
`CometFairMemoryPool` additionally applies a local check before it asks Spark. It divides
254260
`pool_size` by the number of consumers currently registered with the pool and rejects the request if
@@ -289,7 +295,9 @@ Native operators reserve through DataFusion's `MemoryConsumer` / `MemoryReservat
289295
- `try_grow(n)` may fail. Spillable operators (`ExternalSorter`, the grouped hash aggregate,
290296
sort-merge join) respond to a `ResourcesExhausted` error by spilling to disk and retrying. This is
291297
the only mechanism that turns memory pressure into progress rather than failure.
292-
- `grow(n)` is infallible and panics if the pool refuses. It is used where the caller cannot spill.
298+
- `grow(n)` is infallible. It is used for memory that already exists and the caller cannot spill,
299+
such as a spilled batch read back from disk. The Comet pools record it even when Spark grants
300+
less, as overcommit (see [The unified pools](#the-unified-pools)).
293301
- `shrink(n)` returns bytes to the pool.
294302

295303
An operator that never calls `try_grow` is invisible to the pool no matter how much memory it uses.
@@ -367,6 +375,12 @@ diverge for several structural reasons:
367375
Imported JVM batches are reserved only while a reserving operator holds them, and exported native
368376
batches have usually been released by the time the JVM receives them yet stay resident until the
369377
JVM closes them (see [Crossing the FFI boundary](#crossing-the-ffi-boundary)).
378+
- **Overcommit.** Unlike everything above, this is not accidental. When Spark grants less than a
379+
`grow` asked for, Comet holds the difference anyway and deliberately does not tell Spark about it
380+
(see [The unified pools](#the-unified-pools)). `reserved()` includes it, but Spark's memory
381+
manager does not, so until it is repaid Spark can hand the same bytes to another consumer or task.
382+
The `overcommit` figure in the pool's `Display` output and `try_grow` errors shows how much is
383+
outstanding.
370384

371385
The practical consequence is that `reserved()` is a lower bound on Comet's real footprint, and the
372386
gap is workload-dependent. `spark.comet.exec.memoryPool.fraction` exists purely so operators can

‎native/core/src/execution/memory_pools/fair_pool.rs‎

Lines changed: 59 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ use std::{
2222

2323
use jni::objects::{Global, JObject};
2424

25-
use crate::{errors::CometResult, jvm_bridge::JVMClasses};
25+
use super::spark_memory::SparkMemory;
2626
use datafusion::common::resources_err;
2727
use datafusion::execution::memory_pool::MemoryConsumer;
2828
use datafusion::{
@@ -34,7 +34,7 @@ use parking_lot::Mutex;
3434
/// A DataFusion fair `MemoryPool` implementation for Comet. Internally this is
3535
/// implemented via delegating calls to [`crate::jvm_bridge::CometTaskMemoryManager`].
3636
pub struct CometFairMemoryPool {
37-
task_memory_manager_handle: Arc<Global<JObject<'static>>>,
37+
spark: SparkMemory,
3838
pool_size: usize,
3939
state: Mutex<CometFairPoolState>,
4040
}
@@ -51,6 +51,7 @@ impl Debug for CometFairMemoryPool {
5151
.field("pool_size", &self.pool_size)
5252
.field("used", &state.used)
5353
.field("num", &state.num)
54+
.field("overcommit", &self.spark.overcommit())
5455
.finish()
5556
}
5657
}
@@ -59,44 +60,37 @@ impl CometFairMemoryPool {
5960
pub fn new(
6061
task_memory_manager_handle: Arc<Global<JObject<'static>>>,
6162
pool_size: usize,
63+
task_attempt_id: i64,
6264
) -> CometFairMemoryPool {
65+
Self::with_spark(
66+
SparkMemory::new(task_memory_manager_handle, task_attempt_id),
67+
pool_size,
68+
)
69+
}
70+
71+
fn with_spark(spark: SparkMemory, pool_size: usize) -> CometFairMemoryPool {
6372
Self {
64-
task_memory_manager_handle,
73+
spark,
6574
pool_size,
6675
state: Mutex::new(CometFairPoolState { used: 0, num: 0 }),
6776
}
6877
}
69-
70-
fn acquire(&self, additional: usize) -> CometResult<i64> {
71-
let handle = self.task_memory_manager_handle.as_obj();
72-
JVMClasses::with_env(|env| unsafe {
73-
jni_call!(env,
74-
comet_task_memory_manager(handle).acquire_memory(additional as i64) -> i64)
75-
})
76-
}
77-
78-
fn release(&self, size: usize) -> CometResult<()> {
79-
let handle = self.task_memory_manager_handle.as_obj();
80-
JVMClasses::with_env(|env| unsafe {
81-
jni_call!(env, comet_task_memory_manager(handle).release_memory(size as i64) -> ())
82-
})
83-
}
8478
}
8579

8680
impl Display for CometFairMemoryPool {
8781
fn fmt(&self, f: &mut Formatter<'_>) -> FmtResult {
8882
let state = self.state.lock();
8983
write!(
9084
f,
91-
"CometFairMemoryPool(pool_size={}, used={}, num={})",
92-
self.pool_size, state.used, state.num
85+
"CometFairMemoryPool(pool_size={}, used={}, num={}, overcommit={})",
86+
self.pool_size,
87+
state.used,
88+
state.num,
89+
self.spark.overcommit()
9390
)
9491
}
9592
}
9693

97-
unsafe impl Send for CometFairMemoryPool {}
98-
unsafe impl Sync for CometFairMemoryPool {}
99-
10094
impl MemoryPool for CometFairMemoryPool {
10195
fn name(&self) -> &str {
10296
"CometFairMemoryPool"
@@ -118,8 +112,15 @@ impl MemoryPool for CometFairMemoryPool {
118112
.expect("unexpected amount of unregister happened");
119113
}
120114

115+
/// Records memory that already exists, so it must not fail and ignores the fair limit.
116+
/// See [`SparkMemory`].
121117
fn grow(&self, _reservation: &MemoryReservation, additional: usize) {
122-
self.try_grow(_reservation, additional).unwrap();
118+
if additional == 0 {
119+
return;
120+
}
121+
let mut state = self.state.lock();
122+
self.spark.acquire(additional);
123+
state.used = state.used.saturating_add(additional);
123124
}
124125

125126
fn shrink(&self, _reservation: &MemoryReservation, subtractive: usize) {
@@ -134,7 +135,8 @@ impl MemoryPool for CometFairMemoryPool {
134135
state.used
135136
)
136137
}
137-
self.release(subtractive)
138+
self.spark
139+
.release(subtractive)
138140
.unwrap_or_else(|_| panic!("Failed to release {subtractive} bytes"));
139141
state.used = state.used.checked_sub(subtractive).unwrap();
140142
}
@@ -158,21 +160,18 @@ impl MemoryPool for CometFairMemoryPool {
158160
let used = state.used;
159161
if limit < used + additional {
160162
return resources_err!(
161-
"Failed to acquire {additional} bytes where {used} bytes already reserved and the fair limit is {limit} bytes, {num} registered"
163+
"Failed to acquire {additional} bytes where {used} bytes already reserved ({} bytes overcommitted) and the fair limit is {limit} bytes, {num} registered",
164+
self.spark.overcommit()
162165
);
163166
}
164167

165-
let acquired = self.acquire(additional)?;
166-
// If the number of bytes we acquired is less than the requested, return an error,
167-
// and hopefully will trigger spilling from the caller side.
168-
if acquired < additional as i64 {
169-
// Release the acquired bytes before throwing error
170-
self.release(acquired as usize)?;
171-
168+
// A partial grant is handed back and refused, which triggers spilling in the caller.
169+
if let Err(refusal) = self.spark.try_acquire(additional)? {
172170
return resources_err!(
173-
"Failed to acquire {} bytes, only got {} bytes. Reserved: {} bytes",
171+
"Failed to acquire {} bytes plus {} bytes overcommitted, only got {} bytes. Reserved: {} bytes",
174172
additional,
175-
acquired,
173+
refusal.overcommit,
174+
refusal.granted,
176175
state.used
177176
);
178177
}
@@ -188,3 +187,28 @@ impl MemoryPool for CometFairMemoryPool {
188187
self.state.lock().used
189188
}
190189
}
190+
191+
#[cfg(test)]
192+
mod tests {
193+
use super::super::spark_memory::fake::FakeSpark;
194+
use super::*;
195+
196+
#[test]
197+
fn grow_past_the_fair_limit_is_recorded_and_refuses_the_next_try_grow() {
198+
let fake = FakeSpark::with(100);
199+
let pool: Arc<dyn MemoryPool> =
200+
Arc::new(CometFairMemoryPool::with_spark(fake.memory(), 100));
201+
let reservation = MemoryConsumer::new("smj").register(&pool);
202+
203+
// Past both the fair limit and what Spark will grant.
204+
reservation.grow(150);
205+
assert_eq!(pool.reserved(), 150);
206+
assert_eq!(fake.held(), 100);
207+
assert!(reservation.try_grow(1).is_err());
208+
209+
drop(reservation);
210+
assert_eq!(pool.reserved(), 0);
211+
// Spark gets back exactly the 100 bytes it granted.
212+
assert_eq!(fake.released(), vec![100]);
213+
}
214+
}

‎native/core/src/execution/memory_pools/mod.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
mod config;
1919
mod fair_pool;
2020
pub mod logging_pool;
21+
mod spark_memory;
2122
mod task_shared;
2223
mod unified_pool;
2324

@@ -66,6 +67,7 @@ pub(crate) fn create_memory_pool(
6667
tracked(CometFairMemoryPool::new(
6768
comet_task_memory_manager,
6869
pool_size,
70+
task_attempt_id,
6971
))
7072
}),
7173
MemoryPoolType::GreedyTaskShared => acquire_task_shared_pool(task_attempt_id, || {

0 commit comments

Comments
 (0)