Skip to content

Commit e900844

Browse files
authored
fix: check each fair_unified reservation against its own share (#6205)
* fix: check each fair_unified reservation against its own share Since the DataFusion 53 upgrade (#3629), CometFairMemoryPool::try_grow has compared the pool's total reserved bytes against pool_size / num_consumers. That caps the whole task at one consumer's share, and every new registration tightens the ceiling on the consumers already running. The comment that justified the change holds for shrink but not for try_grow. MemoryReservation::try_grow calls pool.try_grow before adding to its size, so reservation.size() is the size before the request. Check the reservation against its share again, and separately keep the pool's total within pool_size, since the shares alone do not bound it once a consumer registers late or one consumer holds sibling reservations. Update the tuning guide, the memory management guide and the memory PR review skill, which described the pool-wide comparison. Closes #5961. * fix: charge sibling fair_unified reservations to their consumer's share Checking each reservation against the share let an operator that holds several reservations under one consumer, such as a sort's streaming merge with one per batch, take other consumers' shares, bounded only by the pool total. Keep a running total for each consumer id instead, so all of a consumer's reservations count against one share. The pool no longer reads reservation.size(), so the check also stops depending on when DataFusion updates it. Note in the tuning guide that tasks can reserve more before they spill than they could in Comet 0.15.0 through 1.0.0.
1 parent 570a97c commit e900844

4 files changed

Lines changed: 206 additions & 54 deletions

File tree

‎.ai/skills/review-comet-memory-pr/SKILL.md‎

Lines changed: 13 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -100,11 +100,17 @@ configuration, from the inside out:
100100

101101
- [ ] A new decorator forwards **every** `MemoryPool` method to its inner pool. A partial
102102
implementation makes `reserved()` disagree between levels.
103-
- [ ] **`fair_unified` compares against the shared total, not a per-consumer quota.** It divides
104-
`pool_size` by the number of registered consumers and rejects if the pool's total reserved
105-
plus the request exceeds that quotient. With two consumers and an 8 GiB pool, once one holds
106-
3 GiB a 2 GiB request from the other is refused. If the PR changes this comparison it changes
107-
when every query spills, so it needs benchmark evidence, not reasoning.
103+
- [ ] **`fair_unified` checks the requesting consumer against its share, and the total against
104+
the pool.** It divides `pool_size` by the number of registered consumers and rejects if what
105+
the consumer holds plus the request exceeds that quotient, or if the pool's total reserved
106+
plus the request exceeds `pool_size`. With two consumers and an 8 GiB pool, once one holds
107+
3 GiB the other can still reserve up to 4 GiB. The pool keeps a running total for each
108+
consumer id, so the sibling reservations that `new_empty()`, `split()` and `take()` create
109+
count against one share. A PR that checks `reservation.size()` instead lets an operator with
110+
several reservations, such as a sort's streaming merge, take other consumers' shares. It
111+
also depends on when DataFusion updates the size, which is after it calls `try_grow` but
112+
before it calls `shrink`. If the PR changes either comparison it changes when every query
113+
spills, so it needs benchmark evidence, not reasoning.
108114
- [ ] **`num_consumers` counts every consumer in the task**, across every native plan, because the
109115
pool is task-shared.
110116
- [ ] **Task-shared pool lifetime.** `acquire_task_shared_pool` keeps a process-wide
@@ -194,7 +200,8 @@ will not catch. Check:
194200
- The **pool stack** diagram, if a decorator is added, removed, or reordered
195201
- The **pool type** table, if a pool type is added, removed, or changes how it is sized
196202
- The `memory_limit` formula, if the budget calculation changes
197-
- The **fair pool** description, which spells out the shared-total comparison and its consequence
203+
- The **fair pool** description, which spells out the share and pool-total checks and their
204+
consequences
198205
- The **Crossing the FFI boundary** section, which names the operators that reserve for imported
199206
batches
200207
- The **Open problems** list. If the PR closes one of these gaps, the entry must be removed or

‎docs/source/contributor-guide/memory_management.md‎

Lines changed: 27 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -266,21 +266,34 @@ JNI, which goes through Spark's ordinary `TaskMemoryManager`. That means:
266266
is refused unless Spark can cover both, so operators spill until the debt is repaid. Both pools'
267267
`Display` output and their `try_grow` errors report the current overcommit.
268268

269-
`CometFairMemoryPool` additionally applies a local check before it asks Spark. It divides
270-
`pool_size` by the number of consumers currently registered with the pool and rejects the request if
271-
the pool's _total_ reserved bytes plus the request would exceed that quotient. Two details matter
272-
for tuning:
273-
274-
- The comparison is against the shared total (`state.used`), not against the requesting consumer's
275-
own reservation. With an 8 GiB pool and two registered consumers, once one of them holds 3 GiB a
276-
2 GiB request from the other is rejected, even though neither would exceed 4 GiB. In effect the
277-
usable pool shrinks to `pool_size / num_consumers` in aggregate as soon as more than one consumer
278-
is registered.
279-
- The pool is task-shared (see below), so `num_consumers` counts every registered consumer across
280-
every native plan in the task, not just the plan making the request.
269+
`CometFairMemoryPool` additionally applies two local checks before it asks Spark, and refuses the
270+
request without calling Spark if either fails:
271+
272+
- **The requesting consumer against its share.** The share is `pool_size` divided by the number of
273+
consumers currently registered with the pool. What the consumer already holds plus the request
274+
must not exceed it. That counts all of the consumer's reservations, including the sibling
275+
reservations that `new_empty()`, `split()` and `take()` create, so an operator that holds several
276+
reservations still gets one share. A sort's streaming merge, for example, creates one for each
277+
batch it reads. The pool keeps a running total for each consumer, because `reservation.size()`
278+
covers only one reservation. With an 8 GiB pool and two registered consumers, each share is
279+
4 GiB, so once one consumer holds 3 GiB the other can still reserve up to 4 GiB.
280+
- **The pool's total against `pool_size`.** The shares alone do not bound the total, because a
281+
consumer keeps what it reserved while fewer consumers were registered. `pool_size` is computed
282+
for the executor but applied to each task's pool, so Spark's own limit on the task is at least as
283+
tight unless the deprecated `spark.comet.exec.memoryPool.fraction` is below `1.0`.
284+
285+
Two details matter for tuning:
281286

282-
This is why `fair_unified` spills earlier than `greedy_unified`. It is not a per-consumer quota, and
283-
reading it as one overstates the memory a multi-operator task can use.
287+
- The pool is task-shared (see below), so `num_consumers` counts every registered consumer across
288+
every native plan in the task, not just the plan making the request. Registering a consumer
289+
lowers every other consumer's share, but does not take back memory they already hold.
290+
- Every registered consumer gets a share, whether or not it can spill. DataFusion's
291+
`FairSpillPool` divides the pool among spillable consumers only.
292+
[Issue #5465](https://github.com/apache/datafusion-comet/issues/5465) tracks doing the same here.
293+
294+
This is why `fair_unified` can spill earlier than `greedy_unified`: a consumer at its share is
295+
refused even when the rest of the pool is free, which keeps that memory for the task's other
296+
consumers.
284297

285298
### Task-shared pools and their lifetime
286299

‎docs/source/user-guide/latest/tuning.md‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -133,10 +133,17 @@ pre-shuffle operators and one for the shuffle writer). The shared pool ensures t
133133
memory usage stays within the per-task limit.
134134

135135
The `fair_unified` pool prevents operators from using more than an even fraction of the available memory
136-
(i.e. `pool_size / num_reservations`). This pool works best when you know beforehand
136+
(i.e. `pool_size / num_consumers`, where `num_consumers` counts the memory consumers registered by all of the task's
137+
native plans). This pool works best when you know beforehand
137138
the query has multiple operators that will likely all need to spill. Sometimes it will cause spills even
138139
when there is sufficient memory in order to leave enough memory for other operators.
139140

141+
Comet 0.15.0 through 1.0.0 capped the memory of all of a task's operators combined at one operator's share, because of a
142+
bug ([#5961](https://github.com/apache/datafusion-comet/issues/5961)). Tasks with several operators can now reserve more
143+
memory before they spill than they could in those releases. The difference is largest on executors that run few tasks
144+
at once, where Spark's own limit on each task is loosest. If you sized executor memory against one of those releases,
145+
check that executors still have enough headroom; see [Sizing the Overhead from the Memory Usage Log].
146+
140147
The `greedy_unified` pool type implements a greedy first-come first-serve limit. This pool works well for queries that do not
141148
need to spill or have a single spillable operator.
142149

‎native/core/src/execution/memory_pools/fair_pool.rs‎

Lines changed: 158 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
// under the License.
1717

1818
use std::{
19+
collections::HashMap,
1920
fmt::{Debug, Display, Formatter, Result as FmtResult},
2021
sync::Arc,
2122
};
@@ -41,7 +42,21 @@ pub struct CometFairMemoryPool {
4142

4243
struct CometFairPoolState {
4344
used: usize,
44-
num: usize,
45+
/// Bytes held by each registered consumer, keyed by [`MemoryConsumer::id`]. The sibling
46+
/// reservations that `new_empty()`, `split()` and `take()` create belong to the same consumer,
47+
/// so they count against one share. The pool keeps these totals itself because
48+
/// `reservation.size()` covers only one reservation, and DataFusion updates it after calling
49+
/// `try_grow` but before calling `shrink`.
50+
consumers: HashMap<usize, usize>,
51+
}
52+
53+
impl CometFairPoolState {
54+
/// The bytes held by `reservation`'s consumer across all of its reservations.
55+
fn consumer_used(&mut self, reservation: &MemoryReservation) -> &mut usize {
56+
self.consumers
57+
.get_mut(&reservation.consumer().id())
58+
.expect("reservation's consumer is not registered with the pool")
59+
}
4560
}
4661

4762
impl Debug for CometFairMemoryPool {
@@ -50,7 +65,7 @@ impl Debug for CometFairMemoryPool {
5065
f.debug_struct("CometFairMemoryPool")
5166
.field("pool_size", &self.pool_size)
5267
.field("used", &state.used)
53-
.field("num", &state.num)
68+
.field("num", &state.consumers.len())
5469
.field("overcommit", &self.spark.overcommit())
5570
.finish()
5671
}
@@ -72,7 +87,10 @@ impl CometFairMemoryPool {
7287
Self {
7388
spark,
7489
pool_size,
75-
state: Mutex::new(CometFairPoolState { used: 0, num: 0 }),
90+
state: Mutex::new(CometFairPoolState {
91+
used: 0,
92+
consumers: HashMap::new(),
93+
}),
7694
}
7795
}
7896
}
@@ -85,7 +103,7 @@ impl Display for CometFairMemoryPool {
85103
"CometFairMemoryPool(pool_size={}, used={}, num={}, overcommit={})",
86104
self.pool_size,
87105
state.used,
88-
state.num,
106+
state.consumers.len(),
89107
self.spark.overcommit()
90108
)
91109
}
@@ -96,74 +114,76 @@ impl MemoryPool for CometFairMemoryPool {
96114
"CometFairMemoryPool"
97115
}
98116

99-
fn register(&self, _: &MemoryConsumer) {
100-
let mut state = self.state.lock();
101-
state.num = state
102-
.num
103-
.checked_add(1)
104-
.expect("unexpected amount of register happened");
117+
fn register(&self, consumer: &MemoryConsumer) {
118+
self.state.lock().consumers.insert(consumer.id(), 0);
105119
}
106120

107-
fn unregister(&self, _: &MemoryConsumer) {
108-
let mut state = self.state.lock();
109-
state.num = state
110-
.num
111-
.checked_sub(1)
112-
.expect("unexpected amount of unregister happened");
121+
fn unregister(&self, consumer: &MemoryConsumer) {
122+
// DataFusion unregisters a consumer after its last reservation has dropped and released
123+
// its bytes. If a release panicked, this runs while unwinding, so it must not panic too.
124+
self.state.lock().consumers.remove(&consumer.id());
113125
}
114126

115-
/// Records memory that already exists, so it must not fail and ignores the fair limit.
127+
/// Records memory that already exists, so it must not fail and ignores the fair and pool
128+
/// limits.
116129
/// See [`SparkMemory`].
117-
fn grow(&self, _reservation: &MemoryReservation, additional: usize) {
130+
fn grow(&self, reservation: &MemoryReservation, additional: usize) {
118131
if additional == 0 {
119132
return;
120133
}
121134
let mut state = self.state.lock();
122135
self.spark.acquire(additional);
123136
state.used = state.used.saturating_add(additional);
137+
let consumer_used = state.consumer_used(reservation);
138+
*consumer_used = consumer_used.saturating_add(additional);
124139
}
125140

126-
fn shrink(&self, _reservation: &MemoryReservation, subtractive: usize) {
141+
fn shrink(&self, reservation: &MemoryReservation, subtractive: usize) {
127142
if subtractive > 0 {
128143
let mut state = self.state.lock();
129-
// We don't use reservation.size() here because DataFusion 53+ decrements
130-
// the reservation's atomic size before calling pool.shrink(), so it would
131-
// reflect the post-shrink value rather than the pre-shrink value.
132-
if state.used < subtractive {
144+
let consumer_used = *state.consumer_used(reservation);
145+
if consumer_used < subtractive {
133146
panic!(
134-
"Failed to release {subtractive} bytes where only {} bytes tracked by pool",
135-
state.used
147+
"Failed to release {subtractive} bytes where only {consumer_used} bytes tracked for the consumer"
136148
)
137149
}
138150
self.spark
139151
.release(subtractive)
140152
.unwrap_or_else(|_| panic!("Failed to release {subtractive} bytes"));
141153
state.used = state.used.checked_sub(subtractive).unwrap();
154+
*state.consumer_used(reservation) -= subtractive;
142155
}
143156
}
144157

145158
fn try_grow(
146159
&self,
147-
_reservation: &MemoryReservation,
160+
reservation: &MemoryReservation,
148161
additional: usize,
149162
) -> Result<(), DataFusionError> {
150163
if additional > 0 {
151164
let mut state = self.state.lock();
152-
let num = state.num;
165+
let num = state.consumers.len();
153166
let limit = self
154167
.pool_size
155168
.checked_div(num)
156169
.expect("overflow in checked_div");
157-
// We use state.used instead of reservation.size() because DataFusion 53+
158-
// calls pool.try_grow() before incrementing the reservation's atomic size,
159-
// so reservation.size() would not include prior grows.
160-
let used = state.used;
161-
if limit < used + additional {
170+
let consumer_used = *state.consumer_used(reservation);
171+
if limit < consumer_used.saturating_add(additional) {
162172
return resources_err!(
163-
"Failed to acquire {additional} bytes where {used} bytes already reserved ({} bytes overcommitted) and the fair limit is {limit} bytes, {num} registered",
173+
"Failed to acquire {additional} bytes where this consumer already holds {consumer_used} bytes and the fair limit is {limit} bytes, {num} registered ({} bytes overcommitted)",
164174
self.spark.overcommit()
165175
);
166176
}
177+
// The shares alone do not bound the pool's total, because a consumer keeps what it
178+
// reserved before another consumer registered.
179+
let used = state.used;
180+
if self.pool_size < used.saturating_add(additional) {
181+
return resources_err!(
182+
"Failed to acquire {additional} bytes where {used} bytes already reserved ({} bytes overcommitted) and the pool limit is {} bytes",
183+
self.spark.overcommit(),
184+
self.pool_size
185+
);
186+
}
167187

168188
// A partial grant is handed back and refused, which triggers spilling in the caller.
169189
if let Err(refusal) = self.spark.try_acquire(additional)? {
@@ -179,6 +199,7 @@ impl MemoryPool for CometFairMemoryPool {
179199
.used
180200
.checked_add(additional)
181201
.expect("overflow in checked_add");
202+
*state.consumer_used(reservation) += additional;
182203
}
183204
Ok(())
184205
}
@@ -211,4 +232,108 @@ mod tests {
211232
// Spark gets back exactly the 100 bytes it granted.
212233
assert_eq!(fake.released(), vec![100]);
213234
}
235+
236+
#[test]
237+
fn each_consumer_is_limited_to_its_own_share() {
238+
// Spark grants everything, so only the pool's own checks refuse.
239+
let fake = FakeSpark::with(usize::MAX);
240+
let pool: Arc<dyn MemoryPool> =
241+
Arc::new(CometFairMemoryPool::with_spark(fake.memory(), 100));
242+
let first = MemoryConsumer::new("first").register(&pool);
243+
let second = MemoryConsumer::new("second").register(&pool);
244+
245+
// Each consumer's share is 50 bytes, whatever the other one holds.
246+
first.try_grow(40).unwrap();
247+
second.try_grow(20).unwrap();
248+
first.try_grow(10).unwrap();
249+
assert!(first.try_grow(1).is_err());
250+
second.try_grow(30).unwrap();
251+
assert!(second.try_grow(1).is_err());
252+
assert_eq!(pool.reserved(), 100);
253+
assert_eq!(fake.held(), 100);
254+
}
255+
256+
#[test]
257+
fn a_consumer_registered_late_is_limited_by_the_pool_total() {
258+
let fake = FakeSpark::with(usize::MAX);
259+
let pool: Arc<dyn MemoryPool> =
260+
Arc::new(CometFairMemoryPool::with_spark(fake.memory(), 90));
261+
let first = MemoryConsumer::new("first").register(&pool);
262+
// Alone, the first consumer's share is the whole pool.
263+
first.try_grow(60).unwrap();
264+
265+
// A second consumer halves both shares, but the first keeps the 60 bytes it holds.
266+
let second = MemoryConsumer::new("second").register(&pool);
267+
assert!(first.try_grow(1).is_err());
268+
second.try_grow(30).unwrap();
269+
// The second consumer is 15 bytes under its share, but the pool is full.
270+
assert!(second.try_grow(1).is_err());
271+
assert_eq!(pool.reserved(), 90);
272+
273+
// Once the first consumer releases memory, the second can use the rest of its share.
274+
first.shrink(30);
275+
second.try_grow(15).unwrap();
276+
assert!(second.try_grow(1).is_err());
277+
278+
// Unregistering the second consumer gives the first the whole pool again.
279+
drop(second);
280+
first.try_grow(60).unwrap();
281+
assert_eq!(pool.reserved(), 90);
282+
assert_eq!(fake.held(), 90);
283+
}
284+
285+
#[test]
286+
fn sibling_reservations_draw_on_one_share() {
287+
let fake = FakeSpark::with(usize::MAX);
288+
let pool: Arc<dyn MemoryPool> =
289+
Arc::new(CometFairMemoryPool::with_spark(fake.memory(), 100));
290+
let first = MemoryConsumer::new("first").register(&pool);
291+
// Like the reservation that a sort's streaming merge creates for each batch it reads.
292+
let sibling = first.new_empty();
293+
let second = MemoryConsumer::new("second").register(&pool);
294+
295+
// Both of the first consumer's reservations draw on its one 50-byte share.
296+
first.try_grow(40).unwrap();
297+
assert!(sibling.try_grow(40).is_err());
298+
sibling.try_grow(10).unwrap();
299+
assert!(first.try_grow(1).is_err());
300+
assert!(sibling.try_grow(1).is_err());
301+
302+
// So the second consumer can still reserve its whole share.
303+
second.try_grow(50).unwrap();
304+
assert_eq!(pool.reserved(), 100);
305+
306+
// Dropping a reservation returns its bytes to the consumer's share. The consumer stays
307+
// registered while its other reservation lives.
308+
drop(first);
309+
sibling.try_grow(40).unwrap();
310+
assert!(sibling.try_grow(1).is_err());
311+
assert_eq!(pool.reserved(), 100);
312+
assert_eq!(fake.held(), 100);
313+
}
314+
315+
#[test]
316+
fn split_and_take_keep_the_bytes_on_their_consumer() {
317+
let fake = FakeSpark::with(usize::MAX);
318+
let pool: Arc<dyn MemoryPool> =
319+
Arc::new(CometFairMemoryPool::with_spark(fake.memory(), 100));
320+
let mut first = MemoryConsumer::new("first").register(&pool);
321+
let _second = MemoryConsumer::new("second").register(&pool);
322+
323+
first.try_grow(50).unwrap();
324+
let split = first.split(20);
325+
let taken = first.take();
326+
// The consumer's 50 bytes now sit in split and taken, and the emptied reservation gets no
327+
// share of its own.
328+
for reservation in [&first, &split, &taken] {
329+
assert!(reservation.try_grow(1).is_err());
330+
}
331+
332+
// Shrinking one reservation makes room in the share for another.
333+
split.shrink(10);
334+
first.try_grow(10).unwrap();
335+
assert!(taken.try_grow(1).is_err());
336+
assert_eq!(pool.reserved(), 50);
337+
assert_eq!(fake.held(), 50);
338+
}
214339
}

0 commit comments

Comments
 (0)