You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit e900844
Browse filesBrowse the repository at this point in the historyBrowse files
fix: check each fair_unified reservation against its own share (#6205)
* fix: check each fair_unified reservation against its own share
Since the DataFusion 53 upgrade (#3629), CometFairMemoryPool::try_grow has
compared the pool's total reserved bytes against pool_size / num_consumers.
That caps the whole task at one consumer's share, and every new registration
tightens the ceiling on the consumers already running.
The comment that justified the change holds for shrink but not for try_grow.
MemoryReservation::try_grow calls pool.try_grow before adding to its size, so
reservation.size() is the size before the request. Check the reservation
against its share again, and separately keep the pool's total within
pool_size, since the shares alone do not bound it once a consumer registers
late or one consumer holds sibling reservations.
Update the tuning guide, the memory management guide and the memory PR review
skill, which described the pool-wide comparison.
Closes#5961.
* fix: charge sibling fair_unified reservations to their consumer's share
Checking each reservation against the share let an operator that holds
several reservations under one consumer, such as a sort's streaming merge
with one per batch, take other consumers' shares, bounded only by the pool
total. Keep a running total for each consumer id instead, so all of a
consumer's reservations count against one share. The pool no longer reads
reservation.size(), so the check also stops depending on when DataFusion
updates it.
Note in the tuning guide that tasks can reserve more before they spill
than they could in Comet 0.15.0 through 1.0.0.
Copy file name to clipboardExpand all lines: docs/source/contributor-guide/memory_management.md
+27-14Lines changed: 27 additions & 14 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -266,21 +266,34 @@ JNI, which goes through Spark's ordinary `TaskMemoryManager`. That means:
266
266
is refused unless Spark can cover both, so operators spill until the debt is repaid. Both pools'
267
267
`Display` output and their `try_grow` errors report the current overcommit.
268
268
269
-
`CometFairMemoryPool` additionally applies a local check before it asks Spark. It divides
270
-
`pool_size` by the number of consumers currently registered with the pool and rejects the request if
271
-
the pool's _total_ reserved bytes plus the request would exceed that quotient. Two details matter
272
-
for tuning:
273
-
274
-
- The comparison is against the shared total (`state.used`), not against the requesting consumer's
275
-
own reservation. With an 8 GiB pool and two registered consumers, once one of them holds 3 GiB a
276
-
2 GiB request from the other is rejected, even though neither would exceed 4 GiB. In effect the
277
-
usable pool shrinks to `pool_size / num_consumers` in aggregate as soon as more than one consumer
278
-
is registered.
279
-
- The pool is task-shared (see below), so `num_consumers` counts every registered consumer across
280
-
every native plan in the task, not just the plan making the request.
269
+
`CometFairMemoryPool` additionally applies two local checks before it asks Spark, and refuses the
270
+
request without calling Spark if either fails:
271
+
272
+
-**The requesting consumer against its share.** The share is `pool_size` divided by the number of
273
+
consumers currently registered with the pool. What the consumer already holds plus the request
274
+
must not exceed it. That counts all of the consumer's reservations, including the sibling
275
+
reservations that `new_empty()`, `split()` and `take()` create, so an operator that holds several
276
+
reservations still gets one share. A sort's streaming merge, for example, creates one for each
277
+
batch it reads. The pool keeps a running total for each consumer, because `reservation.size()`
278
+
covers only one reservation. With an 8 GiB pool and two registered consumers, each share is
279
+
4 GiB, so once one consumer holds 3 GiB the other can still reserve up to 4 GiB.
280
+
-**The pool's total against `pool_size`.** The shares alone do not bound the total, because a
281
+
consumer keeps what it reserved while fewer consumers were registered. `pool_size` is computed
282
+
for the executor but applied to each task's pool, so Spark's own limit on the task is at least as
283
+
tight unless the deprecated `spark.comet.exec.memoryPool.fraction` is below `1.0`.
284
+
285
+
Two details matter for tuning:
281
286
282
-
This is why `fair_unified` spills earlier than `greedy_unified`. It is not a per-consumer quota, and
283
-
reading it as one overstates the memory a multi-operator task can use.
287
+
- The pool is task-shared (see below), so `num_consumers` counts every registered consumer across
288
+
every native plan in the task, not just the plan making the request. Registering a consumer
289
+
lowers every other consumer's share, but does not take back memory they already hold.
290
+
- Every registered consumer gets a share, whether or not it can spill. DataFusion's
291
+
`FairSpillPool` divides the pool among spillable consumers only.
292
+
[Issue #5465](https://github.com/apache/datafusion-comet/issues/5465) tracks doing the same here.
293
+
294
+
This is why `fair_unified` can spill earlier than `greedy_unified`: a consumer at its share is
295
+
refused even when the rest of the pool is free, which keeps that memory for the task's other
Copy file name to clipboardExpand all lines: docs/source/user-guide/latest/tuning.md
+8-1Lines changed: 8 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -133,10 +133,17 @@ pre-shuffle operators and one for the shuffle writer). The shared pool ensures t
133
133
memory usage stays within the per-task limit.
134
134
135
135
The `fair_unified` pool prevents operators from using more than an even fraction of the available memory
136
-
(i.e. `pool_size / num_reservations`). This pool works best when you know beforehand
136
+
(i.e. `pool_size / num_consumers`, where `num_consumers` counts the memory consumers registered by all of the task's
137
+
native plans). This pool works best when you know beforehand
137
138
the query has multiple operators that will likely all need to spill. Sometimes it will cause spills even
138
139
when there is sufficient memory in order to leave enough memory for other operators.
139
140
141
+
Comet 0.15.0 through 1.0.0 capped the memory of all of a task's operators combined at one operator's share, because of a
142
+
bug ([#5961](https://github.com/apache/datafusion-comet/issues/5961)). Tasks with several operators can now reserve more
143
+
memory before they spill than they could in those releases. The difference is largest on executors that run few tasks
144
+
at once, where Spark's own limit on each task is loosest. If you sized executor memory against one of those releases,
145
+
check that executors still have enough headroom; see [Sizing the Overhead from the Memory Usage Log].
146
+
140
147
The `greedy_unified` pool type implements a greedy first-come first-serve limit. This pool works well for queries that do not
141
148
need to spill or have a single spillable operator.
// We use state.used instead of reservation.size() because DataFusion 53+
158
-
// calls pool.try_grow() before incrementing the reservation's atomic size,
159
-
// so reservation.size() would not include prior grows.
160
-
let used = state.used;
161
-
if limit < used + additional {
170
+
let consumer_used = *state.consumer_used(reservation);
171
+
if limit < consumer_used.saturating_add(additional){
162
172
returnresources_err!(
163
-
"Failed to acquire {additional} bytes where {used} bytes already reserved ({} bytes overcommitted) and the fair limit is {limit} bytes, {num} registered",
173
+
"Failed to acquire {additional} bytes where this consumer already holds {consumer_used} bytes and the fair limit is {limit} bytes, {num} registered ({} bytes overcommitted)",
164
174
self.spark.overcommit()
165
175
);
166
176
}
177
+
// The shares alone do not bound the pool's total, because a consumer keeps what it
0 commit comments