@@ -100,19 +100,27 @@ configuration, from the inside out:
100100
101101- [ ] A new decorator forwards ** every** ` MemoryPool ` method to its inner pool. A partial
102102 implementation makes ` reserved() ` disagree between levels.
103- - [ ] ** ` fair_unified ` checks the requesting consumer against its share, and the total against
104- the pool.** It divides ` pool_size ` by the number of registered consumers and rejects if what
105- the consumer holds plus the request exceeds that quotient, or if the pool's total reserved
106- plus the request exceeds ` pool_size ` . With two consumers and an 8 GiB pool, once one holds
107- 3 GiB the other can still reserve up to 4 GiB. The pool keeps a running total for each
108- consumer id, so the sibling reservations that ` new_empty() ` , ` split() ` and ` take() ` create
109- count against one share. A PR that checks ` reservation.size() ` instead lets an operator with
110- several reservations, such as a sort's streaming merge, take other consumers' shares. It
111- also depends on when DataFusion updates the size, which is after it calls ` try_grow ` but
112- before it calls ` shrink ` . If the PR changes either comparison it changes when every query
113- spills, so it needs benchmark evidence, not reasoning.
114- - [ ] ** ` num_consumers ` counts every consumer in the task** , across every native plan, because the
115- pool is task-shared.
103+ - [ ] ** ` fair_unified ` holds the consumers that can spill to a share of the task's share.** The
104+ task's share is ` spark.memory.offHeap.size ` divided by the tasks on the executor running
105+ native plans, which ` RunningTasks ` counts while each task holds its task-shared pool, or
106+ ` pool_size ` if that is smaller. A consumer that can spill is
107+ refused if what it holds plus the request exceeds what the consumers that cannot spill leave
108+ of the task's share, divided by the registered consumers that can spill, or if the task's
109+ total plus the request exceeds the task's share. A consumer that cannot spill is refused only
110+ if the task's total plus the request exceeds ` pool_size ` , which binds before Spark's own
111+ limit only while the deprecated fraction is below ` 1 / N ` . With eight running tasks, a 16 GiB
112+ pool and two consumers that can spill, each may hold 1 GiB, or 768 MiB while a merge holds
113+ 512 MiB. The pool keeps a running total for each consumer id, so the sibling reservations
114+ that ` new_empty() ` , ` split() ` and ` take() ` create count against one share. A PR that checks
115+ ` reservation.size() ` instead lets an operator with several reservations, such as a sort's
116+ streaming merge, take other consumers' shares. It also depends on when DataFusion updates the
117+ size, which is after it calls ` try_grow ` but before it calls ` shrink ` . A PR that limits a
118+ consumer that cannot spill to a share, or sizes the shares from the whole pool, brings back
119+ #7088 : one operator takes the task's share from Spark and a merge or hash join build fails
120+ the task. If the PR changes any of these comparisons it changes when every query spills, so
121+ it needs benchmark evidence, not reasoning.
122+ - [ ] ** The consumers that divide the task's share are every consumer that can spill in the
123+ task** , across every native plan, because the pool is task-shared.
116124- [ ] ** Task-shared pool lifetime.** ` acquire_task_shared_pool ` keeps a process-wide
117125 ` HashMap<task_attempt_id, Weak<TaskSharedMemoryPool>> ` . The returned ` Arc ` is the only
118126 lifetime handle, so there is no explicit release to forget and a ` createPlan ` that fails
@@ -143,14 +151,16 @@ reclamation, that is a significant change and needs to say what it does to in-fl
143151## 5. Budget and Configuration
144152
145153` CometExecIterator.getMemoryConfig ` computes the budget once per executor and passes it across JNI
146- as ` memory_limit ` :
154+ as ` memory_limit ` , along with ` spark.memory.offHeap.size ` itself as ` off_heap_size ` , which
155+ ` fair_unified ` divides among the tasks running native plans:
147156
148157``` text
149158memory_limit = spark.memory.offHeap.size * spark.comet.exec.memoryPool.fraction
159+ off_heap_size = spark.memory.offHeap.size
150160```
151161
152162- [ ] A new memory config goes through ` getMemoryConfig ` rather than being read separately in
153- native code.
163+ native code, and a new value crossing JNI is added to ` createPlan ` on both sides .
154164- [ ] Changing ` memoryPool.fraction ` semantics affects every deployment that tuned it as a haircut
155165 for the accounting gap.
156166- [ ] On Kubernetes, ` spark.memory.offHeap.size ` is ** part of** the pod limit, not headroom on top
@@ -204,8 +214,8 @@ will not catch. Check:
204214- The ** pool stack** diagram, if a decorator is added, removed, or reordered
205215- The ** pool type** table, if a pool type is added, removed, or changes how it is sized
206216- The ` memory_limit ` formula, if the budget calculation changes
207- - The ** fair pool** description, which spells out the share and pool-total checks and their
208- consequences
217+ - The ** fair pool** description, which spells out the task share, the share and task-total
218+ checks, and their consequences
209219- The ** Crossing the FFI boundary** section, which names the operators that reserve for imported
210220 batches
211221- The ** Open problems** list. If the PR closes one of these gaps, the entry must be removed or
0 commit comments