Retain execution-memory task registration while allocations wait

XMLWordPrintableJSON

      Problem

      ExecutionMemoryPool can remove a task from memoryForTask while an acquireMemory call for the same task is waiting. The waiter later indexes the missing entry and throws NoSuchElementException. This is an execution-memory accounting/lifecycle bug, independent of any external execution engine.

      Minimal interleaving

      Use a 1,000-byte execution pool:

      1. Task B acquires 900 bytes, then task A acquires the remaining 100 bytes.
      2. Another consumer belonging to A requests 300 bytes. It waits because A cannot reach the minimum fair share.
      3. The first consumer of A releases its last 100 bytes. releaseMemory removes A from memoryForTask, even though the other acquisition is still waiting.
      4. The waiter wakes and reads memoryForTask(A), which now throws NoSuchElementException.

      Expected: releasing bytes must not unregister a task with an outstanding waiting acquisition. Once B releases 300 bytes, A should acquire 300 bytes normally.

      The interleaving is reproducible for both ON_HEAP and OFF_HEAP using Spark MemoryConsumer and TaskMemoryManager. It requires concurrent memory operations within one task; ordinary single-threaded allocations do not trigger it. This report does not claim an end-to-end SQL or Python query reproducer.

      Proposed fix

      Track acquisitions only after they actually wait, under the existing memory-manager monitor. Retain a zero-byte task entry while any waiter remains, then remove it when the final waiter exits without a reservation. Clean up the waiter count on success, interruption, and exceptions. Count each task once for fairness, even when it has multiple waiting acquisitions. Keep the existing allocation limits, lock ordering, and storage-reclamation callbacks unchanged.

      Releasing all currently charged bytes is not cancellation of outstanding acquisitions; document that distinction.

      Regression coverage

      Cover last-byte release and releaseAllMemoryForTask, two simultaneous waiters, partial release, interruption, an exception after waiting, and two public MemoryConsumer instances sharing one TaskMemoryManager. Run each scenario in both memory modes.

      The same vulnerable map-removal pattern is present in the inspected Apache Spark master and 3.5/4.0/4.1 branch sources. The proposed patch targets master.

            Assignee:
            Unassigned
            Reporter:
            Chao Sun
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 40m
                40m