Repository navigation
Pool.imap(buffersize=...) blocks all other tasks on the pool until the iterator is consumed #158677
Description
Bug report
Bug description:
While an imap() or imap_unordered() iterator created with buffersize is waiting to be consumed, no other task submitted to the same pool is sent to the workers. If the program needs that other work before it reads more of the iterator, it deadlocks.
import multiprocessing as mp
def sq(x):
return x * x
def neg(x):
return -x
if __name__ == '__main__':
with mp.Pool(2) as pool:
a = pool.imap(sq, range(10), buffersize=4)
b = pool.imap(neg, range(10), buffersize=4)
print(list(zip(a, b))) # hangs foreverWithout buffersize, this prints the ten pairs immediately.
A second form, with a single buffered iterator:
with mp.Pool(2) as pool:
a = pool.imap(sq, range(10), buffersize=4)
pool.apply_async(sq, (7,)).get(timeout=5) # raises TimeoutErrorThreadPool and imap_unordered() behave the same way.
Cause: the pool has a single task handler thread (Pool._handle_tasks). It iterates the generator returned by _guarded_task_generation(), and that generator blocks in sema.acquire() when the buffer is full. While it is blocked there, the thread cannot take anything else from _taskqueue.
Thread dump during the zip hang:
Thread [Thread-2 (_handle_tasks)]:
File "Lib/threading.py", line 515 in acquire
File "Lib/multiprocessing/pool.py", line 405 in _guarded_task_generation
File "Lib/multiprocessing/pool.py", line 551 in _handle_tasks
Main thread:
File "Lib/multiprocessing/pool.py", line 922 in next
In the zip example, next(b) waits for a task that is never submitted, and the task handler waits for a to be read, which never happens.
Possible directions:
- Do not throttle inside the task handler thread. For example, submit the next item from the consumer's thread in
IMapIterator.next(), similar to whatExecutor.map(buffersize=...)does inconcurrent.futures. - Or keep the current design and document that a pool should not be shared between a
buffersizeiterator and other work.
I would like to work on this. Before I write a patch, I would like to know which direction maintainers prefer.
Related: gh-158675 (results dropped after close()), which comes from the same blocking sema.acquire() in the task handler.
buffersize was added in gh-64192 and is new in 3.16, so no released version is affected.
CPython versions tested on:
CPython main branch
Operating systems tested on:
macOS
Activity
PrakharAgarwal17 commentedon Oct 3, 2026
oyiakoumis commentedon Oct 4, 2026
@PrakharAgarwal17 thanks for looking into this.
I had a patch in progress too and have just opened it as a draft: gh-158792.
I started with the same approach as you, submitting from the consumer's thread, but moved away from it.
next()then has to read the next input item before returning, so it can hold a ready result while the iterable is slow, and it deadlocks if the next item depends on that result being handled.The draft keeps the iterable in the task handler thread and makes the handler skip an iterator whose buffer is full instead of waiting on it.
On your questions:
close()unchanged and keptPool.imap(buffersize=...)silently drops results after close() #158675 separate, so each fix can be reviewed on its own.imap()returns immediately as before.Your review and testing on the draft would be very welcome. ;)
oyiakoumis commentedon Oct 4, 2026
@picnixz, I closed my draft PR gh-158792 so we can agree on the approach here first.
The approach I used there is different from what I originally suggested in the report.
The task handler keeps consuming the input, as it does today. If one iterator hits its buffer limit, the handler skips it for now, processes other work, and comes back to it once a result is consumed. So from the caller's side, nothing changes.
I initially tried having
next()submit the next task, similar toExecutor.map(buffersize=...). The issue is that this also makesnext()consume the input in the caller's thread. If the input is slow,next()can block on the iterable even though a result is already ready.I'll wait for your take before pushing anything.
pythongh-158677: Fix Pool.imap(buffersize=...) blocking other tasks o…