You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
FusedCodecPipeline: forked worker hangs on a multi-chunk read; reset_resources_after_fork misses codec_pipeline._pool #4478
Description. With codec_pipeline.path = FusedCodecPipeline and max_workers > 1 (the
default), multi-chunk I/O in a parent creates the module-level zarr.core.codec_pipeline._pool. reset_resources_after_fork (zarr/core/sync.py) resets loop, iothread and _executor in
a forked child, but not _pool / _pool_size, so the child reuses an executor whose threads do
not exist. submit consumes an idle permit the parent left rather than starting a thread; a read
of no more chunks than the inherited permits never returns from read_sync's pool.map (a
larger read happens to start a thread and completes, so the hang depends on read size).
Reproducer (fork start method, the Linux default before Python 3.14):
$ python repro.py
zarr 3.4.0 | parent pool idle permits: 10
HANG: forked workers did not return in 30 s
$ python repro.py 1 # codec_pipeline.max_workers = 1
zarr 3.4.0 | parent pool idle permits: None
children: [37900, 37900]
Hangs in 3 of 3 runs with the default pool; completes in 3 of 3 with max_workers = 1. We first hit
this on Zarr v2-format arrays with a numcodecs Blosc compressor, in a fork-pool build reading
multi-chunk bands.
Suggested fix: reset _pool = None and _pool_size = 0 in the child (in reset_resources_after_fork, or an os.register_at_fork in codec_pipeline.py). Workaround:codec_pipeline.max_workers = 1, which never creates the pool.
Activity
test(codec_pipeline): pin the pool size in the fork regression test (#…