Skip to content

FusedCodecPipeline: forked worker hangs on a multi-chunk read; reset_resources_after_fork misses codec_pipeline._pool #4478

Closed

Description

@explodecomputer

Versions: zarr 3.4.0, numcodecs 0.17.0, numpy 2.4.6, Python 3.12.14, Linux.

Description. With codec_pipeline.path = FusedCodecPipeline and max_workers > 1 (the
default), multi-chunk I/O in a parent creates the module-level zarr.core.codec_pipeline._pool.
reset_resources_after_fork (zarr/core/sync.py) resets loop, iothread and _executor in
a forked child, but not _pool / _pool_size, so the child reuses an executor whose threads do
not exist. submit consumes an idle permit the parent left rather than starting a thread; a read
of no more chunks than the inherited permits never returns from read_sync's pool.map (a
larger read happens to start a thread and completes, so the hang depends on read size).

Reproducer (fork start method, the Linux default before Python 3.14):

import multiprocessing as mp, os, sys, tempfile
from concurrent.futures import ProcessPoolExecutor, TimeoutError
import numpy as np, zarr
import zarr.core.codec_pipeline as cp

cfg = {"codec_pipeline.path": "zarr.core.codec_pipeline.FusedCodecPipeline"}
if len(sys.argv) > 1:
    cfg["codec_pipeline.max_workers"] = int(sys.argv[1])
zarr.config.set(cfg)
path = os.path.join(tempfile.mkdtemp(), "a.zarr")
a = zarr.create_array(path, shape=(40, 40), chunks=(10, 10), dtype="i2")
a[:] = np.arange(1600, dtype="i2").reshape(40, 40)  # 16 chunks: parent uses cp._pool
a[:]
print("zarr", zarr.__version__, "| parent pool idle permits:",
      cp._pool._idle_semaphore._value if cp._pool else None)

def child(_):
    return int(zarr.open_array(path, mode="r")[:10, :20].sum())  # 2 chunks

pool = ProcessPoolExecutor(2, mp_context=mp.get_context("fork"))
try:
    print("children:", list(pool.map(child, range(2), timeout=30)), flush=True)
except TimeoutError:
    print("HANG: forked workers did not return in 30 s", flush=True)
    for proc in pool._processes.values():
        proc.kill()
    os._exit(1)

Output:

$ python repro.py
zarr 3.4.0 | parent pool idle permits: 10
HANG: forked workers did not return in 30 s
$ python repro.py 1   # codec_pipeline.max_workers = 1
zarr 3.4.0 | parent pool idle permits: None
children: [37900, 37900]

Hangs in 3 of 3 runs with the default pool; completes in 3 of 3 with max_workers = 1. We first hit
this on Zarr v2-format arrays with a numcodecs Blosc compressor, in a fork-pool build reading
multi-chunk bands.

Suggested fix: reset _pool = None and _pool_size = 0 in the child (in
reset_resources_after_fork, or an os.register_at_fork in codec_pipeline.py).
Workaround: codec_pipeline.max_workers = 1, which never creates the pool.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions