Skip to content

Commit 5bf3390

Browse files
committed
Make zarr.storage picklable by reference
zarr.storage replaces its module class with VerboseModule to warn when default_compressor is set. Because that instance is not a plain module, pickle and cloudpickle try to serialize its state and fail with "cannot pickle 'VerboseModule' object". This breaks sending objects that reference zarr.storage to Dask workers. Define __reduce__ on VerboseModule to pickle the module by name, as is done for ordinary modules, and add a regression test. Fixes zarr-developers#4029.
1 parent df18415 commit 5bf3390

3 files changed

Lines changed: 17 additions & 0 deletions

File tree

‎changes/4029.bugfix.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
`zarr.storage` can now be pickled by reference, so arrays backed by stores in `zarr.storage` can be sent to Dask workers without failing with `cannot pickle 'VerboseModule' object`.

‎src/zarr/storage/__init__.py‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import importlib
12
import sys
23
import warnings
34
from types import ModuleType
@@ -29,6 +30,12 @@
2930

3031

3132
class VerboseModule(ModuleType):
33+
def __reduce__(self) -> tuple[Any, ...]:
34+
# Pickle by reference, as for a plain module, instead of trying to serialize
35+
# this module's instance state. Without this, pickling fails with
36+
# "cannot pickle 'VerboseModule' object".
37+
return importlib.import_module, (self.__name__,)
38+
3239
def __setattr__(self, attr: str, value: Any) -> None:
3340
if attr == "default_compressor":
3441
warnings.warn(

‎tests/test_api.py‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2018,3 +2018,12 @@ def test_unimplemented_kwarg_warnings(kwarg_name: str) -> None:
20182018
kwargs = {kwarg_name: 1}
20192019
with pytest.warns(RuntimeWarning, match=".* is not yet implemented"):
20202020
zarr.create(shape=(1,), **kwargs) # type: ignore[arg-type]
2021+
2022+
2023+
def test_storage_module_is_picklable() -> None:
2024+
"""`zarr.storage` is pickled by reference, so it can be sent to Dask workers (#4029)."""
2025+
import pickle
2026+
2027+
import zarr.storage
2028+
2029+
assert pickle.loads(pickle.dumps(zarr.storage)) is zarr.storage

0 commit comments

Comments
 (0)