Skip to content

Errors with Zarr v3 and da.to_zarr() #11691

New issue
New issue
Closed
Closed
Errors with Zarr v3 and da.to_zarr()#11691
Labels
needs triageNeeds a response from a contributor

Description

@will-moore
will-moore
opened on Jan 23, 2025

I'm having various issues and errors with da.to_zarr() using:

dask==2025.1.0
zarr==3.0.1
fsspec==2024.12.0

from skimage import data
import dask.array as da
import zarr

dask_data = da.from_array(data.coins(), chunks=(64, 64))

da.to_zarr(dask_data, "test_dask_to_zarr.zarr", compute=True, storage_options={"chunks": (64, 64)})

# Traceback (most recent call last):
#   File "/Users/wmoore/Desktop/python-scripts/zarr_scripts/test_dask_to_zarr.py", line 7, in <module>
#     da.to_zarr(dask_data, "test_dask_to_zarr.zarr", compute=True, storage_options={"chunks": (64, 64)})
#   File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/site-packages/dask/array/core.py", line 3891, in to_zarr
#     store = zarr.storage.FsspecStore.from_url(
#             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#   File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/site-packages/zarr/storage/_fsspec.py", line 182, in from_url
#     return cls(fs=fs, path=path, read_only=read_only, allowed_exceptions=allowed_exceptions)
#            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#   File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/site-packages/zarr/storage/_fsspec.py", line 96, in __init__
#     raise TypeError("Filesystem needs to support async operations.")
# TypeError: Filesystem needs to support async operations.

Trying to use a local store has a different error:

store = zarr.storage.LocalStore("test_dask_to_zarr.zarr", read_only=False)
da.to_zarr(dask_data, store, compute=True, storage_options={"chunks": (64, 64)})

#   File "/Users/wmoore/Desktop/python-scripts/zarr_scripts/test_dask_to_zarr.py", line 46, in <module>
#     da.to_zarr(dask_data, store, compute=True, storage_options={"chunks": (64, 64)})
#   File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/site-packages/dask/array/core.py", line 3891, in to_zarr
#     store = zarr.storage.FsspecStore.from_url(
#             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#   File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/site-packages/zarr/storage/_fsspec.py", line 174, in from_url
#     fs, path = url_to_fs(url, **opts)
#                ^^^^^^^^^^^^^^^^^^^^^^
#   File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/site-packages/fsspec/core.py", line 403, in url_to_fs
#     chain = _un_chain(url, kwargs)
#             ^^^^^^^^^^^^^^^^^^^^^^
#   File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/site-packages/fsspec/core.py", line 335, in _un_chain
#     if "::" in path:
#        ^^^^^^^^^^^^
# TypeError: argument of type 'LocalStore' is not iterable

And also tried with FsspecStore:

store = zarr.storage.FsspecStore("test_dask_to_zarr.zarr", read_only=False)
da.to_zarr(dask_data, store, compute=True, storage_options={"chunks": (64, 64)})

# Traceback (most recent call last):
#   File "/Users/wmoore/Desktop/python-scripts/zarr_scripts/test_dask_to_zarr.py", line 32, in <module>
#     store = zarr.storage.FsspecStore("test_dask_to_zarr_v3.zarr", read_only=False)
#             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#   File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/site-packages/zarr/storage/_fsspec.py", line 95, in __init__
#     if not self.fs.async_impl:
#            ^^^^^^^^^^^^^^^^^^
# AttributeError: 'str' object has no attribute 'async_impl'

Many thanks for your help

Activity

  1. github-actions
    added
    needs triageNeeds a response from a contributor
    on Jan 23, 2025

  2. joshmoore commented on Jan 28, 2025

    @joshmoore
    joshmoore
    on Jan 28, 2025

    TypeError: Filesystem needs to support async operations.

    zarr-developers/zarr-python#2554 would imply that you need a new release of zarr-python with zarr-developers/zarr-python#2533

  3. will-moore commented on Jan 29, 2025

    @will-moore
    will-moore
    on Jan 29, 2025
    Author

    Thanks @joshmoore

    I updated to use main branch of zarr-python:

    $ pip freeze | grep zarr-python
-e git+ssh://[email protected]/zarr-developers/zarr-python.git@fb37ff0942c7417a7767cc4d537de7173a9b3329#egg=zarr

    dask==2025.1.0
fsspec==2024.12.0

    This:

    from skimage import data
import dask.array as da

dask_data = da.from_array(data.coins(), chunks=(64, 64))
da.to_zarr(dask_data, "test_dask_to_zarr.zarr", compute=True, storage_options={"chunks": (64, 64)})

    now gives me:

    $ python test_dask_to_zarr.py 
Traceback (most recent call last):
  File "/Users/wmoore/Desktop/python-scripts/zarr_scripts/test_dask_to_zarr.py", line 17, in <module>
    da.to_zarr(dask_data, "test_dask_to_zarr.zarr", compute=True, storage_options={"chunks": (64, 64)})
  File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/site-packages/dask/array/core.py", line 3901, in to_zarr
    z = zarr.create(
        ^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/synchronous.py", line 707, in create
    sync(
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/sync.py", line 142, in sync
    raise return_result
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/sync.py", line 98, in _runner
    return await coro
           ^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/api/asynchronous.py", line 1044, in create
    return await AsyncArray._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/array.py", line 610, in _create
    result = await cls._create_v3(
             ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/array.py", line 726, in _create_v3
    await array._save_metadata(metadata, ensure_parents=True)
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/core/array.py", line 1315, in _save_metadata
    await gather(*awaitables)
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/abc/store.py", line 487, in set_or_delete
    await byte_setter.set(value)
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/storage/_common.py", line 144, in set
    await self.store.set(self.path, value)
  File "/Users/wmoore/Desktop/ZARR_PYTHON/zarr-python/src/zarr/storage/_fsspec.py", line 276, in set
    await self.fs._pipe_file(path, value.to_bytes())
  File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/site-packages/fsspec/implementations/asyn_wrapper.py", line 27, in wrapper
    return await asyncio.to_thread(func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/site-packages/fsspec/spec.py", line 787, in pipe_file
    with self.open(path, "wb", **kwargs) as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/site-packages/fsspec/spec.py", line 1310, in open
    f = self._open(
        ^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/site-packages/fsspec/implementations/local.py", line 200, in _open
    return LocalFileOpener(path, mode, fs=self, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/site-packages/fsspec/implementations/local.py", line 364, in __init__
    self._open()
  File "/Users/wmoore/opt/anaconda3/envs/zarrv3_py312/lib/python3.12/site-packages/fsspec/implementations/local.py", line 369, in _open
    self.f = open(self.path, mode=self.mode)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/Users/wmoore/Desktop/python-scripts/zarr_scripts/test_dask_to_zarr.zarr/zarr.json'

  4. will-moore commented on Jan 30, 2025

    @will-moore
    will-moore
    on Jan 30, 2025
    Author

    The other 2 tests above, using LocalStore or FsspecStore still have the same errors as before (no change wtih zarr-developers/zarr-python#2533)

  5. will-moore commented on Mar 13, 2025

    @will-moore
    will-moore
    on Mar 13, 2025
    Author

    If I use "auto_mkdir": True in storage_options as suggested at zarr-developers/zarr-python#2808 (comment) then this looks good!

    dask_data = da.from_array(data.coins(), chunks=(64, 64))
da.to_zarr(dask_data, "test_dask_to_zarr.zarr", compute=True, storage_options={"chunks": (64, 64), "auto_mkdir": True})
  6. will-moore
    closed this as completedon Mar 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs triageNeeds a response from a contributor

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions