Skip to content

Support for zarr 3.0.0+ #297

Closed
Closed

Description

@BrianWhitneyAI

Description

When trying to read a tiff file with the newest dask tifffile and zarr there seems to be some typing discrepancies. Dask doesn't seem to like ZarrTiffStore anymore. This is blocking me from using some of the features of new zarr versions. I see from your docs that zarr 3 is not supported. Is there a plan to support zarr 3?

Dependencies

dask                      2025.3.0
tifffile                  2025.3.30
zarr                      3.0.7

Python 3.11.2

Code that is failing

import dask.array as da
from tifffile.tifffile import TiffFile, TiffFileError, TiffTags, imread

 def _get_image_data(
      fs: AbstractFileSystem,
      path: str,
      scene: int,
      retrieve_indices: Tuple[Union[int, slice]],
      transpose_indices: List[int],
  ) -> np.ndarray:
      with fs.open(path) as open_resource:
          with imread(
              open_resource,
              aszarr=True,
              series=scene,
              level=0,
              chunkmode="page",
              is_mmstack=False,
          ) as store:
              arr = da.from_zarr(store)

Error output

Unsupported type for store_like: 'ZarrTiffStore'
  File "/Users/brian.whitney/Desktop/Repos/bioio-ome-tiff/bioio_ome_tiff/reader.py", line 586, in _get_image_data
    arr = da.from_zarr(store)
          ^^^^^^^^^^^^^^^^^^^
TypeError: Unsupported type for store_like: 'ZarrTiffStore'

Activity

  1. added
    duplicateThis issue or pull request already exists
    on May 14, 2025
  2. cgohlke commented on May 14, 2025

    @cgohlke
    Owner

    Zarr 3 is unfortunately not able to open Zarr 2 stores, such as tifffile's ZarrTiffStore and ZarrFileSequenceStore.

    See also #282.

  3. BrianWhitneyAI commented on May 14, 2025

    @BrianWhitneyAI
    Author

    Check out this branch. I had some success reading V2 files from zarr 3+. Maybe we could wrap some of that reading to help with this case? Totally possible I am using it incorrectly though...

  4. cgohlke commented on May 14, 2025

    @cgohlke
    Owner

    The immediate issue is that Zarr 3 requires new implementations of stores derived from zarr.abc.store. It's not so much about reading version 2 file format with Zarr 3.

  5. BrianWhitneyAI commented on May 15, 2025

    @BrianWhitneyAI
    Author

    @cgohlke I'm trying to understand the missing piece here. Is there a previous store method that hasn't been ported to Zarr 3? Does the existing ZarrTiffStore need to be updated to the new store implementation? If so, is that a very complex process?

  6. cgohlke commented on May 17, 2025

    @cgohlke
    Owner

    Is there a previous store method that hasn't been ported to Zarr 3?

    Who knows. The v3 store API is completely different from v2 (not a single method is the same), there is no migration guide or wrapper for v2 stores.

    AFAICT, support for byte ranges, key ranges, buffer types, partial writes, and listing need to be implemented newly. Some inefficient implementations of default v3 store methods should be replaced too.

    At that point it may be worth switching to v3 zarr file format to avoid another full breakage when v2 gets dropped.

    Then, the fsspec reference writer functions would need to be ported too, which requires dozens of imagecodecs codecs to be wrapped for zarr.abc.codec.

    Tifffile must not directly depend on zarr or async, so the v3 store implementations have to be separated from core tifffile code.

    I can't tell yet if all of this can be implemented without breaking the tifffile API. If not, tests, docstrings, examples, benchmarks, downstream code need to be updated, users supported...

  7. cgohlke commented on May 21, 2025

    @cgohlke
    Owner

    It took a couple of days to port the ZarrTiffStore and ZarrFileSequenceStore to Zarr 3's new zarr.abc.store.Store interface. It turns out that some new features like byte ranges, key ranges, partial writes, and listing apparently do not need to be implemented, at least for stores using zarr_format 2. All of tifffile's tests are passing now (except remote fsspec; see below) with minimal changes.

    However:

    1. I could not get fsspec reference files to work over HTTP with Zarr 3. Those tests are disabled for now.
    2. Zarr 3 no longer provides a LRUStoreCache.
    3. The Zarr 3 stores are several times slower than the Zarr 2 stores, even though almost all code is unchanged. The tifffile test suite takes 30 minutes to complete with Zarr 3 compared to 8 minutes with Zarr 2. Zarr 2 stores were already inefficient, but this is unacceptable.

    I am hesitant to support zarr 3 at this time or even spend more time on this.

  8. d-v-b commented on May 21, 2025

    @d-v-b

    Who knows. The v3 store API is completely different from v2 (not a single method is the same), there is no migration guide or wrapper for v2 stores.

    insofar as the v2 store api is basically MutableMapping, and since we have a v3 store class (MemoryStore) that wraps an arbitrary MutableMapping, I suspect we could pretty easily throw a v2-compatible store into MemoryStore without much fuss.

    This needs to be tested of course, and the v2-compatible store could not inherit from a class defined in zarr-python 2, unless we make a zarr-python-2 namespace in zarr-python 3. see zarr-developers/zarr-python#3076 for more discussion about making the v2 -> v3 transition easier. we are definitely open to ideas there.

  9. rabernat commented on May 21, 2025

    @rabernat

    @cgohlke - I just wanted to thank you for your candid feedback. Clearly there is work to do to make the developer experience for store maintainers better. This was not something we devoted enough care too during the V3 design process.

  10. ianhi commented on May 21, 2025

    @ianhi

    Hi @cgohlke , I’m Ian, a developer at Earthmover, and a long-time user of tifffile.

    Thanks for providing a clear and actionable list of ways that Zarr 3 could be improved to support Tifffile better. I’m going to be working on these issues with the goal of making it feasible and even desirable for Tifffile to support Zarr 3, starting with the performance issue you described. Do you have a Tifffile branch you can point me to with your WIP for the Zarr 3 support? I’d like to be able to pick that up and reproduce your issues. From there, I’ll work on addressing the problems you identified upstream in Zarr.

  11. cgohlke commented on May 21, 2025

    @cgohlke
    Owner

    Thanks all!

    I made some progress on the store performance this morning: by removing a layer of abstraction in the store, the time to run tifffile tests reduced from 30 to 9 minutes (compared to 7 minutes with Zarr 2). Not entirely sure yet why that made such a huge difference, but it is a big relief. Another issue I noticed is that dask.array.from_zarr using an array from the zarr 3 store doesn't multi-thread any longer...

  12. cgohlke commented on May 22, 2025

    @cgohlke
    Owner

    Fixed in v2025.5.21.

    The stores are now in tifffile.zarr.

    Zarr 3 is now required in order to use the stores. Zarr 2 no longer works.

    All but two tests pass (LRUStoreCache and tiffslide package). The reference files do work over HTTP with zarr 3. I will look more into performance and dask multi-threading later.

  13. cgohlke commented on May 22, 2025

    @cgohlke
    Owner

    Re performance regressions seen during tifffile tests:

    • In general, the tifffile Zarr 3 stores perform comparable to the Zarr 2 stores.

    • Stores that needs to open/close files, like stores for multi-file OME-TIFF or file sequences, take a significant performance hit. I observed up to 4x slowdown. That is not limited to the tifffile stores but is also apparent in the stores created by fsspec/kerchunk.

    • Reading from dask arrays created from tifffile stores (dask.array.from_zarr(store, zarr_format=2)) is significant slower with Zarr 3 than Zarr 2. About 2-3 times slower for the files I benchmarked.

  14. cgohlke commented on May 23, 2025

    @cgohlke
    Owner

    Here are two scripts and results that demonstrate the performance regression using Zarr 3 vs 2 stores. These are using minimal stores, basically just returning pre-allocated chunks. No tifffile, I/O, and decoding involved.

    Practically, the performance differences between Zarr 2 and 3 stores are quite noticeable (for example) when lazy-loading and viewing TIFF files in napari.

    Zarr 3 results

    Python 3.13.3 (tags/v3.13.3:6280bb5, Apr  8 2025, 14:47:33) [MSC v.1943 64 bit (AMD64)]
    numpy 2.2.6
    zarr 3.0.8
    dask 2025.5.1
    
    ScanImage
    shape (5980, 512, 512)
    chunk shape (1, 8, 512)
    chunk number 382,720
    size 3,135,242,240
    zarr_array[:] 13.610396 s
    dask_array.compute() 1:16.208372 s
    
    OME-TIFF
    shape (2, 43, 10, 512, 512)
    chunk shape (1, 1, 1, 1, 512)
    chunk number 440,320
    size 225,443,840
    zarr_array[:] 15.305893 s
    dask_array.compute() 1:30.137136 s
    
    NDPI
    shape (51840, 92160, 3)
    chunk shape (8, 3840, 3)
    chunk number 155,520
    size 14,332,723,200
    zarr_array[:] 8.433944 s
    dask_array.compute() 1:53.307927 s
    

    Zarr 2 results

    Python 3.13.3 (tags/v3.13.3:6280bb5, Apr  8 2025, 14:47:33) [MSC v.1943 64 bit (AMD64)]
    numpy 2.2.6
    zarr 2.18.7
    dask 2025.5.1
    
    ScanImage
    shape (5980, 512, 512)
    chunk shape (1, 8, 512)
    chunk number 382,720
    size 3,135,242,240
    zarr_array[:] 2.878095 s
    dask_array.compute() 42.523847 s
    
    OME-TIFF
    shape (2, 43, 10, 512, 512)
    chunk shape (1, 1, 1, 1, 512)
    chunk number 440,320
    size 225,443,840
    zarr_array[:] 3.437307 s
    dask_array.compute() 51.737325 s
    
    NDPI
    shape (51840, 92160, 3)
    chunk shape (8, 3840, 3)
    chunk number 155,520
    size 14,332,723,200
    zarr_array[:] 3.910766 s
    dask_array.compute() 1:25.085403 s
    

    Zarr 3 script

    """Benchmark minimal Zarr 3 store."""
    
    from __future__ import annotations
    
    import json
    from typing import TYPE_CHECKING
    
    import numpy
    import zarr
    
    try:
        from zarr.abc.store import ByteRequest, Store
    except ImportError as exc:
        raise ValueError(f'zarr {zarr.__version__} < 3 is not supported') from exc
    
    
    if TYPE_CHECKING:
        from collections.abc import AsyncIterator, Iterable
        from typing import Any
    
        from numpy.typing import NDArray
        from zarr.core.buffer import Buffer, BufferPrototype
        from zarr.core.common import BytesLike
    
    
    class ZarrStore(Store):
        """Minimal Zarr 3 store implementation."""
    
        _store: dict[str, Any]
        _chunk: NDArray[Any]
    
        def __init__(
            self,
            shape: tuple[int, ...],
            chunks: tuple[int, ...],
            dtype: str,
        ) -> None:
            super().__init__()
    
            self._store = {
                '.zattrs': b'{}',
                '.zarray': json.dumps(
                    {
                        'zarr_format': 2,
                        'shape': shape,
                        'chunks': chunks,
                        'dtype': dtype,
                        'order': 'C',
                        'fill_value': 0,
                        'filters': None,
                        'compressor': None,
                    },
                    indent=1,
                    sort_keys=True,
                    ensure_ascii=True,
                    separators=(',', ': '),
                ).encode('ascii'),
            }
            self._chunk = numpy.zeros(chunks, dtype).reshape(-1).view('B')
    
        def __eq__(self, other: object) -> bool:
            return isinstance(other, type(self)) and self._store == other._store
    
        async def exists(self, key: str) -> bool:
            print(key)
            if key in self._store:
                return True
            return key[0].isnumeric()
    
        async def get(
            self,
            key: str,
            prototype: BufferPrototype,
            byte_range: ByteRequest | None = None,
        ) -> Buffer | None:
            if byte_range is not None:
                raise NotImplementedError
            if key in self._store:
                return prototype.buffer.from_bytes(self._store[key])
            if key[0].isnumeric():
                return prototype.buffer(self._chunk)
            return None
    
        async def get_partial_values(
            self,
            prototype: BufferPrototype,
            key_ranges: Iterable[tuple[str, ByteRequest | None]],
        ) -> list[Buffer | None]:
            return [
                await self.get(key, prototype, byte_range)
                for key, byte_range in key_ranges
            ]
    
        @property
        def supports_writes(self) -> bool:
            return False
    
        def _set(self, key: str, value: Buffer, /) -> None:
            raise NotImplementedError
    
        async def set(self, key: str, value: Buffer) -> None:
            raise NotImplementedError
    
        @property
        def supports_deletes(self) -> bool:
            return False
    
        async def delete(self, key: str) -> None:
            raise NotImplementedError
    
        @property
        def supports_partial_writes(self) -> bool:
            return False
    
        async def set_partial_values(
            self, key_start_values: Iterable[tuple[str, int, BytesLike]]
        ) -> None:
            raise NotImplementedError
    
        @property
        def supports_listing(self) -> bool:
            return False
    
        def list(self) -> AsyncIterator[str]:
            raise NotImplementedError
    
        def list_prefix(self, prefix: str) -> AsyncIterator[str]:
            raise NotImplementedError
    
        def list_dir(self, prefix: str) -> AsyncIterator[str]:
            raise NotImplementedError
    
        def __repr__(self) -> str:
            return f'{self.__class__.__name__}'
    
    
    if __name__ == '__main__':
        import math
        import sys
    
        import dask
        import dask.array
        import tifffile
    
        print('Python', sys.version)
        print('numpy', numpy.__version__)
        print('zarr', zarr.__version__)
        print('dask', dask.__version__)
        print()
    
        for name, shape, chunks, dtype in [
            ['ScanImage', (5980, 512, 512), (1, 8, 512), '<i2'],
            ['OME-TIFF', (2, 43, 10, 512, 512), (1, 1, 1, 1, 512), '|u1'],
            ['NDPI', (51840, 92160, 3), (8, 3840, 3), '|u1'],
        ]:
            print(name)
            print('shape', shape),
            print('chunk shape', chunks),
            print('chunk number', f'{math.prod(shape) // math.prod(chunks):,}')
            print('size', f'{math.prod(shape) * numpy.dtype(dtype).itemsize:,}')
    
            store = ZarrStore(shape, chunks, dtype)
    
            zarr_array = zarr.open(store, zarr_format=2)
            # print(zarr_array.info)
            # print()
    
            with tifffile.Timer('zarr_array[:]'):
                numpy_array = zarr_array[:]
            del numpy_array
    
            dask_array = dask.array.from_zarr(zarr_array)
            with tifffile.Timer('dask_array.compute()'):
                numpy_array = dask_array.compute()
            del numpy_array
            del dask_array
            del zarr_array
            del store
            print()

    Zarr 2 script

    """Benchmark minimal Zarr 2 store."""
    
    from __future__ import annotations
    
    import json
    from collections.abc import MutableMapping
    from typing import TYPE_CHECKING
    
    import numpy
    
    if TYPE_CHECKING:
        from typing import Any
    
        from numpy.typing import NDArray
    
    
    class ZarrStore(MutableMapping):
        """Minimal Zarr 2 store."""
    
        _store: dict[str, Any]
        _chunk: NDArray[Any]
    
        def __init__(
            self,
            shape: tuple[int, ...],
            chunks: tuple[int, ...],
            dtype: str,
        ) -> None:
            self._store = {
                '.zattrs': b'{}',
                '.zarray': json.dumps(
                    {
                        'zarr_format': 2,
                        'shape': shape,
                        'chunks': chunks,
                        'dtype': dtype,
                        'order': 'C',
                        'fill_value': 0,
                        'filters': None,
                        'compressor': None,
                    },
                    indent=1,
                    sort_keys=True,
                    ensure_ascii=True,
                    separators=(',', ': '),
                ).encode('ascii'),
            }
            self._chunk = numpy.zeros(chunks, dtype)
    
        def flush(self):
            raise PermissionError
    
        def clear(self):
            raise PermissionError
    
        def keys(self):
            return self._store.keys()
    
        def items(self):
            return self._store.items()
    
        def values(self):
            return self._store.values()
    
        def __iter__(self):
            return iter(self._store.keys())
    
        def __len__(self):
            return len(self._store)
    
        def __delitem__(self, key):
            raise PermissionError
    
        def __setitem__(self, key, value):
            raise PermissionError
    
        def __contains__(self, key):
            if key in self._store:
                return True
            return key[0].isnumeric()
    
        def __getitem__(self, key):
            if key in self._store:
                return self._store[key]
            if key[0].isnumeric():
                return self._chunk
            return None
    
        def __repr__(self) -> str:
            return f'{self.__class__.__name__}'
    
    
    if __name__ == '__main__':
        import math
        import sys
    
        import dask
        import dask.array
        import tifffile
        import zarr
    
        print('Python', sys.version)
        print('numpy', numpy.__version__)
        print('zarr', zarr.__version__)
        print('dask', dask.__version__)
        print()
    
        for name, shape, chunks, dtype in [
            ['ScanImage', (5980, 512, 512), (1, 8, 512), '<i2'],
            ['OME-TIFF', (2, 43, 10, 512, 512), (1, 1, 1, 1, 512), '|u1'],
            ['NDPI', (51840, 92160, 3), (8, 3840, 3), '|u1'],
        ]:
            print(name)
            print('shape', shape),
            print('chunk shape', chunks),
            print('chunk number', f'{math.prod(shape) // math.prod(chunks):,}')
            print('size', f'{math.prod(shape) * numpy.dtype(dtype).itemsize:,}')
    
            store = ZarrStore(shape, chunks, dtype)
    
            zarr_array = zarr.open(store)
            # print(zarr_array.info)
            # print()
    
            with tifffile.Timer('zarr_array[:]'):
                numpy_array = zarr_array[:]
            del numpy_array
    
            dask_array = dask.array.from_zarr(zarr_array)
            with tifffile.Timer('dask_array.compute()'):
                numpy_array = dask_array.compute()
            del numpy_array
            del dask_array
            del zarr_array
            del store
            print()
  15. cgohlke commented on May 24, 2025

    @cgohlke
    Owner

    These results demonstrate that dask multi-threading can not speed up reading Zarr 3 stores. A ~1ms synchronous delay (time.sleep(1e-3)) was added before returning chunks:

    Zarr 3 results

    BBBC006_v1
    shape (16, 24, 33, 2, 2, 520, 696)
    chunk shape (1, 1, 1, 1, 1, 520, 696)
    chunk number 50,688
    size 36,690,001,920
    sleep 0.001 s
    zarr_array[:] 1:37.371663 s
    dask_array.compute() 2:05.181760 s
    

    Zarr 2 results

    BBBC006_v1
    shape (16, 24, 33, 2, 2, 520, 696)
    chunk shape (1, 1, 1, 1, 1, 520, 696)
    chunk number 50,688
    size 36,690,001,920
    sleep 0.001 s
    zarr_array[:] 1:25.622381 s
    dask_array.compute() 24.606557 s
    
  16. psobolewskiPhD commented on May 24, 2025

    @psobolewskiPhD

    I'm using:

    def tifffile_reader(tif: TiffFile) -> List[LayerData]:
        """Return napari LayerData from image series in TIFF file."""
        nlevels = len(tif.series[0].levels)
        if nlevels > 1:
            import zarr
            store = tif.aszarr(multiscales=True)
            group = zarr.open_group(store=store, mode='r')
            data = [group[str(key)] for key in range(nlevels)]
            # assert array shapes are in descending order for napari multiscale image
            shapes = [arr.shape for arr in data]
            assert shapes == list(reversed(sorted(shapes)))
        else:
            data = tif.asarray()
         # ... napari layer stuffs
    

    I'm not sure how better/more properly to access the arrays, because when I try arrays() or keys() they all give an error due to not being listable.
    I've tested a few svs and ndpi and concur that at least subjectively, performance is poorer than with the equivalent using zarr2, when visualizing in napari.

    I played around with

    zarr.config.set({
                'async.concurrency': 100,
                'threading.max_workers': None,
            })
    

    tried 10, 64, too.
    but not much of an effect by eye at least on my trusty m1 mbpro.
    did some tests with array[:] and also not seeing more than +/- 1 s with different concurrency/threadworker settings (e.g. 12-13 s)

  17. cgohlke commented on May 24, 2025

    @cgohlke
    Owner

    I'm not sure how better/more properly to access the arrays, because when I try arrays() or keys() they all give an error due to not being listable.

    Can you provide a short, self-contained example that no longer works for you? The old Zarr 2 stores were not really listable either, at least not fully. The tifffile tests did not fail without the stores being listable. I rather avoid returning iterators over potentially many millions of keys if that is not necessary. To query multiscale keys, I use zgroup.attrs['multiscales'][0]['datasets']. It's an ngff extension.

  18. psobolewskiPhD commented on May 24, 2025

    @psobolewskiPhD

    Old code used data = [arr for _, arr in group.arrays()]
    I tried a few variants.
    your way works though, thanks!

    data = [group[path_dict['path']] for path_dict in group.attrs['multiscales'][0]['datasets']]
    

    Edit: and in fact it works with both older and newer tifffile and both zarr!

    Edit2: I don't have any super large ndpi handy at home, but using CMU-1.ndpi and:

    def read_tiff_as_zarr(path):
         with TiffFile(path) as tif:
             store = tif.aszarr(multiscales=True)
             group = zarr.open_group(store=store, mode='r')
             array = group['0']
             array[:]
    

    (38144, 51200, 3)
    (8, 2048, 3)
    tifffile 2025.5.21
    zarr 3.0.7
    py313
    %timeit returns 11.8 s ± 355 ms per loop with default concurrency
    with zarr.config.set({'async.concurrency': 64}): 11 s ± 343 ms per loop

    tifffile 2025.5.10
    zarr 2.18.3
    py310
    10.3 s ± 118 ms per loop
    and with py313
    10 s ± 104 ms per loop
    (this all on a 2020 MacBook Pro M1)

  19. added
    enhancementNew feature or request
    and removed
    duplicateThis issue or pull request already exists
    on May 26, 2025
  20. psobolewskiPhD commented on May 29, 2025

    @psobolewskiPhD

    bleh, finally had a chance to test with a larger ndpi, like Hamamatsu-1.ndpi (from https://openslide.cs.cmu.edu/download/openslide-testdata/Hamamatsu/). ~7Gb on disk, so ~70Gb uncompressed?
    I can replicate the performance regression reported above using updated napari-tiff.
    And with zarr3 visualizing in napari is brutally bad compared to zarr2. At least 5x worse (4-6-second delays vs. choppiness). Even the lowest res is horrible, possibly because:

    viewer.layers[0].data[5].shape
    Out[1]: (3168, 5880, 3)
    
    viewer.layers[0].data[5].chunks
    Out[3]: (8, 120, 3)
    

    :freeze:

    Edit: wrapping the arrays in a dask.array.from_zarr() with chunks='1 MiB' gives:

    viewer.layers[0].data[5]
    Out[1]: dask.array<from-zarr, shape=(3168, 5880, 3), dtype=uint8, chunksize=(152, 2280, 3), chunktype=numpy.ndarray>
    

    And now it's pretty alright with zarr3. Still worse than the equivalent with zarr2, but the user experience is passable for sure.

  21. rabernat commented on May 30, 2025

    @rabernat

    Thanks @psobolewskiPhD for reporting this performance regression. It's very helpful. I'm confident we can understand and fix whatever is causing this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions