You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When trying to read a tiff file with the newest dasktifffile and zarr there seems to be some typing discrepancies. Dask doesn't seem to like ZarrTiffStore anymore. This is blocking me from using some of the features of new zarr versions. I see from your docs that zarr 3 is not supported. Is there a plan to support zarr 3?
Dependencies
dask 2025.3.0
tifffile 2025.3.30
zarr 3.0.7
Python 3.11.2
Code that is failing
import dask.array as da
from tifffile.tifffile import TiffFile, TiffFileError, TiffTags, imread
def _get_image_data(
fs: AbstractFileSystem,
path: str,
scene: int,
retrieve_indices: Tuple[Union[int, slice]],
transpose_indices: List[int],
) -> np.ndarray:
with fs.open(path) as open_resource:
with imread(
open_resource,
aszarr=True,
series=scene,
level=0,
chunkmode="page",
is_mmstack=False,
) as store:
arr = da.from_zarr(store)
Error output
Unsupported type for store_like: 'ZarrTiffStore'
File "/Users/brian.whitney/Desktop/Repos/bioio-ome-tiff/bioio_ome_tiff/reader.py", line 586, in _get_image_data
arr = da.from_zarr(store)
^^^^^^^^^^^^^^^^^^^
TypeError: Unsupported type for store_like: 'ZarrTiffStore'
Check out this branch. I had some success reading V2 files from zarr 3+. Maybe we could wrap some of that reading to help with this case? Totally possible I am using it incorrectly though...
The immediate issue is that Zarr 3 requires new implementations of stores derived from zarr.abc.store. It's not so much about reading version 2 file format with Zarr 3.
@cgohlke I'm trying to understand the missing piece here. Is there a previous store method that hasn't been ported to Zarr 3? Does the existing ZarrTiffStore need to be updated to the new store implementation? If so, is that a very complex process?
Is there a previous store method that hasn't been ported to Zarr 3?
Who knows. The v3 store API is completely different from v2 (not a single method is the same), there is no migration guide or wrapper for v2 stores.
AFAICT, support for byte ranges, key ranges, buffer types, partial writes, and listing need to be implemented newly. Some inefficient implementations of default v3 store methods should be replaced too.
At that point it may be worth switching to v3 zarr file format to avoid another full breakage when v2 gets dropped.
Then, the fsspec reference writer functions would need to be ported too, which requires dozens of imagecodecs codecs to be wrapped for zarr.abc.codec.
Tifffile must not directly depend on zarr or async, so the v3 store implementations have to be separated from core tifffile code.
I can't tell yet if all of this can be implemented without breaking the tifffile API. If not, tests, docstrings, examples, benchmarks, downstream code need to be updated, users supported...
It took a couple of days to port the ZarrTiffStore and ZarrFileSequenceStore to Zarr 3's new zarr.abc.store.Store interface. It turns out that some new features like byte ranges, key ranges, partial writes, and listing apparently do not need to be implemented, at least for stores using zarr_format 2. All of tifffile's tests are passing now (except remote fsspec; see below) with minimal changes.
However:
I could not get fsspec reference files to work over HTTP with Zarr 3. Those tests are disabled for now.
Zarr 3 no longer provides a LRUStoreCache.
The Zarr 3 stores are several times slower than the Zarr 2 stores, even though almost all code is unchanged. The tifffile test suite takes 30 minutes to complete with Zarr 3 compared to 8 minutes with Zarr 2. Zarr 2 stores were already inefficient, but this is unacceptable.
I am hesitant to support zarr 3 at this time or even spend more time on this.
Who knows. The v3 store API is completely different from v2 (not a single method is the same), there is no migration guide or wrapper for v2 stores.
insofar as the v2 store api is basically MutableMapping, and since we have a v3 store class (MemoryStore) that wraps an arbitrary MutableMapping, I suspect we could pretty easily throw a v2-compatible store into MemoryStore without much fuss.
This needs to be tested of course, and the v2-compatible store could not inherit from a class defined in zarr-python 2, unless we make a zarr-python-2 namespace in zarr-python 3. see zarr-developers/zarr-python#3076 for more discussion about making the v2 -> v3 transition easier. we are definitely open to ideas there.
@cgohlke - I just wanted to thank you for your candid feedback. Clearly there is work to do to make the developer experience for store maintainers better. This was not something we devoted enough care too during the V3 design process.
Reacted by Juan Nunez-Iglesias, Brian Whitney and Josh Moore
Hi @cgohlke , I’m Ian, a developer at Earthmover, and a long-time user of tifffile.
Thanks for providing a clear and actionable list of ways that Zarr 3 could be improved to support Tifffile better. I’m going to be working on these issues with the goal of making it feasible and even desirable for Tifffile to support Zarr 3, starting with the performance issue you described. Do you have a Tifffile branch you can point me to with your WIP for the Zarr 3 support? I’d like to be able to pick that up and reproduce your issues. From there, I’ll work on addressing the problems you identified upstream in Zarr.
Reacted by Christoph Gohlke, toloudis and Josh Moore
I made some progress on the store performance this morning: by removing a layer of abstraction in the store, the time to run tifffile tests reduced from 30 to 9 minutes (compared to 7 minutes with Zarr 2). Not entirely sure yet why that made such a huge difference, but it is a big relief. Another issue I noticed is that dask.array.from_zarr using an array from the zarr 3 store doesn't multi-thread any longer...
Reacted by Brian Whitney, Josh Moore and Juan Nunez-Iglesias
Zarr 3 is now required in order to use the stores. Zarr 2 no longer works.
All but two tests pass (LRUStoreCache and tiffslide package). The reference files do work over HTTP with zarr 3. I will look more into performance and dask multi-threading later.
Reacted by Josh Moore, Juan Nunez-Iglesias and Doug Shepherd
Re performance regressions seen during tifffile tests:
In general, the tifffile Zarr 3 stores perform comparable to the Zarr 2 stores.
Stores that needs to open/close files, like stores for multi-file OME-TIFF or file sequences, take a significant performance hit. I observed up to 4x slowdown. That is not limited to the tifffile stores but is also apparent in the stores created by fsspec/kerchunk.
Reading from dask arrays created from tifffile stores (dask.array.from_zarr(store, zarr_format=2)) is significant slower with Zarr 3 than Zarr 2. About 2-3 times slower for the files I benchmarked.
Here are two scripts and results that demonstrate the performance regression using Zarr 3 vs 2 stores. These are using minimal stores, basically just returning pre-allocated chunks. No tifffile, I/O, and decoding involved.
Practically, the performance differences between Zarr 2 and 3 stores are quite noticeable (for example) when lazy-loading and viewing TIFF files in napari.
Zarr 3 results
Python 3.13.3 (tags/v3.13.3:6280bb5, Apr 8 2025, 14:47:33) [MSC v.1943 64 bit (AMD64)]
numpy 2.2.6
zarr 3.0.8
dask 2025.5.1
ScanImage
shape (5980, 512, 512)
chunk shape (1, 8, 512)
chunk number 382,720
size 3,135,242,240
zarr_array[:] 13.610396 s
dask_array.compute() 1:16.208372 s
OME-TIFF
shape (2, 43, 10, 512, 512)
chunk shape (1, 1, 1, 1, 512)
chunk number 440,320
size 225,443,840
zarr_array[:] 15.305893 s
dask_array.compute() 1:30.137136 s
NDPI
shape (51840, 92160, 3)
chunk shape (8, 3840, 3)
chunk number 155,520
size 14,332,723,200
zarr_array[:] 8.433944 s
dask_array.compute() 1:53.307927 s
Zarr 2 results
Python 3.13.3 (tags/v3.13.3:6280bb5, Apr 8 2025, 14:47:33) [MSC v.1943 64 bit (AMD64)]
numpy 2.2.6
zarr 2.18.7
dask 2025.5.1
ScanImage
shape (5980, 512, 512)
chunk shape (1, 8, 512)
chunk number 382,720
size 3,135,242,240
zarr_array[:] 2.878095 s
dask_array.compute() 42.523847 s
OME-TIFF
shape (2, 43, 10, 512, 512)
chunk shape (1, 1, 1, 1, 512)
chunk number 440,320
size 225,443,840
zarr_array[:] 3.437307 s
dask_array.compute() 51.737325 s
NDPI
shape (51840, 92160, 3)
chunk shape (8, 3840, 3)
chunk number 155,520
size 14,332,723,200
zarr_array[:] 3.910766 s
dask_array.compute() 1:25.085403 s
These results demonstrate that dask multi-threading can not speed up reading Zarr 3 stores. A ~1ms synchronous delay (time.sleep(1e-3)) was added before returning chunks:
Zarr 3 results
BBBC006_v1
shape (16, 24, 33, 2, 2, 520, 696)
chunk shape (1, 1, 1, 1, 1, 520, 696)
chunk number 50,688
size 36,690,001,920
sleep 0.001 s
zarr_array[:] 1:37.371663 s
dask_array.compute() 2:05.181760 s
Zarr 2 results
BBBC006_v1
shape (16, 24, 33, 2, 2, 520, 696)
chunk shape (1, 1, 1, 1, 1, 520, 696)
chunk number 50,688
size 36,690,001,920
sleep 0.001 s
zarr_array[:] 1:25.622381 s
dask_array.compute() 24.606557 s
def tifffile_reader(tif: TiffFile) -> List[LayerData]:
"""Return napari LayerData from image series in TIFF file."""
nlevels = len(tif.series[0].levels)
if nlevels > 1:
import zarr
store = tif.aszarr(multiscales=True)
group = zarr.open_group(store=store, mode='r')
data = [group[str(key)] for key in range(nlevels)]
# assert array shapes are in descending order for napari multiscale image
shapes = [arr.shape for arr in data]
assert shapes == list(reversed(sorted(shapes)))
else:
data = tif.asarray()
# ... napari layer stuffs
I'm not sure how better/more properly to access the arrays, because when I try arrays() or keys() they all give an error due to not being listable.
I've tested a few svs and ndpi and concur that at least subjectively, performance is poorer than with the equivalent using zarr2, when visualizing in napari.
tried 10, 64, too.
but not much of an effect by eye at least on my trusty m1 mbpro.
did some tests with array[:] and also not seeing more than +/- 1 s with different concurrency/threadworker settings (e.g. 12-13 s)
I'm not sure how better/more properly to access the arrays, because when I try arrays() or keys() they all give an error due to not being listable.
Can you provide a short, self-contained example that no longer works for you? The old Zarr 2 stores were not really listable either, at least not fully. The tifffile tests did not fail without the stores being listable. I rather avoid returning iterators over potentially many millions of keys if that is not necessary. To query multiscale keys, I use zgroup.attrs['multiscales'][0]['datasets']. It's an ngff extension.
Old code used data = [arr for _, arr in group.arrays()]
I tried a few variants.
your way works though, thanks!
data = [group[path_dict['path']] for path_dict in group.attrs['multiscales'][0]['datasets']]
Edit: and in fact it works with both older and newer tifffile and both zarr!
Edit2: I don't have any super large ndpi handy at home, but using CMU-1.ndpi and:
def read_tiff_as_zarr(path):
with TiffFile(path) as tif:
store = tif.aszarr(multiscales=True)
group = zarr.open_group(store=store, mode='r')
array = group['0']
array[:]
(38144, 51200, 3)
(8, 2048, 3)
tifffile 2025.5.21
zarr 3.0.7
py313
%timeit returns 11.8 s ± 355 ms per loop with default concurrency
with zarr.config.set({'async.concurrency': 64}): 11 s ± 343 ms per loop
tifffile 2025.5.10
zarr 2.18.3
py310
10.3 s ± 118 ms per loop
and with py313
10 s ± 104 ms per loop
(this all on a 2020 MacBook Pro M1)
Thanks @psobolewskiPhD for reporting this performance regression. It's very helpful. I'm confident we can understand and fix whatever is causing this issue.
Activity
cgohlke commentedon May 14, 2025
Zarr 3 is unfortunately not able to open Zarr 2 stores, such as tifffile's
ZarrTiffStoreandZarrFileSequenceStore.See also #282.
BrianWhitneyAI commentedon May 14, 2025
Check out this branch. I had some success reading V2 files from zarr 3+. Maybe we could wrap some of that reading to help with this case? Totally possible I am using it incorrectly though...
cgohlke commentedon May 14, 2025
The immediate issue is that Zarr 3 requires new implementations of stores derived from zarr.abc.store. It's not so much about reading version 2 file format with Zarr 3.
BrianWhitneyAI commentedon May 15, 2025
@cgohlke I'm trying to understand the missing piece here. Is there a previous store method that hasn't been ported to Zarr 3? Does the existing ZarrTiffStore need to be updated to the new store implementation? If so, is that a very complex process?
cgohlke commentedon May 17, 2025
Who knows. The v3 store API is completely different from v2 (not a single method is the same), there is no migration guide or wrapper for v2 stores.
AFAICT, support for byte ranges, key ranges, buffer types, partial writes, and listing need to be implemented newly. Some inefficient implementations of default v3 store methods should be replaced too.
At that point it may be worth switching to v3 zarr file format to avoid another full breakage when v2 gets dropped.
Then, the fsspec reference writer functions would need to be ported too, which requires dozens of imagecodecs codecs to be wrapped for
zarr.abc.codec.Tifffile must not directly depend on zarr or async, so the v3 store implementations have to be separated from core tifffile code.
I can't tell yet if all of this can be implemented without breaking the tifffile API. If not, tests, docstrings, examples, benchmarks, downstream code need to be updated, users supported...
cgohlke commentedon May 21, 2025
It took a couple of days to port the
ZarrTiffStoreandZarrFileSequenceStoreto Zarr 3's new zarr.abc.store.Store interface. It turns out that some new features like byte ranges, key ranges, partial writes, and listing apparently do not need to be implemented, at least for stores using zarr_format 2. All of tifffile's tests are passing now (except remote fsspec; see below) with minimal changes.However:
I could not get fsspec reference files to work over HTTP with Zarr 3. Those tests are disabled for now.The Zarr 3 stores are several times slower than the Zarr 2 stores, even though almost all code is unchanged. The tifffile test suite takes 30 minutes to complete with Zarr 3 compared to 8 minutes with Zarr 2. Zarr 2 stores were already inefficient, but this is unacceptable.I am hesitant to support zarr 3 at this time or even spend more time on this.
d-v-b commentedon May 21, 2025
insofar as the v2 store api is basically
MutableMapping, and since we have a v3 store class (MemoryStore) that wraps an arbitraryMutableMapping, I suspect we could pretty easily throw a v2-compatible store intoMemoryStorewithout much fuss.This needs to be tested of course, and the v2-compatible store could not inherit from a class defined in zarr-python 2, unless we make a zarr-python-2 namespace in zarr-python 3. see zarr-developers/zarr-python#3076 for more discussion about making the v2 -> v3 transition easier. we are definitely open to ideas there.
rabernat commentedon May 21, 2025
@cgohlke - I just wanted to thank you for your candid feedback. Clearly there is work to do to make the developer experience for store maintainers better. This was not something we devoted enough care too during the V3 design process.
ianhi commentedon May 21, 2025
Hi @cgohlke , I’m Ian, a developer at Earthmover, and a long-time user of tifffile.
Thanks for providing a clear and actionable list of ways that Zarr 3 could be improved to support Tifffile better. I’m going to be working on these issues with the goal of making it feasible and even desirable for Tifffile to support Zarr 3, starting with the performance issue you described. Do you have a Tifffile branch you can point me to with your WIP for the Zarr 3 support? I’d like to be able to pick that up and reproduce your issues. From there, I’ll work on addressing the problems you identified upstream in Zarr.
cgohlke commentedon May 21, 2025
Thanks all!
I made some progress on the store performance this morning: by removing a layer of abstraction in the store, the time to run tifffile tests reduced from 30 to 9 minutes (compared to 7 minutes with Zarr 2). Not entirely sure yet why that made such a huge difference, but it is a big relief. Another issue I noticed is that
dask.array.from_zarrusing an array from the zarr 3 store doesn't multi-thread any longer...cgohlke commentedon May 22, 2025
Fixed in v2025.5.21.
The stores are now in tifffile.zarr.
Zarr 3 is now required in order to use the stores. Zarr 2 no longer works.
All but two tests pass (LRUStoreCache and tiffslide package). The reference files do work over HTTP with zarr 3. I will look more into performance and dask multi-threading later.
pin tifffile
Pin tifffile for zarr-python 2 compatibility (#303)
cgohlke commentedon May 22, 2025
Re performance regressions seen during tifffile tests:
In general, the tifffile Zarr 3 stores perform comparable to the Zarr 2 stores.
Stores that needs to open/close files, like stores for multi-file OME-TIFF or file sequences, take a significant performance hit. I observed up to 4x slowdown.That is not limited to the tifffile stores but is also apparent in the stores created by fsspec/kerchunk.Reading from dask arrays created from tifffile stores (
dask.array.from_zarr(store, zarr_format=2)) is significant slower with Zarr 3 than Zarr 2. About 2-3 times slower for the files I benchmarked.cgohlke commentedon May 23, 2025
Here are two scripts and results that demonstrate the performance regression using Zarr 3 vs 2 stores. These are using minimal stores, basically just returning pre-allocated chunks. No tifffile, I/O, and decoding involved.
Practically, the performance differences between Zarr 2 and 3 stores are quite noticeable (for example) when lazy-loading and viewing TIFF files in napari.
Zarr 3 results
Zarr 2 results
Zarr 3 script
Zarr 2 script
cgohlke commentedon May 24, 2025
These results demonstrate that dask multi-threading can not speed up reading Zarr 3 stores. A ~1ms synchronous delay (
time.sleep(1e-3)) was added before returning chunks:Zarr 3 results
Zarr 2 results
psobolewskiPhD commentedon May 24, 2025
I'm using:
I'm not sure how better/more properly to access the arrays, because when I try
arrays()orkeys()they all give an error due to not being listable.I've tested a few svs and ndpi and concur that at least subjectively, performance is poorer than with the equivalent using zarr2, when visualizing in napari.
I played around with
tried 10, 64, too.
but not much of an effect by eye at least on my trusty m1 mbpro.
did some tests with
array[:]and also not seeing more than +/- 1 s with different concurrency/threadworker settings (e.g. 12-13 s)cgohlke commentedon May 24, 2025
Can you provide a short, self-contained example that no longer works for you? The old Zarr 2 stores were not really listable either, at least not fully. The tifffile tests did not fail without the stores being listable. I rather avoid returning iterators over potentially many millions of keys if that is not necessary. To query multiscale keys, I use
zgroup.attrs['multiscales'][0]['datasets']. It's an ngff extension.psobolewskiPhD commentedon May 24, 2025
Old code used
data = [arr for _, arr in group.arrays()]I tried a few variants.
your way works though, thanks!
Edit: and in fact it works with both older and newer tifffile and both zarr!
Edit2: I don't have any super large ndpi handy at home, but using CMU-1.ndpi and:
(38144, 51200, 3)
(8, 2048, 3)
tifffile 2025.5.21
zarr 3.0.7
py313
%timeit returns 11.8 s ± 355 ms per loop with default concurrency
with zarr.config.set({'async.concurrency': 64}): 11 s ± 343 ms per loop
tifffile 2025.5.10
zarr 2.18.3
py310
10.3 s ± 118 ms per loop
and with py313
10 s ± 104 ms per loop
(this all on a 2020 MacBook Pro M1)
Support both zarr2/tifffile<2025.5.21 and zarr3/tifffile>2025.5.21 (#48)
psobolewskiPhD commentedon May 29, 2025
bleh, finally had a chance to test with a larger ndpi, like Hamamatsu-1.ndpi (from https://openslide.cs.cmu.edu/download/openslide-testdata/Hamamatsu/). ~7Gb on disk, so ~70Gb uncompressed?
I can replicate the performance regression reported above using updated napari-tiff.
And with zarr3 visualizing in napari is brutally bad compared to zarr2. At least 5x worse (4-6-second delays vs. choppiness). Even the lowest res is horrible, possibly because:
:freeze:
Edit: wrapping the arrays in a
dask.array.from_zarr()withchunks='1 MiB'gives:And now it's pretty alright with zarr3. Still worse than the equivalent with zarr2, but the user experience is passable for sure.
rabernat commentedon May 30, 2025
Thanks @psobolewskiPhD for reporting this performance regression. It's very helpful. I'm confident we can understand and fix whatever is causing this issue.