Repository navigation
da.from_zarr ignores Zarr's dimension_separator #7673
Description
What happened:
Zarr arrays that have been stored with "/" as the dimension_separator attribute (in .zarray) have no data in them when opened with da.from_zarr. See zarr-developers/zarr-python#715
What you expected to happen:
All Zarr stores in the zarr-python repository were updated in zarr-developers/zarr-python#716 to read this new metadata. The two options I can think of are either using a full store like FSStore in from_zarr or adding support for dimension_separator directly.
cc: @martindurant
Minimal Complete Verifiable Example:
import zarr as zr
import dask.array as da
z = zr.storage.FSStore(
"foo.zarr",
dimension_separator="/",
auto_mkdir=True,
mode="w",
)
a = zr.creation.array(store=z, data=[range(4) for x in range(4)])
# Passes
base = da.from_zarr(z, "")
m = base.max().compute()
assert m == 3, m
# Fails
base = da.from_zarr("foo.zarr")
m = base.max().compute()
assert m == 3, mAnything else we need to know?:
Originally reported in: zarr-developers/zarr-python#709 (comment)
Environment:
- Dask version: 2021.5.0
- Python version: 3.9.4
- Operating System: OSX 11.2.3
- Install method (conda, pip, source):
conda create -n dask dask zarr fsspec
Activity
[-]da.from_zarr ignores dimension_separator[/-][+]da.from_zarr ignores Zarr's dimension_separator[/+]martindurant commentedon Jun 14, 2021
In the end, I think we've decided that this can be solved elsewhere, so I suggest this issue be closed.
joshmoore commentedon Jun 14, 2021
@martindurant: I may have missed something. What's the solution?
martindurant commentedon Jun 14, 2021
I though to either pass the options to, and implement the logic in fsstore, or make a new store that drives from it. I didn't mean to imply that it was solved, only that it could be. Or did I misunderstand and this really is Dask's fault?
joshmoore commentedon Jun 14, 2021
Requiring the user to set the
dimension_separatoris really a non-starter, since the user does not know beforehand. The issue is that none of the detection code that has gone into zarr-python is used by dask. If dask, e.g., used FSStore, then I think this would be solved.martindurant commentedon Jun 15, 2021
Following
if I do
I get only zeros. Is this wrong? This is on zarr "2.8.4.dev6"
joshmoore commentedon Jun 15, 2021
No, I'm seeing that as well. It should be fixed in zarr-developers/zarr-python#773
martindurant commentedon Jun 15, 2021
OK, so when that's in, I'll fix dask. It will simply pass all arguments to zarr and have exactly the same behaviour.
joshmoore commentedon Aug 20, 2021
The fix above has hit the mainline and should be released as 2.9.0 ASAP: zarr-developers/zarr-python#812
joshmoore commentedon Aug 23, 2021
https://pypi.org/project/zarr/2.9.0/ is released, @martindurant
joshmoore commentedon Mar 13, 2025
With the zarr-python v3 line dimension separation (i.e. chunk key encoding) has been significantly overhauled.